OpenSim unfinished symphony
Odd thing this; falling out of OpenSim development I feel like I imagine HAL did, parts of my self falling off and in the end I will probably do the programmer equivalent of droning "daaaisssyyy, daaaaaaiiisss..." and come to a grinding halt.
Before that actually happens, I'll just make sure to squeeze out this blog, listing the projects (aka 'really good and low hanging fruit kind of ideas') I wanted to do with OpenSim, but never got a chance to. If only for posterity. Here goes:
Identifying assets with URL, not GUID
The main idea would be to implement a model where the viewer and region communicated guids as asset id's, but that the regions communicated urls as asset identifiers to the back ends. This would lead to general awesomeness, as assets could truly be served by simple web servers. The region would then simply serve as an asset streamed redistribution cache, fetching data in arbitrary format and converting them locally to the format best suited for the given client. Of course, in the case of the LL stack, because asset data can reference other assets, you would have to substitute guids to urls and back in the asset data: the region would thus have to have a system to keep track of guid<->url mapping, but it is my firm belief that this is doable.
Status: I did start on this in trunk, but stopped; now it's been removed, but I believe I proved the guid substitution in asset data approach is doable.
Avatar Appearance defined by URL, not GUID
A companion to the above, the avatar appearance would be defined as a http endpoint; an xml file outlining the appearance pointing to assets by urls instead of a guids. The content could then either be assembled procedural, thru a database, or as a simple published xml file. If this would then be expanded to an "aar" (avatar archive) file format that encapsulates all the assets involved, this would take avatar portability home.
Status: Pure dreamland, but probably not that hard given the appearance is already fetched over http with guids, and OpenSim already have oars and iars.
Virtual Inventory Nodes
Each inventory node would be identified with an url, not a guid; this would lead to being able to distribute a users inventory over several services - allowing for services to update inventory nodes in real time, for example after a service upgrade of some inventory asset, interacting with a web page or changing to premium services. Think of the inventory a bit like 3d web 'favorites' links to content that still can change.
Status: Pure dreamland, and the feature could get pretty messy real fast. Still, it's probably the way to go. Arguably not that hard either given that inventory is communicated as xml over http in OpenSim today.
Unified UI Toolbox
The main idea would be to implement a shell executable that loads Mono.Addins for functionality separated from user interface, where the latter can be implemented for any environment (winforms, silverlight, ajax, gtk, you name it) whilst sharing the functionality (intelligent ini configuration, connectivity troubleshooter, region launching and monitoring, console command<->gui mapping, log file config and analysis, various launcher url schemes, database management et c) - this model would help developers cooperate on shared functionality and environment-specific UI separately.
Status: Proof of concept lying around on my harddrive, just editing a few ini settings in a WinForm gui. It works, and it's cool as heck.
Tribal One Viewports
In Tribal One, we had a 'viewport', which was a scene object defining a cube, in which content was governed by a http endpoint. In OpenSim, one could do a first implementation by implementing a non-backed up scene object that would "load xml" or "load oar" its content, rotated and displaced with the 'viewport'. The viewport also 'snapped' to other surfaces which let you for example define that a picture always hung on a vertical surface, or that a lamp always was aligned standing on the horizontal surface below it. Pure awesomeness taking building from generic objects so much easier. The idea was that content in a region would be 'assembled' from a great number of web sources, most of them probably programatically generated, procedurally or from databases, and other just simple static xml files.
Status: As with all Tribal One code, based on ~r2000 code, so a total mess to bring up to date.
On-demand regions
Basically, it would not be that hard to have OpenSim turn the solid-state running-region paradigm around, and allowing for on-demand regions that were started and loaded from database when somebody tried to teleport to them, and that closed down when nobody was around in them. This would mean the cpu capacity is not connected to land mass but to land use. Combined with the viewports, we're starting to look at something truly like a modern web page, where a region is started and generated in a response to a user need; you and I might be in very similar regions, but mine is subtly different because I wanted something subtly different.
Status: We did this in Tribal One, but all regions were started and served in one single instance, so didn't scale. Still, the result was amazing. In PIM, we had a 'pool' of region processes that were re-configured (re-loaded) as people wanted access to different 'class rooms'.
Binary de-duplication and scavenging
The idea is to split the asset table as so to extract the actual binary data out of it and into a separate table, with a foreign key. This way, you could make sure to do a SHA-256 hash on all new binary data and normalize the tables by making multiple assets point to the same binary data row and removing the duplicate. The process that computes SHA-256 and normalizes the asset table could run as a separate tool, or as a ROBUST service. On frequent archive (re-)loading, you would still have massive duplication of data rows, but at least the binary storage wouldn't swell just as fast.
Status: There's already a sha field in the asset database, this approach has been discussed back and forth, but nothing general has yet surfaced.
Epilogue: If anybody thinks any of this is a good idea, and want to know more of the thoughts behind it, I'd be more than happy to share. Same goes for code.
Open Source has its place
In his must-read article "The 'wisdom of crowds' loses steam" Matt Asay makes a number of good points:
- As open source projects mature, scarcity of "easy fixes" and heightened thresholds for contributing (as quality and form is being prioritized over features) leads to volunteers leaving.
- A handful of paid programmers can deliver better than a dispersed "community".
- How you treat your "community" is not that different from how a corporation would treat their "partners" and "customers".
- Companies relying on "community" to code for them is a bad idea.
- "Communities" relying on companies to code for them is a bad idea.
What I feel Matt fails to point out is that all his examples shows initiatives that at some time relied on volunteers to a high degree to get started, then slowly tranformed into various corporate-like structures. In effect, there is a "volunteer phase" in some initiatives which get them to market faster. One could argue that some initiatives would probably not get to market at all if it weren't for this phase.
A project does not consist of programmers alone; testers, evangelists, documentators, third party application programmers - with some of these roles you actually want a large and varied group that can approach issues from different angles.
Another point that is not clearly communicated is that there is two different perspectives on open source development communities; whether you're a company like MySql partly crowdsourcing your production, or if you're part of an community, where there is no clear corporate interest in, or control over, the community itself. Or to put it another way, If you're on the outside peeking in (possibly thru agents), or if you're on the inside looking out.
In the case of OpenSimulator, the Open Source project I know best, I would translate this article to the following recommendations:
- As a commercial interest, you cannot rely on the community to function as your development department. If you need development, you need to secure dev resources for yourself.
- As a member of the OpenSim community, you are essentially customer and partner to yourself.
- In the far future, if and when the product has matured, OpenSim will eventually need some kind of governance, and the rate of volunteer effort will dwindle.
- Key is to be getting the balance right between governance and volunteer effort, and getting a sound mix of dependencies on commercial sources. At the time being, I'd say the project have a healthy mix of large and small commercial and non-commercial interests.
Do you agree?
Stepping down from OpenSim core
I've decided to step down from OpenSim core.
Since Tribal Media keeled over, my life situation simply does not allow me neither to participate in the dev community regularly nor contribute code. I have committed exactly three characters since we moved to git.
What I have come to respect the most in OpenSim is the disrespectful attitude against 'talk'.
OpenSim has always been about participating thru effort; testing, promoting, writing, coding, building. So while discussion has always been welcome, any kind of demands have always been met with "do it yourself, or pay someone to do it". A harsh sentiment, but ultimately the only viable attitude in a project like this.
Sean Dague disliked the term 'founding four', feeling that just because you were among the first didn't automatically give you authority. Back then, I didn't agree; for some time now, I have.
So rather than lingering, I'm leaving.
I wish to thank all core devs, past and present; thank you all gridnauts, testers, hosts, customers and partners - It's been an immensely satisfying ride. Hope to see you all on the 3D Web.
Best regards,
Stefan Andersson aka lbsa71
Introducing StyleCop on Legacy Projects using StyleCop for ReSharper
Recently, I've been working with introducing StyleCop on some legacy projects using the StyleCop for ReSharper plug-in.
This Blog aims to collect my findings, and also give some idea of what one can expect from using ReSharper to achieve StyleCop compliance.
First, I'll go thru how I auto-cleaned issues in one project, and after that I'll give some numbers from a complete solution.
A Single Legacy Project
The single legacy project was a measly 16 .cs files big, of which 2 were designer files. Just turning StyleCop onto this yielded 370 issues.
While this may come off as deterring at first, this blog is about how to use StyleCop for ReSharper to bring that down considerably.
Excluding designer files and generated files
Since Visual Studio .Designer.cs files aren't StyleCop compliant (no VS templates are) you probably want to exclude those from StyleCop.
These are your options:
- Doing it project-wide from the "Detailed Settings" on the "Rules" tab of the StyleCop Project Settings.
- Setting it per designer file by including a
//<auto-generated />
comment in the file header.
Handling Documentation Issues
First of all, you need to decide on how to handle the documentation issues, which normally will account for around half the warnings.
These are your options:
- Temporarily turn off all StyleCop rules but the documentation rules, and remedy those manually before going ahead with cleaning up the rest of the code.
- Permanently turn off all StyleCop documentation rules, and concentrate on the code.
- Auto-generate documentation tags by doing a "Code Cleanup" with all documentation cleanup modules enabled.
I would not recommend number three, as it will pollute your code with less-than-meaningful documentation comments.
Handling Header Formatting Documentation Issues
StyleCop is quite strict on how the file header should be formatted. It needs to have the filename in a copyright element, and you're not allowed to wrap the header in a region, for example.
To fix the header, you have these options:
- If your policy dictates an StyleCop-incompatible header content, permanently disable StyleCop rules concerning file header.
- If your policy dictates you have a multi-line copyright + licence text, manually Copy+Paste a compliant file header into each file, applying the quick fix to update the file="" attribute of the copyright element.
- If your policy allows for a single-line copyright statement, let the Stylecop plugin create those for you on running Code Cleanup. Fill in 'Company Information' in the StyleCop Settings, enable "Update file header" and let the StyleCop plugin "Replace entire header" in the Code Cleanup settings.
Taking these measures brought down the number of issues to 302.
Configuring StyleCop Code Cleanup
You need to configure ReSharper to have the built-in Code formatting help auto-format for StyleCop compliance.
Fortunately, the StyleCop for ReSharper comes with a StyleCopCodeStyle.xml resharper codestyle export, that you can import from the ReSharper options dialog.
A couple of notes on applying the StyleCopCodeStyle.xml code styles:
- I set "Blank lines between using groups" to zero; otherwise I get the "usings should be sorted alphabetically" issue as the "sort usings" only do so within using groups.
- I set "Wrap Lines" to zero, as StyleCop really doesn't like that.
- And of course, you need re-specify your StyleCop and Code Cleanup settings for documentation and header file fixes as per above.
Running Code Cleanup
After running the Code Cleanup with these settings and rebuilt, I'm now down to 24 issues - less than a tentht of the original number of warnings.
Running Code Cleanup a second time and rebuilding yields 21 warnings, a third cleanup round gives 16, and finally, doing it a fourth time and rebuilding I land on 10 issues.
This odd behaviour is because each cleanup round only adresses one of several potential issues per code block, and in the process, they sometime introduce new issues.
So just run Code Cleanup and rebuild until the number of issues stabilizes.
While working with this, the StyleCop Fixes page is an invaluable resource to debug your ReSharper Code Cleanup settings.
Manual fixes
The last 10 issues were actual errors or issues where there isn't enough context for Code Cleanup to do automatic refactorings. All of them were easily remedied given the available ReSharper refactorings:
- SA1401: Public fields should be exposed thru properties – if there is a good reason for exposing public fields directly, this warning can be suppressed using the suppression quickfix.
- SA1305: No Hungarian notation-like variable names.
- SA1300: Function names should begin with upper-case letter.
All in all, there were automatic or semi-automatic fixes for all issues.
A Full Solution
For my second test, I used a production solution consisting of 26 projects, all with roughly the same code content as my single project.
Turning StyleCop on yielded 5500+ issues.
Fixing Designer files, Documentation and header issues brought this down to 2924 issues.
- If you want to set StyleCop settings (exclude documentation, no designer files et c) on the full solution, copy/move a suitable Settings.StyleCop into the solution directory. The default for StyleCop is to try and merge the local project Settings.Stylecop with the one in the parent directory.
Applying Code Cleanup iteratively took me down to 264, 261, 256 and finally 255 issues.
During both this test and the single project test I experienced 'hiccups' where a certain issue weren't fixed by a global Code Cleanup, but redoing it on the single file or using the quickfix would.
Conclusion
By using the StyleCop for ReSharper plugin, introducing StyleCop on legacy projects can be reasonably painless. The main part is to comply with the documentation rules; once these are handled, the issues remaining can be fixed reasonably simple and with a small amount of developer work, even for large solutions.
If you have questions, think there is something I've ommitted in this article or just want to share your experiences or good links with the world, please don't hesitate to comment.
Happy refactoring!
Stefan Andersson
The OpenSim lightweight Release Cycle
This is the current (as of the 0.6.5 release) OpenSim lightweight Release Cycle recipe.
The goal is to run a cycle that churns out recommended code snapshots with reasonably regular intervals, and with a minimum of work for any one human resource. It should be a community effort, and anyone should be able to move the cycle forward.
If you're not acquainted with svn revision numbering schemes, see "On revisions, tags and branches" for additional info.
The Cycle:
- Developers and Testers work on /trunk ("testers" being defined as "users feeding off trunk", also lovingly known as "trunkheads")
- Developers and Testers identify suitable release candidate revisions, from the repository history. (for example, people on osgrid.org discuss this on their weekly meet-ups)
- This revision is branched off as a "release candidate" (to /branches/0.6.5-rc1 in this case), and the version number is upped and committed only to this branch. We don't want to up the version number in trunk until we know we have a release.
- Testers now switch their focus to running the rc until they can give feedback on whether "rc sux" or "rc rox".
Key issues:- Has all version numbers been updated?
- Does it play well with last version? Do we need to up the interface version as well?
- Does it feel better or worse than last version?
- Any critical errors that needs fixing first?
- If any critical but fixable errors are found, these are fixed in trunk and the fixing revision is merged into the release candidate branch. (or vice versa, if that works out better)
- If, even after this, testers agree that "rc sux" the project goes back to 1) to wait and look for the next rc (thusly named rc2)
- If testers agree that "rc rox" the current rc revision is tagged as "-release" (to /tags/0.6.5-release in this case)
- The "-rc1" branch is renamed to "-post-fixes" (to /branches/0.6.5-post-fixes in this case) for continued service awaiting the next cycle. The svn rename lets us keep the revision history back thru the rc branch history.
- The version number (and optionally interface version) uppage is merged back into trunk. (this is a minor merge which will probably always succeed without conflict)
- The project goes back to 1)
The output of this would be named revisions for each of these user categories:
- Grid/Region owners running services in production
Feed off "-post-fixes" if they want stability first, or off "-release" if they want a known 'upgrade track'. - Developers
Feed off /trunk, or off -rc or -post-fixes if they want some stability while hunting bugs or doing protoyping. - Testers
Testers are committed to helping the devs find bugs at the prize of instabilities, crashes and loss of data. They therefore feed off the /trunk development branch or -rc depending on whether there is an RC pending release or not. (ie, is the latest version an RC, use that, if not, use /trunk)
Everyone feeding off /trunk are collectively labeled 'testers', and any installation running on anything but a 'release' or 'post-fixes' revision is considered a 'test installation'.
On Revisions, tags and branches
In light of some confusion in the wake of the 0.6.5 release of OpenSimulator, I just thought I'd share a couple of good-to-know points about SVN versioning;
In an SVN repo, the 'revision' is a discrete version of the software. But; the revision number is not an indication of sequential functional increment.
Revisions are based on a revision before it, but that does not have to be the revision immediately before it.
The revision number counter is global to the svn repository, hence it is upped whenever somebody does something within the repo, regardless of what branch the thing happens to.
An example:
- I commit something to trunk, the rev number is upped to, say, 1337, and my new version is given that number.
- I now branch trunk into /branches/mybranch. I do this by doing a remote copy of trunk. This copy of trunk, although identical to 1337, is a new version and is thus given the revision number 1338, the first revision on the new branch.
- Now I commit a number of changes to trunk, spinning past 1339, 1340, 1341 and 1342 - each being separate versions but on the branch called /trunk (think of it as paths)
Now, if I commit some minor change to /branches/mybranch, what revision number will it get? You guessed it - 1343.
If I now commit something to /trunk again, it will get 1344. I then commit something to the really really old branch /branches/reallyoldbranch that was based on, say, revision 666. It will now get 1345.
So, my point is: you should never talk about revision numbers when your're trying to convey an idea of some kind of sequential advancement of state.
Ie, in my example, 1344 is probably the one reflecting the most development, 1343 is essentially based on 1337 and 1345, currently the highest rev number, could basically be six months old, with just a minor modification.
Now on to tags:
Trunk, branches and tags are really all one thing: paths. The only real difference is by convention: there is a certain path called /trunk that serves a certain purpose, namely to be the focal point for developers. There is a certain name used for paths other than trunk, and that is "branches". Some branches by convention aren't meant to be modified, but serve as snapshots of a point in time - they are called "tags".
But to the server, they are all revisions along differerent link paths. There is really nothing stopping you from committing to a tag - it's just the svn client will be reluctant to do so, as it's aware of the convention.
Keep this in mind, when referring to a 'revision' - it might be on a totally different path, so be sure to mention what path you're really talking about. specifying "trunk, between revision 1343 and 1780" or "0.6.2 post-fixes, between revision 1764 and 2006" again lets you compare revision numbers to indicate advancement of state.
Also, don't tell people to check a certain revision out, unless you mean exactly that code snapshot. If you want them to check out a certain version, tell them the logical path to it instead.
I hope this has cleared any potential confusion up.
Tribes and Tribulations
Tribal Media has gone out of the Virtual Worlds business. Quite simply, we never could get a sustainable income out of it so finally we decided to close shop.
Darren and I have now reverted to being 'just' lbsa71 and MW, and we will both continue working with the OpenSim community.
I would like to extend my heart-felt gratitute to the prospects, clients, peers and partners we've had that made the last two years such an exciting ride - and I'm especially thankful for all the 3D webnauts that joined us in Tribal Net. You guys rocked!
Also, I'd like to extend a big thank you to Darren for working so hard with me to make our visions come to life - as head of R&D he produced some pretty amazing stuff that one can only hope will see the light of day some day - and to Jim, who believed in us and our vision so deeply he put his money where our mouth was.
Together, we wrote what will surely be a chapter in the metaverse saga.
Oracle 11gR2 taking 100% CPU utilization
Firts of all, this can have an array of explanations, but I just wanted to share one of them and its solution. (Here's a related problem.)
Oracle has a "Provisioning Daemon" that runs jobs that are pre-populated in the oracle install. Now, apparently, if there is a mismatch in date settings (say, from having the wrong time settings when setting up Oracle) this job template will always be in the past, and consequently, the daemon will try to start a new job, until it consumes 100% and croaks. Fixing the time issue and restarting the server won't help, as the 'bad' jobs are still waiting there when the server comes up.
The solution is to
- Stop the console
- Run
create table mgmt_job_bad as select * from sysman.mgmt_job where job_name = 'PROVISIONING DAEMON';
delete from sysman.mgmt_job where job_name = 'PROVISIONING DAEMON';
commit; - Restart the installation.
- Review the mgmt_job_bad to see what needs to be done.
Caveat: Those jobs were put there for a reason - you should get to know exactly what went wrong before doing this in a production environment.
lbsa71: wb
So, I moved the blog off a standard web host blog to an wordpress installation of my own. Hopefully, this should mean I can get a bit better control over the site content appearance. Thank you for your patience. 
Top ten ways to know you are not doing agile
Alistair Cockburn ('cooburn') is one of my heroes; here's a thought-provoking list to match your team against:
http://alistair.cockburn.us/Top+ten+ways+to+know+you+are+not+doing+agile