I wrote this as part of my work for Cloudsmith. It is a follow up of my earlier posting a few weeks ago.

In summary, Cloudsmith lets you browse and find useful bundles of software components which work together – software playlists – and then download ones of interest. Each one can contain components from different software repositories, and Cloudsmith knows where to go, and how to get to them.

–

Eric S. Raymond wrote a seminal paper in 1997, The Cathedral and the Bazaar, contrasting how Linux emerged from a loosely structured, highly collaborative community or “bazaar” with the traditional approach to developing software (open source or proprietary), in which a select group of cathedral-builders controlled every aspect of design and technology.

Most engineers strive to build at least one great “building” during their career, a monument, a shrine, and a testament to their skill. Today, even “cathedrals” are made from parts found at the bazaars – a huge and growing marketplace for open source components, in which thousands of developers promote parts that many other developers combine into new products. The output of many bazaars — projects and communities such as the Eclipse Foundation, the Apache Foundation, Google Code, SourceForge, etc. – support and publish the efforts of component development teams. Popular components turn up in multiple bazaars, sometimes as identical copies, other times with subtle variations.

Among the challenges development teams, and their co-worker product management and product marketing teams, face when operating within this new ecosystem are:

* What range of components is currently available? Which bazaars have them; what is their status and quality; how popular are they; where can updates and fixes be found; and so on.

* What works with what? What components, and combinations of components, are available? How do the pieces all fit together, and which bazaars have them?

* How popular is this combination of components compared to that alternative one? How do we know when and if we should update a selection of components, as new versions of the constituent parts emerge?

* How can we build playlists which combine components we built ourselves, with components found in public bazaars and that change in ways we don’t control? How can we move to the new version of a public component without breaking what we already have? And how can we keep what we found in the bazaar from getting so intertwined with what we built that we can no longer separate them? What is the best strategy to manage change, when your organisation and your team are increasingly mixing public software components with your proprietary assets?

* Who is going to support us when we use some unique combination which we assembled from public bazaars? Is there anyone out there doing something similar we can learn from?

* We fix and extend components we find in the bazaar, and sometimes create entirely new component playlists of our own. How do we share our work with other developers in our organisation or (assuming our corporate policy allows it) contribute things back to the bazaar for the public good? And assuming we’ve shared it, how do we know who is using it, and for what?

It is, of course, no longer just an issue of providing a stable, managed foundation on which you and your colleagues can build. There is heightened corporate awareness reaching all the way to the audit committees of publicly quoted companies, due to the multiplicity of software licensing policies. The issue of knowing if, when and how public software assets are being used inside a corporation has become a high concern.

The ability to tailor software should be its value rather than its risk. But in todays world, isn’t software componentisation paradoxically slower than it could be, precisely due to the changes, improvements and proliferation offered by the community?

Eric Raymond describes how extremely useful software can result from open collaboration, despite the absence of a clear lead architect directing the project. Today’s software repositories illustrate this principle on a grand scale – they are collections of really good and useful components developed, published, maintained and extended, sometimes by individuals and sometimes by organized teams of collaborators, in a process that can seem almost anarchic compared to conventional internal development.

As bazaars of developed, and contributed, software components have matured, the complexity of fitting together appropriate combinations have increased, as has ensuring that things do not break as each component is maintained.

One example is Eclipse, which is a common integration platform for many components. The recent Ganymede release lists nine application frameworks, six toolsets for embedded and device development, six toolsets for enterprise development, five language IDEs, and five aspects of its rich client platform. All of these elements, in principle, can be used in any combination of choice, although there are seven different official Ganymede packages are listed. Forty-five additional different project downloads are listed. And nine different distributions from member organisations are promoted. It shows an impressive level of community momentum and collective activity, but which of all of the alternatives do you really need for your particular project?

Actually, it is even more complex, because each bazaar stacks up components from its own shelves with components it finds in other bazaars. And you are often building not just one cathedral, but several based on a common set of blueprints. Perhaps you want to develop using Seam rich client Java toolkit? Then you might need a playlist of the Eclipse Classic IDE, JBoss Tools, Seam Core, JBoss AS, and PostgreSQL (with thanks to Stefan Daume for suggesting this particular playlist). But to do so, you may need to visit the Eclipse, JBoss, Seam and Postgres bazaars to put this all together — unless you can happen to find somebody else who has already done this for you. If you want to build an email spam filter, then maybe a playlist of MySQL, qpsmtpd, my qpsmtpd custom modules, php pages (status), and open flash chart run-time files might be just the job (with thanks to Bjorn Freeman-Benson for this playlist).

Finding out what software components are available is a modest challenge: you can use raw Google, or Google CodeSearch, or Koders, or Krugle, or Codase, or something similar. The more significant challenge is finding out what works with what else to form a useful playlist; then how to get hold of the right version of each these pieces from each of the right bazaars concerned; how popular is this specific playlist of components; and how to get notified if any of the pieces are subsequently changed. If you want to be civic-minded, you might also want to find out how best to contribute original or derivative works back to the remainder of your organisation or community at large.

Our industry is maturing: we really soon should reach the equivalent levels of professional practice as our colleagues in other engineering disciplines, such as electronics hardware and civil engineering. There now is – perhaps at long last – a substantial number of re-usable, well-engineered, components available to all of us, being extended and improved on a daily basis. We should all be able to build cathedrals, and other artifacts, from the components we find. But the vast range of components, coupled with the fluidity of material – software – with which to work, has presented our industry with some new challenges,and which are not as apparent in other engineering disciplines.

Henrik Lindberg writes in his blog about Eclipse Spaces, which provides the ability to publish the current workspace as update site. With that you could easily provide tutorials or blogs. Sounds very interesting … I dont know if required plugins would also be published. So you could e.g. create an
XText editor and publish that example. But with all happiness of easily publishing – don’t forget to think about the licenses of the parts you publish …

Confucius was born in the state of Lu. When he received news that the powerful state of Qi was preparing to attack his homeland, he sent his gifted disciple Zi Gong to talk to the rulers of the surrounding states. Zi Gong went first to the state of Qi and observed to the military generals the flaws in a strategy to attack Lu. He succeeded in persuading the generals to first attack the state of Wu instead. Zi Gong subsequently went to Wu and instigated the king of Wu to attack Qi…Thus, Confucius saved Lu.

From ninth chapter of Chang Duan Jing.

At first sight, it would appear that there should be an excellent cultural fit between the open source movement in the West, and Chinese values. Open source emphasizes collective knowledge and sharing of competence. In China, the loyalty to the group is strong; communist philosophy emphasizes sharing, and Confucian teaching emphasizes the latent potential of the individual to attain skilled judgment from the experience of others.

The use of the web is growing fast in China – exceeding the US and growing much faster than the US – as reported in Forbes. Open source collaboration uses the web as a collaboration platform, so you would also expect this to add to the momentum of open source in China.

The Economic Intelligence Unit recently reported a ratio of 100 jobs for every computer science graduate in China, with this number expected to sky-rocket; and Duke University reported 60,000 computer science graduates from 4-year degree programmes in China in 2004, and 292,000 from 3-year programmes. This huge domestic demand, and huge output, of software developers might lead you to expect further momentum for open source in China.

A recent Eclipse Members meeting noted that China has the most number of the downloads globally (over a recent 18 month period) at 21%, followed by the US at 18%, and both Germany and Japan at 8% each.

So: what is the status of open source in China ?

Well, there are a small number of open source projects in China, but apparently not as many as you might expect. XOOPS (a content management framework) is quite well known and Stephen Walli’s blog contains an interesting presentation by Tiaiwen Jiang, the community leader in China, on the project from the Chinese perspective. Huihoo is a leading open source middleware project including a J2EE implementation, JFox. Qianqian at Harvard Medical School initiated, in 2004, a collaborative project Wen Quan Yi to develop an open source font set for the 70,000 Han characters encoded by Unicode.

An interesting development is the merger of ObjectWeb in France and Orientware in China at the end of 2006, to form OW2. They are sharing open source contributions in a variety of middleware technologies and their deliberations are documented in their Board of Directors minutes…

But, perhaps predictably, the main interest in open source in China is Linux: just google ‘china open source’ and you’ll see! In 2003, China enacted the Software Government Procurement Regulation (SGPR) which excluded foreign companies from the federal software market. As recently as the end of 2005, CIO Magazine was discussing China’s federal commitment to Linux – for multiple reasons at the time, including suspicions of US intelligence agencies “trojan horses” in US proprietary code (remember the B-767 government jet delivered in 2002 to Beijing, but with eavesdropping devices discovered on delivery ?); overcoming WTO IP concerns by promoting open source; localization to the Chinese market; and kick-starting a strong domestic software industry.

But then things changed in 2005, despite CIO Magazine’s analysis above: in trade talks, China laid aside the SGPR in favour of concessions on industries such as textile and colour television.

So now, according to Lou Shouqun of the China OSS Promotion Unit (a non-government organisation) in a recent presentation, the Linux revenues in all China last year (2006) were about 218M RMB (20MЄ), with a market share (by revenue) of just 3%. Other UNIX systems were 52% – the financial services and telecommunications industries in China have heavily used Solaris, AIX and HP/UX, amongst others. Windows was 42%. While the overall market is growing about 10% per annum, Shouqun believes Linux in China is growing faster than the market, albeit from a low base.

Things have changed even more significantly since the 2005 abandonment of the SGPR directive: Microsoft seem to have successfully found favour with the federal authorities. Fortune magazine documented Bill Gates recent summer visit to China; the history of Microsoft in China, and how Microsoft has very successfully wooed the Chinese policy makers creating an apparent “win-win” situation. It is a fascinating article, and I recommend it to you if you haven’t already read it.

So, why are there few committers in China, and apparently meek participation in the global open source, particularly when the number of software professionals in China is rising so rapidly ? Why is it that Windows is far more successful in China than Linux, and is Microsoft’s new strategy truly a “win-win” ?

The urgency to make money is IMHO a national obsession in China. Consumerism, and chasing Western fashions and brands are all-consuming. You must remember that, within the life times of those of us in our 40s or more, very many Chinese were incredibly impoverished to appallingly abysmal standards of life. It really is only in the last twenty years or so that national living standards have consistently dramatically improved – while admitting of course that there remain many challenges and disparities today across the huge country. In my own experience, making money is far more important to many Chinese than political discourse.

If your parents and grandparents have supported you, a single child, through your professional education as a software developer, their expectations (and needs in their senior age) will be that you will support them. Your partner’s parents and grandparents will have similar expectations. An engineer, including a software engineer, is considered a respected professional: many policy makers, senior business managers, and senior party members also have engineering backgrounds. As a software engineer, your own, your family’s, and society’s expectations are all that you will be financially successful.

Can you be truly financially successful in China if you are an open source developer ?

In the West, much of the open source activity in fact is carried out within companies, including Intel, Novell, IBM, Sun and Oracle, amongst others. Foreign companies in China with recognized global brands – such as Microsoft, with its very wealthy founder – are highly attractive as employers, since they not only in general pay well by local standards but also potentially open the possibility of international travel.

But, which foreign companies have so far established software development laboratories in China which contribute to open source development ? IBM have a Linux technology centre in Beijing. Intel announced in 2004 development centres in Beijing, Xi’an, and Guandong to help Chinese companies develop desktop applications for Linux. Oracle promotes its products on Linux in China, including via the Oracle technology centre in Beijing. But these centers appear, on the surface, to be solution centers which promote Linux based application solutions perhaps as a response to the apparent promotion of Linux by the Chinese authorities and prior to the SGPR retraction; rather than development centers actively contributing to globally available open source. I of course am very open to correction, but it would appear that to date only Novell has opened an R&D centre in China specifically for Linux system development. It is also noteworthy that IONA (of which I’m Vice-Chairman) has Chinese committers from its Beijing R&D centre on the Eclipse STP and Apache CXF open source projects.

For domestic Chinese activity, the largest player in open source Linux is Red Flag. It is ambitious to develop into an international player in Asian Linux, and has recently announced a specific initiative. However its current sales revenue is still relatively small, even by Chinese national standards, at just 40M RMB (3.6MЄ).

To the extent that open source development is being conducted in China by foreign companies, with Chinese committers, then arguably these initiatives need to be more openly promoted and publicized to the Chinese software development community. The open source movement in China needs major foreign brand name companies to visibly invest and recruit in China for development of open source.

Well, then how about Chinese open source start-ups ? Are there any budding MySQL ABs, SugarCRMs, or xTuples ? Yes, and I mentioned Huihoo and Red Flag earlier, as examples. But IMHO the open source industry in China is currently fragile and considered so both by potential employees (ie software developers) and, as importantly, Chinese corporate customers. It is frankly easier – and perhaps more socially acceptable with one’s parents – to work for an established organization, particularly if it is a foreign brand.

Let’s now look at that second question I posed above: why is Microsoft now being much more successful in China ? Microsoft seems to have overcome concerns by Chinese policy makers by pro-actively taking a number of steps. The “trojan horse” threat has been overcome by allowing access (and hence inspection) under appropriate conditions to Microsoft source code, and China now has a federal laboratory to do exactly so. Microsoft has been very actively investing in the education sector, including rural classrooms and software engineering universities, and so aligning its investments with the federal desire to strengthen software skills nationwide – Microsoft is training 1,000 instructors and 20,000 software engineers, and offering online courses to another 50,000 engineers. It has worked with the federal authorities in the context of WTO obligations to ensure that more Chinese PCs have legally licensed pre-installed copies of Windows. In turn this is overcoming piracy issues since the pre-installed versions are more current, have less bugs and more features than older copies of Windows. It has also dramatically dropped the price of Windows in China. In summary, President Hu Jintao on a visit to Microsoft said that Bill Gates is a friend to China and the Chinese people: in China, this is an incredible endorsement, and it is difficult to find any analogy in the West for such an important and powerful public ratification.

It would appear that Microsoft is on a roll in China. It is very interesting to reflect on the Microsoft strategy and their execution of it within China, and contrast that to the open source industry and initiatives in China. Of which open source project or company will the President of China publicly endorse as a friend of China and of the Chinese people ? If the open source industry globally is to benefit from China’s rapid development, it is clear that investment – perhaps akin to Microsoft’s commitment to China – will be needed.

It is also very interesting to ponder to what extent the prices Windows customers in the West are paying are being used to subsidise Windows customers in the East. Cross-subsidies are of course not at all a new idea in any industry: I find the Microsoft case interesting because it is part of a much broader initiative.

Let me finish by a hypothesis for policy from the Chinese Government perspective. The war is not about whether Windows or Linux will ultimately win. Both are sufficiently low cost here in China to be usable. Microsoft is generously up-skilling the national software engineering talent pool, and the open source industry is also helping by publishing its source code and inner workings. The war is rather about building a vibrant software industry in China, capable not only of satisfying national needs, but also exporting and becoming a world leader.

The main requirement is excellent application development, on whatever foundation systems and middleware technology are de facto in the global industry (it doesn’t matter which, as long as they are low cost in China). Open source by the Chinese industry – and for the Chinese industry – could play a very significant role indeed. By fostering a national repository of re-useable Chinese application components – with documentation and test suites – written by Chinese developers (in Chinese first, and then maybe English), with a framework put in place and re-enforced by Government policy and investment, then the national industry could be rapidly enhanced.

“Therefore the Master concerns himself with the depths and not the surface, with the fruit and not the flower….When his work is done, the people say ‘Amazing: we did it, all by ourselves!’’”

The Tao Te Ching, by Lao Tzu

This was the basis for an invited keynote I was to give at the OS Summit in Hong Kong at the end of this month: but the conference has now been postponed until sometime in 2008.

Like some other teenagers of my generation in the seventies, one of my hobbies at the time was faithfully constructing scale models from plastic kits from Airfix, Revell, Historex etc. As I developed my skills and interest, I took great pride in adding extra levels of detail – particularly for ship kits. Eventually I converted particular kits so as to model an aircraft, ship or vehicle not directly available as a standard kit, by moulding balsa, crafting acetate sheet and so on – for example building a twin engine, triple tail fin Avro Manchester from the Airfix kit of the successor four engine Lancaster. The monthly Airfix Magazine was a great source of designs and examples, and I still have a shelf full of back copies here in my office at home.

When I was an engineering student, I took a similar interest in building my own, albeit fairly simple, analogue electronic circuits – the usual things such as oscillators, radios, fire/smoke detectors, etc. Of course, as with converting plastic kits, having a good stock of spare parts – scavenged from broken electronic circuits – was useful. I used to keep them sorted into empty margarine tubs on my shelves.

In more recent years, I became more and more involved in computers, from the digital gates and circuits up to microcode, assembler, compiler/interpreter systems and finally full systems and applications. I was fascinated when they became affordable as home systems, and played around adding extension cards and interfaces to my first PC. I was initially really impressed by Dell when I saw their range of alternative pre-configured systems for direct order across the internet. Dell used to have a plant in Bray just a few miles away from my home, and I have bought many systems over the years from them for home, school and charitable use.

But I’ve often thought there must be quite a few people out there like me who aren’t quite happy with a specific system they buy from Dell (or indeed any other supplier). Although I carefully chose a configuration before buying, still afterwards I frequently tailor my purchase further with additional cards, expansions and options. I also wonder whether somebody out there has already built the perfect system for creative digital photography, or multi-screen home media management, or whatever. Or whether anyone would be interested if I could publish my own designs and configurations. Or whether any configurations would be commercially interesting, for example for specialist systems such as creative digital media design.

Reflecting on my fads above, the common aspects are a desire to build interesting variations of basic designs; from additional parts either built myself, or stock-piled somewhere (not necessarily in margarine tubs!), or purchased elsewhere; to look at designs published by others, and perhaps even to publish my own.

Well, that’s what Cloudsmith is doing for the world of software. One way to think of Cloudsmith is as a derivation of Dell’s world, for software. You can browse an online catalogue at Cloudsmith and see what “distros” – configurations – of software components are available. If you want to chose any particular one, you click it and it “materializes” – ie downloads its parts, and then automatically assembles them together – onto your machine. And like getting a system from Dell, the different components of your configuration will usually come from different “repos” (repositories – think Dell sub-suppliers) around the world. So rather than just a single download of a binary file, a materialization will usually automatically fetch multiple files, from multiple places, and assemble them automatically for you on your machine as a complete system. The distro is thus “virtual”: it is not monolithic (like an old download) and its parts are not hosted at Cloudsmith (any internet repo can contribute).

If you like what you find, you can send others the “Cloudlink” you used. When they click it, the same components will materialize from the cloud of components available in multiple repos across the internet, and assemble and install on their machine too. So, you can materialize something specific without having to yourself connect to the Cloudsmith site and without having to search the online catalogue for it – just get the Cloudlink for it from someone else via an email or blog or whatever, and the magic will happen when you click.

Whats kinda nice about the Redhats of this world is that they pre-assemble a large collection of software components for you as a full Linux system. Whats nice about Cloudsmith is that if you build your own interesting set of software components, then you can publish that configuration at Cloudsmith, and reliably make it available for other people to use. It need not of course be as complex as a full operating system: just a nice application or tool or subsystem for a particular use which you have put together, using your own design or as an interesting derivation from somebody else’s published distro. You can publish novel and interesting things that you’ve done, for others to take a look at and possibly use – no longer are you or they tied just to the pre-assembled varieties which the software equivalents of Dell provide. And if you have components from a new repo which Cloudsmith does not yet know about, you yourself can simply add that repo to Cloudsmith’s map of the world – it is the Cloudsmith software equivalent to introducing Dell to a new sub-supplier.

Why hasn’t a service like Cloudsmith existed up to now ? One reason is that although there has been a proliferation of useful re-useable software components developed and published around the world in many online repos, there has equally been a proliferation of software version control, make and build systems. It is difficult to justify re-engineering everything globally to use the same version control and build technology. IMHO it is impossible to impose one build or version control technology, or one IDE, on everyone: and it is short-sighted to expect everyone else to use the one that you happen to think is best. However, with some thought and help, it is possible for systems like Cloudsmith to automatically, on the fly, interpret the meta-information in all these version control and build system technologies, and thus automatically derive a global perspective and capability across many different repos.

But how do you know that somebody else’s published configuration is any good, and trustworthy? Well how does a plastic modeler or an electronics hobbyist know that somebody else’s design is any good ? Part of the answer is the popularity of the design – how many others are using it, and what do they say about it – and part of it is the reputation of the designer. Social community sites on the internet in general work because most people are ethical, and many people can watch, observe, comment and mend where necessary information which is incorrect.

Assembling and publishing new configurations of components is fun. Is it however just for hobbyists and software hot-rodding ? I believe that many commercial organizations will find interest in discovering and contributing to software configurations and assemblies that add real value in their business domains. I think enterprise managers will find reassurance not only in the popularity of certain distros, but also because they can be reassured that precisely the same distro (and bill of materials) is reliably installed on every machine under their control, and because component updates can be notified and controlled. I think vendor product managers and technology strategists will find interest in the concept of being able to build their own private “cloudspaces” for their licensed customer communities, to manage and distribute software in a controllable way.

Now, let me put my hand up and confess that all of this isn’t yet as smooth as we would like at Cloudsmith just right at the moment: it is an alpha version for the community to experiment with and give us feedback on. The materialization wizard, the wizard to build a cloudlink are IMHO both pretty good and straightforward to use. Private and public cloudspaces are ready to use. But today, the publishing wizard and the wizard to register a new repo are IMHO a bit clunky: we’re improving them right now and we expect better versions within a couple of weeks. Our online documentation is being improved. However, if you’re interested, contact us and we’ll talk you through online and get you involved in the Cloudsmith community.

Part of the dilemma with the global software community today is that there are so so many open source repositories and projects. I guess amongst the best known community owned initiatives are Apache, eclipse and SourceForge. But there are very many others, as a quick Google will reveal: for example HotScripts, NetLib, RubyForge, and CodeHaus. Most of course are primarily specific to a particular technology but, nevertheless, large scale projects frequently span several technologies.

How can you easily and quickly put together your own project which relies on software components from several different source code repositories ? If your project works out to be cool, how can you easily and quickly share it with other people who may be interested in it ? Now, Linux distribution vendors such as redhat and ubuntu do a great job packaging together a large set of Linux based components – in ubuntu’s case, over 16,000 according to their web site. Great. But what if I myself want to package together some components which I found useful, in a configuration which those Linux distro-vendors don’t currently offer ?

The problem is generally exacerbated if the components which you want to select come from different repositories and projects, having different repository and dependency technologies – CVS, Maven, Subversion, Ant&Ivy, whatever. The problem is also made more difficult because of version dependencies on your installation environment and other components.

In an earlier blog entry, I wrote how assembling software today is reminiscent of childhood experiences with toy building kits, such as Lego. However, there is some danger that in assembling software components from different sources, you end up with a large pile of individual Lego bricks on the floor! There should be a better way of defining and tracking component assemblies across technologies and repositories.

I was excited to see that last week at eclipsecon, AOL’s Lucas McGregor announced the Eclipse Spaces technology project. I guess the essential concept is simple: just in the same way that MySpace provides online storage for friends to share videos and photos and news and events, there is now an opportunity for developers to informally and easily share online interesting component assemblies with one another. The eclipse project process of course offers a formal, process driven approach to collaboratively developing new software between companies, and I guess Apache has something similarly formal between individuals. But Eclipse Spaces suggests something less rigid, in which components which do not necessarily conform to the precise terms of the eclipse.org license, can be informally assembled as personal and individual projects, especially if they come from other repositories outside of eclipse. And since Xdrive is free, this would be a wonderful added service for AOL to provide to the global software community.

I was also delighted to see that Eclipse Spaces is one application of Buckminister’s publishing and materialisation technology for software component assemblies. Henrik Lindberg, Thomas Hallgren and Filip Hrbek are included as the initial set of committers to Eclipse Spaces, all three from Cloudsmith. I’m also delighted to see Malcolm Sparks on the list too, an old IONAian who joined IONA when Ejbhome was acquired in 1999. And Bjorn Freeman-Benson and Ward Cunningham, both from the eclipse foundation. And finally Dennis O’Flynn, the eclipse Corona project lead from Compuware. Looks like a very solid team.

As a global community, we need a far simpler and more elegant way to quickly build, publish, maintain, acquire, materialise and track interesting aggregations of software components, even if they are cross technology. I’ll watch Eclipse Spaces with interest