"The industry has learned by experience that the only software-related standards to fully achieve [their] goals are those which not only permit but encourage open source implementations. Open source implementations are a quality and honesty check for any open standard that might be implemented in software..."
Many open source projects implement open standards. We interviewed five developers who implemented different open standards in open source projects to find out how much interplay there was between implementors and standards developers and how important this communication was as they programmed the details of the specifications. Our somewhat unexpected finding was that developers preferred to work from the printed specification, separate from the standards source. When asked for a reason, most reported that resource constraints prevented them from writing code and specifications at the same time; another factor was the satisfaction that comes from working independently. Most of the developers we spoke to were more than halfway through their development before they even considered reporting specification problems to the source organization. Although this speaks well for the overall quality of computer industry specifications, it also means that feedback from open source developers is not getting back to the specification's authors.
Open Source and Open Standards
Open source and open standards are not the same thing. Open source refers to software whose source code is freely available to users for reference, debugging, modification, and/or extension. Open standards are, typically, specifications: formal descriptions of software or software interfaces. Open standards may have reference implementations, but the description in the formal standard typically takes precedence over the behaviour of a reference implementation.
It is interesting that the two phrases use the word "open" so differently: For open source, open means that the source code must be distributed with every copy of an executable application and every recipient must be allowed to modify and distribute the source code freely to subsequent users. In open standards, open signifies that the standards process is open to participation and that the completed standards are available to everyone. Working documents and drafts are typically kept private to the issuing organization's members, and there may be reasonable conditions for participation such as membership fees, but any person or company may participate as a member at a meaningful level. Many standards organizations give copies of their standards away for free and the right to implement a standard is typically also free and, if not, is available on fair and equitable terms.
In computing, standards enable portability and interoperability. Portability includes: i) application code ports between operating systems or compilers; ii) middleware architecture ports between systems--its source code may also port but not nearly to the same extent; and iii) a developer's skills porting from one platform to another. Each time something ports, someone saves time and money. Interoperability also pays, motivating companies to get together and write standards.
On examination, we see that the two approaches are complementary:
- open standards need implementations to provide: i) confirmation of their suitability; ii) a market presence; and iii) feedback from implementors and users
- open source development projects need guidance and direction regarding their interfaces for interoperability and portability
Each benefits from the products of the other and this synergy can be found in many projects.
We interviewed developers of open source tools based on open standards to investigate the interplay between open source development projects and open standards adoption and maintenance. We had expected that communication would flow liberally in both directions, but found that was not the case. Most of the development projects preferred to regard the specifications as read-only documents and were able to code productively straight from the printout. Our reports from five different projects and our conclusions follow.
Jason Robbins started the ArgoUML project to build a modeling tool based on the Object Management Group's (OMG) standard unified modeling language (UML). He has since moved on to other projects, but stayed with this and a number of other standards-based OSS projects long enough to give us some good feedback. Jason considers this project to be a consumer of standards, and points out that a well-written and well-used standard will have thousands more readers and implementors than authors. The usual case is a group isolated from the specification authors, coding along without a break unless they encounter an inconsistency or ambiguity of such consequence that they contact the issuing organization to have it resolved. Interpretation of words or phrases in OMG specifications is one of the largest sources of filed issues. Jason was the first person to point out something that we ultimately heard from many directions.
Any project that consumes specifications but which doesn't help write them must either accept the limitations of the standards as they are or delay implementing the affected part of the project until the problem is resolved by the issuing organization. Anyone may submit an issue about a specification to OMG, but it typically takes weeks or even months for these to be resolved. This is an eternity in open source development time. OMG has an accelerated resolution process that, for a restricted class of urgent issues, delivers a resolution in only two weeks. Rarely used, this process still lacks the quick response time required by most open source projects.
Marko Boger discovered the ArgoUML project after Jason Robbins had completed his work and moved on. It implemented only four diagrams and Marko needed another five for his work, so he gathered a group of students and started to code. As with our other example developers, the crew worked straight from the specification without contacting any of its authors.
One of the most interesting parts of the project was the building of a MOF-compliant (meta-object facility) repository from the metamodel. In a convincing test-case for code re-use based on standards, they convinced Novosoft to open up the source for its MOF-compliant repository toolxix, allowing them to update it to UML 1.3 and XMI 1.0. Later they switched to MDR, a metadata repository from Sun's open source NetBeans project, enabling a move to UML 1.4 and XMI 1.2. This provided a sophisticated repository system, dynamically updatable from one version of the metamodel to the next, or to a specialized profile. Without an open standard for the repository structure and interfaces, the straightforward repository switch would have been difficult at best.
Partway through the work, they attended an academic conference where they met many of the authors of the specifications they were implementing. Direct contact continued after the conference, as the group stayed in touch as coding continued. Marko reports that this did in fact make a difference, but not in their coding to existing specifications. Instead, it made them more aware of parts of the specification they had overlooked and of new specifications under development.
Around this time, the ArgoUML team split: one group continued the open source development while the other, led by Marko, took a snapshot of the BSD-licensed source as the basis for a company which they named Gentleware. Using a conventional business model, they expanded ArgoUML into a set of modeling tools. They also joined OMG, submitted to the UML 2.0 diagram interchange specification, and chaired the committee finalizing the specification. This is an excellent example of the interplay between open source, open standards, and proprietary extensions.
Matthias Bohlen's AndroMDA implements OMG's Model Driven Architecture (MDA), an ambitious achievement for an open source application. Matthias, a freelance consultant specializing in MDA, decided to write the first version of AndroMDA to show his customers its benefits.
OMG's MDA process starts with a Platform-Independent Model (PIM) of the target application's business functionality and behaviour, typically designed in UML. Two transformations carry this through to a coded application which is ready to make, deploy, and run. The first transformation enhances the PIM meta-objects with behaviour that enables the transformation to the meta-objects of the Platform-Specific Model (PSM). The second transformation takes the PSM meta-objects to code, build script, interface definitions, and whatever other artifacts are needed. AndroMDA uses an open source template engine for this task. The two steps are automated either partially or totally, depending on the application domain and situation. The AndroMDA user finally codes the real business logic mostly by hand, depending on the application domain and how much of the application logic calls for known patterns or uses standard UML profiles. In the process, the different steps, transformations, and code generation may be performed by one or more tools which are not dictated by the specification. XMI, OMG's standard format for model transfer, is used to transfer the various models between tools, enabling tool interoperability.
AndroMDA is rapidly becoming a very flexible and complete implementation of the MDA. A community of highly skilled developers and users has formed around the tool, and taken the product far beyond its first implementation.
Unlike ArgoUML and MICO, AndroMDA has benefited from communication with OMG during development, but not through its prime contributors. Matthias points out that they, like the other open source developers we've described, don't have time to develop code and standards simultaneously and must rely on others to carry their messages. Martin Matula of Sun Microsystems' NetBeans project and Marko Boger participate actively in OMG meetings. They inform the other contributors to the AndroMDA code base, who are not members of OMG, and who do not subscribe to any of the group's email lists.
AndroMDA doesn't mind reacting to changes in the specifications, a task Matthias described as "easy". He claims that it's more difficult for users to adjust than for builders. AndroMDA will use an abstraction layer to hide the differences between UML 1.X and 2.0 where they can, but users will have to adjust to new elements and capabilities, and will have to revise any cartridges they have written in order to use them with the upgraded tool version.
Arno Puder started MICO when he was a graduate student at the University of California at Berkeley. He originally wanted to develop a textbook and course that would teach coding of network middleware. Because design was not part of the process, he looked for a pre-defined system and found it in the Common Object Requesting Broker Architecture (CORBA) specifications. When he started, he was naive about the ways of both open source and open standards. He obtained his copies of CORBA without studying their origin and named his project MICO for "Mini-CORBA". Keeping with the just-in-time philosophy, Arno brushed up on open source licensing as he prepared to post his early code on the web. After all, his goal was to write a textbook and the code was only the means to this end. After posting his CORBA implementation on the web, he found that it was widely used as middleware in software projects whose scope went far beyond the teaching tool originally envisioned. The MICO crew made it available under the original GPL, and then later under the LGPL at some users' requests, and found that its popularity expanded even further. The developers built it into a full implementation of the then-current CORBA specification, forcing a change of its name's significance from the no-longer-true "Mini-CORBA" to the recursive "MICO Is CORBA".
Arno and the other coders on the MICO project worked straight from the OMG documents. They were nearly finished development before they discovered how to submit issues to OMG through our website, meaning they only used the specifications for guidance. How well did they do, in spite of this isolation? Very well, as we'll see.
As the MICO crew was finishing its work, The Open Group (TOG) was developing a CORBA test suite under contract to OMG. The agreement called for three ORBs to be certified and, since few vendors were ready for testing, OMG and TOG looked for open source ORBs. MICO was one of the lucky winners of a free certification, if it could pass the TOG tests. They ran the tests and, after resolving a few issues, qualified for certification. Arno traveled to the next OMG member meeting to receive his certificate and, for the first time, meet the people who wrote the specification he had implemented and find out how the technology adoption process worked.
This was after all of the major coding of MICO was complete, and far too late for the contact to help with coding decisions. Still, Arno doesn't feel that they needed much help--after all, they completed an ORB that passed the compliance test without any contact. Arno continued to attend OMG meetings on behalf of Deutsche Telekom, his new employer, but his work centered on CORBA open source testing (COST) and other projects separate from the basic specifications coded into MICO.
Working at the University of Frankfort, Frank Pilhofer started on MICO by upgrading it to include the Portable Object Adapter (POA) and Objects by Value (OBV) which were both added to CORBA around 1998. Both are substantial and complex specifications with many interworking parts: POA defines the allocation and deallocation of resources as object instances are activated and deactivated, while OBV provides an object-like programming element.
Frank reports that he didn't encounter any serious issues as he implemented the POA and OBV specifications. He and the rest of the MICO crew used Internet sources such as the news group comp.object.corba to resolve questions they couldn't handle themselves, but preferred the independent feeling that came from handling most of these problems without asking outsiders.
Interoperability with other ORBs and language-mapping consistency provided more interesting moments than did pure implementation. CORBA is about interoperability, but open source projects don't have big budgets for software testing. They mainly tested against other free Open Source ORBs: TAO and OmniORB. One incompatibility, encoding of valuetypes, was enough of a problem that they submitted it to OMG as an issue to be resolved by the CORBA Revision Task Force.
Issue resolution is a process that looks entirely different from the inside than from the outside, according to Frank who has worked this process from both ends. From the outside, all you want is a resolution so you can proceed. From the inside, you're faced with resolving an ambiguity in a specification that has already been built into commercial and open source products used by thousands of people. It's likely that a resolution will seriously affect these products, and representatives usually sit on the Task Force that is going to adopt the resolution. Seen in this light, the several weeks or months that it takes to resolve a set of issues takes on a much more reasonable look.
Frank and the MICO crew continued to submit issues as they worked, but stopped expecting resolution to come quickly enough to be built into their current code. Instead, they took stopgap measures to get by, realizing that they'd have to make a suitable change when the issues were eventually resolved.
CORBA Component Model
As Frank completed his work, Arno arranged for funding for a related coding project, the CORBA Component Model (CCM). Originally written by several major system vendors, the CCM needed support in its finalization phase in order to complete the full course of adoption and enter the OMG specification book. Unfortunately, the market had focused almost exclusively on Enterprise JavaBeans (EJBs) and the multi-language CCM superset with its additional features was being ignored. Alcatel was willing to fund Frank's work and he found a few willing helpers when he showed up at OMG meetings ready to work. Alcatel funding covered Frank's coding time and his attendance at meetings. They expected him to feed his experience back and see that a finalized CCM was adopted at the end of the process.
This is our target case: an open source project done by a member of a specifications consortium, in full communication with its other members. However, it lacks many components that make open source special, such as the open organization with its large, widely spread body of voluntary contributors. Frank's MICO CCM implementation is available as open source under the GPL/LGPL, but it was written by a single developer hired by an enterprise to do a particular project. The availability of code as open source was incidental to the rest of the project, which would have run identically if the code had been kept proprietary. The resulting CCM specification benefits from an open source developer, but not from the community of contributors typically associated with the phrase.
Specifications flow from consortia, providing direction for many open source projects, proprietary products, and in-house software development. Because the specifications are downloaded anonymously, no reverse communication channel opens from specification user to supplier. The sources we interviewed for this paper all agreed that communication along this channel will be sparse because developers are too busy to produce code and comment on specifications at the same time.
We don't agree that this is the most productive way for open source projects and open standards organizations to work together. As sophisticated and experienced implementors, open source developers clearly have much to contribute to the design and specification of the applications they ultimately produce, and the industry will benefit from their early and deep involvement in the standards process. We also believe that standards organizations need to recognize and account for open source implementations, a task which requires input from the open source community. Recent cooperation between OMG and Eclipse shows that this can work in a formal way and we look forward to more examples in the future.
Open source projects produce another output besides feedback: well-written applications that faithfully implement the specifications on which they are based. It says something good about a standard, and in turn about the organization that produced it, every time an outside project takes a specification and turns it into an application. Consortia should listen to this message and be pleased.
This article is a shortened version of the OMG publication "Open Source and Open Standards: Working Together for Effective Software Development and Distribution". The full version is available from the OMG website.