"An integral part of our business strategy is to fund FreeBSD developers to continually improve and refine FreeBSD. Our internal infrastructure and services are based on FreeBSD, allowing us to easily identify areas in FreeBSD that need improvement. The BSD license allows us to both use and contribute code freely, and allows our customers to do the same. FreeBSD has benefited from years of open sharing and collaboration which have resulted in a stable, mature, high performance operating system."
Josh Paetzel, Director of Information Technology, iXsystems
The processes that open source projects use to produce new work and maintain the quality of their code base is a subject that comes up infrequently in discussions of open source. One reason for this is that engineers and programmers are usually loathe to deal with issues that are not directly related to the piece of code or technology that they are working on.
Successful businesses know that good processes lead to continued success. The attributes that attract a business to an open source project are stability, reliability, and longevity. Stability gives a business the confidence to invest time into developing products on the project's platform, safe in the knowledge that the next incremental step in development won't be torpedoed by some unforeseen change. Reliability is often not associated with open source and many projects are perceived as being too cutting edge for a business to build upon. Longevity is of value as many businesses are inherently conservative in their approaches, attempting to reduce the risks of adopting any technique or technology. One way to reduce risk is to work with an open source project that has a proven track record of delivering quality products, on schedule.
This article attempts to dispel the myth of the perceived tension between a formally run business and the apparently less formally run open source projects with which a business interacts. We describe how one particular open source project has developed processes which provide its users, customers, and partners with a product that is stable, reliable, and long lived.
The FreeBSD Project develops and supports a complete operating system and set of development tools. The resulting software meets the needs of a wide ranging user community which encompasses software developers, systems integrators, appliance developers, embedded systems designers, Internet service providers (ISPs), web hosting companies and just about anyone who needs a solid, Unix-like, software development and deployment environment. While it is possible to develop a rudimentary operating system for a small community of hobbyists with few rules or regulations, providing a system with the depth and breadth of FreeBSD has lead the FreeBSD community to develop a set of internal processes. These processes serve the community as well as the community's customers, many of whom build their products and businesses on top of FreeBSD.
When discussing an open source project, stability is most relevant to the organization of the project itself. The process by which contributors add to the system, how software releases are made, and how the project governs itself all contribute to whether or not the project as a whole is stable and determines its longevity. The FreeBSD Project has a well defined process by which contributions, and contributors, become a part of the Project itself.
The most common reason for adding code or documentation to the system is to address a user need that is not met by the currently released software. This need may be as simple as a software patch or as complex as a new device driver, network protocol, or compiler. Most users can not directly change the code and documentation in the Project's repositories and need to make their need known to the community. FreeBSD has several groups of people who work directly on the source and documentation that make up the Project, including the source code, third party software (known as ports) and documentation. A committer is someone who has a commit bit, which gives them the ability to commit changes to the Project repository. The process by which someone becomes a member of the community is an intrinsic part of the stability of the project itself.
Let's take a simple example. A developer who is using FreeBSD needs something added to the system. The developer joins a few FreeBSD mailing lists to mention their need and that they are willing to contribute code in order to get the new feature added. The developer becomes known to some segment of the community. The developer sends patches over the mailing lists and one or more Project members check and commit the contributed changes into the system. The developer continues to work with the community until at some point, quite naturally and with little fanfare, the people who have been committing this person's changes decide that they should have direct access to the source code. The developer is "proposed for a commit bit" when someone from the Project sends an email to the Project's core team asking that the developer be made a part of the Project.
The core team is a group of Project members who are periodically elected to help organize the Project. One of the responsibilities of the core team is to accept or reject new members. Because every developer is known by some segment of the community before they are proposed, there are rarely instances of someone being refused a commit bit. One of the important measures of a new committer has to do with temperament. The Project promotes excellent software developers and the community wants members that have the social skills to "play well with others." Developers who are unable to get along with others are not proposed for commit bits, no matter how brilliant their code, because everyone on the Project knows that poisonous programmers are not the people they want to deal with on a daily basis.
Being accepted as a committer does not end with the commit bit being granted. A new committer is given a mentor, usually the person who proposed them for the commit bit, through whom they must pass all their proposed changes. The mentor reviews all changes, ensuring they match the Project's development processes. The FreeBSD Project has a consistent coding and documentation style and the mentorship period is used to pass this knowledge on to the mentee. After some amount of time, the mentee is freed from mentorship by the mentor. This process which governs how a user or outside contributor becomes a member of the Project, with its multiple levels of vetting and its open and accountable nature, has contributed greatly to the Project's stability.
Release Engineering Process
Another process which contributes to the stability of FreeBSD as a platform for development is the Project's release and branch management process. What can be changed between and within releases is governed by a set of rules. There are always two major branches of development in the source code control system: CURRENT and STABLE. All changes go to CURRENT first, meaning that this development branch contains code that is still being tested. The STABLE branch is the one that the Project expects downstream entities, such as systems integrators and appliance developers, to base their products on. If a new feature in CURRENT is also needed in STABLE, that change is merged from CURRENT (MFC'ed) into STABLE. Ensuring that all new code goes into CURRENT first contributes to the stability of the overall system. New changes are fully tested within CURRENT without negatively impacting the broader community.
When a major software release is made, the version numbers for CURRENT and STABLE change. For example, with the release of version 8.0, STABLE is now the 8 branch and CURRENT is the 9 branch. Once a major release has been made, all of the system calls and kernel application programming interfaces (API) will not change within that branch. Any change that would cause binary incompatibility must go into CURRENT and will never be placed into a released branch. As bug fixes and non-destabilizing improvements are made to the system, minor version releases are made. For example, the 6 branch had releases numbered 6.0 through 6.4. The FreeBSD Project takes its commitment to API and kernel programming interface stability seriously, meaning that any code developed within a set of release branches will run on any future release of the same branch.
Releases are officially supported by the Project for two years from their point of release. Due to the pace of development, the only changes being made to an older branch are critical bug fixes, such as security advisories and bugs which might cause system instability. If a branch is still heavily used by systems vendors and other businesses, there is an extended end of life (EOL) process whereby that branch continues to receive bug fixes after the normal two year period has ended.
Code Tracking Process
Systems grow more reliable if they have the support of appropriate infrastructure and processes. The FreeBSD Project has, over time, acquired the resources to provide the community with access to a well maintained source code and documentation management system, as well as build farms and testing facilities. The entire code history of the project can be viewed on line, or checked out from the project's source code repository. All of the changes to the system, going back to the initial import of code from the Berkeley Software Distribution, have been recorded and preserved.
Build clusters, donated by Sentex Data Communications, run an application known as tinderbox. The tinderbox system downloads the operating system code every day, or on some faster machines every few hours, and rebuilds that code. If the build fails, an email is sent to the mailing lists so that the offending problem can be found and fixed quickly. The tinderbox system ensures that development does not move ahead with partially broken code.
No software system can exist for any period of time without being subject to the occasional bug. The FreeBSD project maintains a bug tracking database that is open and accessible to the entire Internet so that any user may record a problem they find with the system. Project members are able to read and respond to problems in the database and there is a bug busting team to make sure that bugs do not get stale from lack of attention.
Security issues are special types of bugs, and these are often treated outside the normal bug tracking system so that they do not become 0 day exploits. The Project has a security officer who is the main point of contact for all security related issues. The security officer works with a security team to investigate security issues and to properly record, report and repair vulnerabilities found in released systems. The policy of the FreeBSD Project favours full disclosure of vulnerability information after a reasonable delay to permit the safe analysis and correction of an issue, testing of the fix, and coordination with other affected parties. The coordination aspect is important to vendors who use the system in their products. While a project should never hide security issues, it has a responsibility to make sure that effected parties are able to properly patch their systems before the vulnerability becomes more widely known.
Other Success Factors
The FreeBSD Project has been producing releases for over fifteen years. What factors have contributed to its longevity? One of the key factors in the success of the FreeBSD Project has been its inclusive and democratic nature. Unlike many other open source projects, FreeBSD has never been owned or controlled by a single individual. The core team is made up of 9 members of the FreeBSD community, chosen through a voting process where anyone who has made a commit to the system in the last year is eligible to vote. The core team wields little direct power: it can not hire or fire project members, has no budget, and manages no schedule.
The main responsibility of core is to act as a focal point for the non-technical aspects of the Project such as thinking strategically about the Project, vetting new developers, tracking commit bits which might be retired or reactivated, acting as a public face for the Project, and helping to work out any non-technical problems that may come up in the day to day running of a large project. The core team also creates "hats" for significant areas of responsibilities within the Project. When a new area of responsibility crops up, core looks within the community to identify the person, or people, who would be best suited to wearing that particular hat. While core can suggest to someone that they put on a hat, they have no ability to force anyone to do so. Much as new developers are part of the community by the time they are proposed for a commit bit, the people who wind up with new hats were usually already doing the job in an unnamed and unrecognized capacity. Giving a person a hat is usually more about recognition than management.
A significant challenge for many open source projects is the need for a public, legally incorporated entity in order to work with vendors, sign contracts, retain copyrights and generally interact with the rest of corporate culture. The FreeBSD Foundation fulfills this role for the FreeBSD Project, acting as a legal entity that can perform all of the duties necessary for the Project to interact with both companies and individuals.
One important area that has lead to the longevity of the Project has been its commitment, from the very beginning, to top notch documentation. While many projects expect new users and integrators to "read the source", the FreeBSD Project has an entire team that works exclusively on documenting the Project, both its internal rules as well as its outward facing manual pages, user guides, FAQs (frequently asked questions) and the like. Documentation is currently produced in twenty different languages including non-Western languages such as Mandarin, Japanese and Korean. The documentation team maintains documentation both outside and within the source tree, such as the manual pages which are available on every installed system. Documentation gives the project coherence to those who build systems on top of it. Without it, the Project would not have been usable to enough people to survive for very long.
An operating system and development tools are not sufficient for all the every day needs of the users of FreeBSD. A tremendous number of external programs are made usable on FreeBSD by the members of the ports team. Like the documentation and the source teams, the ports team is self-organizing and is coordinated by the people who wear the ports hats. The ports team now manages over 20,000 software ports which can be installed on FreeBSD, giving the user community access to a full range of applications, from email, to web browsing, to specialized libraries that help FreeBSD system integrators to build products.
The FreeBSD project provides a stable and reliable platform which systems integrators, ISPs and developers can use to produce their own products. This stability and reliability has been achieved by the careful selection and promotion of an effective set of processes covering all aspects of the Project, including source code control, release management, who can be a Project member, and how the Project itself is governed.
The processes described in this article can act as a guide for businesses who are looking to evaluate the stability of an open source project they are considering interacting with. Members of other open source projects may also find practices that they can integrate into their own processes.