"We intend to integrate free, positive changes from whatever sources will provide them, providing they are well thought-out and increase the usability of the system...Above all, we hope to create a stable and accessible system, and to be responsive to the needs and desires of NetBSD users, because it is for and because of them that NetBSD exists."

Announcement of first release of NetBSD

The NetBSD Project is one of the oldest modern open source software projects. It provides an operating system that runs on over 50 hardware architectures (also called ports), including the IBM PC, Motorola PowerPC, and Sun UltraSPARC machines. Founded in May of 1993, the project has supported the operating system's active development and managed contributions from thousands of individuals.

Prior to the New York City BSD Users Group Conference held in October, 2008, NetBSD developers from across the globe held a face to face meeting for planning and problem solving. Four developers from Sweden, Canada, the US, and Slovakia took a few minutes to think about how the NetBSD community has evolved over the past fifteen years. This article summarizes those perspectives and provides insight into how an open source community maintains development momentum while managing contributions from a large number of volunteers with varying skill levels from across the globe.

Anders Magnusson

Anders is the port maintainer for the port of NetBSD to DEC VAX computers. NetBSD/vax was the first free operating system that ran on the VAX series of computers, and by far runs on the largest number of models of VAX. Within the NetBSD project, there is one port maintainer or a team of two co-maintainers per type of machine that the NetBSD operating system supports. A port's maintainer is the final authority about changes being made to that port, and is responsible for attempting to fix problems with the port and for integrating improvements into the port in a timely manner.

According to Anders, the NetBSD project has always had a focus on the developer. You might say 'by developers, for developers'. Many of the people involved in the project, especially early on, were hardcore kernel hackers. That trend may be for good or for bad. It does have the advantage of attracting people who are very committed to the project.

Since other developers tend to share common interests, discussions are simplified as everyone involved is likely to use a similar set of criteria for evaluating the available options. This kind of like-minded community has the advantage of being very attractive to people with a similar mindset. The disadvantage is that the project may not attract enough people capable of diversifying the skill sets and viewpoints within the project. As an example of the lack of diversity, there have been very few members of the project who were interested in spending time promoting NetBSD, until several years after its first release.

NetBSD has always had a focus on clean, well designed, understandable code. This focus was inherent in the codebase, which started with the public releases of the Unix operating system developed by the Computer Science Research Group (CSRG) at the University of California at Berkeley, as well as 386BSD which was developed by Bill and Lynne Jolitz. The BSD codebase embodied principles from academia such as good choices of algorithms and work that had undergone significant peer review. The quality of the work came from the dedication of the CSRG team, working on computers with limited resources, finding solutions to computing problems that had not been tackled before, and building maintainable, readable source code as a team. As with the developer mindset, having a common desire to maintain and enhance attributes in the code like quality, readability, comprehensibility, portability (to different compilers and target platforms) means that you will attract people who are interested in the same things.

Although it's pretty common these days for new open source projects to incorporate a non-profit foundation with a formal Board of Directors early on, NetBSD did not create its Foundation until around 2000. Until then, the project was driven primarily by the technical group called 'core'. The core membership changes over time, but it is generally a group of about five developers who are considered to have a broad range of knowledge and good skills in design and foresight. The core developers are still part-time volunteers, who spend at least ten hours a week on NetBSD. Having the Board in place has allowed things to get done that couldn't have happened before, but it seems that not having one from the start wasn't detrimental to the NetBSD community.

The project had its early successes, such as having a fully 64-bit clean operating system as early as 1994. It has also learned from challenges, such as balancing the demand for new hardware support with the desire to implement code in a well designed, rather than fast and sloppy fashion. Failing to do this could have cost the project its reputation for clean code.

Having a concurrent versions system (CVS) for a project's developers to share, then later open for the public, may seem commonplace today. In 1993, having that commitment meant educating developers about version control systems and figuring out CVS's quirks. It means that NetBSD has a history of every code change since its inception, which many projects from that era do not.

NetBSD has a mix of younger and older developers, providing a broad range of experience. There is a mix of new ideas and people who have seen new ideas fail and who are able to understand what has changed as what was tried before and didn't work, may well work today. By older, I mean we have a number of developers in their 50s and 60s. Younger developers are attracted to the opportunity of learning from the experience of the older developers.

David Maxwell

David has been developing software for over twenty years and is a former security officer for the NetBSD Project. He states that:

My first exposure to NetBSD was at a new job, with a system running NetBSD 0.8 in production use. I already had a background in Unix that went back almost a decade, so NetBSD had to live up to my expectations for reliability and consistency in a Unix system.

The atmosphere on the NetBSD mailing lists was pleasant, and the signal to noise ratio of the discussions was very high. The primary developers were clearly knowledgeable people from whom I could learn, and that was very appealing to me.

From a technical point of view, the source code for the system was simply amazing. I saw how I could learn from the examples of so many good programming techniques which included layering, reuse, clean code, up to date documentation, and version control. A lot of these methods came from the work of developers at Berkeley, pre-dating NetBSD. The codebase was well tested, time-honed code. The ideas being implemented weren't just theoretical, they had already been in production use for years.

I saw new features being added by the project, showing innovative solutions to problems that I didn't yet have the background to anticipate. Even if I managed to learn everything I could from the source code, there were still new lessons being created on an ongoing basis.

Centralized version control and development of both kernel and userland code taught me about real-world design and maintenance of interfaces between code. The construction of a source tree that builds cleanly for many different platforms also impressed me. I had seen userland code that used configure scripts to adapt to the differences between Unix systems, but here was kernel code that adapted to the differences between CPUs and memory architectures. Even today, after fifteen years of using NetBSD, there are still new things for me to learn.

As previously mentioned, the atmosphere within the project was very good right from the beginning. This was important because changing culture is always hard. Creating an environment that is right in the first place helps, but you can't always be certain what's right in the first place as the community is always learning. NetBSD made many good choices at the beginning, but some percentage of that was simply good luck.

One thing that amuses me regularly is the number of people who have the impression that NetBSD is stagnant, or dying. Those people obviously aren't subscribed to the source-changes mailing list, or they would see new code being committed every day. NetBSD continues to attract new developers and users, and continues to innovate. The NetBSD community-building story isn't finished yet.

Lubomir Sedlacik

Lubomir has been a NetBSD developer since 2002, a former member of the pkgsrc security team, and a current member of the pkgsrc release engineering team. pkgsrc is a framework for building third-party software on NetBSD and other UNIX-like systems. Lubomir speaks:

I started participating in NetBSD after being frustrated by other project communities where some users constantly ask questions but never make an effort to understand the answers.

When I first found NetBSD, I developed an appreciation for NetBSD as a bloat-free, cleanly coded, complete operating system, and started using it on my laptop. Some of the criteria Anders mentioned as goals clearly were a draw for me to use, and then participate in, the project.NetBSD was better thought out and designed when compared to other operating systems I had tried. I saw a high concentration of smart people with lots of experience working on the system and actively participating on the mailing lists. While the rate of code contribution by an individual may vary over time, many of those same smart and experienced people still participate in the project by offering feedback and advice. The people who remain tend to be those who don't have trouble learning for themselves. I was drawn to the system because of the easy to understand, well designed code. I particularly remember the design paper for the rc.d system (which does initialization at boot time), and how well thought out it was.

Individuals need to realize that when you don't just give up on trying to understand and solve a problem, the results can be amazing. If you concentrate and try to figure things out and work through a problem, you'll find that once you develop this attitude everything is easy. The NetBSD community has done a good job of inspiring new developers to build this mindset. While it may sometimes frustrate those who don't make the leap, the project definitely benefits from having a very can-do attitude where individuals share responsibility for the quality and functionality of the code that the NetBSD project distributes.

As a result of that attitude of responsibility, NetBSD has had very little breakage due to API (application programmer interface) changes, and has learned from the breaks that did occur. Even when API changes are needed, the project provides shim layers that reimplement the old API using the new API. These layers mean that old programs continue to work without having to be changed. At the system level, administrators can leave the shims out if they are not needed, to forgo the additional code bloat that would otherwise build up with many layers of backwards compatibility. A similar compatibility mechanism allows NetBSD to run binaries that were compiled for other operating systems which run on the same CPU. For this reason, NetBSD can run Linux, FreeBSD, and Solaris binaries. Taking responsibility for API and ABI (application binary interface) continuity means binaries built for NetBSD 0.8 (the very first release) can still be run on NetBSD 4 (the most recent release). That kind of good design and superior technical leadership in planning and execution attracts technical people who have an appreciation for the aesthetics of a system.

Jeremy C. Reed

Jeremy is a NetBSD and pkgsrc developer. He writes:

I started using NetBSD around 1999 when I wanted a system for running web services. At work, I was already familiar with BSDI, a commercial version of BSD developed by a company founded by former CSRG members. I liked that NetBSD was built to run on many different platforms and that the pkgsrc packaging system could also be used on different operating systems, including Linux, FreeBSD, Solaris, and AIX. This kind of layering and reusable architecture is very attractive to technical people.

I found the NetBSD community to be friendly, helpful, and knowledgeable. In many cases, other communities are either helpful but not knowledgeable, or are knowledgeable but socially abusive to newcomers. The social tone that the NetBSD project maintains on its public mailing lists and forums includes respect, open discussion, and the willingness to disagree. Not counting spam filtering, the only moderated mailing lists are the low volume, official project announcement lists.

I started by filing problem reports for issues I encountered and by submitting potential enhancements to the system; soon I was invited to become a developer. This process of participation and contributing which then leads to membership and CVS commit privileges in the project instills commitment, responsibility, and pride in one's work. For the NetBSD project, that results in high quality that attracts more developers on an ongoing basis.

I appreciated that NetBSD had different technical mailing lists where I could follow what interested me, without trying to track every proposal that was going on system-wide. To a large extent, the NetBSD Project has avoided moving to 'Enterprise 2.0' forums, preferring to document the system in version controlled text files in the source tree, or version controlled HTML documents. With a source tree as large as NetBSD, external discussions would be difficult to match up to the version it relates to and would incur additional work to prune or update periodically, whereas CVS content can only become outdated, not disconnected. The topic-specific lists are also useful for targetting questions and proposals.

Once in the project, I saw that there was plenty of internal discussion and that much occurs behind the scenes in the form of mentorship and design work. Lately, we've pushed for more of that to be done in public forums, so the learning benefit is available to a wider audience. While it's not always practical to open a discussion up to all comers, we have accomplished better transparency in our efforts. Having the problem reporting and tracking system in place from day one was very important. They provide a social history of discussions, why particular decisions were made, and why some solutions can't be implemented until a blocking item is fixed first.

Having public code, CVS, and CVSWeb have been valuable in making it easy to review the code, and to keeping low barriers for new users to see, learn, and experiment. The wiki and IRC presence of the NetBSD Project are examples of communities that were not set up by the project, but have sprung up around it, and are mutually beneficial. Interactive text chat tools allow developers and users to interact in real-time and allow faster dissemination of information and the exchange of ideas. Having a project chat-room also provides a way to introduce new developers to the social rules that are implicit in any group effort.

The NetBSD website has for a long time listed users and providers of NetBSD-based products. The NetBSD In Action gallery lets people see how other members of the community are using NetBSD.

NetBSD welcomed education and research applications early on, which resulted in contributions of useful subsystems such as RAIDframe which reinforced the habit of code based on thorough analysis and has been used as reference material for teaching good programming habits.

Summary

Whether by design or by chance, NetBSD has many attributes that built a strong, vibrant community. We present the following as useful as tips to people starting new projects.

• know your audience, both committers and users, and cater to it
• have a central focus on something that is of high quality, and has attributes that make it unique
• have some form of leadership, but it doesn't have to be overly sophisticated
• learn from your mistakes
• be transparent
• have a broad membership that encourages a variety of perspectives that share common goals
• offer value to your audience, such as learning opportunities
• be friendly
• encourage independence, rather than dependence
• reward good work
• have good communication tools and practices which allow for the archiving of discussions