First Monday
Read related articles on Computer industry, Internet economics, Linux and Software development

Linux and Decentralized Development by Christopher B. Browne

This essay describes the facts and merits of the decentralized form of Linux development and support. It suggests some ways that the continued development and growth of freely redistributable software such as Linux can be encouraged. In many respects, this is not a Linux-specific document; many of the principles should be applicable to other "free software" (or "open source") initiatives such as the free BSD projects and many others that are directed towards applications rather than OS platforms.

Contents

Motivation - Why is Centralization an Issue?
Organization Models in the Linux Community
Mandates and Purposes of a Linux Foundation
In the Interim - Your Fair Share
Sources of Inspiration in the Growth of "Free Software"
Credits

Motivation - Why is Centralization an Issue?

Many people have complained over the last few years that there should be some sort of "central" Linux organization.

Some common reasons that I see include:

There is little agreement as to what organization should be "king" but there is desire for a "king" nonetheless.

I would like to argue that while there are imperfections in the support presently available for Linux, this does not mandate the creation of an authoritative central Linux organization.

I would argue furthermore that the decentralization Linux displays represents a strength in that it allows support to grow simultaneously in many areas, unhindered by any particular controlling agency.

Real Disadvantages to Decentralization: Linux Support is Fragmented

There are disadvantages to the decentralized nature of Linux development, as it causes support arrangements to be somewhat fragmented. Linux does not have a single organization offering all of the sorts of things on the list below, as is the case for most other operating systems. There is no single organization responsible for the various support roles which are presently distributed across various organizations in the Linux community:

Advantages to Decentralization

Despite there being some problems to decentralization, I believe that there is a net advantage for Linux to having the variety of independent organizations fulfilling their various roles. We may view the independent organizations in aggregate as a sort of "virtual corporation."

Here are a number of ways that decentralization benefits Linux:

Other Beneficiaries of Decentralization

Three companies that have seen fairly spectacular levels of growth come out of the "allow independence" approach are:

Organization Models in the Linux Community

The Linux community (and similar "free software" initiatives) have shown quite a number of organizational models that serve different useful purposes in supporting the growth and improvement of Linux and the Linux community.

This section lists different kinds of organizations that we see in the Linux community.

There is room for all of the existing organizations to grow and thrive; in addition, in keeping with the "decentralized" theme already presented, I suggest that if anything there should be more such organizations. This would be particularly valuable when looking at "project-oriented" groups. Groups that are working on independent efforts can and should remain independent.

Linux Development Corporation

This sort of organization develops software/hardware/documentation products, gaining funding from selling Linux "products."

There are many such enterprises (see: Linux Commercial Vendors). Note that most of these organizations do not place under the GPL (GNU Public License) all of the intellectual property that they produce.

Linux Support Consultants Inc.

The purpose would be to sell consulting services to assist people in installing, improving, and supporting Linux systems.

There are a number of consulting firms, with many more findable through the "LinuxConsultants HOWTO that is part of the Linux Documentation Project.

Red Hat Software has an initiative where they seek to create a Red Hat Linux Certification program. There are also other efforts; nothing fully realized thus far.

Russell Nelson of Crynwr Software has suggested the idea of "The Linux Organization" providing services as a "broker" of Linux support. They could accept technical questions and/or contracts to handle technical issues, and "farm them out" to a group of "Linux Consultants".

My personal feeling is that this can probably be most effectively provided by independent "Linux Consultants Inc. (LCI)" organizations that operate explicitly as a service providers of this sort. Recent efforts at Red Hat Software use this approach. If a consulting firm sees fit to support Linux projects, that is well and good, and this may be something that they should present as a marketing tool:

Not only can we provide good service - we are involved actively in making Linux better.

I think, however, that connecting "all of Linux" (in the form of a not-for-profit organization) is likely to dilute the effectiveness of both organizations. I've seen this happen.

An appropriate "free" initiative would be to provide assistance integrating Linux systems into schools initially as mail/network servers, and perhaps ultimately as application servers. This would most sensibly be organized via regional consulting organizations that might grow out of local user groups (See the User Group HOWTO, which contains some comments from this document).

And don't discount Internet-based Linux support; "Linux People on the Internet" won the 1997 InfoWorld B est Technical Support Award.

2.3 Linux Advocacy International

There is an organization called Linux International that does some advocacy work. It would be nice to have such an organization more formally funded for common efforts. Caldera and Red Hat have both sponsored advertising, which has been generally good for Linux. That advertising has been specifically directed to their own products, which isn't necessarily of general value to all Linux folk.

The two primary activities of this sort of organization would be marketing and "information provision." Specific activities could include:

Relevant existing organizations include:

Linux Project Development Foundations

There is an existing model that could usefully be emulated that comes from the example of the Free Software Foundation, in the form of a project-oriented development group.

The FSF has provided a number of things used with Linux, most critically:

The Free Software Foundation (FSF) was founded with a very similar purpose to that which Linux satisfies quite well, that being To create a free operating system environment that can replace UNIX, from whence comes "GNU - GNU's Not UNIX." It started by constructing "system construction" tools, notably GCC and compiler/binary utilities, GNU Emacs, and UNIX-like file utilities.

Unfortunately, by the time they got around to constructing their kernel called Hurd, the organization had hardened into what now looks like a "clique" with what seem (from the outside) to be a strong set of political beliefs that seem rather disparaging of commercial enterprise.

Unfortunately, Richard Stallman, head of the FSF, is very commonly misquoted, and sometimes his actual comments are surprisingly different from what people assume he would say. Many criticisms are based on misguided readings of his statements. This nonetheless doesn't let me agree with everything he says.

Read the texts of interviews with Richard Stallman and Linus Torvalds. Linus' commentary on Stallman and his comments is rather interesting.

Political preferences be as they may; there are most definitely some problems at the FSF. They can be seen most visibly in:

These situations parallel the situation where some people split off of the NetBSD project to start the OpenBSD project.

The root causes for these various splits are diverse, but consider the common features:

I would argue that this shows that there are serious problems with the way the FSF is working.

The world has changed in some substantial ways since the FSF was founded in the early '80s. Their purpose was to build a free version of UNIX. In 1985, the world needed a free UNIX variant, as none existed. In 1997, the presence of Linux and *BSD OSes that are both powerful and robust leaves many wondering why the still highly experimental Hurd is necessary. Hurd expresses some novel ideas, but I think it unlikely that it will ever be more than a curiosity.

I suspect that part of the reason why the FSF has run into trouble is that they have become "old revolutionaries" that are having trouble renewing purpose as the environment has changed around them.

In the GNU Bead Project Portal "manifesto", Lyno Sullivan describes a

GNU project responsible for designing, building and improving a working model of the formal GNU volunteer organization and its software infrastructure.

His commentary on a previous version of this document was that he

was saddened because (I) expressed a lot of frustration with the perceived rigidity of the FSF

That is a fair assessment; his comments to the effect that some degree of "rigidity" is necessary early in the design process is also fair. I would agree that a good design must indeed start with the creation of something of a "Cathedral" by a few people, and that only once the overall design is reasonably stable is it practical to start people working in "Bazaar" mode.

My response to Lyno is that there is a point at which the Cathedral can and should be transformed into a Bazaar, and that the FSF:

Linux VARS Limited

There are quite a number of Linux systems integrators that sell preconfigured Linux boxes,generally providing built-to-spec-on-demand machines for their customers.

There have been some proposals for user groups to assist local PC vendors in constructing "Linux Box Specs" so that there can be local storefront PC vendors that sell Linux systems as well as the Linux VARS that generally sell by mail order.

Linux Documentation/Information Exchange

The World Wide Web has made it possible to implement virtually any conceivable scheme for organizing Linux information.

Most notable as a source of up-to-date system documentation is the Linux Documentation Project from which Linux HOWTO documents are distributed. I rather like My View of Linux; it contains links to various news and information sources that take many approaches to organizing information about Linux.

Publishers such as SSC, O'Reilly and Red Hat Software (amongst many ) are perhaps the most notable providers of printed manuals and books, including both commercially copyrighted material as well as an increasing variety of copylefted documentation.

A lot of the information for which other companies create "help desks" gets transmitted in the assorted forms of:

This does not provide the systematic coverage that many need; the fact that there are a whole lot of monkeys out there banging on keyboards means that on the one hand, there are some dumb answers given, but there often are useful answers given out.

Commercial help desk organizations today too often represent expensive 1-900 services staffed by unknowledgeable people that don't have material as good as the Linux HOWTOs to work with.

The Linux Documentation Project is nicely providing organization for documentation so that documentation work is not excessively duplicated. This came via the creation of an SGML DTD originally called "LinuxDOC" now known as SGML-Tools. This document has been maintained using these tools.

Something similar should be done to documen t "packages" on the Web. For almost any given information classification, there are many people replicating virtually the same information several different ways. For instance, there are at least five independent Web pages documenting databases (as with my RDBMS - Relational Database Management Systems page).

One of the best is SAL - Scientific Applications on Linux.

A "souped-up" version of the LSM format ( The Linux Software Map (LSM) used on the Sunsite archive to automatically collect basic information about Linux software packages) could be used to encourage creation of more reusable information in this area.

The New LSM format should contain sufficient information to be able to represent all of the critical information about software packages shown on such pages as My Word Processor Page, the relational database page, my Linux for Finance page, my spreadsheet page and more massive collections of application-oriented links such as Scientific Applications for Linux .

New fields not already provided for in the current LSM format should include such things as:

The utility rpm2html is a generator of Web pages for RPM packages. It takes RPM files and generates a fairly sizable set of index information, providing various indexed views. Other efforts have been springing up to try to collect and organize these sorts of information, and I anticipate substantial improvement over the next year.

Conclusions

There are a number of efforts corresponding to many of the organizational models models listed in this section.

For some "organization types", extensive support is already coming from software companies, VARs, and other such sources of assistance. Improvement is almost always possible, but in many cases there are good organizations that are growing that require little attention for us to see further improvement.

Commercial products are sponsored by sales because ultimately information and assistance have value.

An area of particular weakness is that of Development Projects for significant pieces of free "infrastructure".

This sort of effort could be strengthened by the introduction what I would call Linux Foundations. The remainder of this document is directed at describing how such organizations might be organized.

Mandates and Purposes of a Linux Foundation

By using a project-oriented mandate rather than having a fixed goal ("to develop the Linux operating system and related tools"), this may allow the goals of the organization to more readily change as public needs change. If the project becomes complete, the purpose of the organization might well go away.

Regular CVS-based releases would help to maintain the accountability of programmers; it will be quite clear whether or not bozo@lsf.org is working on code based on the daily or weekly archive updates.

In some cases, existing projects and tools could be improved if some people had financially sponsored time to spend concentrating on them.

Desirable Projects for a Linux Foundation

As the organization's mandate is to sponsor projects, it makes sense to think of a list of possible candidates. Here's a few examples including things that are fairly common on wish-lists.

These are generally large projects that are a bit too big to be handled by people doing "evening" work.

Equally importantly, some of these things involve tasks that are tedious so that people that are working to build something "good enough for me to use" won't find it worthwhile to put in the additional effort to polish the results for general use.

If such projects are formally funded by a "Linux Foundation", this encourages creation of some of the fiddly things that require more work than people can do in spare time.

Some would argue that these represent products that can be sold commercially. This is indeed true. Most of the products on the list are already available for Linux in some commercial form. The fact of commercial support establishes that people are willing to put money into such applications.

Some would also argue that free software will discourage the creation of more and better commercial software. Free software has not discouraged the development of commercial databases, Web servers, text editors, and many other sorts of programs.

The presence of good and free software raises everyone's expectations.

It presents commercial enterprises with the question:

Why should I pay for your product when I can get blah for free?

To some "free software extremists," the answer is

If there is a faintly acceptable "free" product, I will not pay for your product.

To others that feel less strongly, a useful answer is more like: like:

Buy our product because we offer better functionality or make it easier for you to use.

The availability of free software tools makes it easier and cheaper to develop good software, both free and commercial. Commercial enterprises can benefit from this.

Economics: How would a "Linux Foundation" be Funded?

We could try to have a Linux Foundation have "departments" involved in various sorts of commercial activities in order to provide funding, thus having "subsidiaries" that are of the various sorts of other organizations previously mentioned. Consulting, Internet Service, selling "Linux PCs", or selling Linux software.

I believe, however, that doing these things would direct attention away from the project activities of a LF.

The simple funding answer: Grants/Donations

A Linux Foundation should be incorporated in much the same form as similar organizations as the Free Software Foundation and The XFree86 Project, as a tax-exempt non-profit organization that can receive grants on a tax-deductible basis from both companies and individuals.

I believe that free development efforts need to be sponsored by freely given grants of funds, materials, and services.

This can include encouraging other Linux-related enterprises to sponsor free software projects either directly or by providing funding or other resources to organizations like a LF or directly to development projects. For instance, a variety of Linux-related companies (as well as some less-related organizations) have been known to sponsor projects under the GNU Public License. To cite just a few examples:

Based on these examples, it is quite reasonable to expect companies that profit from Linux to contribute something back to developme nt efforts. Whether that comes in the form of funding or of freely redistributable software is not too critical.

As a not so trivial aside, I would think it preferable for commercial contributors to prefer the form of the GNU Public License (or the LGPL, for libraries) to that of the BSD style of licensing. The GPL requires that any further contributions to code also beGPLed, which means that the contributing organization can expect to themselves benefit from other peoples' contributions. In contrast, Berkeley-style licenses allow competitors to take the code, improve on it, and keep the improvements private.

Moving beyond commercial enterprises, government agencies would be perhaps the most interesting "targets" of Linux sponsorship, as various tools (COBOL compiler, office productivity software) are of a nature that it might make sense for a governmental organization to provide some funding. Here are a few ideas:

These are all pretty big ideas. Almost surely not all of them will happen. But none of them are particularly outrageous.

I have heard arguments that people do not want government involved because of "government inefficiency." So long as usage of software that results from such projects is relatively unrestricted, I do not see a problem with this. If the software is reusable, any initial inefficiency quickly fades to irrelevance. If it were to cost $100 Million "too much" to develop GNU COBOL, but this still leads to millions of dollars worth of later savings in spending on software licenses, this may still be an attractive tradeoff.

Donations? Grants? Commissions?

It has been pointed out to me that contributions to such projects should be viewed as grants rather than as charitable donations.

Pure charitable donations go out as assistance to people in poverty, with no reasonable expectation of much return beyond gratefulness (note that the English word "charity" was, in the early days, synonymous with "love").

Grants, in contrast, are given to people and organizations w= ith an expectation of valuable results.

The purpose to contributing to free software is definitely not well-characterized as "pure charity". It is appropriate to expect contributions to turn into valuable results.

In cases where someone has substantial amounts of funding to provide, it is reasonable to be even more particular about the desired result, and set up a specific commissioning. Aladdin Software, for instance, has been commissioned to produce a Display Postscript "server" particularly for use with the GNUStep project.

Alternative Funding Sources

Here are some other funding methods that I feel are unlikely to provide substantial funding. While they may defray some costs, I do not believe that they are feasible ways to substantially fund development efforts. The Free Software Foundation, for instance, has not gotten wealthy by the sales of CD-ROMs. The fact that there are a goodly 20 vendors selling Linux CD-ROM products and publications means that there is little opportunity to bring in substantial money by starting up another not-for-profit alternative.

The major alternative to "freely given contributions" is to create a "captive commercial enterprise" to provide funding, and in my experience, that doesn't work very well. Student service organizations are not typically competent to also run businesses; churches should not run bingo halls; governments should not run lotteries. This divides the organizations into pieces that have substantially differing goals, to the detriment of all.

There is, nonetheless, some limited value in having small sideline operations, so long as it is kept clear that they are not intended to do much more than defray the direct costs of providing associated public services.

Licensing Approaches to Intellectual Property

There are occasional "wars" between those that think that the GNU Public License is the ideal way of distrbuting "free" software and those that think that the BSDL model is "freer". And I have room only to mention that there are other variants of the GPL such as the "Aladdin Ghostscript" license and the "Artistic License" from Perl.

The critical difference between the BSD License and the GNU Public License:

The GPL specifies that all derivative software must also be made freely available in source form.

On the other hand,

The BSD license specifies that people are free to do whatever they want with "BSD Licensed" code, including reselling derivative works as commercial products without any legal obligation to return the changes to the community at large.

The GPL demands the permanent freeness of any code released under it. In effect, it makes the assumption that people want to make "free" intellectual property proprietary, and seeks to prevent that. It and its most vocal proponents seek to make a political statement to "change the world".

The BSD license approach, in contrast, makes the assumption that it is worthwhile enough to make some intellectual property "free" that people and organizations will support a useful amount of "free" software. If the code is really good, then the world is free, able, and encouraged to adopt it for all kinds of purposes. Hopefully enough people will feel morally obligated to also contribute.

As an example of advantage that comes from using the GPL, Willows have released their TWIN Windows emulation system under the GPL, which means that if anyone else improves it, and wishes to redistribute the improved system, the changes must also be made available under the GPL. This means that Willows (as well as the community at large) gains benefit of other peoples' improvements to the system. Those changes could remain proprietary with the "BSD" approach.

In contrast, by virtue of the fact that it can be used to derive "proprietary" software, BSD networking code has had a tremendous impact in the commercial world; the fact that companies may take and freely make use of the code has had the result that most TCP/IP network implementations include some components of the BSD "reference implementation" code. The GPL blocks that use of code.

There are people that are strong proponents of each approach. Interest in the GPLed Willows system has been limited, while there appear to be far more people working on the WINE system that uses a BSD License. On the other hand, far more people appear to be working on the GPLed Linux kernel than on any of the BSD operating system projects.

My personal belief is that the world has room for both the GPL and BSDL models. There is value to both; in different environments, each approach has advantages. Having the multiple approaches to intellectual property permits people who want to give different things to the community to indeed do so.

GPL proponents should allow the BSDL licensing model to be free to either succeed or fail at providing useful free software to the world, and vice-versa.

If someone wants to give away different aspects of their intellectual property, that's their problem. If they prefer the GPL, that's OK. If they prefer the BSDL licensing model, that's OK.

In the Interim - Your Fair Share

If you have benefited from using Linux or other freely produced software, it is entirely fair to expect that you return back Your Fair Share to help improve future releases. Contributions might reasonably be given in a variety of forms, notably including the following:

I think it fair to expect people to contribute Their Fair Share in some fashion in all of these areas, and in particular, in the financial area. The United Way (a charitable program that operates in the USA and Canada, and elsewhere, as a "clearing house" for donations) asks people to contribute what they suggest to be a "fair share" in order to support community service efforts. They typically suggest contributing on the order of one percent of one's income. For Linux, I would suggest that the reader consider the alternative valuation metric of:

How much would you have paid for commercial versions of the software?

I would argue that $100 per year is a pretty "fair share" for a home user of Linux. Some people (particularly students) honestly can't afford that, which is fair enough. I would still urge having some financial participation, however nominal. Contributions that come in the form of services are valuable and are also highly necessary; combining the various kinds of contributions strengthens the community.

If the gentle reader has received benefit and has not given something back, I would suggest that you consider where contributions can be made in the following section.

Financial Grants

With the multiple millions of Linux users, it would be entirely plausible for grateful users to individually contribute a little.

That has the potential to add up to hundreds of millions of dollars towards development of improved tools.

That being said, getting funds to developers and other worthy recipients may realistically represent something much like the old story about Belling the Cat. At this time there isn't any single "Linux Software Foundation" to which we can expect to see such contributions go. And I don't expect that the present organizations are realistically prepared to receive millions of dollars.

There are, nonetheless, a number of possible routes for Linux users to take to make financial contributions to efforts to improve Linux, and I would urge that people consider contributing to them.

Having a single "umbrella" organization ("Linux Development Foundation") would have some value as a focal point, but is not necessary. It is useful for individuals for there to be a way of turning contributions into tax deductible expenditures, so as to maximize the value of those contributions. I can contribute more money "pre-tax" than I can "post-tax."

I have contributed to some of the following organizations and not to others. Inclusion in the list doesn't mean that I agree with all their aims, merely that they have some reasonable relevance to Linux and free software and that they appear to me to be worth considering.

Note that many of these organizations are also able to use contributions that come in the forms of computer hardware and services.

Build Needed Software

It's pretty obvious that code doesn't simply "get written". There need to be authors of new software.

Eric Raymond's article, The Cathedral and the Bazaar, describes the widely cited "Linux Bazaar Approach" to software development.

It can be simply characterized with three principles:

The Linux kernel typically has a more-or-less weekly release, whether it's ready or not. It has accepted patches from thousands of people. This huge body of participants has become a tremendous strength.

With a growing world population of Linux users, expanding that strength to encompass other system components such as compilers, libraries, utilities, applications, and even documentation allows us to harness the "smarts" of potentially millions of developers of one variety or another.

Ten thousand developers looking at the kernel is quite enough to keep it improving even if most individually only examine small portions of the code in their spare time.

Coordinating efforts in the wake of there being literally millions of participants requires a truly distributed approach. "The Bazaar" has proved to be a potent approach for software development.

The difficult thing that needs to happen is for there to be some compromises in the interests of unity.

Providing Information/Linux Support

If you aren't a "software developer," it may be reasonable to contribute to general as well as specific support efforts. Here are a few ideas:

Sources of Inspiration in The Growth of "Free Software"

The earliest references to "free" software can be found in Richard Stallman's The GNU Manifesto.

The Perl community has produced a document looking at "freely available things" entitled Information Wants to be Valuable that takes a somewhat different perspective.

Another paper that is well worth reading in analyzing the adoption and nonadoption of systems is Lisp: Good News, Bad News, and How to Win Big "Worse is Better". Its thesis is that while LISP provides extensive functionality, typical LISP-based attempts to provide "100% solutions" fail because the UNIX principle of looking for the 90% solution provides something "good enough" more quickly.

This may be further extended to the notion that Good Enough is Best. That essay makes the conclusion that Windows NT is "good enough," and will probably win the "OS Wars". That of course assumes that NT is in fact "good enough," which often stirs up controversy.

Inspiration for ideas in this essay can be traced back to Eric Raymond's The Cathedral and the Bazaar paper. Both Eric Raymond's "Cathedral/Bazaar" paper and this one were recently cited in Technology News from Wired News in connection with the recent reports that Netscape Communications Corp plans to make sources to Netscape Navigator/Communicator freely availableunder a license similar to the GNU Public License.

As a result of Eric Raymond's discussions with Netscape about how they might most appropriately deal with licenses like the GNU Public License, a group of advocates has come up with the proposal that the new term "open source" be used rather than "free software" to more readily draw corporate interest.

The English word "free" is unfortunately very ambiguous, which is a problem that proponents of "free software" have to wrestle with almost continually. "Free" has two main meanings, both of which can separately apply to "free software":

There are already two documents detailing what "Open Source" means, and why it is a good idea.

Credits

A document like this does not just happen by chance; it has certainly built in general on the past work of many people who have contributed to efforts to make Linux useful.

Thanks in particular go to the following people who have contributed comments, suggestions, and corrections to make this document better:

About the Author

Christopher B. Browne is a Consultant with The Sabre Group. He formerly was a Systems Engineer for SHL Systemhouse in Toronto. A graduate of the Universities of Ottawa and Waterloo, he has been working with Linux since 1993, and involved in the Internet from UNIX and UNIX-like platforms since 1985. E-mail: cbbrowne@hex.net
WWW: http://www.hex.net/~cbbrowne/

Contents Index

Copyright © 1998, ƒ ¡ ® s † - m ¤ ñ d @ ¥