Release criteria for the Linux kernel by David G. GlanceBefore software is released to its users, software developers will ensure that the software has met specified functional and technical requirements and that it is as free from bugs as possible. Users should be able to have a high degree of confidence that the software will perform as specified and without fault. With open source development practices such as those employed on the Linux kernel project, there are no detailed specifications and little formal testing processes. The questions, then, are what criteria, if any, are used in determining the suitability for release of a particular version of this software, and do users have any degree of confidence in the quality of that release of software? These questions were examined in this study using information from the Linux Kernel Mailing List (LKML), the primary forum for discussion of development issues of the Linux kernel, and change logs submitted with version releases of the Linux kernel. It was determined that very little planning is employed in determining the release of a particular version of the software and that a version of the software is essentially a collection of source patches released at regular intervals with some stabilisation of the code base before each release. Very little attempt is made to verify that the code is bug free, and consequently, the code released is of a largely unknown level of quality. End users are left to decide for themselves the suitability and robustness of a particular version of the software.
Contents
Introduction
Linux kernel development process
Conclusion
Introduction
From a software engineering perspective, most software projects incorporate some criteria that have to be met before software is released to users. Typically, this will be a combination of a number of factors including the fulfilment of the functional and technical requirements, the testing and removal of bugs and defects in the code and other less software engineering–focused pressures including user demand and business opportunities. Traditional commercial software projects typically use detailed specifications and perform systematic validation of the software against those specifications to assess whether software is ready to be released (Sommerville, 2000). This process can be made more objective by utilising statistical analysis of software metrics to judge when testing should be considered complete (Pressman, 2000). Using these techniques, the software producer can get a reasonable idea of the overall quality of the software when it is released to customers.
In open source software projects such as the development of the Linux kernel, the process methodologies are not so formal and the elicitation of requirements and the validation process relies principally on communications on mailing lists (Iannaci, 2003). Erenkrantz (2003) has described more formal testing procedures in projects such as the Subversion and Apache which is largely absent in the development of the Linux kernel.
The rationale for examining the release criteria for an open source product such as Linux is that this is one form of objective assessment of the level of quality of the product when it is released to the public. There is also the possibility that missing criteria could be added into the development process of the Linux kernel as part of process improvement.
Linux kernel development process
Linux kernel versions are distinguished between stable versions which are even numbered (currently version 2.4.x and now 2.6.x) and odd numbered test versions (version 2.7.x). The difference between the versions is not well defined. Ostensibly the test versions are for experimental code and major changes. However, it is clear that the definition of experimental and major is not hard–and–fast and as I will show, development on the 2.4.x series accelerated dramatically starting with version 2.4.19 (see Table 1), arguably altering its so–called stable nature. Although there is no direct evidence for this, it probably reflected in part, contributions of software changes being submitted for the 2.5.x kernel being backported to 2.4.x.
In order to determine what criteria, if any, are used in the Linux kernel development process for determining the readiness of a particular version, two different approaches were taken. The first was to examine the change logs [ 10, 11, 12] that are submitted with the public release of a version of the kernel on www.kernel.org [ 1]. Observations were made on the release nomenclature used in both the 2.4.x series and the 2.5.x (which later became the 2.6.x series). The number of patches incorporated into each release and the elapsed time between releases was recorded. The second approach was to review the Linux kernel mailing list [ 2] for any evidence of criteria for performing a particular release including any metrics used, tests carried out or bug databases referenced.
Version release numbering
Version numbering for 2.4.x series
Versions for the even numbered series, the so–called stable kernel, follow the scheme of x.y.z. where x is the major kernel version, y is the minor number and z is the patch number of this release (Erenkrantz, 2003).
In the 2.4.x kernel series, the original scheme was to make periodic prepatches for each 2.4.x version. Numbering follows the scheme 2.4.16–pre1, 2.4.16–pre2, 2.4.16–preN. Prior to version 2.4.17, a final release would then be made. Starting with version 2.4.17, prepatches would be followed by release candidates, numbered 2.4.17–rcX, where X would be 1, 2, 3, etc. [ 8] These release candidates were supposed to reflect the attempt to stabilise the version before release. The number of prepatches and per version release was variable with a mean of eight and a standard deviation of three. The number of release candidates was on average six with a standard deviation of two.
The rationale and purpose of the release candidate releases has been discussed on LKML where the role was summarised as new code being added during the prepatch releases and then stabilised during the release candidate release [ 3]. In this context, stabilisation consists of seeing if any major bugs surface and are posted to the mailing list.
Version numbering for 2.5.x series
The version numbering for the odd numbered series, the so–called development kernel, of the Linux kernel is similar to the even numbered series, following the same major.minor.patch level numbering. What differs however is the meaning of the patch level since the patch levels increase far more quickly and with fewer patches than the stable series.
In this series, the prepatch releases were only used up to version 2.5.8 and thereafter abandoned for discreet patch version releases. Up to version 2.5.8, the average number of prepatch releases was six with a standard deviation of four.
On moving from 2.5.x to 2.6.x to signify the stabilisation of the kernel, a different numbering scheme was adopted where test releases were made of the format 2.6.0–testX. For this version, there were 11 test releases and three release candidate releases.
It was not possible to find any rationale for the numbering scheme on the LKML for this series.
Patch submission 2.4.x kernel
In order to investigate the release process more closely, the change logs for each patch version were analysed to count the number of submitted patches per patch version and the elapsed time between versions.
Table 1: Number of patches per patch version of the Linux 2.4 series kernel and the development time for each patch version.
Patch version number Number of patches Development time (days) 2.4.1 59 26 2.4.2 55 22 2.4.3 48 38 2.4.4 54 28 2.4.5 63 28 2.4.6 89 38 2.4.7 49 16 2.4.8 41 21 2.4.9 38 5 2.4.10 132 37 2.4.11 50 16 2.4.12 7 2 2.4.13 44 13 2.4.14 48 11 2.4.15 57 18 2.4.16 4 3 2.4.17 159 25 2.4.18 186 64 2.4.19 1,255 158 2.4.20 1,467 115 2.4.21 1,737 195 2.4.22 1,392 72 2.4.23 956 93
Table 1 shows the number of patches for each patch version release of the Linux 2.4 series and the amount of time in days each version took to release. As can be seen, a significant change occurred starting with version 2.4.19 with 1,255 patches incorporated into the kernel compared to an average of 66 patches +/- 48 for the previous 18 patch version releases.
The relationship between the number of patches submitted per patch version release and the number of days is linear with a correlation coefficient of 0.91 (p < 0.01).
Patch Submission 2.5.x kernel
In order to investigate the release process more closely, the change logs for each patch version were analysed to count the number of submitted patches per patch version and the elapsed time between versions.
Table 2: Number of patches per patch version of the Linux 2.5/2.6 series kernel and the development time for each patch version.
Patch version number Number of patches Development time (days) Patch version number Number of patches Development time (days) 2.5.1 41 2.5.44 342 3 2.5.2 55 28 2.5.45 538 12 2.5.3 69 15 2.5.46 292 4 2.5.4 230 11 2.5.47 378 7 2.5.5 192 9 2.5.48 280 7 2.5.6 288 18 2.5.49 280 4 2.5.7 290 10 2.5.50 332 5 2.5.8 568 27 2.5.51 636 13 2.5.9 198 7 2.5.52 153 6 2.5.10 35 2 2.5.53 292 8 2.5.11 152 5 2.5.54 327 8 2.5.12 56 2 2.5.55 404 7 2.5.13 173 2 2.5.56 130 1 2.5.14 44 3 2.5.57 220 3 2.5.15 176 3 2.5.58 120 1 2.5.16 101 9 2.5.59 212 3 2.5.17 99 3 2.5.60 540 23 2.5.18 174 4 2.5.61 286 5 2.5.19 156 4 2.5.62 158 2 2.5.20 202 4 2.5.63 427 7 2.5.21 190 6 2.5.64 390 11 2.5.22 211 8 2.5.65 546 12 2.5.23 128 2 2.5.66 590 7 2.5.24 139 1 2.5.67 460 13 2.5.25 138 15 2.5.68 487 13 2.5.26 248 11 2.5.69 652 15 2.5.27 64 4 2.5.70 1,232 22 2.5.28 203 4 2.5.71 1,165 17 2.5.29 149 3 2.5.72 125 3 2.5.30 212 4 2.5.73 444 5 2.5.31 246 10 2.5.74 258 10 2.5.32 436 16 2.5.75 256 8 2.5.33 305 4 2.6.0–test1 186 4 2.5.34 141 9 2.6.0–test2 528 13 2.5.35 330 7 2.6.0–test3 605 12 2.5.36 76 2 2.6.0–test4 757 14 2.5.37 296 2 2.6.0–test5 827 15 2.5.38 84 2 2.6.0–test6 1,055 20 2.5.39 207 5 2.6.0–test7 614 10 2.5.40 273 4 2.6.0–test8 284 9 2.5.41 337 6 2.6.0–test9 146 8 2.5.42 427 5 2.6.0–test10 291 29 2.5.43 308 4 2.6.0–test11 23 2
The average number of development days was eight with a standard deviation of six days. Up until version 2.5.69 the average number of patches was 263 with a standard deviation of 164. The surge in patches for 2.5.70 was because of an announced stability and feature freeze.
The relationship between the number of patches submitted per patch version release and the number of days is linear with a correlation coefficient of 0.56 (p < 0.01).
Linux kernel bug database
The principle bug database for the Linux 2.5.x/2.6.x kernel series is hosted by OSDL [ 5]. Accessed on 17 February 2004, the bug database and the status of the bugs are detailed in Table 3.
Table 3: Bugs and their status in OSDL Linux kernel database.
Bug status Number of bugs New 541 Assigned 253 Resolved 83 Verified 2 Rejected 440 Deferred 14 Closed 784 Total 2,117
There is no evidence on the LKML that the number or type of bugs was used in assessment of the stability or suitability for release of the kernel.
A "must fix list" is also maintained which has a number of issues which were considered important to fix before 2.6 was officially released [ 4].
Conclusion
Analysis of the release criteria for the Linux kernels shows a process which is dynamic and whose nature depends on the developer in charge of the administration of a particular version. There is a basic trend to accepting functionality for a period and then stabilising by not accepting any further changes, except for fixes to the already submitted changes. The extent and nature of the changes is governed by time and not necessarily any other criteria, i.e. there is no detailed plan of what changes are expected nor what changes would or would not be acceptable at any given point.
Patches are accepted and released at regular intervals. What functionality goes into a particular version is therefore determined simply by what is accepted during a particular time window. There is some filtering of functionality performed in other forums — such as Linux USB, for example [ 9] — but there is no planning in terms of its further submission to the Linux kernel. Again, a similar process is adopted here whereby patches are accepted, collected and then submitted periodically by the subsystem maintainer to the kernel.
There has been some discussion on the LKML about the nature of the release process [ 3], version numbering schemes and their meaning. There is little evidence that there is any systematic, recorded planning for a particular version of the software.
The stabilisation of the kernel prior to release is again largely ad hoc in nature. In essence, it relies on the absence of any major problems posted to the LKML and on no other criteria. Partly this is because there is no real concept of a "Gold" or "Generally Available (GA)" type release common to other commercial software, but instead there is simply a release of the current work–in–progress. The problem with this approach is that it does not perhaps match end users’ expectations and says nothing about the quality of the particular version release. Use of the software is accompanied by a disclaimer that is the end users’ responsibility to assess the software for suitability of use.
The difficulty with the approach taken for the Linux kernel is that there is no measure of quality other than ad hoc feedback on the LKML. With the absence of formal planning and processes for measuring quality and the suitability of a release, end users are left to their own devices to assess whether a Linux kernel release will be sufficiently stable, robust or secure for their purposes. It is not clear that an end user would be capable of making this assessment, and so the tendency will be to essentially ignore potential problems until they occur and upgrade to fix a specific bug or problem — which of course may introduce other bugs. That the kernel contains numerous bugs when released is beyond doubt.
The introduction of a bug database to record bugs for the 2.5/2.6 kernel was a good step. The adoption of this by all developers and so its use as the central means for tracking defects has been patchy. Part of the difficulty with using bug databases with the general public — rather than with testers — is that the quality of the bugs reported is not necessarily high and so again, this feeds back into the lack of adoption of the bug database and the process surrounding fault–and–change management.
It would be possible to improve the release process of the Linux kernel and still preserve the flexibility of the open source development process. More formal testing — perhaps through the Linux test project [ 6] with bugs arising from this testing being recorded in the OSDL bug database — would be a start. The rates of new bugs entering versus the rates of the bugs being closed could then be used as a measure of the stability of the product. At present, the absence of anyone reporting bugs to the LKML is not really an indication of anything other than the fact that no one has bothered to report any bugs!
An unofficial report stated that Microsoft Windows 2000 was released with 63,000 bugs [ 7]. Although it is clear that most, if not all of these bugs were not necessarily critical, it gives an indication of the order of the number of bugs one would expect in a modern operating system. Given that there are 541 bugs recorded as being open against the 2.6 Linux kernel, one suspects that there are large number of undocumented and unknown bugs in the kernel. This would inevitably lead to questions about the quality of the software that is being released.
About the Author
David G. Glance is Senior Lecturer at the School of Computer Science and Software Engineering at the University of Western Australia.
Web: http://www.csse.uwa.edu.au/~david/
E–mail: david@csse.uwa.edu.au
Notes
1. Source: http://www.kernel.org.
2. Source: http://www.ussg.iu.edu/hypermail/linux/kernel.
3. Source: http://www.ussg.iu.edu/hypermail/linux/kernel/0111.3/0498.html.
4. Source: ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/must-fix.
5. Source: http://bugme.osdl.org.
6. Source: http://ltp.sourceforge.net/.
7. Source: http://www.cnn.com/2000/TECH/computing/02/17/windows.2000/.
8. Source: http://www.ussg.iu.edu/hypermail/linux/kernel/0111.3/0490.html.
9. Source: http://sourceforge.net/mailarchive/forum.php?forum_id=5398.
10. Source: http://www.kernel.org/pub/linux/kernel/v2.4/.
11. Source: http://www.kernel.org/pub/linux/kernel/v2.5/.
12. Source: http://www.kernel.org/pub/linux/kernel/v2.6/.
References
J.R. Erenkrantz, 2003. "Release management within open source projects," at http://www.ics.uci.edu/~wscacchi/Papers/Open-Source-Research/OSSE3-Erenkrantz.pdf, accessed 20 November 2003.
F. Iannaci, 2003. "The Linux managing model," First Monday, volume 8, number 12 (December), at http://www.firstmonday.org/issues/issue8_12/iannacci/, accessed 16 January 2004.
I. Sommerville, 2000. Software engineering. Sixth edition. New York: Addison–Wesley.
R.S. Pressman, 2000. Software engineering: A practitioner’s approach. Fifth edition. Boston, Mass.: McGraw–Hill.
Editorial history
Paper received 26 February 2004; accepted 15 March 2004.
Copyright ©2004, First Monday
Copyright ©2004, David G. Glance
Release criteria for the Linux kernel by David G. Glance
First Monday, volume 9, number 4 (April 2004),
URL: http://firstmonday.org/issues/issue9_4/glance/index.html
