Last week I released the 2.6.32.58 kernel and said it would be the last one of this series that I was releasing. Given that this was one of the most successful kernel series out there, by number of users, I figured it was worth a brief history of how this came to be, and what I have learned from it.

Stable kernel releases

The first stable kernel release, under the "new" model of development, happened with the 2.6.11.1 release, way back on March 4, 2005, almost 7 years ago to today day. Since then, the stable series has been very successfully released for every kernel that Linus releases, following the rules outlined in the file Documentation/stable _ kernel _ rules.txt in the kernel tree.

For more details about how the stable kernel series came to be about, and how it all works, see the presentation I gave a few years ago at the Tokyo LinuxCon conference, 2010.

Originally both me and Chris Wright started releasing the stable kernels, but a few years back, Chris retired from this, and it's been just me ever since.

One important part of the stable releases was the idea of "throw it on the floor" when Linus would release a new kernel. This worked really well for the community users of the stable kernel, but how about the "enterprise" Linux distros?

Enterprise Kernels

My day job at the time was at Novell/SUSE, and we were about to release a product based on the 2.6.16 kernel, SLE10. As engineers, we were staring down the long hallway of pain we were going to have to endure in order to maintain this specific kernel version for the next 5-7 years for our customers. A large part of maintaining an enterprise kernel is digging through the upstream kernel.org bugfixes and backporting them to the kernel where needed. As this was essentially the same thing that I was already doing as part of my upstream stable kernel work, and I had all of the scripts and workflow already created, I decided to try to see how well I could maintain the 2.6.16 kernel for a longer period of time than just the 2-3 months I currently was.

So, with the 2.6.16.1 kernel release, done on March 27, 2006, I started the 2.6.16 kernel series that would go on to live longer than any kernel release I had ever managed. In the end, I put up with that kernel for 855 days, longer than I had ever imagined possible.

I don't remember if I publicly announced this anywhere, but as time went on, I just kept the 2.6.16 kernel alive, backporting patches to it, and doing releases. This base on which to do the SLE10 releases proved invaluable to me and to the users of the SLE10 product. They gained a lot more bugfixes than we had ever found previously, and it made managing the kernel easier in that we now had a base that was "known good" to constantly build upon. To me, this experiment paid off very well, and others noticed, with community users of the 2.6.16 kernel relying on it for their systems as well, for they too wanted a longterm kernel for some machines that could be supported by the community.

Over time, I noticed that as the kernel.org releases moved on, the amount of patches that actually applied to the older kernel dropped off more and more, with a steep drop-off somewhere around 2 years.

The work that went into keeping this kernel alive, and the experience gained from keeping it working for such a long time for enterprise customers, made me write up a proposal about The Future of Enterprise Linux Kernels in June of 2007. Also, at the same time, I gave a talk at the Linux Kernel summit (I think it was that year) about this same topic. I pushed hard at that meeting, saying something like "Living at one kernel version for the lifetime of an enterprise Linux release is wrong."

Despite my goal of getting rid of enterprise Linux distro kernels sticking at a single release, my job went on, and work started at Novell to plan for the SLE11 SP1 release.

The Cabal meets in secret

The kernel developer community is a very tight-knit one. Despite working for companies that compete with each other, we work together daily through email, make fun of each other on IRC, and drink beer together in different cities around the world every few months at various conferences.

During a few of these meetings, in mid to late 2009, the kernel developers working for all of the various distros quickly figured out that the timeline for the next major releases of a number of products appeared to be lining up to happen all near the same timeframe. Because of the success of the 2.6.16 kernel, and how it worked to provide a solid base for a distro to work off of for a long time, we all agreed, informally, to push for a specific kernel release within our communities/companies that I would then maintain in the kernel.org community in the same way I had done for the 2.6.16 kernel release.

We all drifted back to our companies, and planted the seeds that maybe something like the 2.6.32 kernel would be a nice one to do our product on. This planting worked so well, I had to refrain from fits of laughter in one meeting where a project manager got up and said, "We decided that the 2.6.32 kernel would be the best for our product, what does engineering think about this?"

This successfully cumulated in the release of SLE11 SP1, Debian "Squeeze", RHEL 6, Oracle Linux 6, and Ubuntu 10.4 LTS, all based on the 2.6.32 kernel.

Hacking the business models of these different and competing groups, to coordinate on this specific kernel, was one of the (previously) unsung successes of how the community really can achieve remarkable things if they decide to do it.

We did it, now to get down to work keeping this kernel alive.

Success is almost the death of us

Because it was a widespread "secret" that the 2.6.32 kernel was going to be the base of all of these different enterprise releases, a lot of development groups all started rushing to complete things in time for this kernel release. In the end, when 2.6.32 was released on December 2, 2009, it was the third largest kernel ever developed, by number of changes, with 10,998 of them resulting in the rate of 5.46 patches per hour being applied to it over its release cycle (2.6.29 and 2.6.30 had beaten it with 11,718 and 11,989 patches respectively.)

Because of this fixed deadline, it seemed that a number of areas of the kernel were a bit more "unstable" than normal, but the release and testing process of the enterprise distros soon shook all of that mess out by the time they were finally released a number of months later. The worries that we had chosen wrong and rushed things, was now gone.

How long does it live?

With the 2.6.32 kernel being the base of these longterm enterprise distros, it was originally guessed that it would be hanging around for many many years to come. But, my old argument about how moving a kernel forward for an enterprise distro finally sunk in for 2 of the 3 major players. Both Oracle Linux and SLES 11, in their latest releases these few months, have moved to the 3.0 kernel as the base of them, despite leaving almost all other parts of the distro alone. They did this to take advantage of the better support for hardware, newer features, newer filesystems, and the hundreds of thousands of different changes that has happened in the kernel.org releases since way back in 2009.

Debian is planning on their next stable release, and it will be on a newer kernel, and Ubuntu of course has long moved off of the 2.6.32-based kernel for their (monthly?) releases.

So, with the big users of the 2.6.32 kernel moving on, I've decided to no longer support the 2.6.32 kernel. That's not to say it's going to be dead now. The ever capable Willy Tarreau (the 2.4 kernel maintainer) is going to be keeping it alive, slowly, with a very reduced release schedule as time sees fit.

What about RHEL?

Yes, I know, what about RHEL? It turned out that Red Hat wasn't interested in taking advantage of the stable kernel releases of the 2.6.32 kernel that I made. They have their reasons, and they are valid, I'm not trying to say they aren't, but it turned out that these releases I did really didn't provide them much value from their normal operation of how they maintain their kernel. They are sticking with the 2.6.32 kernel for their RHEL 6 product for the near future as far as I can tell, and it works quite well for them and their customers, with one of the largest installed base of any distro out there. I think they pick and choose pieces of the 2.6.32-stable releases as needed, but due to them only releasing a large "one giant patch" for their kernel package, it's a bit hard to tell what they are really doing there.

The numbers

So, how did the 2.6.32 kernel stack up?

It lived (in my hands) for 823 days (only the 2.6.16 kernel lived longer at 855 days). It ended up containing 3,349 patches that matched up with patches in Linus's kernel tree, the most of any stable kernel release by far (2.6.16 only had 991 patches, the next closest was 2.6.27 with 1,596 patches, 3.0 already has 1448 patches after 133 days, so it might end up being the largest eventually.)

The 2.6.32 kernel was the basis of all of the enterprise distros of the time, still running, and will still supported by the major enterprise Linux vendors for many years in the future, so it will live on.

The future

The lessons learned in maintaining the 2.6.32 kernel have cumulated in the proposal I made last year for the longterm kernel.

With the proposed longterm kernel releases being chosen once a year, it's obvious that I'm not giving up on the model of maintaining specific kernel versions for longer than a few months. But the Linux user and developer community has spread way beyond the enterprise Linux distributions over the past few years. They are no longer the primary consumer of the kernel, and it's obvious that the embedded market also needs this type of support. Because of that, I'm working with the LTSI project, to provide a base that a wider range of distros and products can base themselves on.

Will we ever see all of the major distros ever end up basing themselves on the same kernel version? The odds are now less than before, given their shifting release cycles (some faster, some slower than before), and the move forward to newer kernel releases instead of trying to patch together a single kernel for many many years, a move I strongly support.

But, you never know. If you see a group of kernel developers sitting in the corner at a conference, laughing and drinking beer, perhaps they are really plotting how to convince their managers that the idea was really theirs on what kernel they should be picking for the next product...

Thanks

I would personally like to thank the Debian kernel developers, specifically Ben Hutchings, Maximilian Attems, Dann Frazier, Bastian Blank, and Moritz Muehlenhoff. They went above and beyond what any "normal" developer would have done, ferreting patches out of the kernel.org releases and the different vendor kernels and bug tracking systems, backporting them to the 2.6.32 kernel, testing, and then forwarding them on to me. Their dedication to their user community is amazing for such a "volunteer" group of developers.

I firmly believe that without their help, the 2.6.32 kernel would not have been the success that it was. The users of Red Hat and SuSE products owe them a great debt.

Buy them a beer the next time you see them, they more than deserve it.

[Update May 17, 2013]

Above I mentioned the 2.6.16 kernel, while forgetting to acknowledge the wonderful work that Adrian Bunk did in maintaining it, well after I let it go. I gave it up on the 2.6.16.27 release, on July 17, 2006, and Adrian took it and ran with it for much longer, releasing 35 kernels, for two more years, until July of 2008 when it was retired.

My apologies, I never ment to ignore Adrian's work.

posted Thu, 08 Mar 2012 in [/linux]