Here's some thoughts about some hardware I was going to use, hardware I use daily, and hardware I'll probably use someday in the future.

Thunderbolt is dead, long live Thunderbolt.

Seriously, it's dead, use it as a video interconnect and don't worry about anything else.

Ok, some more explanation is probably in order...

Back in October of 2012, after a meeting with some very smart Intel engineers, I ended up the proud owner of a machine with Thunderbolt support, some hard disks with Thunderbolt interfaces, and most importantly, access to the super-secret Thunderbolt specification on how to make this whole thing work properly on Linux. I also had a MacBook Pro with a Thunderbolt interface which is what I really wanted to get working.

So I settled in and read the whole spec. It was fun reading (side note, it seems that BIOS engineers think Windows kernel developers are lower on the evolutionary scale than they are, and for all I know, they might be right...), and I'll summarize the whole super-secret, NDA-restricted specification, when it comes to how an operating system is supposed to deal with Thunderbolt, shhh, don't tell anyone that I'm doing this:

Thunderbolt is PCI Express hotplug, the BIOS handles all the hard work.

Seriously, it's that simple, at least from the kernel point of view. So, it turns out that Linux should work just fine with Thunderbolt, no changes needed at all, as we have been supporting PCI hotplug in one form or another for 15+ years now (you remember CardBus, right?)

Some patches were posted to get the one known motherboard with Thunderbolt support to work properly by the engineers at Intel (it seems that the ACPI tables were of course wrong, so work-arounds were needed), and that should be it, right?

Wrong.

It turns out that that Apple, in their infinite wisdom, doesn't follow the specification, but rather, they require a kernel driver to do all of the work that the BIOS is supposed to be doing. This works out well for them as they can share the same code from their BIOS with their kernel, but for any other operating system, that doesn't know how to talk directly to the hardware at that level, you are out of luck. So, no Thunderbolt support on Apple hardware for Linux (at least through May 2013, maybe newer models will change this, but I'm not counting on it.)

But wait, what about Thunderbolt support on other hardware? I was in Hong Kong in early 2013, and of course found the chance to find the local computer stores. I saw, on one wall of a shop, all of the latest motherboards that were brand new, and would be sold all around the world for the next 6+ months. None of them had Thunderbolt support on them. It's almost impossible to find Thunderbolt on a motherboard these days, and that doesn't look to change any time soon.

Then I read this interesting article that benchmarked Thunderbolt mass-storage devices with USB ones. It turns out that the speeds are the same. And that's with the decades-old USB storage specification that is so slow it's not funny. Wait for manufacturers to come out supporting the latest UAS specification (and the USB host controller drivers to support it as well, Linux doesn't yet because there is no hardware out there, wonderful chicken-and-egg problem...) When that happens, USB storage speeds are going to be way above Thunderbolt.

So Thunderbolt is dead, destined for the same future that FireWire ended up as, a special interconnect that almost no one outside of Apple hardware circles use, with USB ending up taking over the mass-market instead.

Note, all of this is for Thunderbolt the PCI interconnect, not the video connection. That works just fine on Linux as it isn't PCI Express, but just a video pass-through. No problems there.

Present

I've been lucky to be using a Chromebook Pixel for the past few months, thanks to some friends at Google who got it for me. It's the best laptop I've used in a very long time, and I love the thing. I also hate it, and curse it daily, but wouldn't give it up at all.

I'm running openSUSE Tumbleweed on it, not Chrome OS, so of course that is the main reason I'm having the problems listed below with it. If you stick with Chrome OS, it's wonderful, seriously great. My day-job (Linux kernel work) means that I can't use Chrome OS as I can't change the kernel, but almost everyone else can use Chrome OS, especially if your company uses Google Apps for email and the like. Chrome OS is really good, I like it, and I think it is the way forward for a large segment of laptop users. My daughter weekly asks me if I'm willing to give the laptop to her to reinstall Chrome OS on it, as that's her desktop of choice, and this laptop runs it better than anything I've seen.

Here's the things that drive me crazy:

• small disk size. It's ok for normal kernel work, but when I was trying to build some full-system virtual machines for testing, I quickly ran out of space.
• slow disk speed. It's a "SSD", but I'm used to a real SSD speed, not this slow thing, where I can easily max out the I/O path doing kernel builds, as the processor quickly outraces it.
• USB 2 ports, I could get around the disk size and speed if I had USB 3.0, and I totally understand why there are only USB 2 ports in the laptop, but hey, I can wish, right?
• various EC issues, the Embedded Controller in the laptop is "odd" and when you run a different operating system than Chrome OS, the quirks come out. I've learned to live with them, but I would love to see an update for the BIOS that fixes the known problems that are already resolved within the code trees. It's just up to Google to push that out publicly.

Here's the things that make me love this laptop:

• the screen
• the screen
• the screen
• seriously, the screen. It's beautiful, and is worth any problem I've had with this laptop.
• wireless just works, no issues at all, great Atheros driver / hardware.
• it's the best presentation laptop I've ever had. Gnome 3 works wonderfully with it, the external display adaptor can easily handle a different resolution. LibreOffice's presentation mode, with the speaker notes on the laptop, with it's huge screen looks wonderful, and the slides at a much lower resolution is just great. No problems at all with this, just plug the laptop into the projector and go.
• very fast processors. Full kernel builds in less than 5 minutes, no problem.

There are some things that originally bothered me, but have been fixed, or I'm now used to:

• suspend / resume didn't work, that's fixed in 3.10-rc kernels.
• resume used to throttle the CPU to only half speed, again, fixed in 3.10-rc kernels.
• keyboard backlights don't survive suspend/resume, there are fixes out there that hopefully will get into 3.11, it doesn't bother me at all.
• lack of PgUp/PgDown/Home/End/Delete keys. The ever-talented Dirk Hohndel made a patch for the PS/2 driver (seriously, a PS/2 keyboard?) that overloads the right Alt key and arrow keys to provide this fix, so this is solved, but it would be good to get it merged upstream, hopefully one day this will get there for others to use.
• trackpad was annoying at first, but now I'm used to the three-finger tap for middle click. Oh, and I got a good wireless mouse to make it easier.

It's a great laptop, built really solid. I'd recommend it to anyone who uses Chrome OS, and for anyone else if you like tinkering with your own kernels (a small market, I know.) Later this year new hardware should be coming out, with the same type of high-resolution display, and beefier processors and bigger storage devices. When that happens, I'll get one of them, and my daughter will greedily grab this laptop and install Chrome OS, but until then, this is what I use to travel the world with.

The future is glass

A few weeks ago, a friend of mine came over with a newly acquired Google Glass device. I played with it for a few minutes, and was instantly amazed at the possibilities it will provide. I, like probably lots of you, have been reading books that describe different types of heads-up or "embedded computers" for many many years, and I've always been waiting for the day that this will become a reality.

Google Glass might not be the device described in science fiction books, but it's the closest I've seen so far. The interface is completely natural, the display is amazing, and the potential is huge.

And yes, you do look like a dork while wearing them, but that will either become acceptable, or the device will shrink over time. I'm betting on a combination of both of them.

But what I found even more amazing is what happened when the kids put them on. The youngest put them on, and, as I explained on Google+ after it happened, his responses went, in order:

• "You could watch movies with this in class!"
• "Google Glass, what is Iron Man?"
• "Google Glass, what is 7 * 24"

So that was YouTube time waster, to to movie background information, to homework solver in a matter of minutes. Total acceptance, no hesitation at all, I think that's proof of just how big this will be eventually.

Later that day, we went to a neighborhood yogurt shop, and my friend ended up stalling the checkout line for a long time as the teenagers running the store insisted on trying them out and taking pictures of each other and doing google searches to see just how popular their store was (hint, it wasn't the highest ranking, which was funny.) After we finally paid for our dessert, my friend was stuck demoing the device for about everyone who came in the shop for the next 20 minutes. People of all ages, kids to retirees, all instantly got the device and enjoyed it.

So, if you've made fun of Google Glass in the past, try one out, and consider the potential of it.

And of course, it runs Linux, which makes me happy.

posted Thu, 20 Jun 2013 in [/linux]

A few years ago, I gave a history of the 2.6.32 stable kernel, and mentioned the previous stable kernels as well. I'd like to apologize for not acknowledging the work of Adrian Bunk in maintaining the 2.6.16 stable kernel for 2 years after I gave up on it, allowing it to be used by many people for a very long time.

I've updated the previous post with this information in it at the bottom, for the archives. Again, many apologies, I never meant to ignore the work of this developer.

posted Fri, 17 May 2013 in [/linux]

I said this last week on Google+ when I was at a conference, and needed to get it out there quickly, but as I keep getting emails and other queries about this, I might as make it "official" here. For no other reason that it provides a single place for me to point people at.

Anyway, I would like to announce that the 3.8 Linux kernel series is NOT going to be a longterm stable kernel release. I will NOT be maintaining it for long time, and in fact, will stop maintaining it right after the 3.9 kernel is released.

The 3.0 and 3.4 kernel releases are both longterm, and both are going to be maintained by me for at least 2 years. If I were to pick 3.8 right now, that would mean I would be maintaining 3 longterm kernels, plus whatever "normal" stable kernels are happening at that time. That is something that I can not do without loosing even more hair than I currently have. To do so would be insane to attempt.

Hopefully this puts to rest all of the rumors.

posted Wed, 27 Feb 2013 in [/linux]

I've now been with the Linux Foundation for just over a year. When I started, I posted a list of how you can watch to see what I've been doing. But, given that people like to see year-end-summary reports, the excellent graphic designers at the Linux Foundation have put together an image summarizing my past year, in numbers:

posted Thu, 14 Feb 2013 in [/linux]

There's been a lot of information scattered around the internet about these topic recently, so here's my attempt to put them all in one place to (hopefully) settle things down and give my inbox a break.

Last week I spent a number of days at the GNOME Developer Hackfest in Brussels, with the goal to help make the ability to distribute applications written for GNOME (and even more generally, Linux) in a better manner. A great summary of what happened there can be found in this H-Online article. Also please read Alexander Larsson's great summary of what we discussed and worked on for another view of this.

Both of these articles allude to the fact that I'm working on putting the D-Bus protocol into the kernel, in order to help achieve these larger goals of proper IPC for applications. And I'd like to confirm that yes, this is true, but it's not going to be D-Bus like you know it today.

Our goal (and I use "goal" in a very rough term, I have 8 pages of scribbled notes describing what we want to try to implement here), is to provide a reliable multicast and point-to-point messaging system for the kernel, that will work quickly and securely. On top of this kernel feature, we will try to provide a "libdbus" interface that allows existing D-Bus users to work without ever knowing the D-Bus daemon was replaced on their system.

"But Greg!" some of you will shout, "What about the existing AF_BUS kernel patches that have been floating around for a while and that you put into the LTSI 3.4 kernel release?"

The existing AF_BUS patches are great for users who need a very low-latency, high-speed, D-Bus protocol on their system. This includes the crazy automotive Linux developers, who try to shove tens of thousands of D-Bus messages through their system at boot time, all while using extremely underpowered processors. For this reason, I included the AF_BUS patches in the LTSI kernel release, as that limited application can benefit from them.

Please remember the LTSI kernel is just like a distro kernel, it has no relation to upstream kernel development other than being a consumer of it. Patches are in this kernel because the LTSI member groups need them, they aren't always upstream, just like all Linux distro kernels work.

However, given that the AF_BUS patches have been rejected by the upstream Linux kernel developers, I advise that anyone relying on them be very careful about their usage, and be prepared to move away from them sometime in the future when this new "kernel dbus" code is properly merged.

As for when this new kernel code will be finished, I can only respond with the traditional "when it is done" mantra. I can't provide any deadlines, and at this point in time, don't need any additional help with it, we have enough people working on it at the moment. It's available publicly if you really want to see it, but I'll not link to it as it's nothing you really want to see or watch right now. When it gets to a usable state, I'll announce it in the usual places (linux-kernel mailing list) where it will be torn to the usual shreds and I will rewrite it all again to get it into a mergable state.

In the meantime, if you see me at any of the many Linux conferences I'll be attending around the world this year, and you are curious about the current status, buy me a beer and I'll be glad to discuss it in person.

If there's anything else people are wondering about this topic, feel free to comment on it here on google+, or email me.

posted Fri, 08 Feb 2013 in [/linux]

I'm looking for someone to help me out with the stable Linux kernel release process. Right now I'm drowning in trees and patches, and could use some one to help me sanity-check the releases I'm doing.

Specifically, I'm looking for someone to help with:

• test boot the -rc stable kernels to make sure I didn't do anything foolish.
• dig through the Linux kernel distro trees and send me the git commit ids, or the backported patches, of things they are shipping that are not in the stable and longterm kernel releases.
• do code review of the patches going into the stable releases.

If you can help out with this, I'd really appreciate it.

Note, this is not a long-term position, only 6 months or so, I figure you'll be tired of it by then and want to move on to something else, which is fine.

In return, you get:

• your name in the stable releases as someone who has signed-off-by on patches going into it.
• better knowledge of more kernel subsystems than you ever have in the past, and probably really want.
• free beverages of your choice at any Linux conference you attend that I am at (given my travel schedule, seems to be just about all of them.)

If anyone is interested in this, here are the 5 steps you need to do to "apply" for the position:

• email me with the subject line starting with "[Stable tree help]"
• email me "proof" you are running the latest stable -rc kernel at the moment.
• send a link to some kernel patches you have done that were accepted into Linus's tree.
• send a link to any Linux distro kernel tree where they keep their patches.
• say why you want to do this type of thing, and what amount of time you can spend on it per week.

I'll close the application process in a week, on November 7, 2012, after that I'll contact everyone who applied and do some follow-up questions through email with them. I'll also post something here to say what the response was like.

posted Tue, 30 Oct 2012 in [/linux]

As I posted to the linux-kernel mailing list, the 3.4 kernel tree will be the next -longterm kernel that I will be maintaining for at least 2 years.

Currently I'm maintaining the following stable kernel trees for the following amount of time:

• 3.0 - for at least one more year
• 3.4 - for at least two years
• 3.5 - until 3.6.1 is out

Hope this helps clear up any rumors floating around. If anyone has any please let me know.

posted Mon, 20 Aug 2012 in [/linux]

I've been writing an occasional "Ask a kernel maintainer" column on the lwn.net weekly kernel page. It's been a while since I last wrote one, so I figured it's time to start it up again.

So, consider this an open request for questions that you've always wondered, but never knew who to ask, or how to find the answer to, that you have had about the Linux kernel.

Note, any question that can easily be answered by reading either the Documentation/HOWTO or Documentation/SubmittingPatches or Documentation/CodingStyle files in the Linux kernel source code are not eligible. You should read them first before asking.

Please email them to me or post them in this Google+ thread, and I'll work over the next few weeks to answer them in the column.

posted Wed, 25 Jul 2012 in [/linux]

At LinuxCon Japan a few weeks ago I gave a talk entitled, "Linux Kernel Maintainers, What they do and how you can help them."

The video of the talk is now online here if you want to see what I said, and the full slides, and text of what I said can be found in the presentation's repository.

If you have ever wondered why a maintainer of an open source project could be a bit grumpy about anything you have sent them, I strongly suggest you go read the notes I wrote for that talk, or the slides, which includes all speaker notes.

Also, if you ever want to know what to do, in order to get your patches accepted, please go read the slides, I'm not going to repeat the information here.

Well, with one exception.

But first, it seems that this talk has kicked off a bunch of discussion. First off was Jake Edge's excellent summary of the talk on lwn.net, which kicked off a bunch of comments by people who didn't seem to have read the slides, or seen the talk, but it sparked a bunch of interest anyway, as everyone likes watching when people argue.

Then Jon Corbet weighed in on the topic of mocking which was a very good summary of some of the issues involved when maintainers of open source projects are overwhelmed by bad submissions, or just beaten down by constantly having to answer the same questions every single week, and no one seeming to read documentation where things like that are answered. Again, highly recommended, go read both of them, and the comments for the articles.

(What, you aren't a lwn.net subscriber? Why not! Shame on you, go fix that right now.)

After that, the Linux Kernel Summit Call For Participation went out, and lots of the proposals that are being submitted deal with maintainers, our workloads, and how we can handle the issues involved. Go read them here for an idea of what we are currently grappling with if you are interested.

The part I wanted to repeat from my talk is this, what I, as a Linux kernel subsystem maintainer pledge to kernel developers who send me patches for the various parts of the kernel that I maintain:

• I will review your patch within 1-2 weeks (see details below).
• I will offer semi-constructive criticism of your patches.
• I will let you know the status of your patch if it is rejected, or if it is accepted, what tree it has gone into, where you can find it, and when you can expect to see it merged into Linus's tree.

That's it. And from you, I expect to get well-formatted, fully documented, cleanly applying patches that do useful things, and everyone involved will be happy, right?

Note about 1-2 week response time:

Of course, if I'm sick, or traveling on insane Asia trips, my response time will be delayed, but feel free to email me asking the status of a patch you have sent me, I'm happy to respond back that I just haven't gotten to it. I would much rather field these kinds of inquiries (after waiting a few weeks) then loose patches and make people mad.

Also, consider the kernel merge window requirements, during the merge window, I can't accept new patches into my tree that are not bugfixes for this specific kernel release, so that means there is usually a 3 week window starting about 1 week before Linus's release, lasting until after a -rc1 kernel is released, before I can get to your patch. During that time period hundreds of patches accrue, so please give me some time to dig out from all of them. Usually by -rc3 I'm caught up, if not, email me and ask.

posted Mon, 18 Jun 2012 in [/linux]

It's been about a year since I did a status report of what's going on in the openSUSE:Tumbleweed repo, let me know if you find this actually useful or not so that I can determine if I should keep it up.

• As everyone knows, Tumbleweed is running on top of openSUSE:12.1, the transition to 12.1 was rocky for some people who thought that Tumbleweed was somehow a "full" distro, and not just an add-on on top of a stable openSUSE release. To make things easier for future updates of the base openSUSE release, please point to the "current" repo, not the explicitly numbers repo. For more details how to do this, see the Tumbleweed wiki page.
• kernel 3.3.0 is in Tumbleweed, and seems to be working well so far.
• KDE 4.8 is now in Tumbleweed, be careful if you previously had added the KDE repo manually to your system, you should now remove it as I have no idea how well it will interact with this.
• Because of the KDE 4.8 update, LibreOffice was dropped from Tumbleweed. This is due to build issues with the package, not any runtime issue that I can determine. LibreOffice fails to build on Factory at the moment as well, and a bug is open about this, hopefully it gets resolved soon.
• XFCE has been updated in Tumbleweed to the latest version
• vim finally showed up, after a brief breakage that I caused, sorry about that, all should be good now.
• To preempt any questions about a GNOME update in Tumbleweed, I am looking into it, but it will not happen until it stabilizes in Factory first.

As always, if anyone knows of any packages they wish to see added to Tumbleweed, please let me know.

Please read the wiki page for Tumbleweed if you have any basic questions about what it is or how to use it. Any other questions, please ask them on the opensuse-factory mailing list.

posted Mon, 26 Mar 2012 in [/linux]

Last week I released the 2.6.32.58 kernel and said it would be the last one of this series that I was releasing. Given that this was one of the most successful kernel series out there, by number of users, I figured it was worth a brief history of how this came to be, and what I have learned from it.

Stable kernel releases

The first stable kernel release, under the "new" model of development, happened with the 2.6.11.1 release, way back on March 4, 2005, almost 7 years ago to today day. Since then, the stable series has been very successfully released for every kernel that Linus releases, following the rules outlined in the file Documentation/stable _ kernel _ rules.txt in the kernel tree.

For more details about how the stable kernel series came to be about, and how it all works, see the presentation I gave a few years ago at the Tokyo LinuxCon conference, 2010.

Originally both me and Chris Wright started releasing the stable kernels, but a few years back, Chris retired from this, and it's been just me ever since.

One important part of the stable releases was the idea of "throw it on the floor" when Linus would release a new kernel. This worked really well for the community users of the stable kernel, but how about the "enterprise" Linux distros?

Enterprise Kernels

My day job at the time was at Novell/SUSE, and we were about to release a product based on the 2.6.16 kernel, SLE10. As engineers, we were staring down the long hallway of pain we were going to have to endure in order to maintain this specific kernel version for the next 5-7 years for our customers. A large part of maintaining an enterprise kernel is digging through the upstream kernel.org bugfixes and backporting them to the kernel where needed. As this was essentially the same thing that I was already doing as part of my upstream stable kernel work, and I had all of the scripts and workflow already created, I decided to try to see how well I could maintain the 2.6.16 kernel for a longer period of time than just the 2-3 months I currently was.

So, with the 2.6.16.1 kernel release, done on March 27, 2006, I started the 2.6.16 kernel series that would go on to live longer than any kernel release I had ever managed. In the end, I put up with that kernel for 855 days, longer than I had ever imagined possible.

I don't remember if I publicly announced this anywhere, but as time went on, I just kept the 2.6.16 kernel alive, backporting patches to it, and doing releases. This base on which to do the SLE10 releases proved invaluable to me and to the users of the SLE10 product. They gained a lot more bugfixes than we had ever found previously, and it made managing the kernel easier in that we now had a base that was "known good" to constantly build upon. To me, this experiment paid off very well, and others noticed, with community users of the 2.6.16 kernel relying on it for their systems as well, for they too wanted a longterm kernel for some machines that could be supported by the community.

Over time, I noticed that as the kernel.org releases moved on, the amount of patches that actually applied to the older kernel dropped off more and more, with a steep drop-off somewhere around 2 years.

The work that went into keeping this kernel alive, and the experience gained from keeping it working for such a long time for enterprise customers, made me write up a proposal about The Future of Enterprise Linux Kernels in June of 2007. Also, at the same time, I gave a talk at the Linux Kernel summit (I think it was that year) about this same topic. I pushed hard at that meeting, saying something like "Living at one kernel version for the lifetime of an enterprise Linux release is wrong."

Despite my goal of getting rid of enterprise Linux distro kernels sticking at a single release, my job went on, and work started at Novell to plan for the SLE11 SP1 release.

The Cabal meets in secret

The kernel developer community is a very tight-knit one. Despite working for companies that compete with each other, we work together daily through email, make fun of each other on IRC, and drink beer together in different cities around the world every few months at various conferences.

During a few of these meetings, in mid to late 2009, the kernel developers working for all of the various distros quickly figured out that the timeline for the next major releases of a number of products appeared to be lining up to happen all near the same timeframe. Because of the success of the 2.6.16 kernel, and how it worked to provide a solid base for a distro to work off of for a long time, we all agreed, informally, to push for a specific kernel release within our communities/companies that I would then maintain in the kernel.org community in the same way I had done for the 2.6.16 kernel release.

We all drifted back to our companies, and planted the seeds that maybe something like the 2.6.32 kernel would be a nice one to do our product on. This planting worked so well, I had to refrain from fits of laughter in one meeting where a project manager got up and said, "We decided that the 2.6.32 kernel would be the best for our product, what does engineering think about this?"

This successfully cumulated in the release of SLE11 SP1, Debian "Squeeze", RHEL 6, Oracle Linux 6, and Ubuntu 10.4 LTS, all based on the 2.6.32 kernel.

Hacking the business models of these different and competing groups, to coordinate on this specific kernel, was one of the (previously) unsung successes of how the community really can achieve remarkable things if they decide to do it.

We did it, now to get down to work keeping this kernel alive.

Success is almost the death of us

Because it was a widespread "secret" that the 2.6.32 kernel was going to be the base of all of these different enterprise releases, a lot of development groups all started rushing to complete things in time for this kernel release. In the end, when 2.6.32 was released on December 2, 2009, it was the third largest kernel ever developed, by number of changes, with 10,998 of them resulting in the rate of 5.46 patches per hour being applied to it over its release cycle (2.6.29 and 2.6.30 had beaten it with 11,718 and 11,989 patches respectively.)

Because of this fixed deadline, it seemed that a number of areas of the kernel were a bit more "unstable" than normal, but the release and testing process of the enterprise distros soon shook all of that mess out by the time they were finally released a number of months later. The worries that we had chosen wrong and rushed things, was now gone.

How long does it live?

With the 2.6.32 kernel being the base of these longterm enterprise distros, it was originally guessed that it would be hanging around for many many years to come. But, my old argument about how moving a kernel forward for an enterprise distro finally sunk in for 2 of the 3 major players. Both Oracle Linux and SLES 11, in their latest releases these few months, have moved to the 3.0 kernel as the base of them, despite leaving almost all other parts of the distro alone. They did this to take advantage of the better support for hardware, newer features, newer filesystems, and the hundreds of thousands of different changes that has happened in the kernel.org releases since way back in 2009.

Debian is planning on their next stable release, and it will be on a newer kernel, and Ubuntu of course has long moved off of the 2.6.32-based kernel for their (monthly?) releases.

So, with the big users of the 2.6.32 kernel moving on, I've decided to no longer support the 2.6.32 kernel. That's not to say it's going to be dead now. The ever capable Willy Tarreau (the 2.4 kernel maintainer) is going to be keeping it alive, slowly, with a very reduced release schedule as time sees fit.

What about RHEL?

Yes, I know, what about RHEL? It turned out that Red Hat wasn't interested in taking advantage of the stable kernel releases of the 2.6.32 kernel that I made. They have their reasons, and they are valid, I'm not trying to say they aren't, but it turned out that these releases I did really didn't provide them much value from their normal operation of how they maintain their kernel. They are sticking with the 2.6.32 kernel for their RHEL 6 product for the near future as far as I can tell, and it works quite well for them and their customers, with one of the largest installed base of any distro out there. I think they pick and choose pieces of the 2.6.32-stable releases as needed, but due to them only releasing a large "one giant patch" for their kernel package, it's a bit hard to tell what they are really doing there.

The numbers

So, how did the 2.6.32 kernel stack up?

It lived (in my hands) for 823 days (only the 2.6.16 kernel lived longer at 855 days). It ended up containing 3,349 patches that matched up with patches in Linus's kernel tree, the most of any stable kernel release by far (2.6.16 only had 991 patches, the next closest was 2.6.27 with 1,596 patches, 3.0 already has 1448 patches after 133 days, so it might end up being the largest eventually.)

The 2.6.32 kernel was the basis of all of the enterprise distros of the time, still running, and will still supported by the major enterprise Linux vendors for many years in the future, so it will live on.

The future

The lessons learned in maintaining the 2.6.32 kernel have cumulated in the proposal I made last year for the longterm kernel.

With the proposed longterm kernel releases being chosen once a year, it's obvious that I'm not giving up on the model of maintaining specific kernel versions for longer than a few months. But the Linux user and developer community has spread way beyond the enterprise Linux distributions over the past few years. They are no longer the primary consumer of the kernel, and it's obvious that the embedded market also needs this type of support. Because of that, I'm working with the LTSI project, to provide a base that a wider range of distros and products can base themselves on.

Will we ever see all of the major distros ever end up basing themselves on the same kernel version? The odds are now less than before, given their shifting release cycles (some faster, some slower than before), and the move forward to newer kernel releases instead of trying to patch together a single kernel for many many years, a move I strongly support.

But, you never know. If you see a group of kernel developers sitting in the corner at a conference, laughing and drinking beer, perhaps they are really plotting how to convince their managers that the idea was really theirs on what kernel they should be picking for the next product...

Thanks

I would personally like to thank the Debian kernel developers, specifically Ben Hutchings, Maximilian Attems, Dann Frazier, Bastian Blank, and Moritz Muehlenhoff. They went above and beyond what any "normal" developer would have done, ferreting patches out of the kernel.org releases and the different vendor kernels and bug tracking systems, backporting them to the 2.6.32 kernel, testing, and then forwarding them on to me. Their dedication to their user community is amazing for such a "volunteer" group of developers.

I firmly believe that without their help, the 2.6.32 kernel would not have been the success that it was. The users of Red Hat and SuSE products owe them a great debt.

Buy them a beer the next time you see them, they more than deserve it.

[Update May 17, 2013]

Above I mentioned the 2.6.16 kernel, while forgetting to acknowledge the wonderful work that Adrian Bunk did in maintaining it, well after I let it go. I gave it up on the 2.6.16.27 release, on July 17, 2006, and Adrian took it and ran with it for much longer, releasing 35 kernels, for two more years, until July of 2008 when it was retired.

My apologies, I never ment to ignore Adrian's work.

posted Thu, 08 Mar 2012 in [/linux]

With my recent job change, I'm starting to run into a bunch of people asking "What exactly are you going to be doing now?"

I've tried responding by describing the kernel related stuff I've been doing for the past years, and it turns out that a lot of people didn't even realize I was doing that.

So, here's a short list of some of the things that I'm going to be doing at my new job, and most importantly, how you can track what I do yourself, so that I never have to write a status report again...

Stable kernel releases

I've been releasing the Linux kernel stable releases since way back when they first started up, in mid 2005. Early on, the most excellent kernel developer Chris Wright helped out with this task, but for the past few years, I've been doing this on my own.

These releases take the last kernel released by Linus and add any needed bugfixes and other related patches that have gone into Linus's development tree, and package it all up in a format that users can use themselves during the 2-3 month development cycle time while the kernel developers are madly working on creating the next kernel release.

For a description of what entails a change that is acceptable into the stable kernel releases, and how to get a patch accepted, please see the file Documentation/stable _ kernel _ rules.txt in the kernel source tree.

Every year I pick a specific kernel version and declare that as "longterm". That kernel gets support from me for bugfixes and related things for two years before it is gracefully retired to a more leisurely release cycle by the capable extra-extra longterm maintainer. For details on how the longterm kernel works, and how it is picked, see this older post I wrote on the topic.

If you want to be notified of when these kernels are released, you can do one of the following:

• read lwn.net, they post the announcements mere hours after they happen. They also post lots of other wonderful things, if you aren't reading this site already, you are missing out.
• subscribe to the linux-kernel or stable mailing lists. Note, you will get a lot of other traffic, but it's all good, you wanted to know what was going on in Linux kernel development directly from the developers themselves, right?
• subscribe to my twitter feed. You might get other random blatherings there, but I do post the announcements to it.
• watch the Linux G+ feed, the releases are all announced there.
• subscribe to the google calender feeds of the kernel releases. This is maintained by the talented Tsugikazu Shibata (high powered executive by day, Linux kernel developer by night) and can be found here for the stable kernel releases, here for the main Linux kernel releases, and here for the kernel development releases.

Kernel subsystem maintainer

When I'm not releasing stable kernels, I also maintain a number of different kernel subsystems. These entail USB, driver core, staging, tty, and a variety of other bits of the kernel. Being a maintainer means you read patches from submitters, handle questions from both developers and users about things related to the subsystem (usually bug reports). If a patch looks acceptable, you test it if possible, and apply it to the relevant git tree and push it publicly, and notify the author that it was accepted. Every weekday, these git trees get merged together in the linux-next release, and inevitably, problems are reported and it's up to the maintainer to fix them when they affect their portion of the kernel.

If you are curious as to exactly what portions of the kernel I maintain, look at the MAINTAINERS file in the kernel source tree and search for my name. Those entries will show you exactly where the git tree for the subsystem lives, as well as the proper mailing list to contact if you have questions in those areas.

If you want to follow the development done in these various areas, and what patches I apply, you can subscribe to the RSS feed of the individual git trees listed in the MAINTAINERS file, or you can follow along on the various different mailing lists.

Kernel development

When not releasing kernels or reviewing patches from others, I occasionally get time to fix bugs, rework existing code to solve problems or extend it in various ways, or even rarer, write a new driver for some random hardware device. This is one area that I should be doing more of now that I have extra time available.

Right now I'm working on a driver for a USB to serial device that Linux doesn't support, and I have some ideas for how portions of the driver core can be reworked to handle some areas better (most of that has been suggested by Kay years ago, I really should get around to implementing them...) I also have some ideas on cleaning up some cruftier portions of the kernel that haven't seen any love for many years, but that's more of a long-term goal, no specifics yet.

If you want to follow along with this development, just watch the main kernel tree for commits by me. That can be done by either subscribing to the rss feed for the kernel tree, or just using git and doing simple searches.

I keep my kernel development and maintainership scripts and directory structure in a public github repo, if you are curious about how this type of thing works. There's lots of scripts helpfully named "do.sh" which I really should rename to be a bit more descriptive, but make sense to me relative to the directory they are located in.

I also have lots of talks, scripts, and other minor projects in my public github repo, if you are curious as to other things I work on over time.

Linux Driver Project

Despite the creaky web page, the Linux Driver project is continuing on quite well. We have written a number of new drivers now included in the main kernel tree, as well as maintaining the staging portion of the kernel. I'll be working on revamping the web site to make it a bit more obvious as to what is going on here, but again, the best way to follow this work is to watch the mailing list.

LTSI kernel maintainership

As has been announced in various places, the LTSI project (Long Term Support Initiative) has started up with the goal to provide a kernel that the consumer electronic companies can use to help reduce their maintenance burden, and to provide a common area where they can learn how to get involved in upstream kernel development.

I'm helping in setting the kernel tree for that project up, and getting some of the procedures and processes in place for it to succeed in the long run. For now, until it really gets up and going, I'm also going to be maintaining the tree myself, handling the patches and working on the support scripts to make it easier to develop using it. If you want to track this work, watch the kernel tree, or join the public mailing list.

I'm also talking with lots of different companies that create chips used in consumer devices that have traditionally been out of the main kernel tree, and with others that are active upstream developers, to try to get them all working better together. I'm also working with the Yocto project to see how the two projects can work together in sharing their kernel needs.

To follow the development of this kernel, you can subscribe to the mailing list, read the archives, or just watch the git tree.

Distribution work

I'm still going to continue my maintenance of the openSUSE Tumbleweed distro, as I've come to rely on it, and it really takes almost no time at all to keep up and working properly. To follow along with any Tumbleweed questions/concerns, please read the openSUSE-Factory mailing list. The scripts used to maintain the Tumbleweed distro, and the list of packages in it, can be seen, and watched, in the tumbleweed github repo.

I'm also going to continue to remain a Gentoo developer, and will have time to do more package maintenance there, which I have not had the opportunity over the past few years.

Both of these are distros that I use every day on my development systems and my servers, and are great community-based distributions.

Travel

"You traveled last year as much as people think you do." -- my wife

As usual, I'll be attending all of the various Linux Foundation events held all around the world, as well as other different conferences that I'm invited to and can find time to get to. Odds are I'll also be traveling to different companies to work with their kernel developers on how to get them to integrate better with the upstream kernel community, or how the LTSI kernel can help them out.

So once again, my frequent flier miles status will probably not be downgraded this year, much to my very patient family's despair.

Is that all?

So, hopefully that explains a bit of what I'll be doing in the near future for the upcoming years. Needless to say, I'm thrilled to be working for the Linux Foundation and that they are supporting me in all of this. If there's anything that anyone is thinking I should be doing but seem not to be, please let me know. I want to make Linux succeed and thrive, and whatever I can do to help that out, I will.

posted Mon, 20 Feb 2012 in [/linux]

I thought it would be easier to do a round of stable kernel releases in the middle of the larger kernel merge window, to prevent the next round from being so big (given that there are a lot of patches usually applying during the -rc1 merge window cycle).

So, I've now done:

• Linux 2.6.32.54-rc1 release
• Linux 3.0.17-rc1 release
• Linux 3.1.9-rc1 release
• Linux 3.2.1-rc1 release

Please go test and let me know if there are any problems with any of these kernels. If I've missed any patches that you feel should be in them, also please let me know.

Note, this is most likely going to be the LAST 3.1.y kernel release, so please move off to the 3.2 kernel at this point in time. Maintaining so many different kernel branches all at once is not trivial, and I want to minimize it if at all possible.

posted Tue, 10 Jan 2012 in [/linux]

As 3.2 is now out, here's a note as to the current status of the different stable/longterm kernel trees.

First off, please everyone remember to mark any patch that you want to have applied to the stable kernel trees with a simple:

Cc: stable <stable@vger.kernel.org>

marking in the Signed-off-by: area. Once the patch hits Linus's tree, I will automatically be notified of it and it will be applied if possible. If it does not applied, you will be notified of that.

Note that the address is stable@vger.kernel.org, not the older address that used to be used before October of 2011.

At this time, all stable and longterm kernel trees are being maintained in one big git tree, located at:

git.kernel.org:/pub/scm/linux/kernel/git/stable/linux-stable.git

There are different branches for every different major kernel version.

Here's the different active kernel versions that I am maintaining at the moment:

• 3.2.y - this will be maintained until 3.3 comes out
• 3.1.y - there will be only one, maybe two, more releases of this tree
• 3.0.y - this is the new "longterm" kernel release, it will be maintained for 2 years at the minimum by me.
• 2.6.32.y - this is the previous "longterm" kernel release. It is approaching it's end-of-life, and I think I only have another month or so doing releases of this. After I am finished with it, it might be picked up by someone else, but I'm not going to promise anything.

All other longterm kernels are being maintained in various forms (usually quite sporadically, if at all), by other people, and I can not speak for their lifetime at all, that is up to those individuals.

If anyone has any questions about any of this, please let me know.

posted Mon, 09 Jan 2012 in [/linux]

tl;dr;

• -stable kernel releases stay the same
• this proposal is how we pick the -longterm releases
• -longterm kernels will be picked every year, and maintained for 2 years before being dropped.
• the same Documentation/stablekernelrules.txt will apply for -longterm kernels, as before.

History:

2.6.16 became a "longterm" kernel because my day job (at SUSE) picked the 2.6.16 kernel for its "enterprise" release and it made things a lot easier for me to keep working at applying bugfixes and other stable patches to it to make my job simpler (applying a known-good bunch of patches in one stable update was easier than a set of smaller patches that were only tested by a smaller group of people.)

Seeing that this worked well, a cabal of developers got together at a few different Linux conferences and determined that based on their future distro release cycles, we could all aim for standardizing on the 2.6.32 kernel, saving us all time and energy in the long run. We turned around and planted the proper seeds within the different organizations and low-and-behold, project managers figured that this was their idea and sold it to the rest of the groups and made it happen. Right now all of the major "enterprise" and "stable" distro releases are based on the 2.6.32 kernel, making this trial a huge success.

Last year, two different community members (Andi and Paul) asked me if they could maintain the 2.6.34 and 2.6.35 kernels as -longterm kernel releases as their companies needed this type of support. I agreed, and they have done a great job at this.

Andi reports that the 2.6.35 kernel is being used by a number of different distros, but they will be phased out as their support lifetime expires. There are also a number of embedded users of the kernel as well as some individual ones. So that -longterm kernel is having a lot of benefit for a wide range of users.

Today:

Now that 2.6.32 is over a year and a half, and the enterprise distros are off doing their thing with their multi-year upgrade cycles, there's no real need from the distros for a new longterm kernel release. But it turns out that the distros are not the only user of the kernel, other groups and companies have been approaching me over the past year, asking how they could pick the next longterm kernel, or what the process is in determining this.

To keep this all out in the open, let's figure out what to do here. Consumer devices have a 1-2 year lifespan, and want and need the experience of the kernel community maintaining their "base" kernel for them. There is no real "enterprise" embedded distro out there from what I can see. montaVista and WindRiver have some offerings in this area, but they are not that widely used and are usually more "deep embedded". There's also talk that the CELF group and Linaro are wanting to do something on a "longterm" basis, and are fishing around for how to properly handle this with the community to share the workload. Android also is another huge player here, upgrading their kernel every major release, and they could use the support of a longterm kernel as well.

Proposal:

Here's a first cut at a proposal, let me know if you like it, hate it, would work for you and your company, or not at all:

• a new -longterm kernel is picked every year.
• a -longterm kernel is maintained for 2 years and then dropped.
• -stable kernels keep the same schedule that they have been (dropping the last one after a new release happens.) These releases are best for products that require new hardware updates (desktop distros, community distros, fast-moving embedded distros (like Yocto)).
• the normal -stable rules apply to these -longterm kernels as described in Documentation/stablekernelrules.txt

This means that there are 2 -longterm kernels being maintained at the same time, and one -stable kernel. I'm volunteering to do this work, as it's pretty much what I'm doing today anyway, and I have all of the scripts and workflow down.

Public Notifications:

The current kernel.org site doesn't properly show what is and is not being maintained as a -stable and -longterm kernel. I have a proposal for how to fix this involving 'git notes', I just need to sit down and do the work with the kernel.org admins to get this running properly.

Thoughts?

Feel free to comment on the google+ thread about this, or on the lkml thread.

posted Sun, 14 Aug 2011 in [/linux]