No discussion of Linux is complete without a look at the sprawling yet wonderfully efficient operation at the center of it all: the kernel development project. For an inside look, we chased down one of the leading insiders: Greg Kroah-Hartman is a Linux Foundation fellow and the maintainer of the kernel stable branch. He also created the udev device manager, founded the Linux Driver project, and worked on many other drivers and subsystems in the Linux space. We asked Greg how he got started – and how an aspiring kernel hacker who is new in the field could take their first steps.

Linux Magazine: For readers who don't know you, how long have you been working on the Linux kernel, what are you working on, and how does that influence your daily routine?

Greg Kroah-Hartman: I started contributing to Linux pretty late compared to many other core kernel developers, with my first patches getting merged I think in 1997 or 1998. I had been using Linux for quite a while before that, but I just did not have the time to contribute to it before then.

I'm currently working on the stable Linux kernel releases that I do about once or twice a week, as well as serving as the subsystem maintainer for a number of different driver subsystems, and sometimes I get to write code for cleanups or new features that come up. I'm also one of the kernel security team members, so I do a lot of triage and small bug fixes for issues that are reported to that group.

My daily routine is reading lots and lots of emails. I think I average about 1,000 a day that I need to do something with, either just skim and file away or sort and put off to review at a later time.

After sorting everything, I either work on a specific thing like stable kernel patches or subsystem patches, so I bring up the specific mailbox and process them. I've documented how I do the review and handling of patches in way too much detail in an older blog post at my website [1].

LM: Suppose I work for a device manufacturer who wants to add Linux support for a device. I'm the poor soul who is supposed to write the driver and talk to the kernel people. Do I ask for directions first, or do I read intensively about the topic before? What would you recommend?

GKH: If you have no experience in Linux kernel development at all, then yes, read intensively! We have lots of documentation that goes into the whole development process we follow, starting with a simple HowTo that provides links to almost everything you could want to know [2].

Also good reading is a summary of how the whole development process works [3], which is important for understanding what kernel trees to test against, what the release cycle is, and how everything fits together.

After consuming all of that, I recommend subscribing to the developer mailing list for the subsystem that corresponds to the device that you are working on. A list of all of these mailing lists is available in the MAINTAINERS file in the root directory of the kernel source code.

Reading the messages posted there will give you a sense of how to submit code, what the format is, who gives good reviews, and other subsystem-specific things to be aware of as you create the changes you need for your device.

And finally, once you have some changes you want to get accepted, reading about the patch process is essential [4].

LM: The kernel is mainly written in C, and C is a pretty complicated and low-level language compared to others.

GKH: C is a very simple language; I don't think I've heard it called "complicated" in a long time. C is type-safe. Yes, you can override it if "you know what you are doing," like any good low-level language, but we almost never do that in the kernel. And yes, other languages do allow you to not worry about things like garbage collection and memory management, but when you are writing the core code for the system that has to deal with memory management, you do have to pay attention to stuff like that; you can't just ignore it and "hope" your language runtime authors did the work for you.

LM: Is this lack of automation in C a disadvantage for finding new kernel developers, or could it become one in the future?

GKH: For finding new developers, no, I do not think, because it is a simple language that is easy to pick up. As for the future, I'm not making any guesses.

LM: Would you say that it is enough to take a few lessons in C before starting with kernel development? Or would you rather leave the kernel work to more experienced programmers with solid C knowledge?

GKH: You do need solid C knowledge to do kernel development; the kernel should not be the first project you do learning the language. Learn the language on userspace programs that provide more infrastructure, common libraries, easier debugging, and a much more forgiving environment. A mistake in a userspace C program can't cause your machine to reboot, unlike a mistake in the kernel.

LM: In your perspective, what is the biggest challenge that aspiring kernel hackers face today?

GKH: Just finding something to do! Lots of beginner tasks and tutorials are out there, but it is difficult to make the leap from "I did a basic coding style clean" to "here's a real task to work on." It's hard at times to find what to work on if you don't have a specific task you have been assigned to do.

I recommend subscribing to the subsystem mailing list that you are interested in and watching what people report as problems. That's a great way to see real bug reports and to be able to help out with code reviews and find things to fix and work on.

LM: Computers have more resources today, and the rise of cloud native development puts more emphasis on containers and virtual machines. Does it still make sense for people and companies to compile their own kernel to fit it to their needs? Or are more organizations opting for a vanilla kernel?

GKH: It's always better to build a custom kernel if you know exactly what hardware you need it to support, because distro-provided kernels are built for the least-common-denominator. If you know you have the latest processor type, building for your processor only can sometimes provide a real and measurable gain in performance.

If you have an embedded system, it always makes sense to build in only the support for the devices and hardware that you have on your system, as you do not want to waste storage for drivers and features that you never will use.

LM: Some projects mark bugs or feature requests as "good first issue" to encourage new people to contribute to the project. Does something like this exist for the Linux kernel too?

GKH: Yes, we have a number of "janitorial tasks" that can be found on the kernel newbies website, and we have lots of ToDo items listed in the drivers/staging/*/TODO files in the kernel source tree that are there for beginners to help them get involved in the process.

LM: From your perspective, what are the most interesting construction sites in the kernel right now that will shape its future?

GKH: As I'm sure you have heard by now, eBPF [5] is slowly taking over the kernel and how user space interacts with it. The changes in that subsystem are amazing to watch as more and more of the kernel gets hooked up to it. Another really interesting area is the io_uring work from Jens Axboe that provides a new way for user space to do I/O without the need for syscalls [6]. io_uring creates a way for userspace programmers to do amazingly fast I/O that was never possible before.

LM: Do you think the kernel itself should have flavors like some Linux distributions to please different workloads?

GKH: Do distros still do that? As I mentioned, it's good to build kernels that are tuned for specific hardware types because you can get space savings and performance improvements if you know exactly what you are running on. The kernel provides the ability to customize for those types of situations with a very fine level of control. Some say that it's a much too fine of a level of control, and maybe it would be good to provide a higher-level way to configure the build for a common set of features. No one objects to this idea, so someone just needs to step up and do the real work to make it happen.

LM: If you could change one thing in kernel development today, what would it be?

GKH: If you would contribute!