Bug Tracking

One of the ongoing nightmares of Linux kernel development is how to deal with bug reports. The mailing list receives tens of thousands of messages each month, with developers using all sorts of filters and cc habits to try to get the right emails in front of the right eyeballs.

Bugzilla (http://bugzilla.kernel.org), a web-based bug tracking system developed by the Mozilla corporation, has been available for years, but most kernel developers seem to regard it with a mixture of hatred and fear. Once in awhile someone mentions that it should be improved, but those discussions tend to become messy.

Recently Thorsten Leemhuis mentioned that Bugzilla should be improved, and the developers had a discussion about it.

As a preventative measure, Thorsten first pointed out that the servers running the Bugzilla instance, as well as the Bugzilla instance itself, were all maintained by the infrastructure team at the Linux Foundation, who were doing a fantastic job, and who should in no way be confused with anyone actually configuring or using the instance to assign or track bug reports. The infrastructure team kept the tool operational and that was all they were supposed to do. It was up to the kernel developers to actually use the tool (or not) and to configure it according to their needs.

With that out of the way, Thorsten pointed out that the various categories for bug reports, the people assigned to receive those reports, and a lot of other aspects of Bugzilla configuration were "heavily outdated, incomplete, or wrong." He added that while a few kernel subsystems did actively use Bugzilla, "lots of bug and regression reports (even good ones!) never get a reply from a developer." In many cases, submitting a bug on Bugzilla was equivalent to throwing it into a black hole of infinite death.

Thorsten also pointed out that Bugzilla as a software project in its own right appeared to be dead. The upstream code had not received a significant update for years, which suggested that finding an actively maintained replacement might be a better idea than putting in a lot of work to get developers to use the current tool.

Artem S. Tashkinov replied with some tough love. Not only did he agree with Thorsten that Bugzilla needed improvement, but he said, "Bugzilla must be there whether people like it or not. [...] Subsystem _maintainers_ must be present in the bugzilla by definition. You're a maintainer after all. You're expected to be responsible. [...] Kernel bugzilla must be opt-out, not opt-in. To be honest I'd automatically add everyone who's committed to the kernel in the past 6 months and of course if new developers commit to the kernel, I'll add them as well." Artem concluded, "it's so easy to hate/dismiss bugzilla and say 'use our mailing list instead'. Practice shows that 'your mailing lists' are too often completely dysfunctional and allow [bug] reports to linger and get never addressed which is not good for the kernel. I strongly oppose the idea of kernel bugzilla deprecation."

Greg Kroah-Hartman pointed out that subsystem maintainers often did Linux kernel development as part of their paid corporate job, and their time allocation was not necessarily theirs to control.

But Artem said this was illogical and he refused to accept it. He said, "here's a very common scenario: you work for company X. The company tells you to fix a bug/add a new feature/etc. You write the code, submit it and it results in a regression for other use cases. Are you saying it's alright and shouldn't be addressed? That's _exactly_ how many if not _most_ regressions in the kernel are introduced."

To which Greg replied, "Seriously, many of us have been working with many companies to try to change this, but it's slow going and requires constant retraining of new managers when they get moved around. It's a thankless job, please reach out to any contacts that you have to help out."

Meanwhile, Konstantin Ryabitsev, a member of the Linux Foundation's infrastructure team, gave a summary from the perspective of that team:

"The bugzilla as a software platform is a Mozilla product, not Linux Foundation. Unfortunately, it's pretty much dead:

1. all development has stopped years ago
2. it doesn't even work with recent MySQL servers
3. it is written in perl5 and can only pretty much run with mod_perl

"We're committed to running it as far as we can, but we all must also admit that the platform is near-death and probably will become an ever-increasing burden to keep it operating. Heck, one of our IT staff is currently trying to convert bugzilla.kernel.org to use Postgres just so we can keep operating it past the end of 2022.

"The Linux Foundation IT is in charge of running infrastructure – we're not a development shop. All of our software projects are pretty much 'skunkworks'."

He added that the Linux Foundation was potentially in a position to fund developers in some cases – but that this would require a very clear mandate from the community. Konstantin said, "Before we try to fix/replace bugzilla, we really need to figure out the entire process and pinpoint who is going to be the one in charge of bug reports. If you think that LF [Linux Foundation] should establish a fund for a position like that, then you should probably approach LF fellows (Greg KH, Shuah Khan), who can then talk to LF management. The IT team will be happy to support you with the tooling, but tooling should come second to that – otherwise we'll just be replacing an old and rusty dumpster on fire with a new and shiny dumpster on fire."

Artem suggested converting Bugzilla and all supporting systems to use GitLab instead. He said, "that will solve all the issues immediately." For one thing, he said, all patch committers would be automatically present, so they could be cc'd on bug reports. For another, the kernel source code directory tree could easily be split into separate components, and the relevant developers assigned to those components – again, ensuring they would be cc'd on bug reports.

Also, Artem added, "Linus, as a commander, may continue having his local git repo or using its own git website and get merge requests from gitlab.kernel.org. For him barely anything will change (aside from URLs to fetch from)."

But Greg slammed the door firmly on that idea, remarking, "For loads of reasons that have been stated before, we aren't going to move everything to gitlab, sorry. That's a non-starter for a wide range of reasons, not the least being you are trying to solve a 'we have no one who wants to wrangle bugs in bugzilla' problem with 'move all of our code hosting infrastructure to a totally different thing that can't even provide the basic things that we have today'. Sorry, not going to happen, gitlab is not the solution here."

Konstantin also noted that with GitLab, "you will still have all the exact same problems as long as nobody is in charge of handling incoming bugs. There are plenty of active github/gitlab projects with thousands of open issues that nobody is working on for the exact same reason nobody is working on bugs filed via bugzilla – the right people didn't see it (or are actively ignoring it, because they are working on something else)."

In addition, Konstantin also reminded Artem, "Gitlab is also a commercial open-core project. It is permanently in danger of being swallowed by some $ENTITY_NOBODY_LIKES, who will for sure look to prioritize what kinds of things go in to the 'open core' part and what kinds of things are only available with subscription, in order to improve profit margins."

To which Artem replied, "That leaves us with Bugzilla that no one wants to touch and some people actively want to delete altogether. In other words, no central place to report bugs or keep track of them."

James Bottomley pointed out that there was no truly obvious fix for the situation. He said:

"The sad fact is that a lot of bug reports aren't actionable, meaning the reporter can't give a reproducer and also can't easily test patches[;] sometimes by luck the maintainers can work out what the issue is but a lot of the time they have no idea. Then there are tons of bug reports with responses like 'I think xxx commit fixes your problem, can you test it' for which the conversation dies there. There's also the thundering herd problem: some bugs get reported by many different people (as different bug reports) but usually the subsystem only engages with one to fix the issue. In theory bugzilla can mark the latter as dups, but that requires someone to spend an enormous amount of time on evaluation and admin.

"That's not to say we can't improve our process, it's just to set expectations that we're never going to approach anywhere near a perfect bug process. Most of the improvements that worked so far involve having someone coach bug reporters through the process of either testing patches or reproducing the problem in a more generic environment … which I think most people would agree can't really fall wholly on maintainers."

At around this point, various developers did start to consider potential bug tracking solutions. One idea was to rely exclusively on email – simply track bugs via the mailing list, and use scripts or some other form of infrastructure to supplement that. In fact, Laurent Pinchart objected to this idea, saying that there really was no way to track the status of bug reports and fixes using email alone. But Thorsten replied, "I'd disagree partially with that, as my regression tracking bot 'regzbot' [...] does exactly does that: tracking, by connect the dots (e.g. monitoring replies to a report as well recording when patches are posted or committed that link to the report using Link: tags), while making sure nothing important is forgotten. But sure, it's still very rough and definitely not a full bug-tracker (my goal is/was to not create yet another one) and needs quite a bit of hand holding from my side. And I only use it for regressions and not for bugs (on purpose)."

Laurent replied, "Patchwork does something similar for patches, and I agree that it would be possible to use e-mail to manage and track bug reports with tools on top (and don't worry, I'm not asking for regzbot to be turned into a bug tracker :-)). It however has to rely on lots of heuristics at the moment, as the data we exchange over e-mail is free-formed and lacks structure. I've been dreaming of support for structured data in e-mails, but that's a pipe dream really."

Also, Laurent made another point, saying, "The huge elephant in the room is that most maintainers are overworked. Whether a bug report arrives in my mailbox as an e-mail straight from the reporter or from a bug tracker will make very little difference if I don't have time to look into it (I would even argue that bug trackers are even worse there: if I'm really short of time, I'm more likely to prioritize replying to e-mails instead of having to open a link in a web browser). As long as we don't address the maintainer bottleneck in the kernel, bug tracking will suffer."

Artem, however, was strongly against an email-based solution. He said, "Debian uses an email based bug tracker and you know what? Most people avoid it like a plague. It's a hell on earth to use. Ubuntu's Launchpad which looks and feels like Bugzilla is a hundred times more popular."

To which Laurent replied sardonically, "It would be pretty sad if the only options we could come up with for bug tracking would be either popular with reporters and ignored by maintainers, or the other way around."

Tony Luck defended a web-based approach, saying, "Web interfaces have the advantage that they can be full of boxes which indicate useful details to supply. Like what kernel version? Did this work on an older version, [if] so, which one? Which CPU vendor/model are you using? Is there an error message? Are there warnings in the console log before the error? Can you upload a full console log? Does this happen repeatably? What are the steps to reproduce? Etc.etc. Sometimes it takes a few round trips by e-mail to establish the baseline facts."

Slade Watkins agreed, saying, "Email – imo – is good for discussions, but not for reporting bugs. Web has upsides of being easier to navigate (sometimes faster) with just a few clicks/keyboard shortcuts and some words to describe an issue, steps to reproduce, kernel versions it affects, etc."

On the flip side of the equation, Theodore Ts'o said, "Funny thing. I've largely given up on getting any kind of useful bug report from Launchpad, so I've largely ignored it. In contrast, the bug reports I get for e2fsprogs from Debian are generally far more actionable, with bug reports that have all of the data so I can actually root cause the problem, and help the user."

He added, "So Launchpad may be pretty, but perhaps because of selection bias, the bug reports I've seen there are generally a waste of my time, and if I'm going to choose which users I'm going to help for *free*, it's going to be the one which is far less frustrating to me as the volunteer. '100 times more popular' is not necessarily a feature if what we get is 1000 times the noise."

Theodore further added, "Artem, it seems to me that you are hoping that volunteers will provide a commercial level of support – and that's just never going to happen. The users vastly outnumber us developers by orders of magnitude, and [...] we need to clearly express that any kind of support is best efforts only, and if someone has anything business-, mission-, or life-critical, they should darned well pay $$$ for a proper support contract."

Linus Torvalds also came down – at least for the moment – on the side of preferring email over a web interface. When Artem remarked, "I just want a bugzilla where I can CC _any_ developer _if_ and _only if_ they are willing to work within its confounds. That's it." Linus then replied:

"Guess what that 'add developer to the Cc' is called?

"Email.

"What you do is fill in the bugzilla entry with all the data you want.

"Then you use email to inform people about it.

"Put enough data in the email that the developer knows whether it's even worth looking at the bugzilla entry or not. Don't just put a link to the bugzilla. Most developers will just go 'oh, this looks like spam'. Put the overview in the email, enough information that the developer can go 'Ahh, this is worth my time', _and_ the link to bugzilla.

"That gives you exactly what you ask for: you can CC _any_ developer. And it doesn't force the developer to have to go to some bugzilla web interface unless the developer thinks it actually adds value.

"This is *literally* how I end up using bugzilla. As you say, I actually do end up looking at bugzilla entries in the end, but I only do it once it has hit my mailbox first, and I have some fairly good indication that it's worth my time to look at it.

"And yes, for some projects and for some developers you can do that email integration from within bugzilla itself. That's how people reach me.

"But this is exactly the kind of part of bugzilla that is a TOTAL HORROR-SHOW to manage, and it's impossible to expect every developer to be somebody that can be listed on bugzilla, without bugzilla becoming a prime way to send spam.

"Which is why in the general case, you really should consider email to be the 'lingua franca' of kernel development communication. It doesn't have the fundamental limitations and management issues that bugzilla has. If you want to add more people to the Cc in an email, you just do it."

The discussion is not over by any means. I would expect the debate to be ongoing for years to come, in spite of Linus's stated preference. If Linus said something like, "we will never use a web-based bug tracker," then the debate might end. But it's one of Linus's great features that he won't put a definitive end to a discussion unless he feels it is really truly true that one side is absolutely correct and the other absolutely wrong. He kept the revision control discussion going for years and years, until finally he had no choice but to implement a proper solution himself. I don't think he'll do the same for a bug tracking system, but we can hope.