There are two types of users: 1) Those who want a very stable, rock solid environment where they can run their applications without worrying about breaking things; and 2) Those who want the latest technologies to play with. openSUSE is now targeting both use cases with rock solid Leap and rolling release Tumbleweed. At SUSECon, I sat down with Richard Brown, who has played an instrumental role in openSUSE Tumbleweed. Brown chairs the openSUSE Board and recently joined SUSE (Figure 1).

Figure 1: Richard Brown at SUSECon.

The Brief History of Tumbleweed

The name Tumbleweed has been around for a very long time; kernel developer Greg Kroah-Hartman started the project back in 2010. Kroah-Hartman called it "a repo that is a rolling updated version of openSUSE, containing the latest 'stable' versions of packages for people to use." It wasn't a rolling release distribution in the true sense; in essence, it was actually rolling updates on top of the stable releases of openSUSE.

"Since the Tumbleweed project lacked manpower, Greg ended up doing all the work. I tried Tumbleweed a couple of times and things would break after a while, and it would take Greg to go and fix them," Brown said.

Tumbleweed enjoyed some popularity initially, but because the entire stack was not moving forward, the user interest faded and the user-base declined.

Brown had been championing the idea of a rolling release, though. He used to talk to Kroah-Hartman about it but struggled with Kroah-Hartman's vision of rolling updates. In the end, most conversations ended in typical open source fashion: If you want this change, go ahead and do it.

And, that's what happened in the end. The openSUSE community went ahead and created a rolling release. But, it was not Kroah-Hartman's Tumbleweed; the new Tumbleweed was created from the Factory code branch (Figure 2).

Figure 2: openSUSE Tumbleweed.

The Mutation

"To talk about what is Tumbleweed today, we have to talk about Factory actually," said Brown. Factory was the development code base for openSUSE for over a decade. It was similar to Debian unstable or Fedora Rawhide: Everything went there without any testing. Some brave users would use it; it would work for a while, and then occasionally it would spontaneously combust.

A while ago, openSUSE developers started playing with the Factory development branch, trying to find ways in which they could develop it in a more reliable way. Their work led to the creation of OpenQA, a tool that can automatically run an operating system through rigorous testing.

When you are testing a Linux-based operating system, you are looking at hundreds of different scenarios: how and where a user may install it, what desktop environments will be used, how different components or the entire stack will be updated. Testing all those scenarios is a herculean task if done manually, especially when dealing with a continuously moving target like a rolling release. OpenQA was developed to automate the testing process. OpenSUSE developers soon realized that, with OpenQA, they had inadvertently created a true rolling release distribution. So, for a brief period, openSUSE had two rolling release distributions: Factory and Tumbleweed.

Almost one year ago, the openSUSE community decided that it made no sense to have two rolling release distributions. It was causing user confusion as well as duplication for developers. They decided to keep only one rolling release, which was based on Factory. The idea of openSUSE rolling release came from Kroah-Hartman, though, so they wanted to give him credit. They asked his permission to use the name Tumbleweed for this rolling release.

Brown recalled that Kroah-Hartman was really happy about the idea because this was what he had envisioned; this is what he would have done himself, if maintaining his own version had not been too much work for one person. Thus Factory became the place where all crude development work happens, and Tumbleweed became the rolling distribution associated with the Factory development process. There is no longer a distribution called Factory.

How Stuff Works

Open Build Service (OBS) and OpenQA are the key components of Tumbleweed. The developers have a very simple workflow; everything from Gnome, KDE, Apache, etc. goes to OBS. Package maintainers/developers self-organize into groups looking after a particular stack or group of packages. OpenSUSE developers create a project in the build service called a "devel project," and it's those devel project submissions that are accepted into staging and then the Factory code base.

"Factory can't be used by any human being, but we have a process called staging where we get these incoming pull requests, and then we use the power of OBS to make what in essence is a distribution," said Brown.

OBS basically compiles an openSUSE distribution from these packages for the sole purpose of testing. According to Brown, "it's a special spin of openSUSE Tumbleweed, where we have everything that previously worked in Tumbleweed plus these new experimental changes. So, we create this special distribution where changes have been staged, for the sole purpose of testing. Then, we throw this distribution at our OpenQA, which will actually check it quite extensively, but in a very quick manner. It will check if we could accept the new kernel, new GCC change, or whatever without breaking the distribution. If anything breaks, then those submissions can't get into Tumbleweed."

The moment something breaks, OpenQA sends a message to the developer informing him why the package can't be accepted. The developer can fix it and submit it again.

Once everything is staged, checked, and validated, it goes into Factory proper, where everything goes through extensive testing. Factory proper is the actual project called Factory, where everything is integrated and tested together. Stagings only include a subset of changes, grouped together normally based on which devel project they come from.

For example, suppose both KDE and Gnome want to update at the same time. In that case, KDE's proposed changes will be put into a staging project, let's call it Staging:K, and Gnome's proposed changes will be put into a different one: Staging:G. In each of those staging projects, only the KDE or the Gnome changes exist, so openQA can confirm that, on their own, the changes don't break anything. Thus, these changes can be accepted into Factory.

Factory is where everything mixes together, however, so it's entirely possible that new Gnome might break new KDE, or vice versa, so it's in Factory where the more in-depth testing takes place (Figure 3).

Figure 3: Tumbleweed system info.

OpenQA plays an important role, because changes made to one package might not break something, but when other changes are made to different packages and put together, something else might break. That's what trips up other distributions at the last moment. Other distributions either test things manually "or just cross their fingers that it won't break. Take, for example, the Arch approach, where the philosophy is very much that 'you should be doing this to learn how to fix it yourself'," said Brown.

He added that the openSUSE project considers it extremely important to create a stable and reliable operating system. "We try to make a rolling release that is easier to live with. We have a responsibility as a distribution house. We are still producing a distribution, and we want it to work for us as well because most of us are running Tumbleweed."

Seeing the capabilities of OpenQA, even Fedora and SUSE have started using it for internal testing.