A member of the Fedora Engineering Steering Committee recently asked me what I think a distribution is:

quite a few people have stated what they feel a distribution is which can generally be summarized as “curated, integrated, vetting, testing”

I think that is not a question about what a distribution is, but a question about what a distribution should be. It is not a question about a definition, it is a question about vision.

It doesn’t work as a definition of distributions, because Gentoo is without any doubt a distribution. Qt 6.12 was released yesterday, and it is available for Gentoo builds today. I can’t imagine a definition of “curated” that could possibly describe this development practice. (I hope that it is clear that I think that Gentoo is a very good distribution.) Independent of whether that list of characteristics is good or bad, I don’t think they define what a distribution is.

So that leaves the question of whether that is what a distribution should be.

I have a short answer and I have a long answer, and I can’t really combine them into one, so I’m just going to include them both.

The short(er) answer

Does “curated, integrated, vetted, tested” describe the Fedora distribution?

I think it describes an OS, not a distribution. Some of those characteristics are very important in an OS. The components that make up an OS naturally must be well integrated, for example. But a distribution is more than an OS. A distribution includes a great deal of application software. “Curation” implies careful selection among available options, but most users expect that an application included in a distribution will be the latest stable version available at the time a distribution branches, which doesn’t require any kind of curation.

I think it describes a product, not a project. If you use those terms to describe the distribution as a whole, you are making promises that cannot possibly be delivered consistently for every component, by every volunteer maintainer.

I think you’re describing RHEL, not Fedora. Users who want a system made up of carefully curated, well integrated components that come with promises like “vetting” should look to RHEL. RHEL is in a position to deliver those promises, and one of the important practices that enables Red Hat to deliver those things is choosing what not to ship. When Red Hat branches RHEL from Fedora, they drop about 90% of Fedora’s components so that they can deliver those kinds of product-shaped statements about RHEL.

Part of productization is deciding what you’re not going to do and what you’re not going to serve. And while Fedora cannot be all things to all people, pursuing product characteristics limits what the distribution is and who the distribution serves too much.

And in general, I think that Fedora should usually do the opposite of RHEL. Not because RHEL is a bad model, but because RHEL maintenance practices are a bad model for us. Red Hat is in the business of selling enterprise support contracts. Their maintenance practices reflect the constraints of the promises they’ve made to customers, and support the requirements of their contracts. Fedora does not have those contracts, Fedora does not have those constraints, and Fedora should not imitate maintenance practices that don’t serve its users. Red Hat tells Fedora maintainers that it wants Fedora to fill in “the white space”, to do the things that RHEL isn’t doing. That makes sense to me. For every practice where RHEL and Fedora overlap, there is an audience we leave unserved because of the practices that neither of us adopt.

The even longer answer

An even longer list of reasons that “curated, integrated, vetted, tested” doesn’t work for Fedora:

First, the definition offered is not only not shared by all users, it’s not even shared by all FESCo members. Neal and I have discussed the “right” way to manage Rawhide, and he has expressed the preference that shared components should be updated aggressively and packages that require them that can’t adapt to updates in their dependency set should simply fall out of Fedora. I don’t agree with that approach, and I don’t think our written policies agree with that approach, but the point here is that a set of dependencies that aggressively tracks the latest upstream cannot reasonably be called “curated” or “vetted”. Using those terms to describe a collection made up of the latest release available robs them of any meaning.

Second, that definition doesn’t hold up to scrutiny. Somewhere between 1/6 and 1/3rd of Fedora is significantly out of date and vulnerabilities certainly exist within that set. If we’re charitable to Fedora, we could describe it as 85% fresh, and that sounds pretty good. But 85% of a fence isn’t really a secure fence, and 85% of a foundation isn’t a solid foundation. Describing Fedora’s components as “curated” makes them sound intentional, but there’s a pretty substantial portion of Fedora that’s probably just neglected, or outdated because the effort to update a component and all of the cascading changes required as a result exceeds the available labor.

Third, the concept of a “curated, integrated, vetted, tested” package set is probably valuable for an OS, but the value of those things drops off significantly for applications. The most valuable things for users are applications that are still maintained by their developers. They don’t need “curation”, they just need to follow the release stream they were following at the time Fedora branched (or in many cases the latest release stream, if the upstream doesn’t support branching releases.)

Fourth, release stream selection is work best done by a dependency resolver, and we are doing that work with human labor. We often do not have data that indicates what releases are going to require the least porting efforts, or which ones are going to offer the best security posture, or what our priorities are in the context of a specific component. If we think the characteristics of “curated, integrated, vetted, tested” are actually valuable, we should develop tools to measure them and support our selections. Without data, a selection other than the latest is little more than guesswork.

Fifth, I don’t think this is what desktop users actually want. When I see people in community forums talk about why they recommend Fedora, it’s usually that it offers the latest version of most software, with minimal change relative to the upstream projects.

Sixth, I don’t think this is what users want for production/server use. As an SRE, I think the people best suited to support software are the people who develop it day to day. “Curation” implies that the distribution believes that it is better suited to decide what to release to users than the developers upstream, which I don’t believe is true for most software. It might be true for the OS, but the further up the stack you go, the less justification there is for shipping anything other than the latest available version, and the more harmful it is to delay or filter updates. I want a release that delivers what the upstream projects publish, with minimal friction. I view patching and “curation” downstream as overall negatives. That is a view broadly shared by my professional peers.

Seventh, it’s not what the upstream developers want. They have been asking distributions to patch less for as long as I’ve been working in software development. The practice of producing just one collection with just one version of any given shared component necessitates patching and publishing software configurations that upstream developers have never seen and do not want to support. Any bugs they get resulting from patched build configurations harm our relationship with the developers that we depend on.

Eighth, I don’t think that’s what Red Hat wants. That description frames Fedora as if it were a product, and Red Hat does not want Fedora to present itself as a product. Among other reasons, Products are things that select the target audience they will serve and optimize for that group. I think Red Hat would prefer that Fedora remain open as a space for developers to participate to the greatest extent possible. Numerous package maintainers suggested that Fedora can package and ship complex and actively developed software someday, in the future, when development slows down. I don’t think that Fedora should accept the conclusion that it can or should remain irrelevant to emerging technology fields.

Additionally, “curated” and “vetted” start to sound an awful lot like the promises that products make, and I suspect that Red Hat would prefer that Fedora not present the appearance of having made such promises. In particular, as someone whose work has often focused on compliance and vulnerability management, I find the casual use of the term “vetted” deeply concerning, and I would recommend against using it. Even if there were a handful of components that we felt could be described as such and even if Red Hat did not object to our use of a term that implies certain promises, I don’t think it’s a term that should be used for the distribution as a whole because the vast majority of the software we distribute is definitely not “vetted.”

Ninth, I don’t believe those terms appear in any of our documentation, and the insistence that Fedora Linux is those things is an individual point of view, not a point of view the project appears to have adopted by consensus.

Tenth, that list of characteristics does not include “secure”, which is something I want all distributions to prioritize, and not something I think anyone can assume. In order to be secure, software must be maintained. And it is clear that we are already stretching our maintainers too thin. The people best equipped to provide support are the upstream projects, and the best way for Fedora to pursue security is to ship their patch releases as quickly as possible, not to curate, select, or filter them.

A collaborative place to distribute software

The FESCo member asked, further:

If you want to redefine the idea of a distribution into something that’s more of a ‘collaborative place to build code’ (not ship it) then state that plainly

And to that point, I think that “redefinition” happened a LONG time ago.

Once upon a time, there was a small distribution, Fedora Core. And there was a community-maintained collection of packages, Fedora Extras. And in 2007, almost 20 years ago, the two of them merged. When that happened, the “core” system became more open to community contribution, and the “extras” were built at the core system cadence.

Importing the largest collection of community-maintained applications meant that there was no longer an example for third-party developers to follow, nor a shared place for them to make their software readily available. The merge redefined the idea of the distribution into something more like a “collaborative place to build code”.

That may have been the best solution available at the time, but I don’t think it’s the best way to maintain a large collection today.

For over 30 years, developer-focused software distributions have demonstrated a different model. CPAN came online in 1995. PyPI, in 2003. Rubygems, in 2004. NPM, in 2010. And Crates.io, in 2014. For the most part, there isn’t a need for competing registries the way that there are competing distributions. I think the reason is simple: they don’t impose a release schedule on developers. Every module can publish stable releases on their own release cadence. Stable releases mean that every dependent module is allowed to port forward to new dependency versions asynchronously, and asynchronous work is very efficient because it is not interrupt-driven. Maintaining software in these registries and building software using them is vastly easier that maintaining and building packages in Fedora.

The world of software development has evolved and improved since Fedora Core and Fedora Extras merged, but Fedora’s development and build practices haven’t changed nearly as much. We’re still building an OS and a large collection of applications in a single branch, with synchronous development practices, as if the applications were just features of the operating system. There have been several attempts to decouple the source and build, but they continued to target deployment in a shared FHS prefix, which merely shifted the synchronization problems out to the edge systems. For the most part, I think the registry systems have accepted that this doesn’t work. Python’s pip will warn you that installing packages into the system root is a bad idea, and Python’s developers recommend always using a virtual environment.

I don’t know that all of Fedora’s applications should be decoupled from the OS, but I do believe that there is a need to decouple some applications, and that doing so will make both those applications and the underlying OS easier to develop.

I think that everyone agrees that Fedora is difficult to maintain. And I think that everyone agrees that Fedora is valuable. The question, then, is whether the things that make it difficult are the things that make it valuable. There are a few people who think so, but I do not. There is a small set of components that constitute an OS, and arguably those need the kind of intensive development and maintenance that creates net new value. But that’s not true for every package, and maintainers who believe their package doesn’t fall into that category deserve an option to adopt a build system that allows them to work at their own pace (while maintaining reasonable maintenance standards. I’m not suggesting that we ship derelict software.)