Secure Boot is, effectively, an allow-list. It uses digital signatures to limit the set of software that can run in privileged mode. It’s a much more effective mechanism for blocking malware.

Social media threads on the topic frequently include dismissive comments about Secure Boot. I want to caution social media users to consider how well they know the people who make those comments and how much trust they have earned. They may claim that Secure Boot is not useful, but being unable to describe its use does not mean it has no use.

Understanding the Threat Landscape

Malware on any operating system tends to use software exploits to gain persistence. It typically does not rely on the user to run and authorize its access.

A lot of people look at Secure Boot as protecting the pre-boot environment, as if it is a brief event. It isn’t. In addition to the OS you interact with on a modern x86 system, there are (at least) two and a half other operating systems running at all times, with more control over the system than your primary OS. As this presentation explains, these lower-level systems have privileged access that your regular OS doesn’t have.

Secure Boot’s purpose is to protect your kernel and these lower-level operating systems from malware. This protection is critical because malware that gains kernel access is nearly impossible to detect (though it can usually be eliminated by wiping the drive and reinstalling), and malware that gains firmware access is both nearly impossible to detect and nearly impossible to remove. Real-world examples like this firmware bootkit demonstrate why this protection is essential. Rootkits that embed themselves in firmware are becoming more common, and Secure Boot is one of the recommended mitigations.

How Malware Escalates Privileges

Let’s walk through a concrete attack scenario to see how Secure Boot provides protection.

Once malware gets on your system, the malware is likely to begin execution in your user context. The POSIX multi-user design prevents malware from modifying the system outside what your user has permission to modify, unless it can leverage another exploit to get root.

Imagine that malware manages to gain access to a system, and further is able to use a local exploit to get root access. Maybe it joins a botnet at that point. It’s probably going to take extra steps in order to persist (which is to say that it’ll save itself to a file or download a file to execute in the future after a system reboot, and it’ll modify the boot process to execute that file). Now, unless it takes additional steps, it’s detectable. You can use “ps” to see it in the process list, or “ls” to see its files on disk.

How Malware Hides Itself

Many types of malware will take additional steps to hide themselves. The easy way to do that would be to modify “ps” and “ls” so that they no longer show the malware in their output. Simple, right? But what if you use “find” to look at files, or “top” to look at processes? What if you apply updates and overwrite the modified tools? A more complete hiding effort involves loading a kernel module to that the kernel itself no longer tells user-space about the malware’s files, processes, or network traffic! Now when the operator runs “ls /” or “find /”, the malware’s kernel module filters the responses to readdir(), and never includes files that contain the malware.

A modular kernel like Linux inherently allows loading software that can operate at a very low level, and can prevent anti-virus software from discovering and removing the malware.

How Secure Boot Prevents These Attacks

In a legacy design, root is the highest level of access, and nothing prevents malware from modifying the kernel or the system firmware. Secure Boot adds another level of separation, protecting the system firmware and the kernel from modification by malware.

Linux Secure Boot systems with kernel lockdown will not allow modules to load unless they are signed, and that makes it very difficult if not impossible for an attacker to load a kernel module that can hide malware. Malware can still modify user-space tools directly, to try to hide itself, but it’s much much easier to overcome that to determine if a system is infected or not.

Further Reading

For those interested in the technical details:

• An example malware module demonstrates the hiding techniques described above
• A comprehensive series describing how Linux rootkits work (starting with post 1 and proceeding for 9 total posts)

quite a few people have stated what they feel a distribution is which can generally be summarized as “curated, integrated, vetting, testing”

I think that is not a question about what a distribution is, but a question about what a distribution should be. It is not a question about a definition, it is a question about vision.

It doesn’t work as a definition of distributions, because Gentoo is without any doubt a distribution. Qt 6.12 was released yesterday, and it is available for Gentoo builds today. I can’t imagine a definition of “curated” that could possibly describe this development practice. (I hope that it is clear that I think that Gentoo is a very good distribution.) Independent of whether that list of characteristics is good or bad, I don’t think they define what a distribution is.

So that leaves the question of whether that is what a distribution should be.

I have a short answer and I have a long answer, and I can’t really combine them into one, so I’m just going to include them both.

The short(er) answer

Does “curated, integrated, vetted, tested” describe the Fedora distribution?

I think it describes an OS, not a distribution. Some of those characteristics are very important in an OS. The components that make up an OS naturally must be well integrated, for example. But a distribution is more than an OS. A distribution includes a great deal of application software. “Curation” implies careful selection among available options, but most users expect that an application included in a distribution will be the latest stable version available at the time a distribution branches, which doesn’t require any kind of curation.

I think it describes a product, not a project. If you use those terms to describe the distribution as a whole, you are making promises that cannot possibly be delivered consistently for every component, by every volunteer maintainer.

I think you’re describing RHEL, not Fedora. Users who want a system made up of carefully curated, well integrated components that come with promises like “vetting” should look to RHEL. RHEL is in a position to deliver those promises, and one of the important practices that enables Red Hat to deliver those things is choosing what not to ship. When Red Hat branches RHEL from Fedora, they drop about 90% of Fedora’s components so that they can deliver those kinds of product-shaped statements about RHEL.

Part of productization is deciding what you’re not going to do and what you’re not going to serve. And while Fedora cannot be all things to all people, pursuing product characteristics limits what the distribution is and who the distribution serves too much.

And in general, I think that Fedora should usually do the opposite of RHEL. Not because RHEL is a bad model, but because RHEL maintenance practices are a bad model for us. Red Hat is in the business of selling enterprise support contracts. Their maintenance practices reflect the constraints of the promises they’ve made to customers, and support the requirements of their contracts. Fedora does not have those contracts, Fedora does not have those constraints, and Fedora should not imitate maintenance practices that don’t serve its users. Red Hat tells Fedora maintainers that it wants Fedora to fill in “the white space”, to do the things that RHEL isn’t doing. That makes sense to me. For every practice where RHEL and Fedora overlap, there is an audience we leave unserved because of the practices that neither of us adopt.

The even longer answer

An even longer list of reasons that “curated, integrated, vetted, tested” doesn’t work for Fedora:

1. The definition offered is not only not shared by all users, it’s not even shared by all FESCo members. Neal and I have discussed the “right” way to manage Rawhide, and he has expressed the preference that shared components should be updated aggressively and packages that require them that can’t adapt to updates in their dependency set should simply fall out of Fedora. I don’t agree with that approach, and I don’t think our written policies agree with that approach, but the point here is that a set of dependencies that aggressively tracks the latest upstream cannot reasonably be called “curated” or “vetted”. Using those terms to describe a collection made up of the latest release available robs them of any meaning.

2. That definition doesn’t hold up to scrutiny. Somewhere between 1/6 and 1/3rd of Fedora is significantly out of date and vulnerabilities certainly exist within that set. If we’re charitable to Fedora, we could describe it as 85% fresh, and that sounds pretty good. But 85% of a fence isn’t really a secure fence, and 85% of a foundation isn’t a solid foundation. Describing Fedora’s components as “curated” makes them sound intentional, but there’s a pretty substantial portion of Fedora that’s probably just neglected, or outdated because the effort to update a component and all of the cascading changes required as a result exceeds the available labor.

3. The concept of a “curated, integrated, vetted, tested” package set is probably valuable for an OS, but the value of those things drops off significantly for most applications. The most valuable characteristic for users of those applications is that they are still maintained by their developers. They don’t need “curation”, they just need to follow the release stream they were following at the time Fedora branched (or in many cases the latest release stream, if the upstream doesn’t support branching releases.)

4. Release stream selection is work best done by a dependency resolver, and we are doing that work with human labor. We often do not have data that indicates what releases are going to require the least porting efforts, or which ones are going to offer the best security posture, or what our priorities are in the context of a specific component. If we think the characteristics of “curated, integrated, vetted, tested” are actually valuable, we should develop tools to measure them and support our selections. Without data, a selection other than the latest is little more than guesswork.

5. I don’t think this is what desktop users actually want. When I see people in community forums talk about why they recommend Fedora, it’s usually that it offers the latest version of most software, with minimal change relative to the upstream projects.

6. I don’t think this is what users want for production/server use. As an SRE, I think the people best suited to support software are the people who develop it day to day. “Curation” implies that the distribution believes that it is better suited to decide what to release to users than the developers upstream, which I don’t believe is true for most software. It might be true for the OS, but the further up the stack you go, the less justification there is for shipping anything other than the latest available version, and the more harmful it is to delay or filter updates. I want a release that delivers what the upstream projects publish, with minimal friction. I view patching and “curation” downstream as overall negatives. That is a view broadly shared by my professional peers.

7. It’s not what the upstream developers want. They have been asking distributions to patch less for as long as I’ve been working in software development. The practice of producing just one collection with just one version of any given shared component necessitates patching and publishing software configurations that upstream developers have never seen and do not want to support. Any bugs they get resulting from patched build configurations harm our relationship with the developers that we depend on.

8. I don’t think that’s what Red Hat wants. That description frames Fedora as if it were a product, and Red Hat does not want Fedora to present itself as a product. Among other reasons, Products are things that select the target audience they will serve and optimize for that group. I think Red Hat would prefer that Fedora remain open as a space for developers to participate to the greatest extent possible. Numerous package maintainers suggested that Fedora can package and ship complex and actively developed software someday, in the future, when development slows down. I don’t think that Fedora should accept the conclusion that it can or should remain irrelevant to emerging technology fields.

   Additionally, “curated” and “vetted” start to sound an awful lot like the promises that products make, and I suspect that Red Hat would prefer that Fedora not present the appearance of having made such promises. In particular, as someone whose work has often focused on compliance and vulnerability management, I find the casual use of the term “vetted” deeply concerning, and I would recommend against using it. Even if there were a handful of components that we felt could be described as such and even if Red Hat did not object to our use of a term that implies certain promises, I don’t think it’s a term that should be used for the distribution as a whole because the vast majority of the software we distribute is definitely not “vetted.”

9. I don’t believe those terms appear in any of our documentation, and the insistence that Fedora Linux is those things is an individual point of view, not a point of view the project appears to have adopted by consensus.

10. That list of characteristics does not include “secure”, which is something I want all distributions to prioritize, and not something I think anyone can assume. In order to be secure, software must be maintained. And it is clear that we are already stretching our maintainers too thin. The people best equipped to provide support are the upstream projects, and the best way for Fedora to pursue security is to ship their patch releases as quickly as possible, not to curate, select, or filter them.

A collaborative place to distribute software

The FESCo member asked, further:

If you want to redefine the idea of a distribution into something that’s more of a ‘collaborative place to build code’ (not ship it) then state that plainly

And to that point, I think that “redefinition” happened a LONG time ago.

Once upon a time, there was a small distribution, Fedora Core. And there was a community-maintained collection of packages, Fedora Extras. And in 2007, almost 20 years ago, the two of them merged. When that happened, the “core” system became more open to community contribution, and the “extras” were built at the core system cadence. The application development cycle became tightly coupled with the OS development cycle.

Importing the largest collection of community-maintained applications meant that there was no longer an example for third-party developers to follow, nor a shared place for them to make their software readily available. The merge redefined the idea of the distribution into something more like a “collaborative place to build code”.

That may have been the best solution available at the time, but I don’t think it’s the best way to maintain a large collection today.

For over 30 years, developer-focused software distributions have demonstrated a different model. CPAN came online in 1995. PyPI, in 2003. Rubygems, in 2004. NPM, in 2010. And Crates.io, in 2014. For the most part, there isn’t a need for competing registries the way that there are competing distributions. I think the reason is simple: they don’t impose a release schedule on developers. Every module can publish stable releases on their own release cadence. Stable releases mean that every dependent module is allowed to port forward to new dependency versions asynchronously, and asynchronous work is very efficient because it is not interrupt-driven. Maintaining software in these registries and building software using them is vastly easier that maintaining and building packages in Fedora.

The world of software development has evolved and improved since Fedora Core and Fedora Extras merged, but Fedora’s development and build practices haven’t changed nearly as much. We’re still building an OS and a large collection of applications in a single branch, with synchronous development practices, as if the applications were just features of the operating system. There have been several attempts to decouple the source and build phases, but they continued to target deployment in a shared FHS prefix, which merely shifted the synchronization problems out to the edge systems. For the most part, I think the registry systems have accepted that this doesn’t work. Python’s pip will warn you that installing packages into the system root is a bad idea, and Python’s developers recommend always using a virtual environment.

In 2007, we did not have container runtimes with which to deploy applications, but now we do, which means that we can develop, build, and distribute software, decoupled from the development of the OS.

I don’t know that all of Fedora’s applications should be decoupled from the OS, but I do believe that there is a need to decouple some applications, and that doing so will make both those applications and the underlying OS easier to develop.

I think that everyone agrees that Fedora is difficult to maintain. And I think that everyone agrees that Fedora is valuable. The question, then, is whether the things that make it difficult are the things that make it valuable. There are a few people who think so, but I do not. There is a small set of components that constitute an OS, and arguably those need the kind of intensive development and maintenance that creates net new value. But that’s not true for every package, and maintainers who believe their package doesn’t fall into that category deserve an option to adopt a build system that allows them to work at their own pace (while maintaining reasonable maintenance standards. I’m not suggesting that we ship derelict software.)

Fred Brooks’ The Mythical Man Month is best known for the observation that adding manpower to a late software project makes it later. This is now known as Brooks’ Law. He offered several explanations that contribute to this outcome, including the effects of coordination and communication.

I missed the 50th anniversary of The Mythical Man Month by a bit, but I wanted to talk about one of the biggest changes in software development as it relates to the book. When I started developing software, 30 years ago, version control systems with workflows centered around locking a file for one developer’s exclusive access were still common. If you never worked with those systems, you may not appreciate that along with better testing practices, some of the most significant productivity improvements of the last 50 years has been been driven by branching (in various forms), which significantly reduces the friction that Brooks observed.

Branching is used not just in source code, but in releases as well. I spend a lot of time talking about the stable release model. The modern stable release model is a branching model. It provides overlapping release streams in which the extent of change within a stream is less significant than the extent of change from stream to stream. The overlapping streams allow consumers to test and migrate from stream to stream asynchronously. These developers are more efficient because the availability of multiple supported releases allows them to schedule work based on their own priorities.

Interrupt-driven work is a well known killer of productivity. Imagine that you develop a service that uses shared components for TLS, RPC, PubSub messaging, and KeyVal data storage. In this hypothetical production environment, the TLS library published updates every 4 weeks, and RPC publishes updates every 6 weeks, and PubSub publishes updates once a month, and KeyVal publishes updates once a month but not at the same time as PubSub. If the client library releases for these services are linear and not branching, then you may need to adapt your software to track changes in these libraries in order to continue building and testing your own releases, when those releases occur. Those interrupts can make it difficult to find uninterrupted periods in which to develop features of your own. You will probably find your team padding their schedule in order to provide time for testing and porting efforts that can be unpredictable in magnitude, even if they have a predictable frequency.

If release streams overlap, then developers who use those client libraries can schedule updates when it is convenient. They can do that work any time between the initial availability of a new client library and the end of support for the one they were previously using. This improves productivity for users of the client library by removing uncertainty in the development process, which leads to less schedule padding. It tends to improve productivity for the teams providing those client libraries as well, because the urgency of most support requests drops significantly.

This model requires slightly more work, but in exchange, teams avoid most of the coordination that Brooks observed constraining development velocity. The extra work doesn’t slow development down, it speeds it up. Many people try to make systems more efficient by reducing the amount of work required, but in this case the efficiency gains are significantly greater than the work required to continue maintaining older release series.

The distribution release process may require developers to coordinate and synchronize

Stable release distributions like Fedora aim to deliver the benefits of the stable release model for the collection of software they provide, so that users of the distribution don’t have to manage thousands of components individually. They do this by coordinating updates that affect applications across the distribution. In some cases, that means doing the work required to port components to new interfaces provided by the update. The result is a software collection with minimal duplication of code. But, ironically, because they are working to eliminate multiple release streams for shared components, distribution package maintainers don’t get the benefit of the stable release process for their own work. Just like the hypothetical production network, maintainers may be notified that updates have broken their software, at which point they have to adapt to the change to ensure that users can continue to use their software. In this way, they reintroduce the inefficiency that Brooks observed and reimpose the problem that the stable release process solves.

That work is necessary to produce an integrated OS that implements the concepts described in the Linux FHS. but from the point of view of any given application developer, the shared libraries present in the OS are mostly a coincidence, and one which varies from distribution to distribution. Application developers will tend to be most productive using registries like PyPI, crates.io, or npm which allow them to schedule maintenance work asynchronously from any target operating system.

First, we understand that software is typically versioned, and that a sequence of releases has a beginning and and end.

Furthermore, we understand that applications use shared libraries and other types of dependencies, which share the same basic characteristics. Shared libraries are also versioned, with a finite maintenance window.

In addition to those characteristics, an application’s dependencies share one more characteristic, which is that the dependencies were released before the application.

Dependencies aren’t meaningfully different from applications. They may also have dependencies, and when they do, those dependencies will typically have been released earlier, in turn.

And, like applications, dependencies typically follow stable release practices. So, for example, at any given time, the developers of a hypothetical “libEtc” may be supporting multiple releases.

Which brings us to build system requirements. At a given point in time, there may be applications that rely specifically on libEtc’s 1.2 release series. Even if the 1.3 series is intended to be compatible, there might be changes in undefined behavior which are not compatible with that application.

At the same time, there may be applications that require libEtc’s 2.0 release series.

In some cases, incompatible versions of a single component can be installed in parallel, using distinct path prefixes or distinct file names in the same prefixes. But that’s not always the case. Some languages don’t make parallel installation simple or idiomatic. Even fewer do so for parallel installation of components that are intended to be backward compatible upgrades, like libEtc 1.2 and 1.3.

For those reasons, many applications will bundle the specific versions of components against which they have been tested, and with which they are known to be compatible. Bundling is not merely easiest, but necessary in many cases.

Applications might bundle their dependencies in a shared prefix (i.e. “/opt/…” in FHS), or they may bundle them by shipping as a container image (including a Flatpak image). But in order to support the build process for those applications, a build system must supply the versions they need, which means that a build system must not flatten component release streams at build time.

A while back, a coworker at Red Hat mentioned that it would be useful to have a build of vLLM that supported a broader range of consumer AMD GPUs. Attempting to build vLLM raised the question, “Why is it easier to write a build process from scratch using shell scripts than it is to use Fedora’s build process tools?” Surely build process tools should make build processes easier to write.

Many build process tools do make it easy to write build processes. Most contemporary languages are associated with a registry that makes it trivial to consume shared libraries. One of the most important characteristics of these systems is that developers can publish semantically “stable” releases. The stable release process allows developers to evolve the interface for the software they publish without breaking compatibility in projects that consume it. In order to support this, a system must allow developers to publish multiple interfaces simultaneously. Simultaneous publication means that software compatible with interface version 1 and software compatible with interface version 2 of a shared library can both continue to access an appropriate version of the shared library.

That doesn’t necessarily mean that software using interface version 1 is outdated or unsafe. Developers can continue to publish maintenance releases to the v1 release stream while work on v2 is also active. Continued maintenance of old release series makes it viable to use a feature set and interface other than the newest one until a port to the new interface is finished.

Distributions like Fedora follow a similar process. They also publish “stable” releases with overlapping lifecycles. That fact tends to obscure the thing that makes it very difficult to build complex software using distribution tools. A stable distribution release is just a snapshot of a rolling release, which continues to get updates for some period of time.

The implications of that might be lost on some readers, so I’ll explain that statement a little bit. A rolling release component is one where there is a single version at any given point in time, unlike a stable release which allows users to select multiple releases based on their compatibility needs. It’s possible for distributions to offer stable release components. On Fedora 44, users can install any version of CPython from 3.10 through 3.15. But that’s not usually the case… there’s only one release of PyTorch available, and one ROCm. When more than one release is available it is because the maintainers have taken extraordinary steps to make more than one release available.

That is the thing that makes non-trivial software much more difficult to build in Fedora (and most other distributions!) than it is to build almost any other way. The distribution is developed as a rolling release system, and every release branch inherits the basic characteristics of a rolling release, internally.

vLLM demonstrates the problem building software in a rolling release model. While vLLM integrates support for the shared libraries it uses very quickly, the same is not necessarily true throughout its dependency stack. For example, that means that vLLM requires a very recent release of PyTorch (i.e. release 2.12 which will be featured in the upcoming Fedora 45 release), but its amd-quark dependency supports only much older Python releases (officially, up to 3.13). There is no release of Fedora that provides PyTorch 2.12 (and the rest of the massive collection of modules required by vLLM) for CPython 3.13.

Nor is it easy for a Fedora package maintainer to arrange such a thing. It’s possible to have a build of PyTorch for a CPython release other than the default, as a “compat” package perhaps, but due to the way that Fedora structures it component repos and branches, there will only be one PyTorch compat package for all Fedora release. That means that the owner(s) of the PyTorch package will generally need to manage it. That creates a second major barrier, which is that the maintainers who need compatibility packages aren’t necessarily empowered to provide compatibility packages.

There is a simple and straightforward solution to these problems. These problems are rooted in building a collection of software in which everything is synchronized and every component is flattened to a single version (a rolling release system.)

Instead, Fedora could function more like registries, providing branches that reflect the releases of upstream projects. That doesn’t mean replacing distribution release branches; I expect they would be an addition.

If Fedora dist-git repos had upstream-release branches, I would anticipate:

1. Updates would need to be automated. We don’t want to make the work harder for maintainers.

2. Fewer patches. Maybe none. Most of the patches I find are necessitated by the internal rolling release model. They support adapting to specific versions of dependencies.

3. Packages in dist-git that are not in Fedora releases. Packages that don’t synchronize well with a distribution release are hosted without distribution release branches. Builds of these packages would be decoupled from a distribution release and delivered through some other channel (like Flatpak or other container image registry.)

Solving this problem would allow Fedora to ship complex applications like vLLM that are infeasible for Fedora to build today. But it also enables functionality that I think is far more consequential.

1. Closer to upstream configurations: Decoupling applications from the distribution can in some cases allow us to ship applications with a dependency set closer to the one that has been tested by the upstream projects. Numerous developers have complained about Fedora shipping applications with unsupported dependencies.

2. Fewer packages: Applications are are decoupled from the distribution release can be built and shipped just at any given time instead of once per release. For example, it might not make sense to have multiple builds of OBS Studio, especially if we know that one of them does not work.

3. Fewer packages lost to Fail-To-Build-From-Source: Many of our build failures with each release are the result of building an application before it’s ready for the dependencies that have been rebased for a release. Allowing them to update asynchronously might allow Fedora to keep applications that are actively maintained, but difficult to build in a rolling-release root.

4. Zero-CVE releases: Automating spec updates and eliminating patches will allow many containers to build with low-cost deterministic build systems.

5. Close the registry gap: Language-specific registries are an essential part of most developers workflows, with near universal adoption rates. Yet, despite decades of demonstrated utility for builds, no registry has emerged to provide that service in a language agnostic manner, to fill that role for languages like C and C++ that don’t have a dedicated registry or to tie together projects composed of code in multiple languages. Not only does no individual distribution have that kind of broad adoption, it’s not clear that distributions in general have that kind of adoption. If you ask for help building a CI pipeline that includes the latest version of “OpenSSL < 4” you will most likely end up with a shell script. Fedora’s dist-git can’t replace language-specific registries, but it could provide a language-agnostic equivalent to communities of developers who don’t have similar tools today. Many developers see RPM only as a format for builds delivered to platforms that use RPM to install software. From that point of view, RPM is overhead, not value. Providing a language-independent software registry that developers can use to create declarative CI and build pipelines frames RPM as a tool that enables reusable development workflows. If RPM is a tool that makes development easier, then developers have an incentive to participate that they lack today.

Demo

Because vLLM demonstrates the problems with Fedora’s branching model, I built a prototype vLLM pipeline that illustrates what a better branching model and build tooling might look like. The configuration below builds a sequence of components from source and then assembles a container image with the necessary runtime packages.

Releases for which no package specifies an upper bound might use the “main” branch, while dependencies that do have an upper bound may use some “release-x.y” branch provided by the dist-git repo.

The “packages” listed in this configuration are merely the path to a git repo relative to the “url” in the parent configuration. Much like Go’s decentralized module system, this does not rely on a centralized package registry, only on a sequence of dist-git or source-git repositories. It is both language-agnostic and decentralized.

base: fedora-44-x86_64
localrepo: /home/gmessmer/git/vllm-build/vllm-result/

build:
  - type: dist-git
    url: https://src.fedoraproject.org/rpms/
    packages:
      - python-huggingface-hub:rawhide
      - python-click:f45
      - python-httpx2:f45
      - cpuinfo:f45
      - python-hatchling:main
      - python-jaraco-path:main
      - python-filelock:main
      - python-httpx-aiohttp:main
      - python-anthropic:main

  - type: dist-git
    url: /home/gmessmer/git/vllm-build/vllm-packages/
    packages:
      - python-safetensors:release-0.8
      - python-tokenizers:release-0.23
      - python-starlette:release-1.6
      - xnnpack:f44
      - python-torch:release-2.12
      - python-torchvision:release-0.27
      - python-torchaudio:release-2.11
      - python-torchdata:release-0.11
      - python-torchtext:release-0.17
      - python-torchdiffeq:release-0.2
      - python-torchsde:release-0.2
      - python-opencv-python-headless:main
      - python-partial-json-parser:main
      - python-pybase64:main
      - python-blake3:main
      - python-astor:main
      - python-depyf:release-0.20
      - python-openai-harmony:main
      - python-interegular:main
      - python-lm-format-enforcer:release-0.11
      - python-outlines-core:release-0.2
      - python-llguidance:release-1.7
      - python-model-hosting-container-standards:release-0.1
      - python-prometheus-fastapi-instrumentator:main
      - python-setuptools_scm:release-8.2
      - python-transformers:main
      - python-compressed-tensors:release-0.17
      - python-mistral-common:main
      - python-opentelemetry-semantic-conventions:main
      - python-opentelemetry-sdk:main
      - python-opentelemetry-semantic-conventions-ai:main
      - python-mcp-types:main
      - python-mcp:release-2.1
      - python-apache-tvm-ffi:release-0.1
      - python-xgrammar:release-0.2
      - python-peft:main
      - python-timm:main
      - python-tensorizer:release-2.10
      - python-fastsafetensors:main
      - python-conch-triton-kernels:release-1.2
      - python-llvmlite:release-0.47
      - python-numba:release-0.65
      - python-ml-dtypes:main
      - python-tilelang:release-0.1
      - protobuf:release-33
      - python-protobuf:release-6.33
      - python-proto-plus:release-1.26
      - grpc:release-1.78
      - libarrow:f44
      - python-googleapis-common-protos:release-1.75
      - python-opentelemetry-proto:main
      - python-opentelemetry-exporter-otlp-proto-common:main
      - python-opentelemetry-exporter-otlp-proto-grpc:main
      - python-opentelemetry-exporter-otlp-proto-http:main
      - python-opentelemetry-exporter-otlp:main
      - python-runai-model-streamer:main
      - python-amd-quark:main
      - python-vllm:main

install:
  type: container
  base_image: registry.fedoraproject.org/fedora:44
  tag: vllm:latest
  packages:
    - python-vllm

Additional Illustrations

A hypothetical project might have two stable release branches, for semantic release 1.0 and release 1.1. When Fedora 42 branched, release 1.0 was the “main” stable branch, so “f42” branched from release 1.0.3 and later merged release 1.0.4.

The project published release 1.1.0 after f42 branched, and when that happened, a new branch was created for “release-1.1”. The branches “main” and “release-1” which had been following the 1.0 release series now follow the 1.1 release. When “f43” branched, the “main” branch was at 1.1.2.

Because Fedora-release branches are de-coupled from the upstream-release branches, patches that are specific to Fedora can live in the Fedora branch, which allows automation to manage the upstream-release branches with fewer conflicts.

We might also choose a different strategy for the “main” or “rawhide” branch. Today, “rawhide” doesn’t automatically follow new upstream releases. We might choose to preserve that behavior if the risk of new releases breaking Rawhide is too high. In that case, it would still be useful to represent upstream releases automatically to enable pipelines like the vLLM pipeline illustrated above, which require releases newer than any Fedora branch has merged.

Later, when upstream publishes release 1.2.0, a new “release-1.2” branch is created and “main” follows the new branch. When “f44” branches, it will branch from “rawhide” (/main). But even before any Fedora branch merges it, developers can consume the “release-1.2” branch for their own builds, outside of Fedora.

This process allows application packagers to lead the distribution (probably publishing container images), rather than forcing them to build old releases of their own software to meet the constraints of dependencies in Fedora lagging behind upstream releases.

De-coupling and automating the upstream-release branches allows the dist-git to function like a registry, helping developers publish metadata about their software immediately, without the delays required today to allow the other packages in Fedora to adapt. This gives application developers significantly more incentive to participate.

At the same time, Fedora keeps its release branches, so it still retains distribution maintainer responsibility for the software the distribution publishes as RPM packages.

I’ll also note that Fedora doesn’t need to build any artifacts from the upstream-release branches. They just need to exist to enable application developers to build their own stacks, without being held back by what Fedora has managed to merge, and to allow them to build combinations of release streams that Fedora doesn’t, like PyTorch 2.12 with Python 3.13 and ROCm 7.

Footnotes

1. There is one “cheat” package in this stack. python-runai-model-streamer isn’t built from source, it’s just repackaging their binary wheel. This project uses the bazel build system which Fedora doesn’t ship and doesn’t have established guidelines. Getting bazel support in Fedora is an entire project all its own, so I’ve punted on building that package properly.

Simply put, Secure Boot is an implementation of code signing, usable in the firmware boot environment. In the commercial software world, it is common for software developers to sign their code. Secure production networks can deploy policies that allow only signed code to run. Using an allow-list of signing certificates is highly effective as an anti-malware mechanism. Secure Boot brings that practice to consumer devices to prevent malware in a device’s privileged execution mode.

KEK updates

The Platform Key (PK) is the trust root for a system. It is typically a key owned by the device manufacturer. If the system boots and finds a capsule update for the Key Exchange Key (KEK), the update must be signed by the PK. If the KEK update has a valid signature from the PK, then the system will update its KEK.

DB/DBX updates

The Key Exchange Key (KEK) serves a similar purpose, but is typically a key managed by an OS vendor like Microsoft. The KEK is used to authorize updates to the allowed signature database (db) and revoked signature database (dbx). If the system boots and finds a capsule update for the db/dbx, the update must be signed by the KEK.

Booting a Linux operating system with shim and GRUB2

When a PC boots, the firmware will use the UEFI Boot Manager to select an executable program to run. For many GNU/Linux systems, this will be “shim”. The Boot Manager will provide the location of a file, and the firmware will validate the signature of that file against the dbx, the db, and the MOK. If the file is disallowed by the dbx, then it will not execute. If it is allowed by the db or the MOK, then the firmware will begin executing the code in that file.

shim embeds signing keys of its own in its executable file. When it runs, those keys can be used to verify code that is loaded later on. shim is designed to be as small as possible so that it needs to be signed by a db key infrequently, while allowing the distribution vendor to sign and update a more complete boot loader as often as necessary.

Like the firmware, shim will locate a file containing GRUB2. It will verify that GRUB2 has a valid signature, and that the key that signed GRUB2 is not in the dbx, and that it is in the db, or the MOK, or in the keys that were embedded in shim when it was compiled.

GRUB2 continues this process. It loads its configuration files in order to locate a kernel and initramfs. Similar to earlier stages, GRUB2 will verify that the kernel has a valid signature, that the key that signed the kernel is not in the dbx, and that it is in the db, or the MOK, or in the keys that were embedded in shim.

Finally, if the kernel enforces module signing (which is typical on a Secure Boot system), then it will perform a similar process. Like shim, the kernel embeds additional keys, including keys that were generated when the kernel was compiled, used to sign its set of kernel modules, and then discarded. When it loads a kernel module, it will verify that the module has a valid signature, that the key that signed the module is not in the dbx, and that it is in the db, or the MOK, or the keys that were embedded in shim, or in the keys that were embedded in the kernel image.

References

1. Red Hat Enterprise Linux - Release Notes - Chapter 23. Signing a kernel and modules for Secure Boot
2. Red Hat Customer Portal - What is UEFI Secure Boot and how it works?
3. Microsoft Ignite - Windows Secure Boot Key Creation and Management Guidance
4. Microsoft Hardware Dev Center - Microsoft UEFI Signing Requirements

The standard tools: free and top

The most common process accounting tools are provided by procps. These will be available by default on nearly all Linux systems.

free will give you a quick summary of the system, overall. I’ll use the -h option here (but sometimes it’s easier to spot outliers when numbers aren’t converted to human readable units):

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       9.0Gi       2.8Gi       3.5Gi       6.5Gi       6.4Gi
Swap:          8.0Gi       2.8Gi       5.2Gi

In this example, the host has approximately 16 GiB of RAM, and is configured for 8 GiB of swap on zram.

“Available” is a value provided by the Linux kernel which estimates the amount of memory that could be given to applications without causing the kernel to start searching for memory pages to reclaim or swap, and the “used” value is the “total” minus “available” memory.

A relatively small portion of the page cache (where Linux caches files) will present as “used” memory. In some cases, especially if you have a large amount of memory and a lot of it is currently used for page cache, the amount of memory that appears to be “used” might be larger than you expect. I’ll discuss that later.

top can also give you a summary of the state of the system, but adds detail about individual processes.

Tasks: 379 total, 1 running, 378 sleep, 0 d-sleep, 0 stopped, 0 zombie
%Cpu(s):  1.3 us,  0.8 sy,  0.0 ni, 97.3 id,  0.0 wa,  0.3 hi,  0.3 si,  0.0 st
MiB Mem :  15687.4 total,   2534.7 free,   9103.6 used,   6748.9 buff/cache
MiB Swap:   8192.0 total,   5370.1 free,   2821.9 used.   6583.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   5814 gordon    20   0   13.2g   1.1g 307780 S   2.1   7.0 214:53.65 firefox
 605677 gordon    20   0 3431312 665044 130088 S   0.0   4.1   0:43.69 Isolated Web Co
   6144 gordon    20   0 3780996 649408 117208 S   0.0   4.0  18:30.94 WebExtensions
   4998 gordon    20   0 4438168 608468 194156 S   0.0   3.8  16:09.63 thunderbird

Here we see information about memory use that’s similar to the information provided by free, but we also see a description of how much time CPUs spend in various states and information about specific processes. The resident size (RES) of a process provides information about how much memory a program is using, but it can be misleading, because memory can be shared among multiple processes.

top can also display information about how much of a process’ memory is in swap, but this is not in the default set of columns. If you want to identify processes using swap space, press the f key to open the field manager, use the arrow keys to select the SWAP field, then press the d key (and s if you want to sort based on that field), and finally q to return to the process list.

Making sense of processes sharing memory: smem

The smem tool is not part of procps and users will probably need to install it. In addition to the information that the standard tools can provide, smem adds a “proportional” size field that represents the memory that is unique to one process plus a fraction of its shared memory, divided among the processes sharing that memory segment.

Adding the resident size of several processes may over-estimate their memory use when they share memory segments, but adding their “proportional” size will not.

$ smem -s pss -r | head
PID     User    Command                      Swap    USS     PSS     RSS
5814    gordon  /usr/lib64/firefox/firefox   203036  839880  932231  1174208
6144    gordon  /usr/lib64/firefox/firefox   69852   566560  571748  688328
605677  gordon  /usr/lib64/firefox/firefox   3868    530256  537617  664648
4998    gordon  /usr/lib64/thunderbird/thun  184692  462264  510504  621840
561266  gordon  /usr/lib64/firefox/firefox   126240  398612  406060  531920
606913  gordon  /usr/lib64/firefox/firefox   3796    324420  331144  456280
5985    gordon  /usr/lib64/firefox/firefox   13864   322400  330093  449216
571137  gordon  /usr/lib64/firefox/firefox   20856   295132  301865  426320
23559   gordon  /usr/lib64/firefox/firefox   110424  275128  280471  405600

Examining kernel memory use: slabtop

slabtop is another tool provided by procps-ng. It provides a sorted list, like top, but this list represents different data types allocated in the kernel. Run sudo slabtop --human, and press the c key to sort by the size of each cache.

 Active / Total Objects (% used)    : 2267240 / 2936377 (77.2%)
 Active / Total Slabs (% used)      : 59549 / 59549 (100.0%)
 Active / Total Caches (% used)     : 438 / 535 (81.9%)
 Active / Total Size (% used)       : 386Mi / 502Mi (77.0%)
 Minimum / Average / Maximum Object : 0.01K / 0.17K / 12.50K

    OBJS   ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
   88118    66583  75%    0.93K   2717       34       84Mi btrfs_inode
  149191    97228  65%    0.57K   5347       28       83Mi radix_tree_node
    1975     1925  97%   12.50K    988        2       30Mi task_struct
  146076    97308  66%    0.19K   6956       21       27Mi dentry

In this example, we can see that 84 MiB have been allocated to store btrfs inodes, and 27 MiB have been allocated to store directory entries.

When these values are small, users can usually ignore the specific meanings of each data type. But if something is very large, users can look up how that type is used and whether the size of that cache should be investigated further.

Disk and filesystem cache

Disk and network storage is very slow, relative to a CPU and RAM. Memory that isn’t used for anything more important will be used by the kernel to improve the efficiency of reading and writing to slow storage. The vast majority of this memory is just a copy of information that’s on those devices, so when an application requests a memory allocation, the kernel can simply drop some of that cached memory.

But remember that the “available” memory value is an estimate. The kernel will start searching for pages to swap out before it drops 100% of the filesystem cache. Some of this memory affects the “used” value.

The process of calculating the amount of page cache that is treated as “used” memory is complex, as seen here. You can use the used-page-cache script to estimate how much of your system’s “used” memory is actually cache which the kernel may be able to free without necessarily pushing pages into swap.

You can also tune the kernel to protect less memory from reclamation. This will generally make performance worse later if you use enough memory to cause the system to start swapping, but if you’re trying to observe the amount of memory that is being used for something other than cache, you can do it anyway.

echo 8448 | sudo tee /proc/sys/vm/min_free_kbytes

Furthermore, you can instruct the kernel to immediately drop most cache memory, which may make it easier to get a simple value that represents the amount of memory used overall for things other than cache.

$ grep SReclaim /proc/meminfo ; echo 3 | sudo tee /proc/sys/vm/drop_caches ; grep SReclaim /proc/meminfo
SReclaimable:     272620 kB
3
SReclaimable:      67088 kB

Files stored in RAM

Many Linux systems will use “tmpfs” to create directories that store files in RAM. If you copy files to those directories, free will report more memory used. Use df to examine the amount of space “used” by tmpfs filesystems.

$ df -t tmpfs
Filesystem     1K-blocks  Used Available Use% Mounted on
tmpfs            8031932    92   8031840   1% /dev/shm
tmpfs            3212776  3044   3209732   1% /run
none                1024     0      1024   0% /run/credentials/systemd-journald.service
none                1024     0      1024   0% /run/credentials/systemd-resolved.service
tmpfs            8031932  3924   8028008   1% /tmp
tmpfs            1606384   368   1606016   1% /run/user/556600005

ZRAM

zram is a kernel feature that creates a storage device in RAM. Some systems use this device instead of swap in order to compress memory without swapping to disk. On a system that swaps to disk, memory in swap is not “used”, but on a system that uses swap on zram, the compressed data is still “used” RAM.

Use zramctl to get information about memory used by zram devices.

$ zramctl
NAME       ALGORITHM DISKSIZE  DATA  COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo-rle         8G  2.4G 707.4M  725M         [SWAP]

ZFS

ZFS is a filesystem developed by the OpenZFS project. It is not part of the standard Linux kernel because it uses the CDDL license, which is not compatible with the GPL 2.0 license.

One of the unusual behaviors of ZFS is that its filesystem cache (ARC) appears to be “used” memory in the output of free rather than in the buff/cache column like other filesystems.

Use the arcstat tool to get information about memory “used” by ZFS:

$ arcstat
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
11:58:00     9       0     0       9   100      0    0   125G   126G  52.3G

HugePages

The Linux kernel supports a feature named “HugePages” that allows processes to reserve large blocks of memory, as well as allocating blocks larger than the standard 4k page size.

When memory is reserved for HugePages, it will appear as “used” memory, even when the software that reserved the memory isn’t running.

If you can’t find memory used anywhere else, look for memory that might be reserved for HugePages:

$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:   352256 kB
FileHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

References:

1: https://kernel-internals.org/mm/understanding-proc-meminfo/ 2: https://blog.thalheim.io/2025/10/17/zfs-ate-my-ram-understanding-the-arc-cache/

We cannot treat Linux distributions as if they all offer the same level of security. Distributing software after its maintenance is discontinued puts users at risk.

Ubuntu provides two major security improvements over Debian, but keeps one serious flaw.

First, Ubuntu clearly documents that there is a “main” set of packages they will maintain, and a second (much larger) “universe” set of packages for which they do not promise maintenance during a release. They set clear expectations for security coverage.

Second, Ubuntu publishes a new release every six months. Those releases include updates to the “universe” component, so that security issues get patched at least twice per year. That’s not great, but it’s better than every two years.

The flaw is that the “universe” collection is enabled by default, so most users never learn the difference between the two, and aren’t making informed decisions about their security. Most users believe Ubuntu is one collection with complete security coverage.

Red Hat and Fedora approach the same problem very differently.

Fedora publishes a new release every six months and maintains it for just over one year. This schedule keeps the release aligned with most upstream projects, so that Fedora can ship bug fixes provided by upstream projects.

Red Hat develops a distribution based on Fedora, and they, too, manage security by cutting the size of the distribution down to a size that they are staffed to maintain.

EPEL also extends the core system with a larger collection of packages, similar to Ubuntu’s “universe” repository component. This set is community maintained, so it doesn’t necessarily have the same SLA that RHEL does, but critically, that collection is opt-in. EPEL users make an intentional decision to use the community-maintained collection.

Patch coverage is not the same among these systems. Fedora has excellent coverage because it generally isn’t distributing software beyond its upstream maintenance window. RHEL has excellent coverage because Red Hat is staffed to continue maintenance of a smaller package set, independently. Many users will deem Ubuntu’s interim releases acceptable, and informed users of their LTS releases can disable the “universe” repo. The “main” repo will serve many users’ needs and offer good security coverage. However, users should understand that the default configuration of Ubuntu LTS will offer them an enormous body of unmaintained and potentially insecure packages, and that problem will also affect systems derived from Ubuntu LTS, like Mint and Zorin. Debian provides users with the fewest tools for mitigating vulnerabilities, because they continue to distribute software beyond its upstream maintenance, and they don’t differentiate packages that will get security patches from the ones that will not.

I want Fedora, but I want control over my dependency streams

Developers have built registries that allow them to build applications composed of specific package release series. What if we took the things that are great about systems like crates.io and PyPI, and delivered those advantages for everything else?

Registries are an essential component of modern development

Most modern languages have an associated package registry that allows developers to share their work. The basic build and installation tools used in many languages integrate with registries to allow developers to seamlessly define packages they require, and the versions that are compatible.

There are several such registries for C and C++, but none with the kind of clear default status that is common for other languages. Among Free Software developers, the operating system’s package manager often serves this function, providing a language-agnostic package registry, but one that varies from system to system, where there is also no clear default. More importantly, where language-specific package management tools typically allow consumers to specify a per-component version selector and allow the developers who publish software to determine what release streams are supported at any given point in time, distribution package managers often flatten each component into one version, which makes the distribution release stream a kind of global selector for all components.

Fedora provides a great toolbox

Fedora maintainers have developed a collection of really useful tools for their work. In the finest tradition of Unix tools, they are small tools that each do one thing well. The beauty of a toolbox is that you can use the individual tools to create a workflow all your own. The curse of a toolbox is that you must use the tools to create a workflow all your own.

dist-git-manager and rpm-build-assist aren’t fundamentally new tools, they’ve simply composed the existing tools into a workflow that’s already common among software developers, but which span various input and output types.

Although they use tools typically associated with the Fedora distribution, dist-git-manager and rpm-build-assist arrange the tools in a structure directed more toward developers’ needs rather than the needs of the distribution.

Asynchronous is faster than synchronous

The difference between dist-git-manager and rpm-build-assist and the repos and workflows typical within the distribution is that repos in the distribution branch for the distribution release rather than branching for upstream project releases. The distribution’s dist-git repos synchronize all of the packages within the distribution. Synchronizing everything can make development work more difficult, because a developer may prefer to update one dependency at a time in order to isolate the changes needed for each update, and because it tends to delay upstream feature releases until the next release of the distribution.

An example

dist-git-manager maintains a registry-style definition of a component, with major and minor branches that function like SemVer selectors as much as possible. For such a component, the “main” branch will always refer to the latest patch release of the latest minor of the latest major release (the thing most people usually mean when they say “the newest release”), while “release-${major}” will refer to the latest minor release stream within a major release series, and “release-${major}.${minor}” will refer to a specific minor release.

With components structured this way, you can easily build a Django application that uses the latest releases of PostgreSQL 18 and OpenSSL 3, without waiting for a subsequent release of Fedora, or you could build a specific minor release of Django while you work on a regression in your app that’s observed on a new Django release. And perhaps you want a container image with specific RPM packages resulting from that build. No problem!

# build-assist.yaml
base: fedora-44-x86_64

build:
  - type: dist-git
    url: https://forge.example.com/myproject/rpms/
    packages:
      - postgresql:release-18
      - python-django:release-5.1
      - openssl:release-3
      - nginx:release-1
      - myapp:main

install:
  type: container
  base_image: registry.fedoraproject.org/fedora:44
  tag: myapp:latest
  packages:
    - python3-django
    - openssl
    - nginx
    - myapp

Hear me out…

Reading the history of a now-defunct Fedora feature called “Modularity,” one passage in particular caught my attention. The document described “a classic problem that Linux distributions have faced: the “Too Fast/Too Slow” problem.”

I think the problem is not that some components move too fast or some move too slow, it’s that independent developers work asynchronously, and distributions try to synchronize them. That might sound like a trivial distinction, but I think understanding the difference is key to making the distribution more scalable and expanding the developer community.

Let’s consider the example of a synchronous system.

Red Hat Enterprise Linux is a system that (within several categories) synchronizes the lifecycle of its components. That’s one of the things that Red Hat is selling. Rather than seeking support from thousands of projects individually, customers have one vendor to work with. Rather than thousands of individual release cadences and maintenance windows, customers have one cadence and maintenance window to which thousands of components have been synchronized.

Synchronizing those lifecycles is expensive. Red Hat engineers take on the responsibility of fixing bugs and security issues for the components they support for years past the end of upstream maintenance. Even with thousands of professional developers, Red Hat Enterprise Linux supports a far smaller set of components than Fedora does.

This process is one of the things that makes Red Hat Enterprise Linux a product. The lifecycle and feature set are tuned to a specific purpose, making them useful for a specific market segment.

In many ways, Fedora exists in contrast to that model. Rather than a small, focused set of features, Fedora tries to distribute as much software as possible. Rather than a large professional staff of engineers, much of Fedora is maintained by volunteers. Rather than unifying lifecycles, Fedora uses a rapid release cadence, a fairly short maintenance window, and in some cases rolling release packages to minimize the amount of downstream synchronization necessary to ship a collection of components while their lifecycles naturally overlap.

The purpose of productizing a release is to allow an organization to act as a vendor in place of upstream projects. It makes the organization a middleman, placing them in between users and the upstream projects, so that their customers have one support contract to maintain. Fedora is not that. Fedora should do the opposite of that. As a community project, Fedora should minimize the separation between users and upstream projects. Fedora should bring those people together.

Still, many Fedora policies imitate the superficial aspects of RHEL. A new package must be “the latest version” (as if there is only one). For most languages, components must not bundle their dependencies. Each component will provide only one version unless extraordinary steps are taken. Branches in the dist-git repositories represent Fedora releases, not the upstream project’s release series. These policies create something that’s product shaped, but lacking the staffing that allows RHEL to function as a product.

Those policies contrast with the world of package registries and direct publishing channels used by developers outside the realm of software distributions. Outside of distributions, developers are free to publish multiple simultaneous release streams under one component name (the same way that Fedora ships multiple simultaneous releases of the distribution). Developers can readily use a version from any supported release series of their dependencies. Developers are free to bundle dependencies, for better or worse… Some developers use this to avoid tracking updates, but others bundle to allow tighter integration and faster delivery of features developed in the dependent project. Strictly prohibiting bundling in Fedora ignores the value of collaboration in upstream projects, and actually creates silos which do not naturally exist in the Free Software model and which make small projects less sustainable.

Many of the things that are difficult in Fedora are difficult because we try to synchronize a massive collection of packages, like RHEL, rather than enabling developers to work asynchronously, like a package registry. Synchronization is difficult even for professional developers, but it’s reasonable when creating a product and taking on the maintainer role. Synchronizing the collection isn’t just difficult for volunteer packagers, it’s harmful when responsibility for fixing bugs remains with the upstream projects. Even with a rapid release cycle and a short maintenance window, Fedora maintainers do have to synchronize some packages. In some cases they approach that problem by modifying a project’s dependency information to allow it to build against a dependency version that the developers haven’t tested. That not only annoys developers by periodically creating bug reports for what is effectively a fork, the presence of that patch in Fedora makes it more difficult to automate future patch-level updates. When maintainers don’t modify a package to synchronize it with the dependencies in the release, they may need to create a duplicate version of a package for the version they need. At best that process involves tickets, and typically it involves asking the owner of the dependency to do the work. All of this is redundant work, unnecessary when developers are empowered to publish their own releases.

One of the problems that modularity solved was allowing packages and sets of packages to evolve asynchronously with the rest of the distribution. That’s necessary for lots of non-trivial software, where there are deep integrations between software components. That deep integration often indicates collaboration between projects, and we should encourage that. Fedora’s policies against bundling packages sometimes makes project collaboration more difficult.

For example vllm needs a very recent PyTorch, but a slightly older Python3. That’s not impossible in Fedora. A maintainer could request a fork of PyTorch. But there is a lot of friction. In particular, if the maintainer of the application wants a branch of a dependency owned by someone else, and if that maintainer doesn’t want to manage another package, it may be difficult to move forward.

Illustrations might help explain why Fedora’s synchronization makes their work less useful to developers, and how a minor change could make them significantly more useful.

In upstream projects with stable releases, it’s common to use branches to represent each release series.

In some cases, especially for projects that have a cadence similar to Fedora’s, Fedora’s release branches follow very similar paths.

In other cases, Fedora’s release branches might rebase from one upstream release series to another. Or, Fedora might not have a branch representing an upstream release series at all, because no Fedora release is ready to use it. In either case, Fedora’s repositories don’t lend themselves to reuse as a general purpose source code registry because developers can’t reliably select a release series to follow, without synchronizing to a Fedora release.

If Fedora’s dist-git repos provided branches that represented upstream release series, developers could define a complex build, without being limited to one centralized package registry (like PyPI), and without writing long shell scripts to do it.

This hypothetical example might use Fedora 44 packages where no specific branch is called for, but build specific releases of Python3, PyTorch, and vllm, and the resulting RPMs could then be used to create a container image in which some of those RPMs are installed.

base: fedora-44-x86_64
build:
  - type: dist-git
    url: https://src.fedoraproject.org/rpms/
    packages:
      - python3.11:release-3.11
      - python-torch:release-2.11

  - type: git
    url: https://github.com/vllm-project/
    packages:
      - vllm:releases/v0.20.0

install:
  type: container
  base_image: registry.fedoraproject.org/fedora:44
  tag: vllm:latest
  registry: quay.io/vllm
  packages:
    - vllm

Minor changes to Fedora’s dist-git branching could transform Fedora from a self-contained software distribution into the central hub in a federated, language-agnostic package registry, providing an essential tool that addresses the needs of an underserved developer community.