Diagram of dist-git branches

A while back, a coworker at Red Hat mentioned that it would be useful to have a build of vLLM that supported a broader range of consumer AMD GPUs. Attempting to build vLLM raised the question, “Why is it easier to write a build process from scratch using shell scripts than it is to use Fedora’s build process tools?” Surely build process tools should make build processes easier to write.

Many build process tools do make it easy to write build processes. Most contemporary languages are associated with a registry that makes it trivial to consume shared libraries. One of the most important characteristics of these systems is that developers can publish semantically “stable” releases. The stable release process allows developers to evolve the interface for the software they publish without breaking compatibility in projects that consume it. In order to support this, a system must allow developers to publish multiple interfaces simultaneously. Simultaneous publication means that software compatible with interface version 1 and software compatible with interface version 2 of a shared library can both continue to access an appropriate version of the shared library.

That doesn’t necessarily mean that software using interface version 1 is outdated or unsafe. Developers can continue to publish maintenance releases to the v1 release stream while work on v2 is also active. Continued maintenance of old release series makes it viable to use a feature set and interface other than the newest one until a port to the new interface is finished.

Distributions like Fedora follow a similar process. They also publish “stable” releases with overlapping lifecycles. That fact tends to obscure the thing that makes it very difficult to build complex software using distribution tools. A stable distribution release is just a snapshot of a rolling release, which continues to get updates for some period of time.

The implications of that might be lost on some readers, so I’ll explain that statement a little bit. A rolling release component is one where there is a single version at any given point in time, unlike a stable release which allows users to select multiple releases based on their compatibility needs. It’s possible for distributions to offer stable release components. On Fedora 44, users can install any version of CPython from 3.10 through 3.15. But that’s not usually the case… there’s only one release of PyTorch available, and one ROCm. When more than one release is available it is because the maintainers have taken extraordinary steps to make more than one release available.

That is the thing that makes non-trivial software much more difficult to build in Fedora (and most other distributions!) than it is to build almost any other way. The distribution is developed as a rolling release system, and every release branch inherits the basic characteristics of a rolling release, internally.

vLLM demonstrates the problem building software in a rolling release model. While vLLM integrates support for the shared libraries it uses very quickly, the same is not necessarily true throughout its dependency stack. For example, that means that vLLM requires a very recent release of PyTorch (i.e. release 2.12 which will be featured in the upcoming Fedora 45 release), but its amd-quark dependency supports only much older Python releases (officially, up to 3.13). There is no release of Fedora that provides PyTorch 2.12 (and the rest of the massive collection of modules required by vLLM) for CPython 3.13.

Nor is it easy for a Fedora package maintainer to arrange such a thing. It’s possible to have a build of PyTorch for a CPython release other than the default, as a “compat” package perhaps, but due to the way that Fedora structures it component repos and branches, there will only be one PyTorch compat package for all Fedora release. That means that the owner(s) of the PyTorch package will generally need to manage it. That creates a second major barrier, which is that the maintainers who need compatibility packages aren’t necessarily empowered to provide compatibility packages.

There is a simple and straightforward solution to these problems. These problems are rooted in building a collection of software in which everything is synchronized and every component is flattened to a single version (a rolling release system.)

Instead, Fedora could function more like registries, providing branches that reflect the releases of upstream projects. That doesn’t mean replacing distribution release branches; I expect they would be an addition.

If Fedora dist-git repos had upstream-release branches, I would anticipate:

  1. Updates would need to be automated. We don’t want to make the work harder for maintainers.

  2. Fewer patches. Maybe none. Most of the patches I find are necessitated by the internal rolling release model. They support adapting to specific versions of dependencies.

  3. Packages in dist-git that are not in Fedora releases. Packages that don’t synchronize well with a distribution release are hosted without distribution release branches. Builds of these packages would be decoupled from a distribution release and delivered through some other channel (like Flatpak or other container image registry.)

Solving this problem would allow Fedora to ship complex applications like vLLM that are infeasible for Fedora to build today. But it also enables functionality that I think is far more consequential.

  1. Closer to upstream configurations: Decoupling applications from the distribution can in some cases allow us to ship applications with a dependency set closer to the one that has been tested by the upstream projects. Numerous developers have complained about Fedora shipping applications with unsupported dependencies.

  2. Fewer packages: Applications are are decoupled from the distribution release can be built and shipped just at any given time instead of once per release. For example, it might not make sense to have multiple builds of OBS Studio, especially if we know that one of them does not work.

  3. Fewer packages lost to Fail-To-Build-From-Source: Many of our build failures with each release are the result of building an application before it’s ready for the dependencies that have been rebased for a release. Allowing them to update asynchronously might allow Fedora to keep applications that are actively maintained, but difficult to build in a rolling-release root.

  4. Zero-CVE releases: Automating spec updates and eliminating patches will allow many containers to build with low-cost deterministic build systems.

  5. Close the registry gap: Language-specific registries are an essential part of most developers workflows, with near universal adoption rates. Yet, despite decades of demonstrated utility for builds, no registry has emerged to provide that service in a language agnostic manner, to fill that role for languages like C and C++ that don’t have a dedicated registry or to tie together projects composed of code in multiple languages. Not only does no individual distribution have that kind of broad adoption, it’s not clear that distributions in general have that kind of adoption. If you ask for help building a CI pipeline that includes the latest version of “OpenSSL < 4” you will most likely end up with a shell script. Fedora’s dist-git can’t replace language-specific registries, but it could provide a language-agnostic equivalent to communities of developers who don’t have similar tools today. Many developers see RPM only as a format for builds delivered to platforms that use RPM to install software. From that point of view, RPM is overhead, not value. Providing a language-independent software registry that developers can use to create declarative CI and build pipelines frames RPM as a tool that enables reusable development workflows. If RPM is a tool that makes development easier, then developers have an incentive to participate that they lack today.

Demo

Because vLLM demonstrates the problems with Fedora’s branching model, I built a prototype vLLM pipeline that illustrates what a better branching model and build tooling might look like. The configuration below builds a sequence of components from source and then assembles a container image with the necessary runtime packages.

Releases for which no package specifies an upper bound might use the “main” branch, while dependencies that do have an upper bound may use some “release-x.y” branch provided by the dist-git repo.

The “packages” listed in this configuration are merely the path to a git repo relative to the “url” in the parent configuration. Much like Go’s decentralized module system, this does not rely on a centralized package registry, only on a sequence of dist-git or source-git repositories. It is both language-agnostic and decentralized.

base: fedora-44-x86_64
localrepo: /home/gmessmer/git/vllm-build/vllm-result/

build:
  - type: dist-git
    url: https://src.fedoraproject.org/rpms/
    packages:
      - python-huggingface-hub:rawhide
      - python-click:f45
      - python-httpx2:f45
      - cpuinfo:f45
      - python-hatchling:main
      - python-jaraco-path:main
      - python-filelock:main
      - python-httpx-aiohttp:main
      - python-anthropic:main

  - type: dist-git
    url: /home/gmessmer/git/vllm-build/vllm-packages/
    packages:
      - python-safetensors:release-0.8
      - python-tokenizers:release-0.23
      - python-starlette:release-1.6
      - xnnpack:f44
      - python-torch:release-2.12
      - python-torchvision:release-0.27
      - python-torchaudio:release-2.11
      - python-torchdata:release-0.11
      - python-torchtext:release-0.17
      - python-torchdiffeq:release-0.2
      - python-torchsde:release-0.2
      - python-opencv-python-headless:main
      - python-partial-json-parser:main
      - python-pybase64:main
      - python-blake3:main
      - python-astor:main
      - python-depyf:release-0.20
      - python-openai-harmony:main
      - python-interegular:main
      - python-lm-format-enforcer:release-0.11
      - python-outlines-core:release-0.2
      - python-llguidance:release-1.7
      - python-model-hosting-container-standards:release-0.1
      - python-prometheus-fastapi-instrumentator:main
      - python-setuptools_scm:release-8.2
      - python-transformers:main
      - python-compressed-tensors:release-0.17
      - python-mistral-common:main
      - python-opentelemetry-semantic-conventions:main
      - python-opentelemetry-sdk:main
      - python-opentelemetry-semantic-conventions-ai:main
      - python-mcp-types:main
      - python-mcp:release-2.1
      - python-apache-tvm-ffi:release-0.1
      - python-xgrammar:release-0.2
      - python-peft:main
      - python-timm:main
      - python-tensorizer:release-2.10
      - python-fastsafetensors:main
      - python-conch-triton-kernels:release-1.2
      - python-llvmlite:release-0.47
      - python-numba:release-0.65
      - python-ml-dtypes:main
      - python-tilelang:release-0.1
      - protobuf:release-33
      - python-protobuf:release-6.33
      - python-proto-plus:release-1.26
      - grpc:release-1.78
      - libarrow:f44
      - python-googleapis-common-protos:release-1.75
      - python-opentelemetry-proto:main
      - python-opentelemetry-exporter-otlp-proto-common:main
      - python-opentelemetry-exporter-otlp-proto-grpc:main
      - python-opentelemetry-exporter-otlp-proto-http:main
      - python-opentelemetry-exporter-otlp:main
      - python-runai-model-streamer:main
      - python-amd-quark:main
      - python-vllm:main

install:
  type: container
  base_image: registry.fedoraproject.org/fedora:44
  tag: vllm:latest
  packages:
    - python-vllm

Additional Illustrations

A hypothetical project might have two stable release branches, for semantic release 1.0 and release 1.1. When Fedora 42 branched, release 1.0 was the “main” stable branch, so “f42” branched from release 1.0.3 and later merged release 1.0.4.

The project published release 1.1.0 after f42 branched, and when that happened, a new branch was created for “release-1.1”. The branches “main” and “release-1” which had been following the 1.0 release series now follow the 1.1 release. When “f43” branched, the “main” branch was at 1.1.2.

Because Fedora-release branches are de-coupled from the upstream-release branches, patches that are specific to Fedora can live in the Fedora branch, which allows automation to manage the upstream-release branches with fewer conflicts.

We might also choose a different strategy for the “main” or “rawhide” branch. Today, “rawhide” doesn’t automatically follow new upstream releases. We might choose to preserve that behavior if the risk of new releases breaking Rawhide is too high. In that case, it would still be useful to represent upstream releases automatically to enable pipelines like the vLLM pipeline illustrated above, which require releases newer than any Fedora branch has merged.

Diagram of dist-git branches with Fedora release branches

Later, when upstream publishes release 1.2.0, a new “release-1.2” branch is created and “main” follows the new branch. When “f44” branches, it will branch from “rawhide” (/main). But even before any Fedora branch merges it, developers can consume the “release-1.2” branch for their own builds, outside of Fedora.

This process allows application packagers to lead the distribution (probably publishing container images), rather than forcing them to build old releases of their own software to meet the constraints of dependencies in Fedora lagging behind upstream releases.

De-coupling and automating the upstream-release branches allows the dist-git to function like a registry, helping developers publish metadata about their software immediately, without the delays required today to allow the other packages in Fedora to adapt. This gives application developers significantly more incentive to participate.

At the same time, Fedora keeps its release branches, so it still retains distribution maintainer responsibility for the software the distribution publishes as RPM packages.

Diagram of dist-git branches with Fedora release branches

I’ll also note that Fedora doesn’t need to build any artifacts from the upstream-release branches. They just need to exist to enable application developers to build their own stacks, without being held back by what Fedora has managed to merge, and to allow them to build combinations of release streams that Fedora doesn’t, like PyTorch 2.12 with Python 3.13 and ROCm 7.

Footnotes

  1. There is one “cheat” package in this stack. python-runai-model-streamer isn’t built from source, it’s just repackaging their binary wheel. This project uses the bazel build system which Fedora doesn’t ship and doesn’t have established guidelines. Getting bazel support in Fedora is an entire project all its own, so I’ve punted on building that package properly.