A branching release process frees developers to work independently and asynchronously

A branching release process frees developers to work independently and asynchronously

Fred Brooks’ The Mythical Man Month is best known for the observation that adding manpower to a late software project makes it later. This is now known as Brooks’ Law. He offered several explanations that contribute to this outcome, including the effects of coordination and communication.

I missed the 50th anniversary of The Mythical Man Month by a bit, but I wanted to talk about one of the biggest changes in software development as it relates to the book. When I started developing software, 30 years ago, version control systems with workflows centered around locking a file for one developer’s exclusive access were still common. If you never worked with those systems, you may not appreciate that along with better testing practices, some of the most significant productivity improvements of the last 50 years has been been driven by branching (in various forms), which significantly reduces the friction that Brooks observed.

Branching is used not just in source code, but in releases as well. I spend a lot of time talking about the stable release model. The modern stable release model is a branching model. It provides overlapping release streams in which the extent of change within a stream is less significant than the extent of change from stream to stream. The overlapping streams allow consumers to test and migrate from stream to stream asynchronously. These developers are more efficient because the availability of multiple supported releases allows them to schedule work based on their own priorities.

Interrupt-driven work is a well known killer of productivity. Imagine that you develop a service that uses shared components for TLS, RPC, PubSub messaging, and KeyVal data storage. In this hypothetical production environment, the TLS library published updates every 4 weeks, and RPC publishes updates every 6 weeks, and PubSub publishes updates once a month, and KeyVal publishes updates once a month but not at the same time as PubSub. If the client library releases for these services are linear and not branching, then you may need to adapt your software to track changes in these libraries in order to continue building and testing your own releases, when those releases occur. Those interrupts can make it difficult to find uninterrupted periods in which to develop features of your own. You will probably find your team padding their schedule in order to provide time for testing and porting efforts that can be unpredictable in magnitude, even if they have a predictable frequency.

If release streams overlap, then developers who use those client libraries can schedule updates when it is convenient. They can do that work any time between the initial availability of a new client library and the end of support for the one they were previously using. This improves productivity for users of the client library by removing uncertainty in the development process, which leads to less schedule padding. It tends to improve productivity for the teams providing those client libraries as well, because the urgency of most support requests drops significantly.

This model requires slightly more work, but in exchange, teams avoid most of the coordination that Brooks observed constraining development velocity. The extra work doesn’t slow development down, it speeds it up. Many people try to make systems more efficient by reducing the amount of work required, but in this case the efficiency gains are significantly greater than the work required to continue maintaining older release series.

The distribution release process may require developers to coordinate and synchronize

Stable release distributions like Fedora aim to deliver the benefits of the stable release model for the collection of software they provide, so that users of the distribution don’t have to manage thousands of components individually. They do this by coordinating updates that affect applications across the distribution. In some cases, that means doing the work required to port components to new interfaces provided by the update. The result is a software collection with minimal duplication of code. But, ironically, because they are working to eliminate multiple release streams for shared components, distribution package maintainers don’t get the benefit of the stable release process for their own work. Just like the hypothetical production network, maintainers may be notified that updates have broken their software, at which point they have to adapt to the change to ensure that users can continue to use their software. In this way, they reintroduce the inefficiency that Brooks observed and reimpose the problem that the stable release process solves.

That work is necessary to produce an integrated OS that implements the concepts described in the Linux FHS. but from the point of view of any given application developer, the shared libraries present in the OS are mostly a coincidence, and one which varies from distribution to distribution. Application developers will tend to be most productive using registries like PyPI, crates.io, or npm which allow them to schedule maintenance work asynchronously from any target operating system.