In the curl project we ship new releases based on the master branch of our git repository, in a clean and linear commit history. We have never maintained an old branch for long term or stability etc. Instead we promise to not break user behavior nor the ABI or API. All users should be able to always upgrade to the latest.
A never-ending stream of releases you can always upgrade to; a new one every 8th week.
We build infrastructure you can lean on.
But
Sometimes reality does not match our intentions and we ship regressions.
Sometimes users are too scared that there might be a regression so they refrain from upgrading. risk averse is probably how they view themselves.
Sometimes users, organizations and Linux distros have policies that say they do not upgrade versions. Usually based on how software in general works and there needs to be a single fixed policy for managing software versions and then curl gets treated the same way.
For those situations and other related scenarios, repeating the top paragraphs does not help.
long term branches by others
In practice, just about every major Linux distributor maintains one or more stable curl branches. They backport security fixes to those versions to keep them secure for their users. Some vendors also merge selected bugfixes into their branches.
Every Linux distributor picks the particular curl version they stick to by themselves without coordination with other distros. They all do it at different times and they all have their own specific criteria and work processes for doing this. This, in combination with curl’s frequent releases, tend to make them all pick different versions for their different branches. And keep them alive for different lengths.
Some vendors maintain their stable branches for extended periods of time. Upwards and beyond ten years happen.
These long-lived branches may eventually end up having literally hundreds of patches applied to them. The curl builds done from these branches still report as version x.y.z but in reality they are mutated versions that can be significantly different compared to the original x.y.z version that the curl project shipped in a tarball back in the day.
That’s the comfort you get for picking (and paying?) a Linux distribution. (Yes, you also easily get stuck with an ancient version because of this.)
Some users also simply get stuck on older versions for other reasons and do not security-patch them over time (by ignorance or incompetence), making them more and more insecure over time.
Reality
At the time I write this post, the curl release with the largest number of known security vulnerabilities has 85 published CVEs.
By asking users and by looking in logs in various servers, we know that just about every curl versions we have shipped the last dozen of years or so remain in use somewhere. We can only hope that most of them are security patched.
In reality, every release we do becomes a long term release for someone.
Long term support?
Every once in a while a discussion pops out in or close to the curl project whether we should consider starting to maintain one or more LTS branches.
We have never completely dismissed those ideas. We are however acutely aware of the extra effort and energy such an endeavor requires, so we have so far shrugged it off. But should there come users and sponsors willing to help make it happen, we would not be shy of implementing something.
After all, the ones most interested in LTS branches are usually people and companies with an economic gain to be had; with businesses using and relying on a rock solid curl. They should then also be able to help pay for this.
If you or the company you work for would be interested in something like this, please reach out and we can get the conversation going. Maybe we can do something to improve the lives of people out there?
Until then, we stick to a single release branch.
Credits
Image by Julius Silver from Pixabay
Sounds like you can define version x.y.z to be LTS. Then you just apply current bleeding version to that LTS… 🙂
The key could be that people want something called LTS, and you already provide LTS.
@Kjell: I’m pretty sure that’s not what peeps want.
Hi Daniel,
Please let me share my experience of LTS on HAProxy. What we’ve found over time is that it takes a little bit time but far less than just chasing and fixing bugs (which is needed anyway). Doing maintenance releases essentially requires to backport existing fixes, which is not very long. Also the farther the LTS branch is from mainline, the least likely it is that you’ll need to backport a fix. We maintain our LTS versions for 5 years after release and the non-LTS ones one year, so we effectively have ~6-7 versions at a time. Most often the oldest ones barely see more than a fix per week in the last year because almost nobody’s using them anymore, and we know that their users value extreme stability so let’s not backport risky fixes for bugs that affect nobody.
We’re also fairly open to backport suggestions from the users (best way to deliver ultra-stable versions after all), provided these don’t come with a risk. We tend to accept to backport tiny useful features to the latest LTS branch when that makes users’ lives easier, but never beyond. A new feature *is* one reason to force an upgrade. This works very very well. Usually in 6 months after an LTS release is emitted, many of the users have switched to it (based on bug reports, which can be skewed by the inceased number of bugs in any new version), then maybe 30% for the previous one, 10-15% for the one before, and the rest being within noise. For distros it’s quite convenient, because we know they just pick select patches, and these 5 years of maintenance help them get patches close to their version.
What I noticed overall is that there is a sort of moral contract with users that increases both trust and feature delivery: by being extremely careful not to break old LTS releases, we definitely have a part of the user base, the one most sensitive to bugs, that is perfectly secured by running slow-moving releases. These ones almost never face a regression, and if one ever happens due to a problematic fix, we have no problem instantly emitting another version with only this issue fixed. This allows us to be a bit more aggressive on recent versions. The latest stable that is not LTS can move a little bit and receive a few occasional backports for harmless popular features. This way the share of the population which values features more than stability tests new features early and shares interesting feedback that allows to improve these features before they’re widely adopted. And in parallel sensitive users almost never face a breakage. This means that the pressure on the development team caused by bug reports usually is extremely low: bugs being worked on generally do not affect users in a critical way. It even happens that we develop new debugging features to get more info on a bug, and users generally are happy to participate to the bug resolution because it’s not critical for them either.
Before we had this LTS cycle, I was regularly forced to integrate small features into the “current” branch along with fixes, and users were extremely reluctant to upgrade, even to fix a bug, due to the risk that a change would cause them some trouble. This fear has totally disappeared. We even indicate on a page the list of bugs affecting each and every version, that we use to suggest users to upgrade when they report an issue.
Overall I find that this process works extremely well. The only difficulty with old LTS versions can be the tooling. For example when you build a 5-year old version with your shiny new laptop that comes with a shiny new compiler and you discover that it’s no longer supported by that old version, it’s quite annoying. But that doesn’t happen often, and we all have a collection of native and cross-compilers anyway when working on portable projects.
In our case we alternate non-LTS and LTS (2 versions total per year). Odd versions are not LTS while even ones are LTS, so that users know upfront which one will have what status. And it’s reminded in the version output, with an indication of the date of end of support. This has also helped make older versions disappear completely.
I can only encourage you to go into that direction. The beginning can be more diffcult than the long term due to the change of habits, and most users initially using the same LTS version with different expectations. But after a year, the fog will dissipate and you’ll start to see different groups on different versions, then this will become cool.
Hoping this can help you in your decision.
@willy: thanks a lot for taking the time to share your valuable insights and experiences with LTS. Most helpful!