In 2012 I wrote a blog post titled curling the metalink, describing how we added support for metalink to curl.
Today, we remove that support again. This is a very drastic move, and I feel obliged to explain it so here it goes! curl 7.78.0 will ship without metalink support.
Metalink problems
There were several issues found that combined led us to this move.
Security problems
We’ve found several security problems and issues involving the metalink support in curl. The issues are not detailed here because they’ve not been made public yet.
When working on these issues, it become apparent to the curl security team that several of the problems are due to the system design, metalink library API and what the metalink RFC says. They are very hard to fix on the curl side only.
Unusual use pattern
Metalink usage with curl was only very briefly documented and was not following the “normal” curl usage pattern in several ways, making it surprising and non-intuitive which could lead to further security issues.
libmetalink is abandoned
The metalink library libmetalink was last updated 6 years ago and wasn’t very actively maintained the years before that either. An unmaintained library means there’s a security problem waiting to happen. This is probably reason enough.
XML is heavy
Metalink requires an XML parsing library, which is complex code (even the smaller alternatives) and to this day often gets security updates.
Not used much
Metalink is not a widely used curl feature. In the 2020 curl user survey, only 1.4% of the responders said that they’d are using it. In the just closed 2021 survey that number shrunk to 1.2%. Searching the web also show very few traces of it being used, even with other tools.
The torrent format and associated technology clearly won for downloading large files from multiple sources in parallel.
Violating a basic principle
This change unfortunately breaks command lines that uses --metalink. This move goes directly against one of our basic principles as it doesn’t maintain behavior with previous versions. We’re very sorry about this but we don’t see a way out of this pickle that also takes care of user’s security – which is another basic principle of ours. We think the security concern trumps the other concerns.
Possible to bring back?
The list above contains reasons for the removal. At least some of them can be addressed given enough efforts and work put into it. If someone is willing to do the necessary investment, I think we could entertain the possibility that support can be brought back in a future. I just don’t think it is very probable.
Credits
Image by Ron Porter from Pixabay
Stupid question – why not leave the existing functionality with all of those caveats and force users to explicitly pass in a “–force” or since such flag to acknowledge that they’re making use of a deprecated feature that is backwards compatible for a limited time to allow for users to migrate to another solution if they truly need metaling capability?
It’s backwards-compatible-ish in that they can modify their call easily and track down where they use it since they’ll have to do that to run in a limited time “legacy” mode. And after a few releases of refusing to run without the force flag while displaying and logging a warning, you can in good conscience yank the feature out completely knowing users knew this was coming and had to modify their scripts to enable using the deprecated feature – they had more than enough warning to either change dependencies or package an old embedded version of curl that supports metalinks (for as long as that’ll be viable depending on packaging solutions – at some point the entire dep chain will have to be embedded and cross compiled).
My $0.02 on a reasonable compromise between breaking backwards compatibility between the CLI and removing little used, but effort intensive, features. Take it as you will.
@Scott: if everyone in the world was a mature grown-up who’d read the documentation carefully before adding or using such an option, then maybe it could have worked like that.
Now, we know that a large amount of users will blindly and without reading anything gladly add an option that “makes it work” if one is provided. Like they click on “yes”, “ok” and “make it work” dialogues already in browsers without at all understanding what they agree to. Thus, we would knowingly ship software that could (and probably would) hurt users. I cannot shoulder that burden. I think that would be a betrayal of what I stand for and believe in.
Then there’s the large portion of users who just run a script or pastes someone else’s command lines. They don’t even get to see the initial warning but will use the override flag at start from within the script and will thus be vulnerable all the same.
So no: we should not ship a product with known security problems lingering. Even if the user needs to ask for them with an extra option.
The only way to ship curl with metalink support in the future is if someone steps up and fund development that fixes all the issues.
Why not leave it as is and let the chips fall where they may?
Who cares about it if they don’t use it?