On Thursday March 17 2022 at 09:00 PDT (16:00 UTC, 17:00 CET) I will run this free live webinar.
It is an introduction to doing Internet transfers using libcurl, may 30-35 minutes presentation followed by a Q&A session where I can answer all and any questions you may have.
Do you remember August 26 2002? I can’t say I particularly do but the curl git log remembers for us that it was on that day we added this TODO item:
Add an option that prevents cURL from overwriting existing local files. When used, and there already is an existing file with the target file name (either -O or -o), a number should be appended (and increased if already existing).
That idea hadn’t even been listed for twenty years before it was converted into code by HexTheDragon and landed in curl the other day (with this commit). To get included in the pending curl 7.83.0 release.
--no-clobber
This new command line option (curl’s 247th) is called --no-clobber and it works as suggested already back in 2002. If the output file already exists at the time when curl wants to create it, it will instead append a number to the end of the name. If that file also exists, curl retries iteratively with numbers up to a 100 before it gives up and returns error.
To help you write even cooler scripts. Oh, and the -w variable %{filename_effective} will show this actually used file name.
We have just merged curl’s 246th command line option: --remove-on-error (with this commit). To be included in the upcoming curl 7.83.0 release.
This command line option is quite simple and does exactly what the name suggests. If you tell curl to download something into a local file and something goes wrong in that transfer – that makes curl return an error – this option will make curl remove that file rather than leaving the leftovers on disk in a possibly partial file.
The option is in fact slightly more useful in more complicated cases, like when you want to download lots files in parallel and some of them might fail and you rather only keep the files that actually were transferred successfully:
The curl project has been alive for decades. We gradually introduce new features and options into the command line tool and library over time and we work hard never to break existing behavior and keep the ABI and API stable.
Still, some features and functionalities go out of style sooner or later. Versions get deprecated, third party dependencies go stale and turn unsuitable for use.
How to discard “dead branches”
I like the mental image of curl as a big flourishing tree, with roots, the main trunk, branches and a multitude of thick green leaves.
Every once in a while some leaves or branches die. They turn all brown and dry and we need to do something about them. We trim the tree.
Dependencies go sour
A few times during curl’s life-time we have found ourselves in a position where we supported a third-party dependency for some functionality, but that library was maybe no longer a product we want to recommend our users to actually lean on. For libraries that aren’t being maintained correctly or that fall behind in other aspects and we are made aware of that fact, we need to make a decision.
To keep supporting the library, we indirectly give it our blessing. Products that no longer get updates or we no longer trust are keeping up with the world, we need to “chop off” from the curl family tree. We have done so with a few TLS libraries over the years for example. Users that want to keep doing TLS powered protocols with curl and libcurl then “just” have to switch to a supported TLS library.
We don’t have any special mechanisms or policies to detect this kind of expired products, but we simply have to use our judgement and do the best we can.
Protocol versions go extinct
The most obvious example here are SSL and TLS versions. Back in 1998 there actually existed servers that supported SSL version 2. Since that day, all SSL versions have been phased out from the internet and several TLS versions have. I doubt we have seen the last deprecated protocol version so more are likely to happen going forward.
In curl we follow the internet transfer ecosystem and in many cases we get told by the TLS libraries what curl can and cannot support. The options that we once added that ask for certain specific protocol versions thus no longer actually work for most system installs. This is rarely a problem because even if users could ask for a really old TLS version to be attempted, rarely any server side is actually supporting those so this usually isn’t a cause for concern.
For the most desperate users in niche situations, they can usually go build their own versions of the TLS libraries and re-enable the deprecated versions if they really need.
We might give up on features
Sometimes we add features to libcurl for stuff that then over time never really gets used or work correctly. This can happen because the Internet world decided that the particular feature isn’t cool or even because it doesn’t work perfectly in the curl architecture. In cases where we can then remove the feature without breaking the ABI, like when the option the user sets asks for something to get used if possible, things are good. This is for example how we could remove support for HTTP/1 Pipelining from curl without breaking any promises.
Unused platforms erode
The source code for curl and libcurl is written to be extremely portable and has reportedly run on at least 86 different operating systems at some point. Platforms come and go, and popularity and support for them go up and down over time. We might add support for a popular platform one year, which later, fifteen years or so down the line is basically never used or heard of any longer. Code for platforms that we never build or verify slowly but very surely “rots” and will no longer be possible to build.
After a certain number of years without attention, the cost of keeping the code around that presumably does not work anyway, gets higher than the value of keeping it around. If then nobody raises their hand and says they are in fact using curl on platform ZZ we might rip out the adjustments we have in the code for that platform. We can always bring back support if someone would suddenly emerge with such a desire and plan.
In practical terms
When we figure out that there’s something in the project we think should be deprecated. A feature, a backend, something, we bring it up for discussion on the libcurl mailing list and we add it to the docs/DEPRECATE.md document, to be removed no earlier than six months later. This way, users can check this document and get informed about these plans in several releases ahead.
If someone would object to a particular deprecation plan, we would take that discussion and possibly reconsider or delay the deprecation plans.
Welcome to the 206th curl release, 59 days since we shipped curl 7.81.0. The extra three days because I was away on the day the release would normally have been done. (I call it Impartial Content as a little play on the HTTP 206 response code message.)
the 206th release 2 change 59 days (total: 8,751) 173 bug-fixes (total: 7,691) 266 commits (total: 28,321) 0 new public libcurl function (total: 86) 0 new curl_easy_setopt() option (total: 295) 1 new curl command line option (total: 245) 67 contributors, 39 new (total: 2,597) 43 authors, 24 new (total: 1,014) 0 security fixes (total: 111) 0 USD paid in Bug Bounties (total: 16,900 USD)
Changes
There are only two changes this time around:
The JSON option
With the new --json command line option, curl suddenly made it more convenient to send JSON from command lines and shell scripts.
MesaLink: removed support
curl supports a crazy amount of different TLS libraries, but the amount was now decreased by one (to 13) as we officially drop support for MesaLink. The library is not developed anymore so we don’t want to encourage future users to go down that road.
Bug-fixes
Here are some of my favorite bug-fixes from this release cycle.
bearssl
We landed three notable fixes for the bearssl backend which should make users of it happy. For cert expiration, incomplete CA certs and session resumption.
strlen call removals
After I posted a library-call count to the mailing list that showed quite a large number of calls to strlen(), a cleanup race started that subsequently reduced the number of calls by over 60% in some use cases! Primarily replaced by compile-time constants.
configure requires –with-nss-deprecated
To build curl with the NSS backend using configure, you now need to confirm this choice by also passing on --with-nss-deprecated to make it clear to users what the future looks like for our NSS support.
erase some more sensitive command line arguments
After a lengthy discussion we now hide even more command line arguments from appearing in ps output (on systems that support it). Since the hiding is done by curl itself, there is still a short moment during which they will be visible, plus that we cannot hide everything so there is still a risk that some argument might leak information unwillingly. That is the nature of command line arguments. Use the config file concept or stdin etc to work around that.
NPN is deprecated
The TLS extension NPN is now marked “deprecated” and is scheduled for removal in six months unless someone yells very loudly and explains why not. This extension was once used to negotiate SPDY and early HTTP/2 but have no purpose these days. The browsers removed support for it several years ago.
allow CURLOPT_HTTPHEADER change “:scheme”
The only pseudo header for HTTP/2 and HTTP/3 that couldn’t be modified by a user can now be changed at will.
remove support for TPF, Netware, vxWorks
Support for these platforms for which the code haven’t been modified for the last decade or so, and therefore are highly unlikely to still work, were dropped. After this, I had it confirmed that you can still build curl for vxWorks using the “regular” build!
remove support for CURL_DOES_CONVERSIONS
After support for TPF was dropped, we took the next step and removed support for the charset conversion functions necessary to run curl on non-ASCII platforms such EBCDIC using ones. As TPF was the only/last platform such platform we supported, this cleanup improved lots of code paths.
allow user callbacks to call curl_multi_assign
A regression in 7.81.0 made curl_multi_assign() return error if used from a callback.
http3: quiche and ngtcp2 fixes
We landed several fixes in both HTTP/3 backends, improving the situation for everyone who plays HTTP/3 with curl.
reduce memory use when FTP is disabled
After several cleanups the total memory footprint for builds with FTP and/or proxy support disabled has been reduced.
check for ~/.config/curlrc too
curl now also checks for its default “config file” in the path mentioned above.
DNS options that need c-ares now fail without it
The command line tool offers a set of functions to control DNS specific details, and since those options only work if libcurl was built to use c-ares and not at all if it was built to use another resolver backend, curl will now correctly return error when one of those options are used when libcurl can’t execute them.
keep trailing dot in host name
If there is a trailing dot after the host name in the URL, that dot is now kept in the name when used everywhere internally – except for the SNI field in TLS.
wolfssl: when SSL_read() returns zero, check the error
Even while obviously very rare, curl could wrongly return an “end of transfer” prematurely before this fix.
Next
The next release is scheduled to ship on April 27, 2022.
