We have just merged curl’s 246th command line option: --remove-on-error (with this commit). To be included in the upcoming curl 7.83.0 release.
This command line option is quite simple and does exactly what the name suggests. If you tell curl to download something into a local file and something goes wrong in that transfer – that makes curl return an error – this option will make curl remove that file rather than leaving the leftovers on disk in a possibly partial file.
The option is in fact slightly more useful in more complicated cases, like when you want to download lots files in parallel and some of them might fail and you rather only keep the files that actually were transferred successfully:
The curl project has been alive for decades. We gradually introduce new features and options into the command line tool and library over time and we work hard never to break existing behavior and keep the ABI and API stable.
Still, some features and functionalities go out of style sooner or later. Versions get deprecated, third party dependencies go stale and turn unsuitable for use.
How to discard “dead branches”
I like the mental image of curl as a big flourishing tree, with roots, the main trunk, branches and a multitude of thick green leaves.
Every once in a while some leaves or branches die. They turn all brown and dry and we need to do something about them. We trim the tree.
Dependencies go sour
A few times during curl’s life-time we have found ourselves in a position where we supported a third-party dependency for some functionality, but that library was maybe no longer a product we want to recommend our users to actually lean on. For libraries that aren’t being maintained correctly or that fall behind in other aspects and we are made aware of that fact, we need to make a decision.
To keep supporting the library, we indirectly give it our blessing. Products that no longer get updates or we no longer trust are keeping up with the world, we need to “chop off” from the curl family tree. We have done so with a few TLS libraries over the years for example. Users that want to keep doing TLS powered protocols with curl and libcurl then “just” have to switch to a supported TLS library.
We don’t have any special mechanisms or policies to detect this kind of expired products, but we simply have to use our judgement and do the best we can.
Protocol versions go extinct
The most obvious example here are SSL and TLS versions. Back in 1998 there actually existed servers that supported SSL version 2. Since that day, all SSL versions have been phased out from the internet and several TLS versions have. I doubt we have seen the last deprecated protocol version so more are likely to happen going forward.
In curl we follow the internet transfer ecosystem and in many cases we get told by the TLS libraries what curl can and cannot support. The options that we once added that ask for certain specific protocol versions thus no longer actually work for most system installs. This is rarely a problem because even if users could ask for a really old TLS version to be attempted, rarely any server side is actually supporting those so this usually isn’t a cause for concern.
For the most desperate users in niche situations, they can usually go build their own versions of the TLS libraries and re-enable the deprecated versions if they really need.
We might give up on features
Sometimes we add features to libcurl for stuff that then over time never really gets used or work correctly. This can happen because the Internet world decided that the particular feature isn’t cool or even because it doesn’t work perfectly in the curl architecture. In cases where we can then remove the feature without breaking the ABI, like when the option the user sets asks for something to get used if possible, things are good. This is for example how we could remove support for HTTP/1 Pipelining from curl without breaking any promises.
Unused platforms erode
The source code for curl and libcurl is written to be extremely portable and has reportedly run on at least 86 different operating systems at some point. Platforms come and go, and popularity and support for them go up and down over time. We might add support for a popular platform one year, which later, fifteen years or so down the line is basically never used or heard of any longer. Code for platforms that we never build or verify slowly but very surely “rots” and will no longer be possible to build.
After a certain number of years without attention, the cost of keeping the code around that presumably does not work anyway, gets higher than the value of keeping it around. If then nobody raises their hand and says they are in fact using curl on platform ZZ we might rip out the adjustments we have in the code for that platform. We can always bring back support if someone would suddenly emerge with such a desire and plan.
In practical terms
When we figure out that there’s something in the project we think should be deprecated. A feature, a backend, something, we bring it up for discussion on the libcurl mailing list and we add it to the docs/DEPRECATE.md document, to be removed no earlier than six months later. This way, users can check this document and get informed about these plans in several releases ahead.
If someone would object to a particular deprecation plan, we would take that discussion and possibly reconsider or delay the deprecation plans.
Welcome to the 206th curl release, 59 days since we shipped curl 7.81.0. The extra three days because I was away on the day the release would normally have been done. (I call it Impartial Content as a little play on the HTTP 206 response code message.)
the 206th release 2 change 59 days (total: 8,751) 173 bug-fixes (total: 7,691) 266 commits (total: 28,321) 0 new public libcurl function (total: 86) 0 new curl_easy_setopt() option (total: 295) 1 new curl command line option (total: 245) 67 contributors, 39 new (total: 2,597) 43 authors, 24 new (total: 1,014) 0 security fixes (total: 111) 0 USD paid in Bug Bounties (total: 16,900 USD)
Changes
There are only two changes this time around:
The JSON option
With the new --json command line option, curl suddenly made it more convenient to send JSON from command lines and shell scripts.
MesaLink: removed support
curl supports a crazy amount of different TLS libraries, but the amount was now decreased by one (to 13) as we officially drop support for MesaLink. The library is not developed anymore so we don’t want to encourage future users to go down that road.
Bug-fixes
Here are some of my favorite bug-fixes from this release cycle.
bearssl
We landed three notable fixes for the bearssl backend which should make users of it happy. For cert expiration, incomplete CA certs and session resumption.
strlen call removals
After I posted a library-call count to the mailing list that showed quite a large number of calls to strlen(), a cleanup race started that subsequently reduced the number of calls by over 60% in some use cases! Primarily replaced by compile-time constants.
configure requires –with-nss-deprecated
To build curl with the NSS backend using configure, you now need to confirm this choice by also passing on --with-nss-deprecated to make it clear to users what the future looks like for our NSS support.
erase some more sensitive command line arguments
After a lengthy discussion we now hide even more command line arguments from appearing in ps output (on systems that support it). Since the hiding is done by curl itself, there is still a short moment during which they will be visible, plus that we cannot hide everything so there is still a risk that some argument might leak information unwillingly. That is the nature of command line arguments. Use the config file concept or stdin etc to work around that.
NPN is deprecated
The TLS extension NPN is now marked “deprecated” and is scheduled for removal in six months unless someone yells very loudly and explains why not. This extension was once used to negotiate SPDY and early HTTP/2 but have no purpose these days. The browsers removed support for it several years ago.
allow CURLOPT_HTTPHEADER change “:scheme”
The only pseudo header for HTTP/2 and HTTP/3 that couldn’t be modified by a user can now be changed at will.
remove support for TPF, Netware, vxWorks
Support for these platforms for which the code haven’t been modified for the last decade or so, and therefore are highly unlikely to still work, were dropped. After this, I had it confirmed that you can still build curl for vxWorks using the “regular” build!
remove support for CURL_DOES_CONVERSIONS
After support for TPF was dropped, we took the next step and removed support for the charset conversion functions necessary to run curl on non-ASCII platforms such EBCDIC using ones. As TPF was the only/last platform such platform we supported, this cleanup improved lots of code paths.
allow user callbacks to call curl_multi_assign
A regression in 7.81.0 made curl_multi_assign() return error if used from a callback.
http3: quiche and ngtcp2 fixes
We landed several fixes in both HTTP/3 backends, improving the situation for everyone who plays HTTP/3 with curl.
reduce memory use when FTP is disabled
After several cleanups the total memory footprint for builds with FTP and/or proxy support disabled has been reduced.
check for ~/.config/curlrc too
curl now also checks for its default “config file” in the path mentioned above.
DNS options that need c-ares now fail without it
The command line tool offers a set of functions to control DNS specific details, and since those options only work if libcurl was built to use c-ares and not at all if it was built to use another resolver backend, curl will now correctly return error when one of those options are used when libcurl can’t execute them.
keep trailing dot in host name
If there is a trailing dot after the host name in the URL, that dot is now kept in the name when used everywhere internally – except for the SNI field in TLS.
wolfssl: when SSL_read() returns zero, check the error
Even while obviously very rare, curl could wrongly return an “end of transfer” prematurely before this fix.
Next
The next release is scheduled to ship on April 27, 2022.
In the beginning of every year I jot down a couple of “larger things” I would like to work on during the year. Some of the ideas are more in the maybe category and meant to be tested on the audience and users to see what others think and want, others are features I believe are ripe and ready for addition.
I talk about what I personally plan and consider to work on as I cannot control or decide what volunteers and random contributors will do this year, but of course with the hope that feedback from users and customers will guide me.
If you have use cases, applications or devices with needs or interests in particular topics going forward: let me know! At this webinar or outside of it.
The curl roadmap 2022 webinar will happen on February 17th at 09:00 PST (17:00 UTC, 18:00 CET). Register to attend on this link:
I’m the lead developer in the curl project. We make the command line tool curl and the library libcurl, for doing Internet transfers. The command line tool uses the library for all the internet transfer heavy lifting.
The command line tool is somewhat of a shell binding to access libcurl.
We make these things. We recently surpassed 1,000 authors. I lead the project and I have done the most number of commits per month in curl for the last 79 months, and in fact in 222 of the 267 months we have stats for.
The tool came first
curl was born in 1998 as a command line tool only. Two years in, we (well, mostly me actually) remodeled some internals and shipped the first libcurl to the world in August 2000. The idea was (and still is) to provide the same Internet transfer powers the tool has to any application or device out there that need it.
Code sizes
The library side of things is right now about 85% of the total product code. A little over 120,000 lines right now, including comments.
Pie chart showing code distribution between tool and library
Users at scale
Many users think of the curl project as equivalent to the curl tool and the command line tool certainly has a lot of users. It is available for and on all the popular platforms. It is impossible to count curl command users, but millions should not be an exaggeration.
While it seems likely that more users are using the tool than is writing applications that use libcurl, each product, service or device that uses libcurl can themselves scale up the volumes. A single libcurl user can use libcurl in an application used by billions. A few hundreds or thousands of libcurl users populate the world with things transferring data with our library. (A noticeable share of the current Internet traffic is likely driven by libcurl.)
The net result is therefore that libcurl runs in several thousand times more installations than the tool.
If we visualize the number of curl users as a yellow ball and the libcurl installations as a green ball, putting them next to each other would look something like this:
Planet curl is barely visible next to planet libcurl. Install and user numbers are estimated.
(Image math: 3 million curl users, 10 billion libcurl installations. Yellow sphere radius is 90 vs the green’s 1340)
Complexity
Making a command line tool is much easier than doing a library. The command line tool has just one entry point and the interface is limited. A command line and associated files and pipes to read from. In our case, the tool lets libcurl deal with a lot of platform specifics which makes the command line code generic and mostly identical on all platforms it runs on.
A library has many more entry points, each that needs to be written to care for what the users might pass in to them. libcurl has code to work with millions of build combinations, and (right now) up to 35 different external libraries. (13 different TLS libraries, 3 different SSH libraries, 2 different QUIC/HTTP/3 libraries, 3 different compression libraries etc.)
libcurl is used in many more challenging environments such as niche operating systems, scaled up to thousands of concurrent transfers, systems that never exits and in builds with a creative set of features disabled at build-time.
Compared to the curl tool, libcurl is way more advanced. A bug in the command line tool is often much easier to fix than one in the library. Such bugs are also rarer since it is much simpler and a smaller amount of code.
Where it matters
Because of what I have outlined above, I focus my curl work on library related changes. Both when it comes to bug-fixes and adding features. It scales better to the world, and as one of the designers and architects of most solutions used in libcurl I am in many cases a suitable engineer to work on many of the more complicated problems.
It is smarter for the entire project for me to leave the slightly less complicated problems and more easily understood features to be fixed and added by others. It scales better. After all, I have a finite number of hours per week to spend on curl. I want to make the best use of my time as I can.
Therefore, I tend to leave tool-related things for others to work on.
Where it pays
I work on curl for a living. The companies that pay for support have higher priority than almost any other bug reports, and they tend to be libcurl related. Keeping my paying customers happy is crucial to me. And in a funny way also to others, since that work usually end up benefiting all curl users.
Where its fun
As I work on curl full-time during my workdays and also during a good chunk of my spare time, I need to “lighten up” my work at times and get some variation. Sometimes I can go about and find new little things to work on in the project that maybe isn’t top priority by any means, but are things that could use some polish and are different enough from what I spent the rest of my week on. To give me variation. To keep the fun.
Also, sometimes scratching the surface on a new somewhat forgotten place brings up more important stuff.
Theory meets practice
Years ago I even for a while considered to hand over maintenance of the curl tool to someone else and more distinctly say that I would only work on the library as they are separate entities and could possibly benefit from being worked on independently from each other.
My idea of focusing my work on the more complicated issues, to work on design and architecture and help newcomers find their way into the code doesn’t always work out.
I never gave up maintenance of the tool and a lot of things that someone else could implement or fix in the project aren’t, which makes me eventually get to work on that too anyway. For the good of the project. Also, it makes my work day more varied and fun if I take occasional strolls around the project every now and then and put on some new paint on areas I find could use some.
Summary
Most of my work efforts go into libcurl matters. But I work on the tool too.
The curl “cockpit” is yet again extended with a new command line option: --json. The 245th command line option.
curl is a generic transfer tool for sending and receiving data across networks and it is completely agnostic as to what it transfers or even why.
To allow users to craft all sorts of transfers, or requests if you will, it offers a wide range of command line options. This flexibility has made it possible for a large number of users to keep using curl even as network ecosystems and its (HTTP) use have changed over time.
Craft content to send
curl offers a few convenience options for creating contents to send in HTTP “uploads” . Options such as -F for building multi-part formposts, and --data-urlencode for URL-encoding POST data.
JSON is the new popular kid
When curl was born, JSON didn’t exist. Over the decades since, JSON has grown to become a very commonly used format for structured data, often used in “REST API” calls and more. curl command lines sending and receiving JSON are very common now.
When asking users today what features they would like to see added in curl, and when asking them what good features they see in curl alternatives, a huge chunk of them mention JSON support in one way or another.
Partly because dealing with JSON with curl sometimes make the involved command lines rather long and quirky.
JSON in curl
The discussion has been ignited in the curl community about what, if anything, we should do in curl to make it a smoother and better tool when working with JSON. The offered opinions range from nothing (“curl is content agnostic”) to full-fledged JSON generator, parser and pretty-printer (or a combination in between). I don’t think we are ready to put our collective foot down on where we should go with this.
Introducing --json
While the discussion is ongoing, we still bring a first JSON oriented feature to the tool as a first step. The --json option. This is a new option that basically works as an alias, or shortcut, to sending JSON to an endpoint. You use it like this:
If you want another Content-Type or Accept header, then you can override them as usual with -H. If you use multiple --json options on the same command line, the contents from them will be concatenated before sent off.
Ships!
This new json option was merged in this commit and will be available and present in curl 7.82.0, to be released in early March 2022. Of course you can build it from git or a daily snapshot already now to test it out!
Future JSON stuff
I want to at least experiment a little with some level of JSON creation for curl. To help users who want to create basic JSON on the command line to send off. Lots of users run into problems with this since JSON uses double quotes a lot and then the combination of quoting and unquoting and the shell often make users confused or even frustrated.
Exactly how or to what level is still to be determined. Your opinions and feedback are very valuable and helpful.
jo+curl+jq
This trinity of commands helps you send correctly formatted data and show the returned JSON easily. Like in a command line similar to:
jo name=daniel tool=curl | curl --json @- https://httpbin.org/post | jq
In this presentation I talk about how libcurl’s most important aspect is the stable ABI and API. If we just maintain those, we can change the internals however we like.
libcurl has a system with build-time selected components, called backends. They are usually powered by third party libraries. With the recently added HTTP backend there are now seven “flavors” of backends.
A backend can be provided by a library written in rust, as long as that rust component provides a C interface that libcurl can use. When it does, it being rust or not is completely transparent and a non-issue for libcurl.
curl currently supports components written in rust for three different backends:
The number of commit authors in curl has gradually been increasing over time.
1998-03-20: 1 author recorded in the premiere curl release. It actually took until May 30th 2001 for the second recorded committer (partly of course because we used CVS). 1167 days to bump up the number one notch.
2014-05-06: 250 authors took 5891 days to reach.
2017-08-18: 500 authors required additional 1200 days
2019-12-20: 750 authors was reached after only 854 more days
2022-01:30: reaching 1000 authors took an additional 772 days.
Jan-Piet Mens became the thousandth author by providing pull-request #8354.
These 1,000 authors have done in total 28,163 commits. This equals 28.2 commits on average while the median commit author only did a single one – more than 60% of the authors are one-time committers after all! Only fourteen persons in the project have authored more than one hundred commits.
It took us 8,717 days to reach 1,000 authors, making it one new committer in the project per 8.7 days on average, over the project’s life time so far. The latest 250 authors have “arrived” at a rate of one new author per 3.1 days.
In 2021 we had more commit authors than ever before and also more first-timers than ever. It will certainly be a challenge to reach that same level or maybe even break the record again in 2022.
It took almost 24 years to reach 1,000 authors. How long until we double this?
Visualized on video
To celebrate this occasion, I did a fresh gource video of curl development so far; for the duration of the first 1,000 committers. I spiced it up by adding 249 annotations of source related facts laid out in time.
If you look at the curl/curl repository on GitHub, it appears to be short of a few hundred committers. It says 792 right now.
This is because GitHub can’t count commit authors. The theory is that they somehow ignore committers that do not have GitHub accounts. My author count is instead based on plain git access and standard git commands run against a plain git clone:
On Friday January 21, 2022 I received this email. I tweeted about it and it took off like crazy.
The email comes from a fortune-500 multi-billion dollar company that apparently might be using a product that contains my code, or maybe they have customers who do. Who knows?
My guess is that they do this for some compliance reasons and they “forgot” that their open source components are not automatically provided by “partners” they can just demand this information from.
I answered the email very briefly and said I will be happy to answer with details as soon as we have a support contract signed.
I think maybe this serves as a good example of the open source pyramid and users in the upper layers not at all thinking of how the lower layers are maintained. Building a house without a care about the ground the house stands on.
In my tweet and here in my blog post I redact the name of the company. I most probably have the right to tell you who they are, but I still prefer to not. (Especially if I manage to land a profitable business contract with them.) I suspect we can find this level of entitlement in many companies.
The level of ignorance and incompetence shown in this single email is mind-boggling.
While they don’t even specifically say which product they are using, no code I’ve ever been involved with or have my copyright use log4j and any rookie or better engineer could easily verify that.
In the picture version of the email I padded the name fields to better anonymize the sender, and in the text below I replaced them with NNNN.
(And yes, it is very curious that they send queries about log4j now, seemingly very late.)
Continue down for the reply.
The email
Dear Haxx Team Partner,
You are receiving this message because NNNN uses a product you developed. We request you review and respond within 24 hours of receiving this email. If you are not the right person, please forward this message to the appropriate contact.
As you may already be aware, a newly discovered zero-day vulnerability is currently impacting Java logging library Apache Log4j globally, potentially allowing attackers to gain full control of affected servers.
The security and protection of our customers' confidential information is our top priority. As a key partner in serving our customers, we need to understand your risk and mitigation plans for this vulnerability.
Please respond to the following questions using the template provided below.
1. If you utilize a Java logging library for any of your application, what Log4j versions are running?
2. Have there been any confirmed security incidents to your company?
3. If yes, what applications, products, services, and associated versions are impacted?
4. Were any NNNN product and services impacted?
5. Has NNNN non-public or personal information been affected?
6. If yes, please provide details of affected information NNNN immediately.
7. What is the timeline (MM/DD/YY) for completing remediation? List the NNNN steps, including dates for each.
8. What action is required from NNNN to complete this remediation?
In an effort to maintain the integrity of this inquiry, we request that you do not share information relating to NNNN outside of your company and to keep this request to pertinent personnel only.
Thank you in advance for your prompt attention to this inquiry and your partnership!
Sincerely,
NNNN Information Security
The information contained in this message may be CONFIDENTIAL and is for the intended addressee only. Any unauthorized use, dissemination of the information, or copying of this message is prohibited. If you are not the intended addressee, please notify the sender immediately and delete this message.
Their reply
On January 24th I received this response, from the same address and it quotes my reply so I know they got it fine.
Hi David,
Thank you for your reply. Are you saying that we are not a customer of your organization?
/ [a first name]
My second reply
I replied again (22:29 CET on Jan 24) to this mail that identified me as “David”. Now there’s this great story about a David and some giant so I couldn’t help myself…
Hi Goliath,
No, you have no established contract with me or anyone else at Haxx whom you addressed this email to, asking for a lot of information. You are not our customer, we are not your customer. Also, you didn't detail what product it was regarding.
So, we can either establish such a relationship or you are free to search for answers to your questions yourself.
I can only presume that you got our email address and contact information into your systems because we produce a lot of open source software that are used widely.
Best wishes,
Daniel
The image version of the initial email
The company
Update on February 9: The email came from MetLife.
