Imagine running a trillion dollar company that bundles various open source components into your products, making billions of dollars of profit annually. When one of your users reach out and ask for help, with the product you ship to your customers, you instead refer the user to the open source project. The project which is run by volunteers which you never sponsored with a cent.
Who would do something like that?
Thursday November 18 at 09:00 Pacific Time (17:00 UTC, 18:00 CET)
I will sum up the curl news in the 7.80.0 release with extra focus and details on and about the post quantum features you can now use!
The webinar will be live and we end with a Q&A session where you can ask me anything, in particular about this release and curl post quantum, but not necessarily limited to that…
I still remember the RFC number off the top of my head for the first multipart formdata spec that I implemented support for in curl. Added to curl in version 5.0, December 1998. RFC 1867.
Multipart formdata is the name of the syntax for how HTTP clients send data in a HTTP POST when they want to send binary content and multiple fields. Perhaps the most common use case for users is when uploading a file or an image with a browser to a website. This is also what you fire off with curl’s -F command line option.
RFC 1867 was published in November 1995 and it has subsequently been updated several times. The most recent incarnation for this spec is now known as RFC 7578, published in July 2015. Twenty years of history, experiences and minor adjustments. How do they affect us?
I admit to having dozed off a little at the wheel and I hadn’t really paid attention to the little tweaks that slowly had been happening in the multipart formata world until Ryan Sleevi woke me up.
While I wasn’t looking, the popular browsers have now all switched over to a different encoding style for field and file names within the format. They now all use percent-encoding, where we originally all used to do backslash-escaping! I haven’t actually bothered to check exactly when they switched, primarily because I don’t think it matters terribly much. They do this because this is now the defined syntax in WHATWG’s HTML spec. Yes, another case of separate specs diverging and saying different things for what is essentially the same format.
curl 7.80.0 is probably the last curl version to do the escaping the old way and we are likely to switch to the new way by default starting in the next release. The work on this is done by Patrick Monnerat in his PR.
For us in the curl project it is crucial to maintain behavior, but it is also important to be standard compliant and to interoperate with the big world out there. When all the major browsers have switched to percent-encoding I suspect it is only a matter of time until servers and server-side scripts start to assume and require percent-encoding. This is a tricky balancing act. There will be users who expect us to keep up with the browsers, but also some that expect us to maintain the way we’ve done it for almost twenty-three years…
libcurl users at least, will be offered a way to switch back to use the old escaping mechanism to make sure applications that know they work with older style server decoders can keep their applications working.
This blog post is made public before the PR mentioned above has been merged in order for people to express opinions and comments before it is done. The plan is to have it merged and ship in curl 7.81.0, to be released in January 2022.
Almost 14 months since I celebrated 15,000 commits in curl’s source code repository I have now passed 16,000 commits.
My commit number 16,000 was a minor man page fix.
The official gitstats page shows that I’ve committed changes on almost 4,600 separate days since the year 2000.
16,000 commits is 13,413 commits more than the person with the second-most number of commits: Yang Tse (2587 commits). He has however not committed anything in curl since he vanished from the project in 2013.
I have also done 4,700 commits in the curl-www repository, but that’s another story.
I like figuring out even or somehow particularly aligned numbers and dates to celebrate. Here’s another one: today marks the day when httpget 0.1 was released in 1996.
httpget 0.1 was a tiny command line tool written by Rafael Sagula. It was less than 300 lines of C code. (Today, the product code is 173,000 lines!)
I found httpget just days after it was released when I was searching for a tool to use for downloading currency rates with from an HTTP site. This was the time before Google existed so I assume I used Altavista or something. I can’t remember actually.
The httpget source code was my initial step into the HTTP world. In many ways, reading that code opened my eyes.
Of course there was some issues with the code that I can’t remember any longer, but I very soon sent Rafael my first patches to improve it. A few weeks later I took over maintenance of the little tool. The rest is documented elsewhere. The oldest httpget source code I still have around is httpget 1.3, released sometime between April and August 1997.
I have since been working with HTTP pretty much daily. For twenty-five years.
If someone tries to tell you that HTTP is an easy or simple protocol, don’t believe them.
For a long time I have been wanting to avoid us to ever reach curl version 7.100.0. I strongly suspect that going three-digits in the minor number will cause misunderstandings and possibly even glitches in people’s comparison scripts etc. If nothing else, it is just a very high number to use in a version string and I believe we would be better off by starting over. Reset the clock so to speak.
Given that, a curl version 8.0.0 is inevitably going to have to happen and since we do releases every 8 weeks and we basically bump the version number in just about every release, there is a limited amount of time left to avoid the minor number to reach 100. We just shipped curl 7.80.0, so we have less than 20 release cycles in the worst case; a few years.
A while ago it struck me that we have a rather big anniversary coming up, also within a few years, and that is curl’s 25th birthday.
Let’s combine the two !
On March 20, 2023, curl turns 25 years old. On that same day, the plan is to release curl 8.0.0, the first major number bump in (by then) 23 years.
The major number bump will happen independently of what features we add or not, independently if there are any new bells and whistles to celebrate or “just” a set of bug-fixes landed. As we always do, the release is time based and not feature based, but the different unique property that this particular release will have is that it will also reset the minor version number and increase the major version number!
In the regular curl release cycles we always do releases every eight weeks on Wednesdays and if we ever adjust the cycle, we do it with full weeks and stick to releasing on Wednesdays.
March 20, 2023 however is a Monday. curl 8.0.0 will therefore also be different in the way that it will not be released on a Wednesday. We will also have to adjust the cycle period before and after this release, since they cannot then be eight weeks cycles. The idea is to go back to Wednesdays and the regular cycles again after 8.0.0 has happened.
Image by Tanja Richter from Pixabay
Welcome to another curl release, 7 weeks since the previous one.
the 204th release
6 changes
49 days (total: 8,636)
117 bug-fixes (total: 7,397)
198 commits (total: 27,866)
1 new public libcurl function (total: 86)
4 new curl_easy_setopt() option (total: 294)
1 new curl command line option (total: 243)
78 contributors, 44 new (total: 2,533)
50 authors, 28 new (total: 976)
0 security fixes (total: 111)
0 USD paid in Bug Bounties (total: 16,900 USD)
CURLOPT_MAXLIFETIME_CONNThis new option provides yet another knob for applications to control and limit connection reuse. Using this option, the application sets an upper limit that specifies that connections may not be older than this when being reused. It can for example be used to make sure connections don’t “get stuck” on one single specific load-balancer for extended periods of time.
CURLOPT_PREREQFUNCTIONThis new callback gets called immediately before the request is started (= “Pre req”). Gives the application a heads up and some details about what is just about to start.
This is a bump up from the previous MD5 version. Make sure that curl connections to the correct host with a cryptographically strong fingerprint check.
When a URL API function returns an error it does so using a CURLUcode type. Now there’s a function to convert this error code into an error message.
The URL parser now understands UNC paths when parsing URLs on Windows.
The wolfSSL backend now allows setting of specific curves for TLS 1.3 connections, which allows users to use post quantum algorithms if wolfSSL is built to support them!
This is another release with more than one hundred individual bug-fixes, and here are a selected few I think might be worth highlighting.
I’ve done numerous small improvements in this cycle to take the hyper backend closer to become a solid member of the curl backend family.
--help descriptions aligned rightWhen listing available options with --help or -h, the list is now showing the descriptions right-aligned which makes the output more easy-to-read in my opinion. Illustration:
HTTP/3 uses QUIC and with this bug fixed, the %{remote_ip} variable for --write-out works there as well – as you’d expect. This fixes the underlying CURLINFO_PRIMARY_IP option.
The HTTP response parser no longer accepts response code numbers below 100 as a legitimate protocol. The HTTP protocol has never specified any such code so this should not cause any problems. Lots of other HTTP clients already enforce this requirement too.
If curl runs out of remote HTTP/2 window for a stream while uploading, ie the other end says it can’t receive any data right now, curl would still wait for the socket to be writable which would cause really bad busy-loops.
curl now asks libssh2 for its version string in runtime instead of showing the version that was used back when the curl binary was built, as it might very well be upgraded dynamically after the build!
We tighten the bolts even more to make the libcurl documentation consistent. Now all libcurl man pages have to feature the same set of headers in the same order to not cause test failure. This includes a required example section. We also added an extra check to detect a common backslash-wrong-formatting mistake that we’ve previously done several times in man page examples.
Turns out that the code that is implemented to use OpenSSL for doing NTLM authentication was using a function call that returns error if a “bad” key is used. NTLM v1 being a very weak algorithm in general makes it very easy to end up calling the function with such a weak key and then the NTLM authentication failed…
If peer verification is disabled for a transfer when curl is built to use OpenSSL or one of its forks, curl will now completely skip the loading of the CA cert bundle. It was basically only used for being able to show in the verbose output if there was a match or not – that was then ignored anyway – and by skipping the load step the TLS handshake will use less memory and CPU resources.
The URL parser did not accept percent encoded host names. Now it does. Note however that libcurl will not by default percent-encode the host name when extracting a URL for the purpose of keeping IDN names working. It’s a little complicated.
We switch over to use the “real” QUIC identifier when setting up connections instead of using the identifier made for a previous draft version of the protocol. curl still asks h3-29 for HTTP/3 since that RFC has still not shipped, but includes h3 as well – since it turns out some servers assume plain h3 when the final QUIC v1 version is used for transport.
Specifying a file name to save etags in will from now on only fail those transfers using that file. If you specify more transfers that use another file or not use etags at all, those transfers can still get done.
The custom tool we use for checking code style compliance, checksrc, has become quite advanced and now checks for a lot of source code details, and when we once again improved it in this release cycle we also added a test case for the tool itself to make sure it maintains its functionality even when we keep improving it going forward!
The next release is planned for January 5, 2022. We have several changes queued up as pull requests already so I’d say it is likely that it then will become version 7.81.0.
I offer commercial support and curl related contracting for you and your company!