Welcome to this patch release of curl, shipped only 14 days since the previous version. We decided to cut the release cycle short because of the several security vulnerabilities that were pointed out. See below for details. There are no new features added in this release.
It burns. Mostly in our egos.
Release video
Numbers
the 208th release 0 changes 14 days (total: 8,818) 41 bug-fixes (total: 7,857) 65 commits (total: 28,573) 0 new public libcurl function (total: 88) 0 new curl_easy_setopt() option (total: 295) 0 new curl command line option (total: 247) 20 contributors, 6 new (total: 2,632) 13 authors, 3 new (total: 1,030) 6 security fixes (total: 121) Bug Bounties total: 22,660 USD
Security
Axel Chong reported three issues, Harry Sintonen two and Florian Kohnhäuser one. An avalanche of security reports. Let’s have a look.
curl removes wrong file on error
CVE-2022-27778 reported a way how the brand new command line options remove-on-error and no-clobber when used together could end up having curl removing the wrong file. The file that curl was told not to clobber actually.
cookie for trailing dot TLD
CVE-2022-27779 is the first of two issues this time that identified a problem with how curl handles trailing dots since the 7.82.0 version. This flaw lets a site set a cookie for a TLD with a trailing dot that then might have curl send it back for all sites under that TLD.
percent-encoded path separator in URL host
In CVE-2022-27780 the reporter figured out how to abuse curl URL parser and its recent addition to decode percent-encoded host names.
CERTINFO never-ending busy-loop
CVE-2022-27781 details how a malicious server can trick curl built with NSS to get stuck in a busy-loop when returning a carefully crafted certificate.
TLS and SSH connection too eager reuse
CVE-2022-27782 was reported and identifies a set of TLS and SSH config parameters that curl did not consider when reusing a connection, which could end up in an application getting a reused connection for a transfer that it really did not expected to.
HSTS bypass via trailing dot
CVE-2022-30115 is very similar to the cookie TLD one, CVE-2022-27779. A user can make curl first store HSTS info for a host name without a trailing dot, and then in subsequent requests bypass the HSTS treatment by adding the trailing dot to the host name in the URL.
Bug-fixes
The security fixes above took a lot of my efforts this cycle, but there were a few additional ones I could mention.
In our regular attempts to remove warnings and errors, we fixed this warning that was on the border of a false positive. We want to be able to run with sanitizers warning-free so that every real warning we get can be treated accordingly.
gskit: fixed bogus setsockopt calls
A set of setsockopt() calls in the gskit.c backend was fond to be defective and haven’t worked since their introduction several years ago.
define HAVE_SSL_CTX_SET_EC_CURVES for libressl
Users of the libressl backend can now set curves correctly as well. OpenSSL and BoringSSL users already could.
x509asn1: make do_pubkey handle EC public keys
The libcurl private asn1 parser (used for some TLS backends) did not have support for these before.
As usual, here’s a set of selected favorite bug-fixes of mine from this cycle:
require “see also” for every documented option in curl.1
When the curl command man page is generated at build time, the script now makes sure that there is a “see also” for each option. This will help users find related info. More mandatory information for each option makes us do better documentation that ultimately helps users.
lazy-alloc the table in Curl_hash_add()
The internal hash functions moved the allocation of the actual hash table from the init() function to when the first add() is called to add something to the table. This delay simplified code (when the init function became infallible ) and does even avoid a few allocs in many cases.
enable haproxy support for hyper backend
Plus a range of code and test cases adjusted to make curl built with hyper run better. There are now less than 30 test cases still disabled for hyper. We are closing in!
mbedTLS: add support for CURLOPT_CAINFO_BLOB
Users of this backend can now also use this feature that allows applications to provide a CA cert store in-memory instead of using an external file.
multi: handle errors returned from socket/timer callbacks
It was found out that the two multi interface callbacks didn’t at all treat errors being returned the way they were documented to do. They are now, and the documentation was also expanded to clarify.
nss:set_cipher don’t clobber the cipher list
Applications that uses libcurl built to use NSS found out that if they would select cipher, they would also effectively prevent connections from being reused due to this bug.
openldap: implement STARTTLS
curl can now switch LDAP transfers into LDAPS using the STARTTLS command much like how it already works for the email protocols. This ability is so far limited to LDAP powered by OpenLDAP.
openssl: define HAVE_OPENSSL_VERSION for OpenSSL 1.1.0+
This little mistake made libcurl use the wrong method to extract and show the OpenSSL version at run-time, which most notably would make libcurl say the wrong version for OpenSSL 3.0.1, which would rather show up as the non-existing version 3.0.0a.
sha256/md5: return errors when init fails
A few internal functions would simply ignore errors from these hashing functions instead of properly passing them back to the caller, making them to rather generate the wrong hash instead of properly and correctly returning an error etc.
curl: updated search for a file in the homedir
The curl tool now searches for personal config files in a slightly improved manner, to among other things make it find the same .known_hosts file on Windows as the Microsoft provided ssh client does.
url: check ssl_config when re-use proxy connection
A bug in the logic for checking connections in the connection pool suitable for reuse caused flaws when doing subsequent HTTPS transfers to servers over the same HTTPS proxy.
ngtcp2: verify server certificate
When doing HTTP/3 transfers, libcurl is now doing proper server certificate verification for the QUIC connection – when the ngtcp2 backend is used. The quiche backend is still not doing this, but really should.
urlapi: accept port number zero
Years ago I wrote a blog post about using port zero in URLs to do transfers. Then it turned out port zero did not work like that with curl anymore so work was done and now order is restored again and port number zero is once again fine to use for curl.
urlapi: provide more detailed return codes
There are a whole range of new error codes introduced that help better identify and pinpoint what the problem is when a URL or a part of a URL cannot be parsed or will not be accepted. Instead of the generic “failed to parse URL”, this can now better tell the user what part of the URL that was found out to be bad.
socks5h: use appropriate ATYP for numerical IP address
curl supports using SOCKS5 proxies and asking the proxy to resolve the host name, what we call socks5h. When using this protocol and using a numerical IP address in the URL, curl would use the SOCKS protocol slightly wrong and pass on the wrong “ATYP” parameter which a strict proxy might reject. Fixed now.
Coming up?
The curl factory never stops. There are many pull-requests already filed and in the pipeline of possibly getting merged. There will also, without any doubts, be more ones coming up that none of us have yet thought about or considered. Existing pending topics might include:
the ManageSieve protocol
--no-clobber
CURLMOPT_STREAM_WINDOW_SIZE
Remove Mesalink support
HAproxy protocol v2
WebSockets
Export/import SSL session-IDs
HTTP/3 fixes
more hyper improvements
CURLFOLLOW_NO_CUSTOMMETHOD
Next release
March 2, 2022 is the scheduled date for what will most probably become curl 7.82.0.
This new option provides yet another knob for applications to control and limit connection reuse. Using this option, the application sets an upper limit that specifies that connections may not be older than this when being reused. It can for example be used to make sure connections don’t “get stuck” on one single specific load-balancer for extended periods of time.
This new callback gets called immediately before the request is started (= “Pre req”). Gives the application a heads up and some details about what is just about to start.
libssh2: SHA256 fingerprint support
This is a bump up from the previous MD5 version. Make sure that curl connections to the correct host with a cryptographically strong fingerprint check.
When a URL API function returns an error it does so using a CURLUcode type. Now there’s a function to convert this error code into an error message.
support UNC paths in file: URLs on Windows
The URL parser now understands UNC paths when parsing URLs on Windows.
allow setting of groups/curves with wolfSSL
The wolfSSL backend now allows setting of specific curves for TLS 1.3 connections, which allows users to use post quantum algorithms if wolfSSL is built to support them!
Bug-fixes
This is another release with more than one hundred individual bug-fixes, and here are a selected few I think might be worth highlighting.
more hyper work
I’ve done numerous small improvements in this cycle to take the hyper backend closer to become a solid member of the curl backend family.
print curl --help descriptions aligned right
When listing available options with --help or -h, the list is now showing the descriptions right-aligned which makes the output more easy-to-read in my opinion. Illustration:
Showing off right-aligned –help descriptions
store remote IP address for QUIC connections too
HTTP/3 uses QUIC and with this bug fixed, the %{remote_ip} variable for --write-out works there as well – as you’d expect. This fixes the underlying CURLINFO_PRIMARY_IP option.
reject HTTP response codes < 100
The HTTP response parser no longer accepts response code numbers below 100 as a legitimate protocol. The HTTP protocol has never specified any such code so this should not cause any problems. Lots of other HTTP clients already enforce this requirement too.
do not wait for writable socket if there’s no remote HTTP/2 window
If curl runs out of remote HTTP/2 window for a stream while uploading, ie the other end says it can’t receive any data right now, curl would still wait for the socket to be writable which would cause really bad busy-loops.
get libssh2 version at runtime
curl now asks libssh2 for its version string in runtime instead of showing the version that was used back when the curl binary was built, as it might very well be upgraded dynamically after the build!
require all man pages to use the same section headers in the same order
We tighten the bolts even more to make the libcurl documentation consistent. Now all libcurl man pages have to feature the same set of headers in the same order to not cause test failure. This includes a required example section. We also added an extra check to detect a common backslash-wrong-formatting mistake that we’ve previously done several times in man page examples.
NTLM: use DES_set_key_unchecked with OpenSSL
Turns out that the code that is implemented to use OpenSSL for doing NTLM authentication was using a function call that returns error if a “bad” key is used. NTLM v1 being a very weak algorithm in general makes it very easy to end up calling the function with such a weak key and then the NTLM authentication failed…
openssl: if verifypeer is not requested, skip the CA loading
If peer verification is disabled for a transfer when curl is built to use OpenSSL or one of its forks, curl will now completely skip the loading of the CA cert bundle. It was basically only used for being able to show in the verbose output if there was a match or not – that was then ignored anyway – and by skipping the load step the TLS handshake will use less memory and CPU resources.
urlapi: URL decode percent-encoded host names
The URL parser did not accept percent encoded host names. Now it does. Note however that libcurl will not by default percent-encode the host name when extracting a URL for the purpose of keeping IDN names working. It’s a little complicated.
ngtcp2: use QUIC TLS RFC9001
We switch over to use the “real” QUIC identifier when setting up connections instead of using the identifier made for a previous draft version of the protocol. curl still asks h3-29 for HTTP/3 since that RFC has still not shipped, but includes h3 as well – since it turns out some servers assume plain h3 when the final QUIC v1 version is used for transport.
a failed etag save now only fails that transfer
Specifying a file name to save etags in will from now on only fail those transfers using that file. If you specify more transfers that use another file or not use etags at all, those transfers can still get done.
Added test case for checksrc!
The custom tool we use for checking code style compliance, checksrc, has become quite advanced and now checks for a lot of source code details, and when we once again improved it in this release cycle we also added a test case for the tool itself to make sure it maintains its functionality even when we keep improving it going forward!
Next
The next release is planned for January 5, 2022. We have several changes queued up as pull requests already so I’d say it is likely that it then will become version 7.81.0.
Support?
I offer commercial support and curl related contracting for you and your company!
The curl factory has once again cranked out a new curl release.
Release presentation
Numbers
the 202nd release 3 changes 56 days (total: 8,580) 128 bug-fixes (total: 7,270) 186 commits (total: 27,651) 0 new public libcurl function (total: 85) 0 new curl_easy_setopt() option (total: 290) 0 new curl command line option (total: 242) 62 contributors, 25 new (total: 2,484) 41 authors, 16 new (total: 948) 3 security fixes (total: 111) 3,500 USD paid in Bug Bounties (total: 16,900 USD)
Security
This time we once again announce security advisories in association with the release.
CVE-2021-22945 is a double-free flaw in the MQTT code. Patch your old curl or upgrade to this version if you use it to send MQTT. The reporter of this flaw was awarded 1,000 USD from the curl bug-bounty program.
CVE-2021-22946 is a bug in response handling for several protocols (IMAP, POP3 and FTP) that bypasses the enforced TLS check so that even transfers that are explicitly told to require TLS can accidentally silently be performed in clear text! Rewarded 1,000 USD.
CVE-2021-22947 allows a mitm attacker to inject data into the protocol stream for FTP, IMAP, POP3 or SMTP in a way before the TLS upgrade so that curl accepts that data and uses it after after having upgraded to TLS. The untrusted data slips in and gets treated as trusted! Rewarded 1,500 USD.
These two latter ones came as an indirect result/inspiration from the NO STARTTLS research.
Changes
This release comes with three changes to take note of…
Users of the bearssl TLS backend will appreciate that it too now supports the CURLOPT_CAINFO_BLOB option so that the CA certificate easily can be provided in-memory by applications.
The cookie engine in curl now considers http://localhost to be secure and thus cookies that are marked “secure” will be sent over it – even when not using HTTPS. This is done because curl now since a while back makes sure that localhost is always truly local.
Users of the Secure Transport TLS backend can now use CURLINFO_CERTINFO to extract information about the server’s certificate chain.
Bug-fixes
Some of the most interesting bug-fixes we did this round.
use ares_getaddrinfo()
When you build curl to use the c-ares name resolver backend, curl will now use this function to get improved handling for IPv4+ IPv6. This also ups our requirement on c-ares to 1.16.0.
hyper works better
1xx responses, Transfer-Encoding and more have been fixed. The number of tests that are disabled for hyper builds are even fewer than before, but there’s still plenty of work to do before it can be considered not experimental.
cmake builds: avoid poll() on macOS
We have deliberately not used poll() in macOS builds for a long time when building with configure, and now we realized that cmake builds inadvertently had poll() use enabled, which caused curl to misbehave when for example connecting to a host while that connection got closed by the peer. poll() is now disabled on macOS even when cmake is used.
configure: also check lib64 for the OpenSSL pkg-config file
OpenSSL did a very late change just before they shipped version 3.0.0: they modified the default installation path for the library for 64 bit systems from $prefix/lib to $prefix/lib64, and subsequently we had to update our configure script detection logic accordingly. This helps configure to find OpenSSL v3 installs.
curl.1: provide examples for each option
The documentation now must provide at least one example command line for each command line option curl provides. This is verified in the build and will cause build errors if a file doesn’t comply! Feel free to suggest new, more or better examples when you start to see them in the man page.
HTTP 1.1: disallow >3-digit response codes
The HTTP protocol is defined to only allow three-digit numbers and now curl enforces that check stricter. This was in part made to align behavior when curl is built to use hyper.
HTTP 1.1: ignore content-length if any transfer-encoding is used
Non-chunked transfer-encoded content that also sends Content-Length headers is rare but was incorrectly handled by curl. Found when aligning behavior with hyper builds.
http_proxy: only wait for writable socket while sending request
Due to a mistake in the handling of what socket activity to wait for, curl could accidentally be made to busy-loop from the CONNECT request was sent to the proxy until the first data arrived.
Support mbedTLS 3.0.0
When mbedTLS released a new version with support for TLS 1.3 etc, they also modified the API a bit.
Ban strerror
We’ve had our own internal strerror replacement function for a long time (primarily due to it not being thread-safe), but a recent code review revealed that a lot of uses of this function had still crept in. Starting now, our code check tool (checksrc) will error if strerror is used in libcurl code.
The mailing lists move from cool.haxx.se to lists.haxx.se
Our old decommissioned server hosted 29 mailing lists. We moved most of them and killed off a few. All our mailing lists are now hosted on lists.haxx.se, including all the curl related ones of course! The old server name will simply redirect to the new one if you go there with a browser.
Welcome to another release! We did more bug-fixes than in any previous release (176). We paid more in bug-bounties than during any previous release cycle (4,200 USD) and we thank more contributors in the RELEASE-NOTES than ever before (83).
Release presentation
Numbers
the 201st release 6 changes 56 days (total: 8,524) 176 bug-fixes (total: 7,142) 263 commits (total: 27,465) 0 new public libcurl function (total: 85) 0 new curl_easy_setopt() option (total: 290) 0 new curl command line option (total: 242) 83 contributors, 49 new (total: 2,459) 56 authors, 32 new (total: 933) 5 security fixes (total: 108) 4,200 USD paid in Bug Bounties (total: 13,200 USD)
Security
This time we announce no less than 5 separate security advisories and we are once again setting a new bug-bounty record. This release cycle we spent 4,200 USD on rewarding security researchers.
Let’s do them in numerical order. Click the CVE links to get to the full and much more detailed advisories.
CVE-2021-22922: Wrong content via metalink not discarded
This was one of the problems we found that that all together made us take the drastic decision to completely remove metalink support.
The metalink format has a hash for the content so that a client can detect faulty contents. curl didn’t act properly if the has mismatched and it could easily make users not realize the bad content.
If you download the metalink file using credentials, the subsequent download(s) of the file mentioned in that XML file will also get the same credentials passed to those servers, unexpectedly, thus potentially leaking sensitive information to other parties!
CVE-2021-22924: Bad connection reuse due to flawed path name checks
libcurl keeps previously used connections in a connection pool for subsequent transfers to reuse, if one of them matches the setup.
Due to errors in the logic, the config matching function did not take ‘issuer cert’ into account and it compared the involved paths case insensitively, which could lead to libcurl reusing wrong connections!
Possibly the most embarrassing security flaw in a long time.
When we shipped 7.77.0 we announced CVE-2021-22898, which was a flaw in the telnet code and an associated fix. Know what? The fix was incomplete and plain wrong so the original problem actually remained for a certain set of input.
This is thus the second advisory for the same problem and now we fix this again. Hopefully for real and for good this time…
CVE-2021-22926: CURLOPT_SSLCERT mixup with Secure Transport
When libcurl is built to use the macOS native TLS library Secure Transport, an application can ask for the client certificate by name or with a file name – using the same option. If the name exists as a file, it will be used instead of by name. This could be exploited in rare circumstances.
Changes
The six big changes this time around are:
curl_url_set now rejects spaces in the URL unless specifically asked to allow them.
CURLE_SETOPT_OPTION_SYNTAX is a brand new return code (name) for when libcurl detects an illegally formatted input passed to a setopt(), when it is detected later in the transfer.
Now username and password can be used for MQTT transfers.
Bug-fixes
I’m doing this release in the midst of my vacation so I’m doing this section a little shorter than usual. Here are some bug-fixes to highlight:
Lots of tiny fixes when built to use hyper for HTTP. Now curl built to use hyper can run many more test cases. There’s more to do and more will be done going forward.
GnuTLS: set the preferred TLS versions in correct order. Previously the occasional TLS connection would fail because of the wrong way the code would instruct GnuTLS…
on macOs: free returned memory of SCDynamicStoreCopyProxies. A small memory leak on Apple operating systems, possibly as many as one per name resolve?
HSTS: not experimental anymore. It is now built and provided by default.
netrc: skip ‘macdef’ definitions. The netrc parser is ancient but it turned out this kind of macro use could threw it off.
OpenSSL: don’t remove session id entry in disassociate. We had a regression that basically killed session-id use and made subsequent TLS handshakes to the same host much slower.
Next
The plans says we ship the next release on September 15th 2021. See you then!
Welcome to the 200th curl release. We call it 200 OK. It coincides with us counting more than 900 commit authors and surpassing 2,400 credited contributors in the project. This is also the first release ever in which we thank more than 80 persons in the RELEASE-NOTES for having helped out making it and we’ve set two new record in the bug-bounty program: the largest single payout ever for a single bug (2,000 USD) and the largest total payout during a single release cycle: 3,800 USD.
This release cycle was 42 days only, two weeks shorter than normal due to the previous 7.76.1 patch release.
Release Presentation
Numbers
the 200th release 5 changes 42 days (total: 8,468) 133 bug-fixes (total: 6,966) 192 commits (total: 27,202) 0 new public libcurl function (total: 85) 2 new curl_easy_setopt() option (total: 290) 2 new curl command line option (total: 242) 82 contributors, 44 new (total: 2,410) 47 authors, 23 new (total: 901) 3 security fixes (total: 103) 3,800 USD paid in Bug Bounties (total: 9,000 USD)
Security
We set two new records in the curl bug-bounty program this time as mentioned above. These are the issues that made them happen.
This is a Use-After-Free in the OpenSSL backend code that in the absolutely worst case can lead to an RCE, a Remote Code Execution. The flaw is reasonably recently added and it’s very hard to exploit but you should upgrade or patch immediately.
The issue occurs when TLS session related info is sent from the TLS server when the transfer that previously used it is already done and gone.
The reporter was awarded 2,000 USD for this finding.
When libcurl accepts custom TELNET options to send to the server, it the input parser was flawed which could be exploited to have libcurl instead send contents from the stack.
The reporter was awarded 1,000 USD for this finding.
In the Schannel backend code, the selected cipher for a transfer done with was stored in a static variable. This caused one transfer’s choice to weaken the choice for a single set transfer could unknowingly affect other connections to a lower security grade than intended.
The reporter was awarded 800 USD for this finding.
Changes
In this release we introduce 5 new changes that might be interesting to take a look at!
Make TLS flavor explicit
As explained separately, the curl configure script no longer defaults to selecting a particular TLS library. When you build curl with configure now, you need to select which library to use. No special treatment for any of them!
No more SSL
curl now has no more traces of support for SSLv2 or SSLv3. Those ancient and insecure SSL versions were already disabled by default by TLS libraries everywhere, but now it’s also impossible to activate them even in special build. Stripped out from both the curl tool and the library (thus counted as two changes).
HSTS in the build
We brought HSTS support a while ago, but now we finally remove the experimental label and ship it enabled in the build by default for everyone to use it more easily.
In-memory cert API
We introduce API options for libcurl that allow users to specify certificates in-memory instead of using files in the file system. See CURLOPT_CAINFO_BLOB.
Favorite bug-fixes
Again we manage to perform a large amount of fixes in this release, so I’m highlighting a few of the ones I find most interesting!
Version output
The first line of curl -V output got updated: libcurl now includes OpenLDAP and its version of that was used in the build, and then the curl tool can add libmetalink and its version of that was used in the build!
curl_mprintf: add description
We’ve provided the *printf() clone functions in the API since forever, but we’ve tried to discourage users from using them. Still, now we have a first shot at a man page that clearly describes how they work.
This is important as they’re not quite POSIX compliant and users who against our advice decide to rely on them need to be able to know how they work!
CURLOPT_IPRESOLVE: preventing wrong IP version from being used
This option was made a little stricter than before. Previously, it would be lax about existing connections and prefer reuse instead of resolving again, but starting now this option makes sure to only use a connection with the request IP version.
This allows applications to explicitly create two separate connections to the same host using different IP versions when desired, which previously libcurl wouldn’t easily let you do.
Ignore SIGPIPE in curl_easy_send
libcurl makes its best at ignoring SIGPIPE everywhere and here we identified a spot where we had missed it… We also made sure to enable the ignoring logic when built to use wolfSSL.
Several HTTP/2-fixes
There are no less than 6 separate fixes mentioned in the HTTP/2 module in this release. Some potential memory leaks but also some more behavior improving things. Possibly the most important one was the move of the transfer-related error code from the connection struct to the transfers struct since it was vulnerable to a race condition that could make it wrong. Another related fix is that libcurl no longer forcibly disconnects a connection over which a transfer gets HTTP_1_1_REQUIRED returned.
Partial CONNECT requests
When the CONNECT HTTP request sent to a proxy wasn’t all sent in a single send() call, curl would fail. It is baffling that this bug hasn’t been found or reported earlier but was detected this time when the reporter issued a CONNECT request that was larger than 16 kilobytes…
TLS: add USE_HTTP2 define
There was several remaining bad assumptions that HTTP/2 support in curl relies purely on nghttp2. This is no longer true as HTTP/2 support can also be provide by hyper.
When libcurl (built with libssh2 support) stopped an SFTP transfer because a timeout was triggered, the following SFTP disconnect procedure was subsequently also stopped because of the same timeout and therefore wasn’t allowed to properly clean up everything, leading to a memory-leak!
IRC network switch
We moved the #curl IRC channel to the new network libera.chat. Come join us there!
Next release
On Jul 21, 2021 we plan to ship the next release. The version number for that is not yet decided but we have changes in the pipeline, making a minor version number bump very likely.
I’m happy to once again present a new curl release to the world. This time we decided to cut the release cycle short and do a quick patch release only two weeks since the previous release. The primary reason was the rather annoying and embarrassing HTTP/2 bug. See below for all the details.
Release presentation
Numbers
the 199th release 0 changes 14 days (total: 8,426) 21 bug-fixes (total: 6,833) 30 commits (total: 27,008) 0 new public libcurl function (total: 85) 0 new curl_easy_setopt() option (total: 288) 0 new curl command line option (total: 240) 23 contributors, 10 new (total: 2,366) 14 authors, 6 new (total: 878) 0 security fixes (total: 100) 0 USD paid in Bug Bounties (total: 5,200 USD)
Bug-fixes
This was a very short cycle but we still managed to merge a few interesting fixes. Here are some:
HTTP/2 selection over HTTPS
This regression is the main reason for this patch release. I fixed an issue before 7.76.0 was released and due to lack of covering tests with other TLS backends, nobody noticed that my fix also break HTTP/2 selection over HTTPS when curl was built to use one GnuTLS, BearSSL, mbedTLS, NSS, SChannnel, Secure Transport or wolfSSL!
The problem I fixed for 7.76.0: I made sure that no internal code updates the HTTP version choice the user sets, but that it then updates only the internal “we want this version”. Without this fix, an application that reuses an easy handle could without specifically asking for it, get another HTTP version in subsequent requests if a previous transfer had been downgraded. Clearly the fix was only partial.
The new fix should make HTTP/2 work and make sure the “wanted version” is used correctly. Fingers crossed!
Progress meter final update in parallel mode
When doing small and quick transfers in parallel mode with the command line tool, the logic could make the last update call to get skipped!
file: support getting directories again
Another regression. A recent fix made curl not consider directories over FILE:// to show a size (if -I or -i is used). That did however also completely break “getting” such a directory…
HTTP proxy: only loop on 407 + close if we have credentials
When a HTTP(S) proxy returns a 407 response and closes the connection, curl would retry the request to it even if it had no credentials to use. If the proxy just consistently did the same 407 + close, curl would get stuck in a retry loop…
The fixed version now only retries the connection (with auth) if curl actually has credentials to use!
Next release cycle
The plan is to make the next cycle two weeks shorter, to get us back on the previously scheduled path. This means that if we open the feature window on Monday, it will be open for just a little over two weeks, then give us three weeks of only bug-fixes before we ship the next release on May 26.
The next one is expected to become 7.77.0. Due to the rather short feature window this coming cycle I also fear that we might not be able to merge all the new features that are waiting to get merged.
libcurl did not strip off user credentials from the URL when automatically populating the Referer: HTTP request header field in outgoing HTTP requests, and therefore risks leaking sensitive data to the server that is the target of the second HTTP request.
libcurl automatically sets the Referer: HTTP request header field in outgoing HTTP requests if the CURLOPT_AUTOREFERER option is set. With the curl tool, it is enabled with --referer ";auto".
Rewarded with 800 USD
TLS 1.3 session ticket proxy host mixup
CVE-2021-22890 is a flaw in curl’s OpenSSL backend that allows a malicious HTTPS proxy to trick curl with session tickets and subsequently allow the proxy to MITM the remote server. The problem only exists with OpenSSL and it needs to speak TLS 1.3 with the HTTPS proxy – and the client must accept the proxy’s certificate, which has to be especially crafted for the purpose.
Note that an HTTPS proxy is different than the mode comon HTTP proxy.
The reporter declined offered reward money.
Changes
We list 6 “changes” this time around. They are…
support multiple -b parameters
The command line option for setting cookies can now be used multiple times on the command line to specify multiple cookies. Either by setting cookies by name or by providing a name to a file to read cookie data from.
add –fail-with-body
The command line tool has had the --fail option for a very long time. This new option is very similar, but with a significant difference: this new option saves the response body first even if it returns an error due to HTTP response code that is 400 or larger.
add DoH options to disable TLS verification
When telling curl to use DoH to resolve host names, you can now specify that curl should ignore the TLS certificate verification for the DoH server only. Independently of how it treats other TLS servers that might be involved in the transfer.
read and store the HTTP referer header
This is done with the new CURLINFO_REFERER libcurl option and with the command line tool, --write-out '%{referer}‘.
support SCRAM-SHA-1 and SCRAM-SHA-256 for mail auth
For SASL authentication done with mail-using protocols such as IMAP and SMTP.
A rustls backend
A new optional TLS backend. This is provided via crustls, a C API for the rustls TLS library.
Some Interesting bug-fixes
Again we’ve logged over a hundred fixes in a release, so here goes some of my favorite corrections we did this time:
curl: set CURLOPT_NEW_FILE_PERMS if requested
Due to a silly mistake in the previous release, the new --create-file-mode didn’t actually work because it didn’t set the permissions with libcurl properly – but now it does.
share user’s resolve list with DOH handles
When resolving host names with DoH, the transfers done for that purpose now “inherit” the same --resolve info as used for the normal transfer, which I guess most users already just presumed it did…
bump the max HTTP request size to 1MB
Virtually all internal buffers have length restrictions for security and the maximum size we allowed for a single HTTP request was previously 128 KB. A user with a use-case sending a single 300 KB header turned up and now we allow HTTP requests to be up to 1 MB! I can’t recommend doing it, but now at least curl supports it.
allow SIZE to fail when doing (resumed) FTP upload
In a recent change I made SIZE failures get treated as “file not found” error, but it introduced this regression for resumed uploads because when resuming a file upload and there’s nothing uploaded previously, SIZE is then expected to fail and it is fine.
fix memory leak in ftp_done
The torture tests scored another victory when it proved that when the connection failed at just the correct moment after an FTP transfer is complete, curl could skip a free() and leak memory.
fail if HTTP/2 connection is terminated without END_STREAM
When a HTTP/2 connection is (prematurely) terminated, streams over that connection could return “closed” internally without noticing the premature part. As there was no previous END_STREAM message received for the stream(s), curl should consider that an error and now it does.
don’t set KEEP_SEND when there’s no more HTTP/2 data to be sent
A rare race condition in the HTTP/2 code could make libcurl remain expecting to send data when in reality it had already delivered the last chunk.
With HTTP, use credentials from transfer, not connection
Another cleanup in the code that had the potential to get wrong in the future and mostly worked right now due to lucky circumstances. In HTTP each request done can use its own set of credentials, so it is vital to not use “connection bound” credentials but rather the “transfer oriented” set. That way streams and requests using different credentials will work fine over a single connection even when future changes alter code paths.
lib: remove ‘conn->data’ completely
A rather large internal refactor that shouldn’t be visible on the outside to anyone: transfer objects now link to the corresponding connection object like before, but now connection objects do not link to any transfer object. Many transfers can share the same connection.
adapt to OpenSSL v3’s new const for a few API calls
The seemingly never-ending work to make a version 3 of OpenSSL keeps changing the API and curl is adapting accordingly so that we are prepared and well functioning with this version once it ships “for real” in the future.
Close the connection when downgrading from HTTP/2 to HTTP/1
Otherwise libcurl is likely to reuse the same (wrong) connection again in the next transfer attempt since the connection reuse logic doesn’t take downgrades into account!
Cap initial HTTP body data amount during send speed limiting
The rate limiting logic was previously not correctly applied on the initial body chunk that libcurl sends. Like if you’d tell libcurl to send 50K data with CURLOPT_POSTFIELDS and limit the sending rate to 5K/second.
Celebratory drink
I’ll go for an extra fine cup of coffee today after I posted this. I think I’m worth it. I bet you are too. Go ahead and join me: Hooray for another release!
There’s been another 56 day release cycle and here’s another curl release to chew on!
Release presentation
Numbers
the 197th release 6 changes 56 days (total: 8,357) 113 bug fixes (total: 6,682) 268 commits (total: 26,752) 0 new public libcurl function (total: 85) 1 new curl_easy_setopt() option (total: 285) 2 new curl command line option (total: 237) 58 contributors, 30 new (total: 2,322) 31 authors, 17 new (total: 860) 0 security fixes (total: 98) 0 USD paid in Bug Bounties(total: 4,400 USD)
Security
No new security advisories this time!
Changes
We added --create-file-mode to the command line tool. To be used for the protocols where curl needs to tell the remote server what “mode” to use for the file when created. libcurl already supported this, but now we expose the functionality to the tool users as well.
The CURLOPT_RESOLVE option got an extended format that now allows entries to be inserted to get timed-out after the standard DNS cache expiry time-out.
gophers:// – the protocol GOPHER done over TLS – is now supported by curl.
As a new experimentally supported HTTP backend, you can now build curl to use Hyper. It is not quite up to 100% parity in features just yet.
AWS HTTP v4 Signature support. This is an authentication method for HTTP used by AWS and some other services. See CURLOPT_AWS_SIGV4 for libcurl and --aws-sigv4 for the curl tool.
Bug-fixes
Some of the notable things we’ve fixed this time around…
Reduced struct sizes
In my ongoing struggles to remove “dead weight” and ensure that curl can run on as small devices as possible, I’ve trimmed down the size of several key structs in curl. The memory foot-print of libcurl is now smaller than it has been for a very long time.
Reduced conn->data references
While itself not exactly a bug-fix, this is a step in a larger refactor of libcurl where we work on removing all references back from connections to the transfer. The grand idea is that transfers can point to connections, but since a connection can be owned and used by many transfers, we should remove all code that reference back to a transfer from the connection. To simplify internals. We’re not quite there yet.
Silly writeout time units bug
Many users found out that when asking the curl tool to output timing information with -w, I accidentally made it show microseconds instead of seconds in 7.74.0! This is fixed and we’re back to the way it always was now…
CURLOPT_REQUEST_TARGET works with HTTP proxy
The option that lets the user set the “request target” of a HTTP request to something custom (like for example “*” when you want to issue a request using the OPTIONS method) didn’t work over proxy!
CURLOPT_FAILONERROR fails after all headers
Often used with the tools --fail flag, this is feature that makes libcurl stop and return error if the HTTP response code is 400 or larger. Starting in this version, curl will still read and parse all the response headers before it stops and exists. This then allows curl to act on and store contents from the other headers that can be used for features in combination with --fail.
Proxy-Connection duplicated headers
In some circumstances, providing a custom “Proxy-Connection:” header for a HTTP request would still get curl’s own added header in the request as well, making the request get sent with a duplicate set!
CONNECT chunked encoding race condition
There was a bug in the code that handles proxy responses, when the body of the CONNECT responses was using chunked-encoding. curl could end up thinking the response had ended before it actually had…
proper User-Agent header setup
Back in 7.71.0 we fixed a problem with the user-agent header and made it get stored correctly in the transfer struct, from previously having been stored in the connection struct.
That cause a regression that we fixed now. The previous code had a mistake that caused the user-agent header to not get used when a connection was re-used or multiplexed, which from an outside perspective made it appear go missing in a random fashion…
add support for %if [feature] conditions in tests
Thanks to the new preprocessor we added for test cases some releases ago, we could now take the next step and offer conditionals in the test cases so that we can now better allow tests to run and behave differently depending on features and parameters. Previously, we have resorted to only make tests require a certain feature set to be able to run and otherwise skip the tests completely if the feature set could be satisfied, but with this new ability we can make tests more flexible and thus work with a wider variety of features.
if IDNA conversion fails, fallback to Transitional
A user reported that that curl failed to get the data when given a funny URL, while it worked fine in browsers (and wget):
The host name consists of a heart and a fox emoji in the .ws top-level domain. This is yet another URLs-are-weird issue and how to do International Domain Names with them is indeed a complicated matter, but starting now curl falls back and tries a more conservative approach if the first take fails and voilá, now curl too can get the heart-fox URL just fine… Regular readers of my blog might remember the smiley URLs of 2015, which were similar.
urldata: make magic first struct field
We provide types for the most commonly used handles in the libcurl API as typedef’ed void pointers. The handles are typically declared like this:
CURL *easy; CURLM *multi; CURLSH *shared;
… but since they’re typedefed void-pointers, the compiler cannot helpfully point out if a user passes in the wrong handle to the wrong libcurl function and havoc can ensue.
Starting now, all these three handles have a “magic” struct field in the same relative place within their structs so that libcurl can much better detect when the wrong kind of handle is passed to a function and instead of going bananas or even crash, libcurl can more properly and friendly return an error code saying the input was wrong.
Since you’d ask: using void-pointers like that was a mistake and we regret it. There are better ways to accomplish the same thing, but the train has left. When we’ve tried to correct this situation there have been cracks in the universe and complaints have been shouted through the ether.
SECURE_RENEGOTIATION support for wolfSSL
Turned out we didn’t support this before and it wasn’t hard to add…
openssl: lowercase the host name before using it for SNI
The SNI (Server Name Indication) field is data set in TLS that tells the remote server which host name we want to connect to, so that it can present the client with the correct certificate etc since the server might serve multiple host names.
The spec clearly says that this field should contain the DNS name and that it is case insensitive – like DNS names are. Turns out it wasn’t even hard to find multiple servers which misbehave and return the wrong cert if the given SNI name is not sent lowercase! The popular browsers typically always send the SNI names like that… In curl we keep the host name internally exactly as it was given to us in the URL.
With a silent protest that nobody cares about, we went ahead and made curl also lowercase the host name in the SNI field when using OpenSSL.
I did not research how all the other TLS libraries curl can use behaves when setting SNI. This same change is probably going to have to be done on more places, or perhaps we need to surrender and do the lowercasing once and for all at a central place… Time will tell!
Future!
We have several pull requests in the queue suggesting changes, which means the next release is likely to be named 7.76.0 and the plan is to ship that on March 31st, 2021.
Welcome to another curl release, 56 days since the previous one.
Release presentation
Numbers
the 196th release 1 change 56 days (total: 8,301) 107 bug fixes (total: 6,569) 167 commits (total: 26,484) 0 new public libcurl function (total: 85) 6 new curl_easy_setopt() option (total: 284) 1 new curl command line option (total: 235) 46 contributors, 22 new (total: 2,292) 22 authors, 8 new (total: 843) 3 security fixes (total: 98) 1,600 USD paid in Bug Bounties(total: 4,400 USD)
Security
This time around we have no less than three vulnerabilities fixed and as shown above we’ve paid 1,600 USD in reward money this time, out of which the reporter of the CVE-2020-8286 issue got the new record amount 900 USD. The second one didn’t get any reward simply because it was not claimed. In this single release we doubled the number of vulnerabilities we’ve published this year!
The six announced CVEs during 2020 still means this has been a better year than each of the six previous years (2014-2019) and we have to go all the way back to 2013 to find a year with fewer CVEs reported.
I’m very happy and proud that we as an small independent open source project can reward these skilled security researchers like this. Much thanks to our generous sponsors of course.
CVE-2020-8284: trusting FTP PASV responses
When curl performs a passive FTP transfer, it first tries the EPSV command and if that is not supported, it falls back to using PASV. Passive mode is what curl uses by default.
A server response to a PASV command includes the (IPv4) address and port number for the client to connect back to in order to perform the actual data transfer.
This is how the FTP protocol is designed to work.
A malicious server can use the PASV response to trick curl into connecting back to a given IP address and port, and this way potentially make curl extract information about services that are otherwise private and not disclosed, for example doing port scanning and service banner extractions.
If curl operates on a URL provided by a user (which by all means is an unwise setup), a user can exploit that and pass in a URL to a malicious FTP server instance without needing any server breach to perform the attack.
There’s no really good solution or fix to this, as this is how FTP works, but starting in curl 7.74.0, curl will default to ignoring the IP address in the PASV response and instead just use the address it already uses for the control connection. In other words, we will enable the CURLOPT_FTP_SKIP_PASV_IP option by default! This will cause problems for some rare use cases (which then have to disable this), but we still think it’s worth doing.
CVE-2020-8285: FTP wildcard stack overflow
libcurl offers a wildcard matching functionality, which allows a callback (set with CURLOPT_CHUNK_BGN_FUNCTION) to return information back to libcurl on how to handle a specific entry in a directory when libcurl iterates over a list of all available entries.
When this callback returns CURL_CHUNK_BGN_FUNC_SKIP, to tell libcurl to not deal with that file, the internal function in libcurl then calls itself recursively to handle the next directory entry.
If there’s a sufficient amount of file entries and if the callback returns “skip” enough number of times, libcurl runs out of stack space. The exact amount will of course vary with platforms, compilers and other environmental factors.
The content of the remote directory is not kept on the stack, so it seems hard for the attacker to control exactly what data that overwrites the stack – however it remains a Denial-Of-Service vector as a malicious user who controls a server that a libcurl-using application works with under these premises can trigger a crash.
CVE-2020-8286: Inferior OCSP verification
libcurl offers “OCSP stapling” via the CURLOPT_SSL_VERIFYSTATUS option. When set, libcurl verifies the OCSP response that a server responds with as part of the TLS handshake. It then aborts the TLS negotiation if something is wrong with the response. The same feature can be enabled with --cert-status using the curl tool.
As part of the OCSP response verification, a client should verify that the response is indeed set out for the correct certificate. This step was not performed by libcurl when built or told to use OpenSSL as TLS backend.
This flaw would allow an attacker, who perhaps could have breached a TLS server, to provide a fraudulent OCSP response that would appear fine, instead of the real one. Like if the original certificate actually has been revoked.
Change
There’s really only one “change” this time, and it is an experimental one which means you need to enable it explicitly in the build to get to try it out. We discourage people from using this in production until we no longer consider it experimental but we will of course appreciate feedback on it and help to perfect it.
The change in this release introduces no less than 6 new easy setopts for the library and one command line option: support HTTP Strict-Transport-Security, also known as HSTS. This is a system for HTTPS hosts to tell clients to attempt to contact them over insecure methods (ie clear text HTTP).
One entry-point to the libcurl options for HSTS is the CURLOPT_HSTS_CTRL man page.
Bug-fixes
Yet another release with over one hundred bug-fixes accounted for. I’ve selected a few interesting ones that I decided to highlight below.
enable alt-svc in the build by default
We landed the code and support for alt-svc: headers in early 2019 marked as “experimental”. We feel the time has come for this little baby to grow up and step out into the real world so we removed the labeling and we made sure the support is enabled by default in builds (you can still disable it if you want).
8 cmake fixes bring cmake closer to autotools level
In curl 7.73.0 we removed the “scary warning” from the cmake build that warned users that the cmake build setup might be inferior. The goal was to get more people to use it, and then by extension help out to fix it. The trick might have worked and we’ve gotten several improvements to the cmake build in this cycle. More over, we’ve gotten a whole slew of new bug reports on it as well so now we have a list of known cmake issues in the KNOWN_BUGS document, ready for interested contributors to dig into!
configure now uses pkg-config to find openSSL when cross-compiling
Just one of those tiny weird things. At some point in the past someone had trouble building OpenSSL cross-compiled when pkg-config was used so it got disabled. I don’t recall the details. This time someone had the reversed problem so now the configure script was fixed again to properly use pkg-config even when cross-compiling…
curl: only warn not fail, if not finding the home dir
The curl tool attempts to find the user’s home dir, the user who invokes the command, in order to look for some files there. For example the .curlrc file. More importantly, when doing SSH related protocol it is somewhat important to find the file ~/.ssh/known_hosts. So important that the tool would abort if not found. Still, a command line can still work without that during various circumstances and in particular if -k is used so bailing out like that was nothing but wrong…
curl_easy_escape: limit output string length to 3 * max input
In general, libcurl enforces an internal string length limit that prevents any string to grow larger than 8MB. This is done to prevent mistakes or abuse. Due a mistake, the string length limit was enforced wrongly in the curl_easy_escape function which could make the limit a third of the intended size: 2.67 MB.
only set USE_RESOLVE_ON_IPS for Apple’s native resolver use
This define is set internally when the resolver function is used even when a plain IP address is given. On macOS for example, the resolver functions are used to do some conversions and thus this is necessary, while for other resolver libraries we avoid the resolver call when we can convert the IP number to binary internally more effectively.
By a mistake we had enabled this “call getaddrinfo() anyway”-logic even when curl was built to use c-ares on macOS.
fix memory leaks in GnuTLS backend
We used two functions to extract information from the server certificate that didn’t properly free the memory after use. We’ve filed subsequent bug reports in the GnuTLS project asking them to make the required steps much clearer in their documentation so that perhaps other projects can avoid the same mistake going forward.
libssh2: fix transport over HTTPS proxy
SFTP file transfers didn’t work correctly since previous fixes obviously weren’t thorough enough. This fix has been confirmed fine in use.
make curl –retry work for HTTP 408 responses too
Again. We made the --retry logic work for 408 once before, but for some inexplicable reasons the support for that was accidentally dropped when we introduced parallel transfer support in curl. Regression fixed!
use OPENSSL_init_ssl() with >= 1.1.0
Initializing the OpenSSL library the correct way is a task that sounds easy but always been a source for problems and misunderstandings and it has never been properly documented. It is a long and boring story that has been going on for a very long time. This time, we add yet another chapter to this novel when we start using this function call when OpenSSL 1.1.0 or later (or BoringSSL) is used in the build. Hopefully, this is one of the last chapters in this book.
“scheme-less URLs” not longer accept blank port number
curl operates on “URLs”, but as a special shortcut it also supports URLs without the scheme. For example just a plain host name. Such input isn’t at all by any standards an actual URL or URI; curl was made to handle such input to mimic how browsers work. curl “guesses” what scheme the given name is meant to have, and for most names it will go with HTTP.
Further, a URL can provide a specific port number using a colon and a port number following the host name, like “hostname:80” and the path then follows the port number: “hostname:80/path“. To complicate matters, the port number can be blank, and the path can start with more than one slash: “hostname://path“.
curl’s logic that determines if a given input string has a scheme present checks the first 40 bytes of the string for a :// sequence and if that is deemed not present, curl determines that this is a scheme-less host name.
This means [39-letter string]:// as input is treated as a URL with a scheme and a scheme that curl doesn’t know about and therefore is rejected as an input, while [40-letter string]:// is considered a host name with a blank port number field and a path that starts with double slash!
In 7.74.0 we remove that potentially confusing difference. If the URL is determined to not have a scheme, it will not be accepted if it also has a blank port number!
