Things did not work out the way we had planned. The 7.88.0 release that was supposed to be the last curl version 7 release contained a nasty bug that made us decide that we better ship an update once that is fixed. This is the update. The second final version 7 release.
Release presentation
Numbers
the 214th release 0 changes 5 days (total: 9,103) 25 bug-fixes (total: 8,690) 32 commits (total: 29,853) 0 new public libcurl function (total: 91) 0 new curl_easy_setopt() option (total: 302) 0 new curl command line option (total: 250) 19 contributors, 7 new (total: 2,819) 10 authors, 1 new (total: 1,120) 0 security fixes (total: 135)
Bugfixes
As this is a rushed patch-release, there is only a small set of bugfixes merged in this cycle. The following notable bugs were fixed.
http2 multiplexed data corruption
The main bug that triggered the patch release. In some circumstances , when data was delivered as a HTTP/2 multiplexed stream, curl would get it wrong and cause the saved data to be corrupt. It would get the wrong data from the internal buffer.
This was not a new bug, but recent changes made it more likely to trigger.
make connect timeouts use full duration
In some cases curl would only allow half the given timeout period when doing connects.
runtests: fix “uninitialized value $port”
Running the test suite with verbose mode enabled, it would error out with this message. Since a short while back, we consider warnings in the test script fatal so this then aborts all the tests.
tests: make sure gnuserv-tls has SRP support before using it
The test suite uses gnuserv-tls to verify SRP authentication. It will only use this tool if found at startup, but due to recent changes in the GnuTLS project that ships this tool, it now builds with SRP disabled by default and thus can’t be used for this test. Now, the test script also checks that it actually supports SRP before trying to use it.
setopt: allow HTTP3 when HTTP2 is not defined
A regression made it impossible to ask for HTTP/3 if the build did not also support HTTP/2.
socketpair: allow EWOULDBLOCK when reading the pair check bytes
The fix in 7.88.0 turned out to cause occasional hiccups (on Windows at least) and this is a follow-up improvement for the verification of the socketpair emulation. When we create the pair and verify that it works, we must make sure that the code handles EWOULDBLOCK correctly.
Welcome to the final and last release in the series seven. The next release is planned and intended to become version 8.
Numbers
the 213th release 5 changes 56 days (total: 9,098) 173 bug-fixes (total: 8,665) 250 commits (total: 29,821) 0 new public libcurl function (total: 91) 0 new curl_easy_setopt() option (total: 302) 1 new curl command line option (total: 250) 78 contributors, 41 new (total: 2,812) 42 authors, 18 new (total: 1,119) 3 security fixes (total: 135)
Release presentation
Security
This time we bring you three security fixes. All of them covering cases for which we have had problems reported and fixed before, but these are new subtle variations.
While we count over 140 individual bugfixes merged for this release, here follows a curated subset of some of the more interesting ones.
http/3 happy eyeballs
When asking for HTTP/3, curl will now also try older HTTP versions with a slight delay so that if HTTP/3 does not work, it might still succeed with and use an older version.
An application can now set drastically larger download buffers. For high speed/localhost transfers of some protocols this might sometimes make a difference.
curl: output warning at –verbose output for debug-enabled version
To help users realize when they use a debug build of curl, it now outputs a warning at the top of the --verbose output. We strongly discourage users to ship or use such builds in production.
websocket: multiple bugfixes
WebSocket support remains an experimental feature in curl but it is getting better. Several smaller and bigger bugs were squashed. Please continue to try it and report any problems and we can probably consider removing the experimental label soon.
dict: URL decode the entire path always
If you used a DICT URL it would sometimes do wrong as it previously only URL decoded parts of the path when using it. Now it correctly decodes the entire thing.
URL-encode/decode much faster
The libcurl functions for doing these conversions were sped up significantly. In the order of 3x and 7x.
haxproxy: send before TLS handhshake
The haproxy details are now properly sent before the TLS handshake takes place.
HTTP/[23]: continue upload when state.drain is set
Fixes a stalling problem when data is being uploaded and downloaded at the same time.
http2: aggregate small SETTINGS/PRIO/WIN_UPDATE frames
Optimizes outgoing frames for HTTP/2 into doing more in fewer sends.
openssl: store the CA after first send (ClientHello)
By changing the order of things, curl is better off spending CPU cycles while waiting for the server’s response and thereby making the entire handshake process complete faster.
curl: repair –rate
A regression in 7.87.0 made this feature completely broken. Now back on track again.
HTTP/2 much faster multiplexed transfers
By improving the handling of multiple concurrent streams over a single connection, curl now performs such transfers much faster than before. Sometimes an almost 3x speedup.
noproxy: support for space-separated names is deprecated
The parser that parses the “noproxy” string accepts plain space (without comma) as separators, while hardly any other tool or library does. This matters because it can be set in an environment variable. This accepted space-only separation is now marked as deprecated.
nss: implement data_pending method
The NSS backend was improved to work better for cases when the socket has been drained of data and only the NSS internal buffers has it, which could lead to curl getting stalled or losing data. Note: NSS support is marked for removal later in 2023.
socketpair: allow localhost MITM sniffers
curl has an internal socketpair emulation function for Windows. The way it worked did not allow MITM sniffers, but instead return error if such a thing was detected. It turns out too many users run tools on Windows that do this, so we have changed the logic to accept their presence and use.
tests-httpd: infra to run curl against an apache httpd
An entirely new line of tests that opens up new ways to test and verify our HTTP implementations in ways we could not do before. It uses pytest and an apache httpd server with special test modules.
curl: fix hiding of command line secrets
A regression.
curl: fix error code on bad URL
If you would use an invalid URL for upload, curl would erroneously report the problem as “out of memory” which unsurprisingly greatly confused users.
the 212th release 5 changes 56 days (total: 9,042) 155 bug-fixes (total: 8,492) 238 commits (total: 29,571) 0 new public libcurl function (total: 91) 2 new curl_easy_setopt() option (total: 302) 1 new curl command line option (total: 249) 83 contributors, 40 new (total: 2,771) 42 authors, 20 new (total: 1,101) 2 security fixes (total: 132) Bug Bounties total: 48,580 USD
Release presentation
At 10:00 CET (9:00 UTC) on December 21, Daniel live-streams the release presentation on twitch. This paragraph will later be replaced by a link to the YouTube version of that video.
Security
Two security advisories this time around, severity low and medium.
The HSTS logic could be bypassed if the host name in the given URL first uses IDN characters that get replaced to ASCII counterparts as part of the IDN conversion. Like using the character UTF-8 U+3002 (IDEOGRAPHIC FULL STOP) instead of the common ASCII full stop (U+002E). Then in a subsequent request, it does not detect the HSTS state and makes a clear text transfer. Because it would store the info IDN encoded but look for it IDN decoded.
The struct returned by curl_version_info now returns all built-in features listed by name. This is a preparation to allow applications to adapt slowly and get ready for the future moment when the features can no longer fit in in the 32 bit fields previously used for this purpose.
We fixed a few issues in the hyper backend and are down to just 12 remaining disabled tests to address.
gen.pl: fix the linkifier
This script generates the curl.1 man page and make sure to properly mark references correctly, so that the man page can get rendered as we webpage with correct links etc on the website. This time we made it work better and therefore more cross-references in the man page is now linked correctly in the web version.
tool: override the numeric locale and set “C” by force
In previous curl versions it mistakenly used the locale when parsing floating point numbers, which then made the tool hard to use in scripts which would run in multiple locales. An example is the timeout option specified with -m / --max-time as number of seconds with a fraction. Now it requires the decimal separator to always be a dot/period independently of the user’s locale.
tool: timeout in the read callback
The command line tool can now timeout reading data better, for example when using telnet:// with a timeout option and the user does not press any key and nothing happens over the network.
curl_get_line: allow last line without newline char
Because of a somewhat lazy recent fix, the .netrc parsed and other users of the nternal curl_get_line() function would ignore the last line if it did not end with a newline. This is no more.
support growing FTP files with CURLOPT_IGNORE_CONTENT_LENGTH
If this option is set, also known as --ignore-content-length on the command line, curl will not complain if the size grows from the moment the FTP transfer starts until it ends. Thus allowing it to grow while being transferred.
do not send PROXY more than once
The HAproxy protocol line could get sent more than once and thus break stuff.
feature deprecation warnings in gcc
A number of outdated libcurl options and functions are now tagged as deprecated, which will cause compiler warnings when used in application code for users of gcc 6.1 or later. Deprecated here means that we recommend using other, more modern, alternatives.
parse numbers with fixed known base 10
In several places in curl and libcurl source code we would allow numbers to be specified using octal or hexadecimal while decimal was the only expected and documented base. In order to minimize surprises and for consistency, we now limited them as far as possible to only accepting decimal numbers.
rewind BEFORE request instead of AFTER previous
When curl is used to send a request, for example a POST, and there is reason for it to send it again, like if there is a redirect or an ongoing authentication process, it would previously rewind the stream at the end of that transfer first transfer in order to have it done when the next transfer is about to get done. Now, it instead does the rewind first in the second request. This, because there are times when the second request are not done, and the rewind may not work. So, such a failing rewind can be avoided by not doing it until it is strictly necessary.
noproxy
Several independent regressions were fixed – in spite of the new set of test cases added for testing this feature in the previous release. Noproxy is the support for the NO_PROXY environment variable and related options.
openssl: prefix errors with ‘[lib]/[version]:’
To help users understand errors and their origins a little better, libcurl will now prefix error messages originating from OpenSSL (and forks) with the name of the flavor and its version number.
RTSP auth works again
This functionality was broken a few versions back and now it has finally been fixed again.
runtests: –no-debuginfod now disables DEBUGINFOD_URLS
valgrind and gdb support downloading stuff at the moment of need if this environment variable is set. Previously the curl test running script would unset that variable unconditionally, but now it will not and instead offer an option that unsets it – for the cases where the environment variable causes problems (such as performance slowdowns).
HTTP/3 tests
We finally have the first infrastructure merged for doing and running HTTP/3 specific tests in the curl test suite. Now we can better avoid regressions going forward. This is only the beginning and I expect us to expand and grow these tests going forward.
determine the correct fopen option for -D
When saving response headers into a dedicated file with curl’s -D, –dump-header option, curl would be inconsistent about when to create a new file and when to append do it. Now it acts exactly as documented.
better error message for -G with bad URL
Several users figured out curl showed misleading error messages when -G was used in combination with a malformed URL. This is now improved.
repair IDN for proxies
A recent fix we landed for IDN for host names accidentally simultaneously broke it for proxies…
cmake: set the soname on the shared library
Using cmake to build libcurl as a shared library on Linux and several other systems, will now set the SONAME number correctly in the same style and with the same number that the autotools build uses.
WebSocket polish
fixes for partial frames and buffer updates
now returns CURLE_NOT_BUILT_IN when websockets support is not built in
returns error properly when the connection is closed
TLS goes connection filters => more HTTPS-proxy
As a direct result of the internal refactor and introduction of connection filters also for TLS, curl now supports HTTPS-proxy for a wider selection of TLS backends than previously.
When doing HTTP(S) transfers, libcurl might erroneously use the read callback (CURLOPT_READFUNCTION) to ask for data to send, even when the CURLOPT_POSTFIELDS option has been set, if the same handle previously was used to issue a PUT request which used that callback.
This flaw may surprise the application and cause it to misbehave and either send off the wrong data or use memory after free or similar in the subsequent POST request.
The problem exists in the logic for a reused handle when it is changed from a PUT to a POST.
curl can be told to parse a .netrc file for credentials. If that file ends in a line with consecutive non-white space letters and no newline, curl could read past the end of the stack-based buffer, and if the read works, write a zero byte possibly beyond its boundary.
This will in most cases cause a segfault or similar, but circumstances might also cause different outcomes.
If a malicious user can provide a custom .netrc file to an application or otherwise affect its contents, this flaw could be used as denial-of-service.
If curl is told to use an HTTP proxy for a transfer with a non-HTTP(S) URL, it sets up the connection to the remote server by issuing a CONNECT request to the proxy, and then tunnels the rest of protocol through.
An HTTP proxy might refuse this request (HTTP proxies often only allow outgoing connections to specific port numbers, like 443 for HTTPS) and instead return a non-200 response code to the client.
Due to flaws in the error/cleanup handling, this could trigger a double-free in curl if one of the following schemes were used in the URL for the transfer: dict, gopher, gophers, ldap, ldaps, rtmp, rtmps, telnet
curl’s HSTS check could be bypassed to trick it to keep using HTTP.
Using its HSTS support, curl can be instructed to use HTTPS directly instead of using an insecure clear-text HTTP step even when HTTP is provided in the URL. This mechanism could be bypassed if the host name in the given URL uses IDN characters that get replaced to ASCII counterparts as part of the IDN conversion. Like using the character UTF-8 U+3002 (IDEOGRAPHIC FULL STOP) instead of the common ASCII full stop (U+002E) ..
Like this: http://curl?se?
Changes
This time around we add one and we remove one.
NPN support removed
curl no longer supports using NPN for negotiating HTTP/2. The standard way for doing this has been ALPN for a long time and the browsers removed their support for NPN several years ago.
WebSocket API
There is an experimental WebSocket API included in this release. It comes in the form of three new functions and a new setopt option to control behavior. The possibly best introduction to this new API is in everything curl.
I am very interested in feedback on the API.
Bugfixes
Here some of the fixed issues from this cycle that I think are especially worthy to highlight.
aws_sigv4 header computation
The sigv4 code got a significant overhaul and should now do much better than before. This is a fairly complicated setup and there are more improvements coming for future releases.
curl man page details multi-use for each option
Every command line option is documented in its own file, which is then used as input when the huge curl.1 man page is generated. Starting now, each such file needs to specify how the option functions when specified more than once. So from now on, this information is mentioned in the man page for all supported options.
deprecate builds with small curl_off_t
Starting in this release, we deprecate support for building curl for systems without 64 bit data types. Those systems are extremely rare this days and we believe it makes sense to finally simplify a few internals when it hurts virtually no one. This is still only deprecated so users can still build on such systems for a short while longer if they really want to.
the ngtcp2 configure option defaults to ‘no’
You need to explicitly ask for ngtcp2 to be enabled in the build.
reject cookie names or content with TAB characters
Cookies with tabs in names or content are not interoperable and they caused issues when curl saved them to disk, so we decided to rather reject them.
for builds with gcc + want warnings, set gnu89 standard
Just to make better sure we maintain compatibility.
use -O2 as default optimize for clang in configure
It was just a mistake that it did not already do this.
warn for –ssl use, considered insecure
To better highlight for users that this option merely suggests for curl that it should use TLS for the protocol, while --ssl-reqd is the one that requires TLS.
ctype functions converted to macros-only
We replaced the entire function family with macros.
100+ documentation spellfixes
After a massive effort and new CI jobs, we now regularly run a spellcheck on most man pages and as a result we fixed lots of typos and we should now be able to better maintain properly spelled documentation going forward.
make nghttp2 less picky about field whitespace in HTTP/2
If built with a new enough nghttp2 library, curl will now ask it to be less picky about trailing white space after header fields. The protocol spec says they should cause failure, but they are simply too prevalent in live servers responses for this to be a sensible behavior by curl.
use the URL-decoded user name for .netrc parsing
This regression made curl not URL decode the user name provided in a URL properly when it later used a .netrc file to find the corresponding password.
make certinfo available for QUIC
The CURLOPT_CERTINFO option now works for QUIC/HTTP/3 transfers as well.
make forced IPv4 transfers only use A queries
When asking curl to use IPv4-only for transfers, curl now only resolves IPv4 names. Out in the wide world there is a significant share of systems causing problems when asking for AAAA addresses so having this option to avoid them is needed.
schannel: when importing PFX, disable key persistence
Some operations when using the Schannel backend caused leftover files on disk afterward. It really makes you wonder who ever thought designing such a thing was a good idea, but now curl no longer triggers this effect.
add and use Curl_timestrcmp
curl now uses this new constant-time function when comparing secrets in the library in an attempt to make it even less likely for an outsider to be able to use timing as a feedback as to how closely a guessed user name or password match the secret ones.
curl: prevent over-queuing in parallel mode
The command line tool would too eagerly create and queue up pending transfers in parallel mode, making a command line with millions of transfers easily use ridiculous amounts of memory.
url parser: extract scheme better when not guessing
A URL has a scheme and we can use that fact to detect it better and more reliable when not using the scheme-guessing mode.
fix parsing URL without slash with CURLU_URLENCODE
When the URL encode option is set when parsing a URL, the parser would not always properly handle URLs with queries that also lacked a slash in the path. Like https://example.com?moo.
url parser: leaner with fewer allocs
The URL parser is now a few percent faster and makes significantly fewer memory allocations and it uses less memory in total.
url parser: reject more bad characters from the host name field
Another step on the journey of making the parser stricter.
wolfSSL: fix session management bug
The session-id cache handling could trigger a crash due to a missing reference counter.
Future
We have several pull-requests in the pipe that will add changes to trigger a minor number bump.
Removals
We are planning to remove the following features in a future-:
support for systems without 64 bit data type
support for the NSS TLS library
If you depend on one of those features, yell at us on the mailing list!
Welcome to a new curl release, the result of a slightly extend release cycle this time.
Release presentation
Numbers
the 210th release 3 changes 65 days (total: 8,930) 165 bug-fixes (total: 8,145) 230 commits (total: 29,017) 0 new public libcurl function (total: 88) 2 new curl_easy_setopt() option (total: 299) 0 new curl command line option (total: 248) 79 contributors, 38 new (total: 2,690) 44 authors, 22 new (total: 1,065) 1 security fixes (total: 126) Bug Bounties total: 40,900 USD
Security
We have yet another CVE to disclose.
control code in cookie denial of service
CVE-2022-35252 allows a server to send cookies to curl that contain ASCII control codes. When such cookies subsequently are sent back to a server, they will cause 400 responses from servers that downright refuse such requests. Severity: low. Reward: 480 USD.
Changes
This release counts three changes. They are:
schannel backend supports TLS 1.3
For everyone who uses this backend (which include everyone who uses the curl that Microsoft bundles with Windows) this is great news: now you too can finally use TLS 1.3 with curl. Assuming that you use a new enough version of Windows 10/11 that has the feature present. Let’s hope Microsoft updates the bundled version soon.
These are two new options meant to replace and be used instead of the options with the same names without the “_STR” extension.
While working on support for new future protocols for libcurl to deal with, we realized that the old options were filled up and there was no way we could safely extend them with additional entries. These new functions instead work on text input and have no limit in number of protocols they can be made to support.
This was yet again a cycle packed with bugfixes. Here are some of my favorites:
asyn-thread: fix socket leak on OOM
Doing proper and complete memory cleanup even when we exist due to out of memory is sometimes difficult. I found and fixed this very old bug.
cmdline-opts/gen.pl: improve performance
The script that generates the curl.1 man page from all its sub components was improved and now typically executes several times faster then before. curl developers all over rejoice.
configure: if asked to use TLS, fail if no TLS lib was detected
Previously, the configure would instead just silently switch off TLS support which was not always easy to spot and would lead to users going further before they eventually realize this.
configure: introduce CURL_SIZEOF
The configure macro that checks for size of variable types was rewritten. It was the only piece left in the source tree that had the mention of GPL left. The license did not affect the product source code or the built outputs, but it caused questions and therefore some friction we could easily avoid by me completely writing away the need for the license mention.
close the happy eyeballs loser connection when using QUIC
A silly memory-leak when doing HTTP/3 connections on dual-stack machines.
treat a blank domain in Set-Cookie: as non-existing
Another one of those rarely used and tiny little details about following what the spec says.
configure: check whether atomics can link
This, and several other smaller fixes together improved the atomics support in curl quite a lot since the previous version. We conditionally use this C11 feature if present to make the library initialization function thread-safe without requiring a separate library for it.
digest: fix memory leak, fix not quoted ‘opaque’
There were several fixes and cleanups done in the digest department this time around.
remove examples/curlx.c
Another “victim” of the new license awareness in the project. This example was the only file present in the repository using this special license, and since it was also a bit convoluted example we decided it did not really have to be included.
resolve *.localhost to 127.0.0.1/::1
curl is now slightly more compliant with RFC 6761, follows in the browsers’ footsteps and resolves all host names in the “.localhost” domain to the fixed localhost addresses.
enable obs-folded multiline headers for hyper
curl built with hyper now also supports “folded” HTTP/1 headers.
libssh2:+libssh make atime/mtime date overflow return error
Coverity had an update in August and immediately pointed out these two long-standing bugs – in two separate SSH backends – related to time stamps and 32 bits.
curl_multi_remove_handle closes CONNECT_ONLY transfer
When an applications sets the CONNECT_ONLY option for a transfer within a multi stack, that connection was not properly closed until the whole multi handle was closed even if the associated easy handle was terminated. This lead to connections being kept around unnecessarily long (and wasting resources).
use pipe instead of socketpair on apple platforms
Apparently those platform likes to close socketpairs when the application is pushed into the background, while pipes survive the same happening… This is a change that might be preferred for other platforms as well going forward.
use larger dns hash table for multi interface
The hash table used for the DNS cache is now made larger for the multi interface than when created to be used by the easy interface, as it simply is more likely to be used by many host names then and then it performs better like this.
reject URLs with host names longer than 65535 bytes
URLs actually have no actual maximum size in any spec and neither does the host name within one, but the maximum length of a DNS name is 253 characters. Assuming you can resolve the name without DNS, another length limit is the SNI field in TLS that is an unsigned 16 bit number: 65535 bytes. This implies that clients cannot connect to any SNI-using TLS protocol with a longer name. Instead of checking for that limit in many places, it is now done early.
reduce size of several struct fields
As part of the repeated iterative work of making sure the structs are kept as small as possible, we have again reduced the size of numerous struct fields and rearranged the order somewhat to save memory.
Next
The next release is planned to ship on October 25, 2022.
In the curl project we do timed releases and we try to do them planned and scheduled long in advance. The dates are planned. The content is not.
I have been the release manager for every single curl release done. 209 releases at current count. Having this role means I make sure things are in decent shape for releases and I do the actual mechanic act of running the release scripts etc on the release days.
Since 2014, we make releases on Wednesdays, every eight weeks. We sometimes adjust the date slightly because of personal events (meaning: if I have a vacation when the release is about to happen, we can move it), and we have done several patch releases within a shorter time when the previous release proved to have a serious enough problem to warrant an out-of-schedule release. Even more specifically, I make the releases available at or around 8 am (central euro time) on the release days.
The main objective is to stick to the 56 day interval.
It probably goes without saying but let me be clear: curl is a software project that quite evidently will never be done or complete. It will keep getting fixes, improvements and features for as long as it lives, and as long as it lives we keep making new releases.
Timed releases
We decided to go with a fixed eight weeks release frequency quite arbitrarily (based on past release history and what felt “right”) but it has over time proved to work well. It is short enough for everyone to never have to wait very long for the next release, and yet it is long enough to give us time for both merging new features and having a period of stabilization.
Timed releases means that we ship releases on the predetermined release date with all the existing features and bugfixes that have landed in the master branch in time. If a change is not done in time and it is not merged before the release, it will simply not be included in the release but will get a new opportunity to get included and shipped in the next release. Forever and ever.
I am a proponent of timed releases compared to feature based ones for projects like curl. For the simplicity of managing the releases, for long term planning, for user communication and more.
The cycle has three windows
Within the eight week release cycle, there are three distinct and different windows or phases.
1 – release margin
The cycle can be said to start at the day of a release. The release is created straight from the master git branch. It gets signed, uploaded, blogged about and the news of it is shouted across the globe.
This day also starts the “release margin window”. In this window we still accept and merge bugfixes into the master branch, but we do not merge changes or new features.
The five day margin this speaks of, is that this window gives us a few calendar days to assess and get a feel for how the previous release is being received. Is there a serious bug reported? Did we somehow royally mess up? If we did an important enough snafu, we may decide to do a follow-up patch release soon and if we do, we are in a better position if we have not merged any changes yet into the release branch.
If we decide on doing a patch release, we skip the feature window and go directly to feature freeze.
2 – feature window
If we survived the margin window without anything alarming happening, we open the feature window. On the Monday following the preceding release.
This is the phase during which we merge changes and new features in addition to the regular normal bugfixes (assuming there are any of course). Things that warrant the minor version number for the next release to get bumped.
The feature window is open 23 days. Ideally, people have already been working and polishing on their pull-requests for a while before this window opens and then the work can get merged fairly quickly – and presumably painless.
What exactly can be considered a change and what is a bugfix can of course become a matter of opinion and discussion, but we tend to take the safer approach when in doubt.
3 – feature freeze
In the 28 days before the pending release we only merge bugfixes into the master branch. All features and changes are queued up and will have to wait until the feature window opens again. (Exceptions might be made for experimental and off-by-default features.)
This phase is of course intended for things to calm down, to smooth out rough edges, to fix any leftover mistakes done in the previous feature merges. With the mindset that the pending next release will be the best, most stable, shiniest, glorious release we ever did. With even fewer bugs than before. The next release is always our best release yet.
The release cycle as an image
(Yes, if you squint and roll your chair away from the screen a bit, it looks like the Chrome logo!)
Release frequency graph
The graph below shows the number of days between all curl releases ever done. You can see the eight week release cycle introduced in 2014 visible in the graph, and you can also easily spot that we have done much quicker releases than eight weeks a number of a times since then – all of them actually a sign of some level of failure since that means we felt urged to do a patch release sooner than we had previously anticipated.
Welcome to take the next step with us in this never-ending stroll.
Release presentation
Numbers
the 209th release 8 changes 47 days (total: 8,865) 123 bug-fixes (total: 7,980) 214 commits (total: 28,787) 0 new public libcurl function (total: 88) 2 new curl_easy_setopt() option (total: 297) 1 new curl command line option (total: 248) 51 contributors, 20 new (total: 2,652) 35 authors, 13 new (total: 1,043) 4 security fixes (total: 125) Bug Bounties total: 34,660 USD
Security
This is another release in which scrutinizing eyes have been poking around and found questionable code paths that could be lead to insecurities. We announce four new security advisories this time – all found and reported by Harry Sintonen. This bumps mr Sintonen’s curl CVE counter up to 17; the number of security problems in curl found and reported by him alone.
A malicious server can serve excessive amounts of Set-Cookie: headers in a HTTP response to curl and curl stores all of them. A sufficiently large amount of (big) cookies make subsequent HTTP requests to this, or other servers to which the cookies match, create requests that become larger than the threshold that curl uses internally to avoid sending crazy large requests (1048576 bytes) and instead returns an error.
curl supports “chained” HTTP compression algorithms, meaning that a server response can be compressed multiple times and potentially with different algorithms. The number of acceptable “links” in this “decompression chain” was unbounded, allowing a malicious server to insert a virtually unlimited number of compression steps.
When curl saves cookies, alt-svc and hsts data to local files, it makes the operation atomic by finalizing the operation with a rename from a temporary name to the final target file name.
In that rename operation, it might accidentally widen the permissions for the target file, leaving the updated file accessible to more users than intended.
When curl does FTP transfers secured by krb5, it handles message verification failures wrongly. This flaw makes it possible for a Man-In-The-Middle attack to go unnoticed and even allows it to inject data to the client.
Changes
We have no less than eight different changes logged this time. Two are command line changes and the rest are library side.
These are two options that have not been used by anyone for an extended period of time, and starting now they have no functionality left. Using them has no effect.
The point here is that you can check if global init is thread-safe in your particular libcurl build.
CURLINFO_CAPATH/CAINFO: get default CA paths
As the default values for these values are typically figured out and set at build time, applications might appreciate being able to figure out what they are set to by default.
CURLOPT_SSH_HOSTKEYFUNCTION
For libssh2 enabled builds, you can now set a callback for hostkey verification.
deprecate RANDOM_FILE and EGDSOCKET
The libcurl version of the change mentioned above for the command line. The CURLOPT_RANDOM_FILE and CURLOPT_EGDSOCKET options no longer do anything. They most probably have not been used by any application for a long time.
unix sockets to socks proxy
You can now tell (lib)curl to connect to a SOCKS proxy using unix domain sockets instead of traditional TCP.
Bugfixes
We merged way over a hundred bugfixes in this release. Below are descriptions of some of the fixes I think are particularly interesting to highlight and know about.
improved cmake support for libpsl and libidn2
more powers to the cmake build
address cookie secure domain overlay
Addressed issues when identically named cookies marked secure are loaded over HTTPS and then again over HTTP and vice versa. Cookies are complicated.
make repository REUSE compliant
Being REUSE compliant makes we now have even better order and control of the copyright and licenses used in the project.
headers API no longer EXPERIMENTAL
The header API is now officially a full member of the family.
reject overly many HTTP/2 push-promise headers
curl would accept an unlimited number of headers in a HTTP/2 push promise request, which would eventually lead to out of memory – starting now it will instead reject and cancel such ridiculous streams earlier.
restore HTTP header folding behavior
curl broke the previous HTTP header behavior in the 7.83.1 release, and it has now been restored again. As a bonus, the headers API supports folded headers as well. Folding headers being the ones that are the rare (and deprecated) continuation headers that start with a whitespace.
skip fake-close when libssh does the right thing
Previously, libssh would, a little over-ambitiously, close our socket for us but that has been fixed and curl is adjusted accordingly.
check %USERPROFILE% for .netrc on Windows
A few other tools apparently look for and use .netrc if found in the %USERPROFILE% directory, so by making curl also check there, we get better cross tool .netrc behavior.
There were lots of big and small changes in the HTTP/3 backend powered by ngtcp2.
provide a fixed fake host name in NTLM
curl no longer tries to provide the actual local host name when doing NTLM authentication to reduce information leakage. Instead, curl now uses the same fixed fake host name that Firefox uses when speaking NTLM: WORKSTATION.
return error from “lethal” poll/select errors
A persistent error in select() or poll() could previously be ignored by libcurl and not result in an error code returned to the user, making it loop more than necessary.
After a redirect or if doing multi-stage authentication, the --path-as-is status would be dropped.
support CURLU_URLENCODE for curl_url_get
This is useful when for example you ask the API to accept spaces in URLs and you want to later extract a valid URL with such an embedded space URL encoded
Welcome to this patch release of curl, shipped only 14 days since the previous version. We decided to cut the release cycle short because of the several security vulnerabilities that were pointed out. See below for details. There are no new features added in this release.
It burns. Mostly in our egos.
Release video
Numbers
the 208th release 0 changes 14 days (total: 8,818) 41 bug-fixes (total: 7,857) 65 commits (total: 28,573) 0 new public libcurl function (total: 88) 0 new curl_easy_setopt() option (total: 295) 0 new curl command line option (total: 247) 20 contributors, 6 new (total: 2,632) 13 authors, 3 new (total: 1,030) 6 security fixes (total: 121) Bug Bounties total: 22,660 USD
Security
Axel Chong reported three issues, Harry Sintonen two and Florian Kohnhäuser one. An avalanche of security reports. Let’s have a look.
curl removes wrong file on error
CVE-2022-27778 reported a way how the brand new command line options remove-on-error and no-clobber when used together could end up having curl removing the wrong file. The file that curl was told not to clobber actually.
cookie for trailing dot TLD
CVE-2022-27779 is the first of two issues this time that identified a problem with how curl handles trailing dots since the 7.82.0 version. This flaw lets a site set a cookie for a TLD with a trailing dot that then might have curl send it back for all sites under that TLD.
percent-encoded path separator in URL host
In CVE-2022-27780 the reporter figured out how to abuse curl URL parser and its recent addition to decode percent-encoded host names.
CERTINFO never-ending busy-loop
CVE-2022-27781 details how a malicious server can trick curl built with NSS to get stuck in a busy-loop when returning a carefully crafted certificate.
TLS and SSH connection too eager reuse
CVE-2022-27782 was reported and identifies a set of TLS and SSH config parameters that curl did not consider when reusing a connection, which could end up in an application getting a reused connection for a transfer that it really did not expected to.
HSTS bypass via trailing dot
CVE-2022-30115 is very similar to the cookie TLD one, CVE-2022-27779. A user can make curl first store HSTS info for a host name without a trailing dot, and then in subsequent requests bypass the HSTS treatment by adding the trailing dot to the host name in the URL.
Bug-fixes
The security fixes above took a lot of my efforts this cycle, but there were a few additional ones I could mention.
In our regular attempts to remove warnings and errors, we fixed this warning that was on the border of a false positive. We want to be able to run with sanitizers warning-free so that every real warning we get can be treated accordingly.
gskit: fixed bogus setsockopt calls
A set of setsockopt() calls in the gskit.c backend was fond to be defective and haven’t worked since their introduction several years ago.
define HAVE_SSL_CTX_SET_EC_CURVES for libressl
Users of the libressl backend can now set curves correctly as well. OpenSSL and BoringSSL users already could.
x509asn1: make do_pubkey handle EC public keys
The libcurl private asn1 parser (used for some TLS backends) did not have support for these before.
As usual, here’s a set of selected favorite bug-fixes of mine from this cycle:
require “see also” for every documented option in curl.1
When the curl command man page is generated at build time, the script now makes sure that there is a “see also” for each option. This will help users find related info. More mandatory information for each option makes us do better documentation that ultimately helps users.
lazy-alloc the table in Curl_hash_add()
The internal hash functions moved the allocation of the actual hash table from the init() function to when the first add() is called to add something to the table. This delay simplified code (when the init function became infallible ) and does even avoid a few allocs in many cases.
enable haproxy support for hyper backend
Plus a range of code and test cases adjusted to make curl built with hyper run better. There are now less than 30 test cases still disabled for hyper. We are closing in!
mbedTLS: add support for CURLOPT_CAINFO_BLOB
Users of this backend can now also use this feature that allows applications to provide a CA cert store in-memory instead of using an external file.
multi: handle errors returned from socket/timer callbacks
It was found out that the two multi interface callbacks didn’t at all treat errors being returned the way they were documented to do. They are now, and the documentation was also expanded to clarify.
nss:set_cipher don’t clobber the cipher list
Applications that uses libcurl built to use NSS found out that if they would select cipher, they would also effectively prevent connections from being reused due to this bug.
openldap: implement STARTTLS
curl can now switch LDAP transfers into LDAPS using the STARTTLS command much like how it already works for the email protocols. This ability is so far limited to LDAP powered by OpenLDAP.
openssl: define HAVE_OPENSSL_VERSION for OpenSSL 1.1.0+
This little mistake made libcurl use the wrong method to extract and show the OpenSSL version at run-time, which most notably would make libcurl say the wrong version for OpenSSL 3.0.1, which would rather show up as the non-existing version 3.0.0a.
sha256/md5: return errors when init fails
A few internal functions would simply ignore errors from these hashing functions instead of properly passing them back to the caller, making them to rather generate the wrong hash instead of properly and correctly returning an error etc.
curl: updated search for a file in the homedir
The curl tool now searches for personal config files in a slightly improved manner, to among other things make it find the same .known_hosts file on Windows as the Microsoft provided ssh client does.
url: check ssl_config when re-use proxy connection
A bug in the logic for checking connections in the connection pool suitable for reuse caused flaws when doing subsequent HTTPS transfers to servers over the same HTTPS proxy.
ngtcp2: verify server certificate
When doing HTTP/3 transfers, libcurl is now doing proper server certificate verification for the QUIC connection – when the ngtcp2 backend is used. The quiche backend is still not doing this, but really should.
urlapi: accept port number zero
Years ago I wrote a blog post about using port zero in URLs to do transfers. Then it turned out port zero did not work like that with curl anymore so work was done and now order is restored again and port number zero is once again fine to use for curl.
urlapi: provide more detailed return codes
There are a whole range of new error codes introduced that help better identify and pinpoint what the problem is when a URL or a part of a URL cannot be parsed or will not be accepted. Instead of the generic “failed to parse URL”, this can now better tell the user what part of the URL that was found out to be bad.
socks5h: use appropriate ATYP for numerical IP address
curl supports using SOCKS5 proxies and asking the proxy to resolve the host name, what we call socks5h. When using this protocol and using a numerical IP address in the URL, curl would use the SOCKS protocol slightly wrong and pass on the wrong “ATYP” parameter which a strict proxy might reject. Fixed now.
Coming up?
The curl factory never stops. There are many pull-requests already filed and in the pipeline of possibly getting merged. There will also, without any doubts, be more ones coming up that none of us have yet thought about or considered. Existing pending topics might include:
the ManageSieve protocol
--no-clobber
CURLMOPT_STREAM_WINDOW_SIZE
Remove Mesalink support
HAproxy protocol v2
WebSockets
Export/import SSL session-IDs
HTTP/3 fixes
more hyper improvements
CURLFOLLOW_NO_CUSTOMMETHOD
Next release
March 2, 2022 is the scheduled date for what will most probably become curl 7.82.0.
This new option provides yet another knob for applications to control and limit connection reuse. Using this option, the application sets an upper limit that specifies that connections may not be older than this when being reused. It can for example be used to make sure connections don’t “get stuck” on one single specific load-balancer for extended periods of time.
This new callback gets called immediately before the request is started (= “Pre req”). Gives the application a heads up and some details about what is just about to start.
libssh2: SHA256 fingerprint support
This is a bump up from the previous MD5 version. Make sure that curl connections to the correct host with a cryptographically strong fingerprint check.
When a URL API function returns an error it does so using a CURLUcode type. Now there’s a function to convert this error code into an error message.
support UNC paths in file: URLs on Windows
The URL parser now understands UNC paths when parsing URLs on Windows.
allow setting of groups/curves with wolfSSL
The wolfSSL backend now allows setting of specific curves for TLS 1.3 connections, which allows users to use post quantum algorithms if wolfSSL is built to support them!
Bug-fixes
This is another release with more than one hundred individual bug-fixes, and here are a selected few I think might be worth highlighting.
more hyper work
I’ve done numerous small improvements in this cycle to take the hyper backend closer to become a solid member of the curl backend family.
print curl --help descriptions aligned right
When listing available options with --help or -h, the list is now showing the descriptions right-aligned which makes the output more easy-to-read in my opinion. Illustration:
Showing off right-aligned –help descriptions
store remote IP address for QUIC connections too
HTTP/3 uses QUIC and with this bug fixed, the %{remote_ip} variable for --write-out works there as well – as you’d expect. This fixes the underlying CURLINFO_PRIMARY_IP option.
reject HTTP response codes < 100
The HTTP response parser no longer accepts response code numbers below 100 as a legitimate protocol. The HTTP protocol has never specified any such code so this should not cause any problems. Lots of other HTTP clients already enforce this requirement too.
do not wait for writable socket if there’s no remote HTTP/2 window
If curl runs out of remote HTTP/2 window for a stream while uploading, ie the other end says it can’t receive any data right now, curl would still wait for the socket to be writable which would cause really bad busy-loops.
get libssh2 version at runtime
curl now asks libssh2 for its version string in runtime instead of showing the version that was used back when the curl binary was built, as it might very well be upgraded dynamically after the build!
require all man pages to use the same section headers in the same order
We tighten the bolts even more to make the libcurl documentation consistent. Now all libcurl man pages have to feature the same set of headers in the same order to not cause test failure. This includes a required example section. We also added an extra check to detect a common backslash-wrong-formatting mistake that we’ve previously done several times in man page examples.
NTLM: use DES_set_key_unchecked with OpenSSL
Turns out that the code that is implemented to use OpenSSL for doing NTLM authentication was using a function call that returns error if a “bad” key is used. NTLM v1 being a very weak algorithm in general makes it very easy to end up calling the function with such a weak key and then the NTLM authentication failed…
openssl: if verifypeer is not requested, skip the CA loading
If peer verification is disabled for a transfer when curl is built to use OpenSSL or one of its forks, curl will now completely skip the loading of the CA cert bundle. It was basically only used for being able to show in the verbose output if there was a match or not – that was then ignored anyway – and by skipping the load step the TLS handshake will use less memory and CPU resources.
urlapi: URL decode percent-encoded host names
The URL parser did not accept percent encoded host names. Now it does. Note however that libcurl will not by default percent-encode the host name when extracting a URL for the purpose of keeping IDN names working. It’s a little complicated.
ngtcp2: use QUIC TLS RFC9001
We switch over to use the “real” QUIC identifier when setting up connections instead of using the identifier made for a previous draft version of the protocol. curl still asks h3-29 for HTTP/3 since that RFC has still not shipped, but includes h3 as well – since it turns out some servers assume plain h3 when the final QUIC v1 version is used for transport.
a failed etag save now only fails that transfer
Specifying a file name to save etags in will from now on only fail those transfers using that file. If you specify more transfers that use another file or not use etags at all, those transfers can still get done.
Added test case for checksrc!
The custom tool we use for checking code style compliance, checksrc, has become quite advanced and now checks for a lot of source code details, and when we once again improved it in this release cycle we also added a test case for the tool itself to make sure it maintains its functionality even when we keep improving it going forward!
Next
The next release is planned for January 5, 2022. We have several changes queued up as pull requests already so I’d say it is likely that it then will become version 7.81.0.
Support?
I offer commercial support and curl related contracting for you and your company!
