In my view, wget is not a curl competitor. It is a companion tool that has a feature overlap with curl.
Use the tool that does the job.
Getting the job done is the key. If that means using wget instead of curl, then that is good and I don’t mind. Why would I?
To illustrate the technical differences (and some similarities) between curl and wget, I made the following Venn diagram. Click the image to get the full resolution version.
The curl-wget Venn diagram
I have contributed code to wget. Several wget maintainers have contributed to curl. We are all friends.
If you think there is a problem or omission in the diagram, let me know and I can do updates.
This is a story consisting of several little building blocks and they occurred spread out in time and in different places. It is a story that shows with clarity how our current system with CVE Ids and lots of power given to NVD is a completely broken system.
CVE-2020-19909
On August 25 2023, we got an email to the curl-library mailing list from Samuel Henrique that informed us that “someone” had recently created a CVE, a security vulnerability identification number and report really, for a curl problem.
I wanted to let you know that there's a recent curl CVE published and it doesn't look like it was acknowledged by the curl authors since it's not mentioned in the curl website: CVE-2020-19909
We can’t tell who filed it. We just know that it is now there.
We own our curl issues
In the curl project we work hard and fierce on security and we always work with security researchers who report problems. We file our own CVEs, we document them and we make sure to tell the world about them. We list over 140 of them with every imaginable detail about them provided. We aim at providing gold-level documentation for everything and that includes our past security vulnerabilities.
That someone else suddenly has submitted a CVE for curl is a surprise. We have not been told about this and we would really have liked to. Now there is a new CVE out there reporting a curl issue and we have no details to say about it on the website. Not good.
I bet curl users soon would like to know the details about this.
Wait 2020?
The new CVE has an ID containing 2020 and that is weird. When you register a CVE you typically get it with the year you request it. Unless you get an ID for an old problem of the past. Is that what they did?
Sources seem to indicate that this was published just days ago.
Integer overflow vulnerability in tool_operate.c in curl 7.65.2 via crafted value as the retry delay.
And then the craziest statement of the year. They grade it a 9.8 CRITICAL issue. With 10 as a maximum, this is close to the worst case possible, right?
The code
Let’s pause NVD in their panic state for a moment because I immediately recognized this description. Brief as it is.
I spend a lot of time in the curl security team receiving reports, reviewing reports, reviewing source code, assessing claims and figuring out curl security issues. I had seen this claim before!
On July 27, 2019, a Jason Lee file an issue on hackerone, where he reported that there was an integer overflow problem in curl’s --retry-delay command line option. The option accepts number of seconds and then internally converts to milliseconds by multiplying the value by 1000. The option sets how long time curl should wait until it makes a retry if the previous transfer failed with a transient error.
This means that on a 64 bit machine, if you write
curl --retry-delay 18446744073709552 ...
The number will overflow the math and instead of waiting until the end of the universe, it might retry again within the next few seconds. The above example apparently made it 384 seconds instead. On Windows, which uses 32 bit longs, you can get the problem already by asking for more than two million seconds (roughly 25 days).
A bug, sure. Security problem? No. I told Jason that in 2019 and then we closed the security report. I then filed a pull-request and fixed the bug. Credits to Jason for the report. We moved on. The fix was shipped in curl 7.66.0, released in September 2019.
Grading issues
In previous desperate attempts from me to reason with NVD and stop their scaremongering and their grossly inflating the severity level of issues, they have insisted that they take in all publicly available data about the problem and make an assessment.
It was obvious already before that NVD really does not try very hard to actually understand or figure out the problem they grade. In this case it is quite impossible for me to understand how they could come up with this severity level. It’s like they saw “integer overflow” and figure that wow, yeah that is the most horrible flaw we can imagine, but clearly nobody at NVD engaged their brains nor looked at the “vulnerable” code or the patch that fixed the bug. Anyone that looks can see that this is not a security problem.
The issue listed by NVD even links to my pull request I mention above. There is no doubt that it is the exact same bug they refer to.
Spreading like a virus
NVD hosts a CVE database and there is an entire world and eco system now that pulls the records from them.
NVD now has this CVE-2020-19909 entry in there, rated 9.8 CRITICAL and now this disinformation spreads across the world. Now when we search for this CVE number we find numerous sites that repeats the same data. “This is a 9.8 CRITICAL problem in curl” – when it is not.
I will object
I learned about this slap in my face just a few hours ago (and I write this past Friday midnight), but I intend to do what I can to reject this CVE.
I have the birthday of curl remembered and I often repeat that it was started on March 20 1998. But that’s just the first time we shipped a version of the tool using the name curl. The tool, the code and the idea started before.
httpget
Back in November 1996 I found the tool httpget when I wanted to set up a cronjob to download currency rates automatically. (I can’t remember how I found it, I presume I used Altavista or something, as this was years before Google…)
I contributed back patches to that tool in late 1996 and quickly became its new maintainer when the original author Rafael Sagula handed over the keys to the kingdom to me. httpget was then a single 300 line C file. With admittedly rather crude code.
We did several httpget releases through 1997 until we after httpget 1.5 added support for FTP. The name of the tool then simply was not suitable anymore so we renamed it. To urlget.
urlget
We kept the version numbering and urlget 2.0 shipped in August 1997. We kept doing more releases and adding more features. With the combination of new upload support (no longer just “get”) and the realization that there were other tools out there already named urlget, I decided to rename the project again. So we bumped the version number and shipped curl 4.0 on March 20 1998.
early changes
The changes we did in the project before it was named curl have been a little lost, primarily because we did not use proper version control back then and not all tarballs have survived the many years since. For a long time, the changelog on the website only showed the changes done since 6.0 (Released in September 1999) but in December 2019 I did the job and added all the missing entries back in time to curl 4.0.
Based on early tarballs and their contents I went back and figured out the missing releases. All the httpget and urlget releases we did before the rename to curl. It turns out there were 29 of them – assuming my counting is correct.
The graph below shows releases per year and is now based on all these releases. Turns out that we have done exactly 250 releases to date.
Since this new extended changelog then also presumably contains all changes and bugfixes ever done, the graph over bugfixes and bugfix speed also grew more complete.
By reading the old changelog entries it is clear that we were not documenting all the bugfixes and changes as thoroughly and as detailed back then as we do these days in the project, so that is one obvious piece to the explanation for the growth in the above plot.
Rebased count
In release presentations and elsewhere I sometimes mention which release “number” we do. This number is now bumped significantly. I don’t think it matters much. It is just a number and this is just a correction of it. For the keen observer, it will of course cause a discrepancy that might look odd, but now I have explained it!
The next curl release becomes release 251. Later this year – the year we celebrate curl turning 25 – we can count 27 years since httpget 0.1 shipped.
Rafael
The original author of httpget kept up with the development and stuck around in the project the first year or two and then slowly dwindled away. His work was rewritten many times since and already by the time curl shipped, there were no traces left of the original httpget code. Everything had already been rewritten.
Unfortunately, Rafael died many years ago so we cannot get his words or memories from these days. But we can remember and honor his work and legacy.
For the first time ever, I am going to present a single, very long, video class with the title shown above.
This session will be streamed and recorded live on August 31, starting at 16:00 UTC (18:00 CEST, 09:00 PDT) and is expected to take about two and a half hours. Due to many uncertainties, the stream might of course be longer even if the end recording might get edited down a little.
The agenda for this monster session might still be tweaked a little before it happens but the work in progress version is shown below. It should cover most of what curl can and knows in 2023.
There is no need to sign up. It is entirely free of charge. All you need to do to enjoy it live is to go to the above link at the correct time on the right day. You can participate and ask questions live in the designated chat while the stream is live.
The curl option –write-out is one of my personal favorites and offers users an excitingly powerful way to output information from a transfer. Over time, it has been extended to provide more and more features.
It was for example not that long ago we added the ability to output the content of specific headers with %headers{} to this option.
Now (shipping in the coming curl 8.3.0, merged in commit 1032f56efa) we take the next step and add yet another little nifty function to this option that makes it even more powerful and allows you to use it for more purposes better going forward.
It can now save the selected info to a specific file instead of just outputting to stdout or stderr. Or to multiple files. Or append to files. With %output{}.
Examples
Write the used IP address of the remote host to a file named “remote.txt”:
If you are anything like me, you appreciate solving your every day simple tasks directly from the command line. Creating crafty single shot command lines or a small shell script to solve that special task you figured out you needed and makes your day go a little smoother. A fellow command line cowboy.
Video presentation
Background
To make life easier for curl users, the tool supports “config files“. They are a set of command line options written in a text file that you can point the curl tool to use. By default curl will check for and use such a config file named .curlrc if placed in your home directory.
One day not too long ago, a user over in the curl IRC channel asked me if it was possible to use environment variables in such config files to avoid having to actually store secrets directly in the file.
Variables
This new variable system that we introduce in curl 8.3.0 (commit 2e160c9c65) makes it possible to use environment variable in config files. But it does not stop there. It allows lots of other fun things.
First off, you can set named variables on the command line. Like :
curl --variable name=content
or in the config file:
variable=name=content
A variable name must only consist of a-z, A-Z, 0-9 or underscore (up to 128 characters). If you set the same name twice, the second set will overwrite the first.
There can be an unlimited amount of variables. A variable can hold up to 10M of content. Variables are set in a left to right order as curl parses the command line or config file.
Assign
You can assign a variable a plain fixed string as shown above. Optionally, you can tell curl to populate it with the contents of a file:
curl --variable name@filename
or straight from stdin:
curl --variable name@-
Environment variables
The variables mentioned above are only present in the curl command line. You can also opt to “import” an environment variable into this context. To import $HOME:
curl --variable %HOME
In this case above, curl will exit if there is no environment variable by that name. Optionally, you can set a default value for the case where the variable does not exist:
curl --variable %HOME=/home/nouser
Expand variables
All variables that are set or “imported” as described above can be used in subsequent command line option arguments – or in config files.
Variables must be explicitly asked for, to make sure they do not cause problems for older command lines or for users when they are not desired. To accomplish this, we introduce the --expand- option prefix.
Only when you use the --expand- prefix in front of an option will the argument get variables expanded.
You reference (expand) a variable like {{name}}. That means two open braces, the variable name and then two closing braces. This sequence will then be replaced by the contents of the variable and a non-existing variable will expand as blank/nothing.
Trying to show a variable with a null byte causes error
Examples
Use the variable named ‘content’ in the argument to --data, telling curl what to send in a HTTP POST:
--expand-data “{{content}}”
Create the URL to operate on by inserting the variables ‘host’ and ‘user’.
--expand-url “https://{{host}}/user/{{user}}”
Expand variables
--variable itself can be expanded when you want to create a new variable that uses content from one or more other variables. Like:
the 221st release 0 changes 7 days (total: 9,259) 27 bug-fixes (total: 9,194) 37 commits (total: 30,646) 0 new public libcurl function (total: 91) 0 new curl_easy_setopt() option (total: 303) 0 new curl command line option (total: 255) 20 contributors, 7 new (total: 2,927) 13 authors, 3 new (total: 1,173) 0 security fixes (total: 146)
Bugfixes
Here are some the most important fixes in this release
configure: check for nghttp2_session_get_stream_local_window_size
We use this function now, introduced in nghttp2 1.15.0, released in September 2016.
return IPv6 first for localhost resolves
Resolving “localhost” did not return the (fixed) addresses in the correct order. It now returns IPv6 as the first.
http2 regression on upload EOF handling
When we added an optimization in the previous release we missed a code path that sometimes lead to “hanging” uploads over HTTP/2.
os400: correct EXPECTED_STRING_LASTZEROTERMINATED
curl builds fine for “IBM i” again.
quiche: fix lookup of transfer at multi
Doing multiplexed HTTP/3 over multiple connections with quiche works much better.
mkhelp: strip off escape sequences
The command sequence that generates the man page display code for the --manual option did at some point regress to include escape sequences. Now those sequences are properly filtered out.
fix build when SIZEOF_CURL_OFF_T > SIZEOF_OFF_T
A build problem was fixed for these rare platforms.
do not clear the credentials on redirect to absolute URL
Yet another regression that we allowed because we apparently did not have a test for this! Now we have a test and redirects to the same origin when using -u for credentials now send the credentials even in the redirected request.
Welcome to another curl release. You know how this dance goes…
Numbers
the 220th release 5 changes 50 days (total: 9,252) 122 bug-fixes (total: 9,167) 177 commits (total: 30,606) 0 new public libcurl function (total: 91) 1 new curl_easy_setopt() option (total: 303) 4 new curl command line option (total: 255) 55 contributors, 34 new (total: 2,922) 35 authors, 20 new (total: 1,170) 1 security fixes (total: 146)
Release presentation
Security
fopen race condition (medium)
CVE-2023-32001. libcurl can be told to save cookies, HSTS and/or alt-svc data to files. When doing this, it called stat() followed by fopen() in a way that made it vulnerable to a TOCTOU (Time of Check, Time of Use) race condition problem.
By exploiting this flaw, an attacker could trick the victim to create or overwrite protected files holding this data in ways it was not intended to.
Changes
curl: add –ca-native and –proxy-ca-native
The command line tool (and library) got new options to ask it to use the systems “native” CA storage. Currently only work on Windows when curl is built to use an OpenSSL fork.
curl: add –trace-ids
This option makes the trace log files include connection and transfer identifiers, which greatly helps debugging transfers doing many (parallel) transfers.
Now users of the tool (and library) pass on specific IP addresses instead of simply using the current one.
add CURLINFO_CONN_ID and CURLINFO_XFER_ID
Two options that allows the application to extract the connection and transfer “Id” of the current transfer, presumably from a debugfunction callback and the likes.
Bugfixes
We have again fixed more than a hundred problems in this release cycle. Here follows a subset that I suspect might be among the most interesting ones.
examples: we’ve added and extended numerous
The ambition is to gradually over time provide examples that show use of all curl_easy_setopt options. We are still way off from that.
http2: numerous smaller and larger fixes
Several regressions and cleanups have been done that improves how HTTP/2 works compared to previous releases.
http2: send HEADER and DATA together
When sending POST requests, libcurl now does a better job in putting the initial outgoing HEADER and DATA frames together, most likely in the same TLS frame.
http3: upload EAGAIN handling
EAGAIN handling for HTTP/3 uploads was fixed, like it was for HTTP/2 as well.
http: fix the outgoing Cookie: header length check
The check that would prevent too long outgoing cookie headers was off by up to a few hundred bytes.
libssh2: use custom memory functions (again)
Bring back use of custom memory functions with libssh2 as otherwise it actually cannot be used with a debug build of curl (or when libssh2 is used as a DLL on windows) due to naive presumptions in the libssh2 API.
A regression caused curl misbehave on end of connection using TLS when built to use Secure Transport.
timeval: use CLOCK_MONOTONIC_RAW if available
For platforms with this clock option, curl now prefers that in an effort to avoid a time that can go backwards.
tool_writeout_json: fix encoding of control characters
The output of control codes in the generated JSON with --json now works better.
urlapi: have *set(PATH) prepend a slash if one is missing
Setting a path using the URL API without a leading slash would previously generate a broken URL when it was extracted. Starting now, libcurl will prepend a slash if there is none.
urlapi: scheme must start with alpha
The URL parser would previously allow a few other characters to start a scheme as well. No more.
tool_parsecfg: accept line lengths up to 10M
The config file parser now allows lines to be up to 10 megabytes. For those odd users generating files with huge data components embedded.
The curl user survey 2023 ran for two full weeks in the end of May, in the same fashion we run it every year.
I have then collected all the answers, ran the numbers and looked at the trends and put all the conclusions and graphs into a single document for everyone to enjoy.
Five quick things
If you are in too much of a hurry to read it all, here are five key facts this year’s survey revealed:
curl users leave Twitter and join Mastodon in notable amounts
Windows 11 is growing quickly as a platform curl users are on
HTTP/3 is used by a quarter of all curl users
WebSocket reached the top-10 of most used protocols before its first birthday
The positive comments in section 21 are heart-warming
The document
The final document is a 3MB 36 page PDF with collected data and conclusions. You find it here:
I will do a dedicated live-streamed video presentation of this curl user survey 2023 analysis and talk about how I see the numbers, the trends and maybe also show some additional data that was left out from the final document.
Previous years
This is the 10th year we run the survey. Here are links to five previous analysis documents:
June 15, 2023 10:00 AM PST (19:00 CEST, 17:00 UTC)
Register Here
This will be was an overview by Daniel Stenberg of the new WebSocket support in curl and in particular how to use this API with libcurl in your applications. It is followed by a live Q&A.
The session will be recorded and made available after the fact.
This webinar is done as a registration-only event on Zoom. If this is problematic for you, there will be a separate second version of this webinar done over Twitch at a later date.
