cURL and libcurl

Presentation: curl from start to end

Leave a comment

On Tuesday January 21st 2025, at 16:00 CET (15:00 UTC) I will do a presentation titled as per above. I have not done this one before.

The talk will be a detailed explainer and step-by-step going through exactly what happens when a curl command line is typed into a shell and the return key is pressed. As the image below illustrates.

The talk will be live-streamed on twitch, and will be recorded and hosted on YouTube after the fact. For free of course, no signup. Just show up.

Those who join the live-stream chat can of course also ask questions live in the following Q&A.

cURL and libcurl, Security

Secure Transport support in curl is on its way out

4 Comments

In May 2024 we finally decided that maybe the time has come for curl to drop support of older TLS libraries. Libraries that because they don’t support the modern TLS version (1.3) for many users are maybe not suitable to build upon for the future. We gave the world 12 months to adapt or to object. More than half of that time has passed.

This means that after May 2025, we intend to drop support for Secure Transport and BearSSL unless something changes drastically that makes us reconsider.

This blog post is an attempt to make everyone a little more aware and make sure that those who need to, prepare appropriately.

Secure Transport

Secure Transport is a quite a horrible name, but it is still the name of a TLS library written by Apple, shipped as a component for macOS and all the different flavors of iOS. It has been supported by curl since 2012 but as of a few years back it is considered deprecated and “legacy” by Apple themselves. Secure Transport only supports TLS up to but not later than 1.2.

Once upon the time Apple shipped curl built against Secure Transport with macOS but they switched over to LibreSSL several years ago.

I hear two primary reasons mentioned why people still like using libcurl/Secure Transport on iOS:

  1. It saves them from having to use and bundle a separate third party library that also adds to the footprint.
  2. It gives them easy and convenient use of the iOS certificate store instead of having to manage a separate one..

Network Framework

Continuing on their weird naming trajectory, the thing that Apple recommends the world to use instead of Secure Transport is called Network Framework.

Due to completely new paradigms and a (to me at least) funny way to design their API, it is far from straight-forward to write a backend for curl that uses the Network Framework for TLS. We have not even seen anyone try. Apple themselves certainly seem to be fine to simply not use their own TLS for their curl builds.

I am not sure it is even sensible to try to use the Network Framework in curl.

Options

Without Secure Transport and no prospect of seeing Network Framework support, users of libcurl on macOS and iOS flavors need to decide on what to do next.

I can imagine that there are a few different alternatives to select from.

  1. Stick to an old libcurl. At first an easy and convenient choice, but it will soon turn out to be a narrow path with potential security implications going forward.
  2. Maintain a custom patch. The TLS backends are fairly independent so this is probably not an impossible task, but still quite a lot of work that also takes a certain amount of skill.
  3. Switch off from libcurl. Assuming you find an alternative that offers similar features, stability, portability, speed and that supports the native cert storage fine. Could mean quite some work.
  4. Use libcurl with another TLS library. This is then by itself two sub-categories. A) The easiest route is to accept that you need to maintain a separate CA store and then you can do this immediately and you can use a TLS library that supports the latest standards and that is well supported. B) Use a TLS library that supports use of the native iOS cert store. I believe maybe right now wolfSSL is the only one that does this out of the box, but there is also the option to pay someone or write the code to add such features to another curl TLS backend.
  5. Some other approach

Post removal

After this removed support of two libraries from curl, there is still support for ten different TLS libraries. There should be an adequate choice for everyone – and there is nothing stopping us from adding support for future newcomers on the scene.

Protests are listened to

Part of the deprecation process in curl is that we listen to what possible objections people might have in the time leading up to the actual future date when the code is cut out. Given a proper motivation a deprecation decision can be canceled or at least postponed.

cURL and libcurl

curl with partial files

2 Comments

Back in September 2023, we extended the curl command line tool with a new fairly advanced and flexible variable system. Using this, users can use files, environment variables and more in a powerful way when building curl command lines in ways not previously possible – with almost all existing command line options.

curl command lines were already quite capable before this, but these new variables certainly took it up several additional notches.

Come February 2025

In the pending curl 8.12.0 release, we extend this variable support a little further. Starting now, you can assign a variable to hold the contents of a partial file. Get a byte range from a given file into a variable and use that variable in the command line, instead of using the entire file.

You can get the first few bytes and use as a username, you can get a hundred bytes in the middle of a file and POST that or do countless other things.

Byte range

You ask curl to read a byte range from a file instead of the whole one by appending [n-M] to the variable name, when you assign a variable. Where N and M are the first and the last byte offsets into the file, 0 being the first byte. If you omit the second number, it means until the end of file.

For example, get the first 32 bytes from a file named secret and set as password for daniel:

curl --variable "pwd[0-31]@secret" \
     --expand-user daniel:{{pwd}} \
     https://example.com/

Skip the first thousand bytes from a file named localfile and send the rest of it in a POST:

curl --variable "upload[1000-]@localfile" \
     --expand-post '{{upload}}' \
     https://example.com/

With functions

You can of course also combine the byte offsets with the standard expand functions. For example, get the first hundred bytes from the file called random and send them base64 encoded in a POST:

curl --variable "binary[0-99]@random" \
     --expand-post '{{binary:b64}}' \
     https://example.com/

I hope you will like it.

Update

After his post was first published, we discussed the exact syntax for this feature and decided to tweak it a little to make it less likely that old curl versions could be tricked when trying a new command line options.

This version is now showing the updated syntax.

cURL and libcurl

dropping hyper

5 Comments

The ride is coming to an end. The experiment is done. We tried, but we admit defeat.

Four years ago we started adding support for an alternative HTTP backend in curl. It would use a library written in rust, called hyper. The idea was to introduce an alternative implementation of HTTP internals that you could make curl/libcurl use instead of the native implementation.

This new backend that used a library written in rust would enable users to run a product where a larger piece of the total code than otherwise would be written in a memory-safe language: rust. Memory-safety being all the rage these days.

The initial work was generously sponsored by ISRG, the organization behind such excellent efforts such as Let’s Encrypt, which believes strongly in this concept. I cooperated intensely with Sean McArthur, the lead developer of hyper. We made it work.

We have shipped hyper support in curl labeled EXPERIMENTAL for several years by now, hoping to attract attention and trigger the experimental spirit in users out there. Seeing so many people seem to want more memory-safety, surely the users would come?

95% of the work is the easy part

I mean that we took it perhaps 95% of the way and almost the entire test suite ran identically independently of which backend we built curl to use. The final few percent would however turn out to be friction enough to now eventually make us admit defeat, give up and instead yank it all out again.

There simply were no users asking for it and there were almost no developers interested or knowledgeable enough to work on it. libcurl is written in C, hyper is written in rust and there is a C binding glue layer in between. It takes someone who is interested and good at both languages to dig in, understand the architectures, the challenges and the protocols to drive this all the way through.

But with no user demand, why do it?

It seems quite clear that rust users use hyper but few of them want to work on making it work for a C project like curl, and among existing curl users there is virtually no interest in hyper. The overlap in the Venn diagram of the two universes is not big enough.

With no expectation of seeing this work completed in the short to medium length term, the cost of keeping the hyper code is simply deemed too high. We gain code agility and reduce complexity by trimming this off.

We improved

While the experiment itself is deemed a failure, I think we learned from it and improved curl in the process. We had to rethink and reassess several implementation details when we aligned HTTP behavior with hyper. libcurl parses and handles HTTP stricter now. Better.

I also believe that hyper benefited from this journey and gained experiences and input from us that led to improvements in their end and in their HTTP library. Which then by extension have benefited the hyper users.

When we started this, even rust itself was not ready and over this time rust has improved and it is today a better language and it is better prepared to offer something like this. For us or for other projects.

Backends

With this amputation we are back to no separate HTTP/1 backends. We only provide the single native implementation.

I am not against revisiting the topic and the idea of providing alternative backends for HTTP/1 in the future, but I think we should proceed a little different next time. We also have a better internal architecture now to build on than what we had in 2020 when this attempt started.

Less rust (for now?)

Before this step, we supported three different backends backed up by libraries written in rust. Now we are down to two: rustls (for TLS) and quiche (for QUIC and HTTP/3). Both of them are still marked experimental.

These two backends use better internal APIs in curl and are hooked into libcurl in a cleaner way that makes them easier to support and less of burden to maintain over time.

Of course nothing prevents us from adding support for more and other rust libraries in the future. libcurl is a protocol engine using a plethora of different backends for many different protocols and services hooked into its core. Virtually all of those backends could be provided by a rust library.

Thanks

A big thank you go to Sean and all others who helped us take it as far as we did. You are great. Nothing of this should put any shade on you.

When

The hyper backend code has been removed in git as of December 21. There will be no traces left of it in the curl 8.12.0 release coming in February 2025.

cURL and libcurl, Security

A twenty-five years old curl bug

2 Comments

I have talked about old curl bugs before, but now we have a new curl record.

When we announced the security flaw CVE-2024-11053 on December 11, 2024 together with the release of curl 8.11.1 we fixed a security bug that was introduced in a curl release 9039 days ago. That is close to twenty-five years.

The previous record holder was CVE-2022-35252 at 8729 days.

Now at 161 reported CVEs, the median time a security problem has existed in curl until fixed is 2583 days, a little over seven years.

Age

We know the age of every single curl security problem because every time we have a confirmed one, I spend a significant time and effort digging through the source code history to figure out in which exact commit the problem was introduced.

(This is also how we know that almost every CVE we have ever announced was introduced by my mistakes.)

What’s Wrong?

I don’t think anyone is doing anything wrong here. I think it illustrates the difficulty and challenges involved. There are a lot of people looking at curl code all the time. We run tests and analyzers on the code, all the time. In fact, in November 2024 alone, we had CI jobs running on GitHub alone at 9.17 CPU days per day. Meaning that on average more than nine machines were running curl tests and builds to help us verify that it works as intended.

Apart from that, we of course have all the human individual testers, security researchers and the Google OSS-Fuzz project that is fuzzing curl non-stop and has been doing so for the last 6-7 years.

Security is hard. I mean really really hard.

I have no immediate ideas how to find the next such bug other than the plain old: add more test cases for scenarios and setups not previously tested. That is hard, difficult and quite frankly quite boring work that nobody in particular wants to do nor fund someone else to do.

Enough eyeballs

I think we all agree by now that not all bugs are shallow. Or perhaps we can’t ever truly get enough eyeballs. Or maybe the saying works, just that it needs an addendum

Given enough eyeballs and time, all bugs are shallow

Learn from each mistake

It is often said, and it is true, that you learn from mistakes. The question is only what exactly to learn from each and every reported security vulnerability. Each new one always feels like a unique stupid mistake that was a one-off that surely will not happen again because that situation is now gone and we have no other like that.

Not a C mistake

Let me also touch this subject while talking security problems. This bug, the oldest so far in curl history, was a plain logic error and would not have been avoided had we used another language than C.

Otherwise, about 40% of all security problems in curl can be blamed on us using C instead of a memory-safe language. 50% of the high/critical severity ones.

Almost all of those C mistakes were done before there even existed a viable alternative language – if that even exists now.

Graphs

I decided to not sprinkle graph images in the post this time. You can find data and graphs for all my claims in here in the curl dashboard.

Sad update

After intensive bisecting, it turns out this bug was incorrectly believed to have been introduced in a certain commit, while in fact it was introduced much later. As of January 7th 2025, we have updated the metadata for this CVE and now it is no longer the oldest bug fixed in curl…

cURL and libcurl

curl 8.11.1

Welcome to another curl release. This time we do a bugfix only release, five weeks since the previous version shipped.

Release Presentation

Numbers

the 263rd release
0 changes
35 days (total: 9,763)
79 bugfixes (total: 11,173)
115 commits (total: 33,811)
0 new public libcurl function (total: 94)
0 new curl_easy_setopt() option (total: 306)
0 new curl command line option (total: 266)
51 contributors, 32 new (total: 3,299)
22 authors, 10 new (total: 1,323)
1 security fixes (total: 161)

Security

CVE-2024-11053: netrc and redirect credential leak. (Severity: Low) When asked to both use a .netrc file for credentials and to follow HTTP redirects, curl could leak the password used for the first host to the followed-to host under certain circumstances.

Bugfixes

As usual, here follows some bugfixes I figure could be worth highlighting. See the changelog on the curl site for the full list of changes.

curl

  • –continue-at is mutually exclusive with –no-clobber
  • –continue-at is mutually exclusive with –range
  • –continue-at is mutually exclusive with –remove-on-error
  • use real time in trace timestamps

scripts

  • dmaketgz: use –no-cache when building docker image

libcurl

  • duphandle: also init netrc
  • hostip: don’t use the resolver for FQDN localhost
  • mime: fix reader stall on small read lengths
  • mprintf: fix integer overflow checks
  • multi: fix callback for CURLMOPT_TIMERFUNCTION not being called again
  • netrc: address several netrc parser flaws
  • netrc: support large file, longer lines, longer tokens
  • socket: handle binding to “host!”

http related

  • http_negotiate: allow for a one byte larger channel binding buffer
  • digest: produce a shorter cnonce in Digest headers
  • cookie: treat cookie name case sensitively
  • nghttp2: use custom memory functions

protocols

  • libssh: use libssh sftp_aio to upload file
  • libssh: when using IPv6 numerical address, add brackets
  • OpenSSL: improved error message on expired certificate
  • rtsp: check EOS in the RTSP receive and return an error code
  • schannel: remove TLS 1.3 ciphersuite-list support
  • fixes for wolfSSL OPENSSL_COEXIST

cURL and libcurl

No need to email me about Cisco AnyConnect

3 Comments

My name and email address can be found in the VPN client application made by Cisco called AnyConnect.

They are present there as part of the curl license, because this product – like thousands of others – uses libcurl. My name appears in many products.

Apparently, people often have problems finding an appropriate address to contact when they have issues with this app. This leads a disproportionate amount of them to send emails to me asking for solutions and fixes to their situations.

So far over the years, close to one hundred different persons have emailed me about problems with Cisco’s AnyConnect. I have not been able to help a single one of them because I know nothing about this application.

The reason my email address is shown there is because I am the lead developer of curl, which is but a small component in this application. I am not associated with Cisco nor this product.

This is the support email address you are looking for:

ac-mobile-feedback@cisco.com

See also: other funny emails I got and curl credit screenshots.

Network, Web

Workshop season six, episode three

One positive thing among many others at this version of the HTTP Workshop (day one, day two) is the fact that there have been several new faces showing up here. People who have not previously attended any HTTP Workshops. Getting fresh blood into the mix is great. A chance to maybe lower the average age of the attendees also feels welcome.

This half day was the final session for this time. Three topics were dealt with.

Do you speak HTTP? Getting your HTTP implementation to do right according to the specification can be a challenge. There is a whole range of existing tests for various areas of HTTP but there might still be a place to add HTTP semantic tests in particular for servers. Discussions brought about reflections around testing, doing tests, test formats, other tests, test infrastructure and more. I think the general sense was that yes it would be great. At least if someone else makes it happen…

Every HTTP stack is an intermediary – HTTP semantics is the (requirement for low-level) API. Yes. Lots of nodding around the huge table.

Workshop feedback and thoughts. What is a good cadence for future events, how long should the events be etc. This is probably the maximum amount of attendees we can handle using the same setup. This event was clearly better than several of the past ones in terms of diversity, but I will second our “workshop maestro” in that it could improve further still. We also discussed whether do-arranging together with IETF is good or bad, should it then be before or after IETF?

I think the consensus said that making it biannual event is good. The reasoning for keeping the event in Europe has been because a larger share of the European attendees come from smaller companies compared to the non-Europeans which to a larger degree come from larger companies that might have it easier to pay for longer trips.

My personal take

The HTTP Workshop is a one-of-a-kind event. At these events everything is about and around HTTP with an information density level that is super high. We get to learn how things actually work for people or that do not work. And that we are not alone in whatever struggles or HTTP challenges we have.

Networking with other doers here and absorbing every protocol detail being expressed, gives food for thoughts and lessons to take advantage from in years to come when we for sure are going to take HTTP transfers further. This is in many ways a kind of brain fertilizer event.

Did I mention I enjoyed it? I will certainly try to attend the next one.

Network, Web

The 2024 Workshop, day two

The fun continues. See day one.

In an office building close to the Waterloo station in London, around 40 persons again sat down at this giant table forming a big square that made it possible for us all to see each other. One by one there were brief presentations done with follow-up discussions. The discussions often reiterated old truths, brought up related topics and sometimes went deep down into the weeds about teeny weeny details of the involved protocol specs. The way we love it.

The people around the table represent Ericsson, Google, Microsoft, Apple, Meta, Akamai, Cloudflare, Fastly, Mozilla, Varnish. Caddy, Nginx, Haproxy, Tomcat, Adobe and curl and probably a few more I forget now. One could say with some level of certainty that a large portion of every day HTTP traffic in the world is managed by things managed by people present here.

This morning we all actually understood that the south entrance is actually the east one (yeah, that’s a so called internal joke) and most of us were sitting down, eager and prepared when the day started at 9:30 am.

Capsule. The capsule protocol (RFC 9297) is a way to, put simply, send UDP packets/datagrams over old style HTTP/1 or HTTP/2 proxies.

Cookies. With the 6265bis effort well on its way to ship as an updated RFC, there is an effort and intent to take yet another stab at improving and refreshing the cookie spec situation. In particular to better split off browser management and API related stuff from the more network-oriented over-the-wire details. You know yours truly never ceases an opportunity to voice his opinion on cookies… I approve of this attempt as well, as I think increasing clarity and improving the specification situation can’t but to help improve things.

Declarative web push. There’s an ongoing effort to improve web push – not to be confused with server push, so that it can be done easier and without needing JavaScript to manage it in the client side.

Reverse HTTP. There are origins who want to contact their CDNs without having to listening on any ports/sockets and still be able to provide content. That’s one of the use cases for Reverse HTTP and we got to learn details from internet drafts done on the topics for the last fifteen years and why it might still be a worthwhile effort and why the use cases still exist. But is there an enough demand to put it into HTTP?

Server Stack Detection. A discussion around how someone can detect the origin of the server stack of any given HTTP server implementation. Should there be a better way? What is the downside of introducing what would basically be the server version of the user-agent header field? Lots of productive discussions on how to avoid recreating problems of the past but in a reversed way.

MoQ: What is it and why is it not just HTTP/3? Was an educational session about the ongoing work done in this working group that is wrongly named and would appreciate more input from the general protocol community.

New HTTP stack. A description of the journey of a full HTTP stack rewrite: how components can be chained together and in which order and a follow-up discussion about if this should be included in documentation and if so in which way etc. Lessons included that the spec is one thing, the Internet is another. Maybe not an entirely new revelation.

Multiplexing in the year 2024. There are details in HTTP/2 multiplexing that does not really work, there are assumptions that are now hard to change. To introduce new protocols and features in the modern HTTP stack, things need to be done for both HTTP/2 and HTTP/3 that are similar but still different and it forces additional work and pain.

What if we create a way to do multiplexing over TCP, so called “over streams”, so that the HTTP/3 fallback over TCP could still be done using HTTP/3 framing. This would allow future new protocols to remain HTTP/3-only and just make the transport be either QUIC+UDP or Streams-over TCP+TLS. This triggered a lot of discussions, mostly positive and forward-looking but also a lot of concerns raised about additional work and yet another protocols component to write and implement that then needs to be supported until the end of times because things never truly go away completely.

I think this sounds like a fun challenge! Count me in.

End of day two. I need a beer or two to digest this.

curl, open source and networking