All posts by Daniel Stenberg

cURL and libcurl

curl performance

4 Comments

tldr: the live version is here: https://curl.se/perf/

How fast is “fast” and is it good enough? Does it run as fast now as it did before or was there a regression? What exactly needs to be fast? How fast is it?

These are questions that many projects and products face, and in curl we are no different. Yet, performance testing and comparisons are hard and full of landmines and time-wasting efforts. For many years we have occasionally brought up the idea of a performance test suite for curl only to shut it down again because the challenges seemed hard and no one was volunteering to do this.

This week it changed.

Let’s do this

I started out trying to find existing projects that host performance results for Open Source projects so that we could just feed our results something else and get great visualizations and data management. I did not find any such.

I then took a look at what existing tools there are for this purpose, and most pointers seemed to suggest that Grafana is a popular and maybe even a good solution to build something like this with. But man, that is a complicated machine and it felt more than a little overwhelming just figure out where or how to start with it. I decided to postpone that take as well.

Let me do this

I decided that instead of trying to do this the best and optimal way – I shouldn’t let perfect be the enemy of good – I would start out by doing the things I know how to do and take it as far as I can one step at a time. Something should be better than nothing.

Performance testing needs decently stable system conditions so that repeated runs produce reasonably similar results, when all involved factors remain identical. This is basically impossibly to accomplish using most cloud infrastructure since those are almost always shared with countless other users. At least on the cheap and free tiers we use.

We probably need our own dedicated hardware for this, but instead of trying to figure out where to get that and arrange for that, I would start by running performance tests on my own local development machine. I am a single user on this and it has many cores and runs decently fast. It should be good enough to get this going on.

I created a first shell script that updates the curl source code from git, it configures and builds it. Then it runs a bunch of tests, outputs a bunch of data and logs all the output in a single log file. I started out with a few simple tests. How fast does curl download a 100 GB file from localhost, how many allocations and how big allocations does it need for a single HTTP download?

My second script parses all the test log files from the previous builds and generates summaries and graphs for them. To make it possible for humans to see how the performance changes between builds and ideally to automatically detect when something changes more than what should be tolerated.

As I am a graph addict already since before, and that journey has taught me a little gnuplot, I decided that even while there probably are much better tools and fancy JavaScript things that could be used, I don’t know them and learning them now is an endeavor I rather avoid. So I stick to what I know and can get results with quickly.

A third script is invoked from a crontab every twenty minutes, sets up some variables and invokes the runner script.

Once the basics started to work, I showed my curl friends the early versions and I soon created a new git repository for the code.

It’s live baby

After a little more poking, I soon made my locally produced performance test summary get packaged and automatically transferred to the curl website after each build, and voila, the first public curl performance tests were live and public.

Getting this data available immediate triggered curl developers. It only took hours until we had the first proposed changes to improve some numbers, and soon we had a few merges to that affect. Visibility really helps!

The performance numbers we get are still varying to a certain degree, partially of course because I still use my machine for my daily development things, but also because most of them do real (localhost) networking and that is by its nature a little… varying.

The system builds and runs a new round every twenty minutes and it does that using the latest commits from git. This setup makes it sometimes run many rounds on the same commit and it might also mean that it sometimes updates and get several new commits at once, so it might skip a round for some commits. I might reconsider this design later, but since it is still a twenty minute time window, the number of commits is still limited.

When the script makes multiple build rounds on the same commit, it accumulates the numbers and for the graph it stores the maximum, the median and the minimum value. It helps show the variation per commit and allows us to cram more into the graphs. It is still early days, but there will be a maximum limit to how many commits that can be displayed in a single graph and still be helpful.

HTTP/2 parallel download speed through 261 builds spread over 31 build rounds

Distribution

To help visualize the distribution and data spread per test, I created a separate illustration that shows the minimum, maximum, P25, P75, medium and mean values in a Box-and-Whisker Plot.

A Box-and-Whisker Plot showing the HTTP/2 parallel download speed data distribution.

Changing conditions

An obvious downside with me just storing build logs in files, is that it will not scale up to the millions. I did however decide that I’m not designing this system for that. At least not now.

Performance tests are highly specific and dependent on the exact machine it runs on, the exact third party libraries and their versions that are used, the other components involved in the tests, such as the servers, and more.

I expect that we will change conditions for the tests every once in a while that makes it hard to compare the current numbers with past numbers. Therefore I think the performance test numbers and values are primarily useful in the short term. To help us spot if we land something that subtly and unintentionally degrades something.

Stakes

To detect extremely slow and long-term changes in performance and even making sure we can better survive wiping all the existing build logs etc, I introduced a concept I call stakes. As in a stake pole. A marker. An arbitrary threshold set manually for each specific test. This value can be used to measure performance test results against, now and later. As conditions change and maybe something makes the results go up or down and we are fine with those changes because they are motivated and expected, then we just change the stakes.

If it works out, I might try to have the system automatically detect and maybe highlight tests that deviate too much from its set stake (at least if done in the wrong direction) . It could be a signal that something bad was merged.

Balances

As with everything in life, things are often balanced out. We already ran into this when we eagerly merged several changes to reduce the number of allocations done for a single HTTP download, only to realize that one of the optimizations we did had the side-effect that it expanded the size one of the main structs maybe a little too much…

Improvements in one area might come at an expense in another. With sufficient tests and data we can improve curl for users, and at the same time make sure that our changes don’t come with a cost we are not prepared to pay. Exactly how to make the balance is of course a question we need to deal with, discuss and decide. Possibly for every change we do!

The tests

As I write this, we have 24 tests and a full test round completes in about six minutes on my machine.

We can of course do multiple builds using different hardware, different operating systems, different build options, different third party libraries and different test servers to check more angles of performance, and I am certainly open for and prepared to do that going forward. I will however first let this single-flavor run for a while so that we get more data, get a change to tweak it and make it as usable as possible for curl developers.

As with everything there is no end to what we can make this do. This is a start. I sure we can take it further as we move along. In particular if people join in and help out. Both with ideas and proposals for visualizations, graphs and new tests to add, but also with actual pull-requests and code.

Build volumes and graphs

Over the last year, we have merged, on average, about 10 commits per day. If we keep this pace up and this performance test setup can show 100 build rounds conveniently into a single graph, that is just ten days of development. Probably not enough.

Once we reach one hundred builds or so in the first graphs I need to consider adding separate long term graphs that use select data-points to display data development over a longer time. Some googling told me the Largest-Triangle-Three-Buckets, or LTTB for short, is a fine algorithm to use for this. I now do a separate “long term” graph that “downsamples” the full range down to something that can be shown in a reasonable way. I suppose we will see properly in the future how this works.

Spotting change

The stake thing I mentioned is one way to help us spot gradual performance changes over time. Another googling told me that there’s a Mann-Kendall Test + Sen’s Slope algorithm to use to identify trends in graphs like this and it can be used to plot a trend. It might work as a helper to better identify… yeah, the data trend for each test.

The HTTP/2 parallel download speed trend at a specific moment

Developing

This setup has only existed for a few days. There is lots to do, lots to learn and much more to experiment with.

Your comments, help and pull-requests will be appreciated!

cURL and libcurl

What the bliss taught us

Leave a comment

At this exact moment curl’s summer of bliss 2026 ends.

We (the maintainers of curl) took the entire month of July off from vulnerability reporting and in this post I will try to explain how this went.

(If you feel like skipping the wordy blab below, the single word answer is: fine)

This was possibly our best project decision in a long while.

Zero vulnerability reports

Already before this, we have been refusing to answer emails about vulnerabilities. Partly because we can’t keep track of them that way but even more so because it makes it much harder to properly disclose and publish the entire report sequence after the fact.

On our Hackerone page we informed visitors that we were on pause and that they could come back in August.

We had I believe one vulnerability report sent to my private email address in this period in spite of that messaging, but for all intents and purposes this worked out exactly as good as we hoped it would. I just ignored that email. That was easy.

Bliss

The effect was almost immediate. Just a few days into the bliss, my fellow curl maintainers all agreed with me that we felt a sense of relief, of vacation and that a load had been taken off our chests. We felt free, unchained, and now suddenly able to do what we wanted.

We could now spend time reviewing some of the queued up pull-requests for features and changes we like. We could suddenly again work on code in areas we had been leaving behind lately as vulnerability reports sucked all the air out the room. We polished details on the website, we found document gaps to tighten. It felt like the good old days again. The fun days. We got reminded why we do Open Source and how fun it is.

We took time off, saw some other corners of the world and enjoyed some time away from the keyboards.

We truly healed and re-energized.

CNA

Before we took off on the bliss, we were informed in clear terms that the CNA rules (we are a CNA) mandate that we must respond within 72 hours for some critical vulnerabilities so we can’t just ignore them. I told them sure we can, but in the worst case case our “root” could do some emergency assignments. I figured the risk was minimal and it turns out I was right, Nothing like that was needed and no CVE assignments were necessary during the bliss.

Customers

I got a curious question or two from existing support customers on how the bliss would affect them, but that was easy: it did not affect them. Now, post-bliss, I think they all can confirm that it really did not.

New customers?

As I promised to keep up the contact with and support for paying customers even during the bliss, you could possibly imagine that this would have been an incentive for worried commercial curl users out there to sign up for support contracts.

This did not happen – at all. By this I think we should conclude that (commercial) curl users were not worried either.

The outside world

Lots of fellow open source maintainers and most people in my surrounding have been super positive and downright supportive of our taking some time off. I can’t recall having receiving a single negative comment about the curl summer of bliss!

Fellow blissers

I was moved to see that several other Open Source projects followed our example and also took some time off in order to recharge and relax. In addition to giving us a little vacation, it helps sending a signal and a reminder that Open Source is to a large extent done voluntarily and even maintainers need a break at times.

Major incidents?

Have we opened ourselves up for dangerous attacks and flaws now? Have the bad guys an edge on all curl users out there now because we lived in bliss for a month? We don’t know yet, but it would surprise me.

Queues

During this slow-down, we slowly got more open issues and pull-requests lingering on GitHub than usual. No surprise there. Once we started to come back to life again, we have since managed to return them back to the normal amounts.

Flood gates

Yes, there is an obvious risk that there are now a whole range of queued up reports that will hit us in a short period time as we open up for vulnerability reports again. Presumably the risk for duplicates among these reports should also be significantly higher than usual. I suppose I need to do an update post in a month or two and let you know what happened.

We always treat vulnerability reports and project security with topmost priority and we will continue to do so. We will simply work with what we have and make sure our users and by extension, the world, are safe.

Since I am a member of a few other (non-curl) security teams that did not have a summer of bliss, I have seen that the flood of vuln reports have not really slowed down so it might depend a lot on the details of each specific project.

Some emails were read

All individual curl maintainers of course handled this gift in their own ways. We did not all just disconnect to sit on a remote beach for the whole time. Some of us did that part of the time, but we mostly enjoyed the lower stress level and the absence of pressure. It was mentally relaxing. So, even if some of us kept up with emails, occasionally responded to issues or even submitted some pull requests of our own, it was still vacation. It was still blissful.

Rebliss?

Will we do another summer/winter of bliss? I think yes. It was simply great, with virtually no downsides for the people involved but instead lots of positiveness. Ideally a reduced workload going further will remove the need for another one, but it is not easy to tell what the future holds.

Just transfers

After all, curl just does transfers. Fast. Reliably. Secure.

cURL and libcurl

HTTP Message Signatures with curl

2 Comments

The recently published RFC 9421 describes how to do HTTP Message Signatures, and starting just now, curl experimentally supports them.

Message Signatures

The specification describes this as a mechanism for creating, encoding, and verifying digital signatures or message authentication codes over components of an HTTP message. It is a way to verify that selected parts of the HTTP request arrives unmodified and exactly the same as when the request was created by the client.

These days, it is very common that there are layers of proxies, load balancers, front-ends, CDNs, web firewalls and what not in between the client and the ultimate application. With HTTP Message Signatures, there can be assurances that the headers are components of the request end are unaltered.

Command line

This functionality comes with four new command line options to allow users to use its full power:

--httpsig-algo allows the user to specify which algorithm to use, with ed25519 being used by default. The only other algorithm supported right now is hmac-sha256.

--httpsig-key specifies the key to use when signing the request.

--httpsig-keyid is the key identifier, a string that is passed on in the headers.

--httpsig-headers details exactly which parts of the request and which headers that should be signed. If not set, it defaults to signing the method, authority, path and query.

With these four new flags added to the list, curl supports 278 different command line options.

libcurl

The corresponding options of course also exist as options for curl_easy_setopt:

Experimental

This feature is marked experimental. This means that it need to be explicitly enabled in the build to appear, and that we strongly discourage use of it in production as we reserve the rights to change it before it gets supported for real. We use the experimental phases as a time for people to test it, to tweak it and to learn what we should fix so that we then can support this to the end of time. We do not guarantee any backward compatibility for experimental features.

Please test this feature and tell us how you experienced it! The more tests and more feedback we get, the faster we can get moved out of the experimental phase to have it present for real for everyone.

Ships

This feature is already merged into git and will be part of the pending curl 8.22.0 release. As experimentally supported.

Credits

This feature was graciously brought to us by Sameeh Jubran.

Top image by Antonios Ntoumas from Pixabay

cURL and libcurl

1,500 curl authors

Leave a comment

It takes a village to make curl. A rather big village.

I have not been a solo maintainer of curl for a long time and I don’t even do half of the commits anymore

Since today, the curl git repository holds the accumulated efforts from 1,500 separate and named individuals. Only 4.5 years since we passed 1,000. Yay for us!

Author 1,500 turned out to be Sameeh Jubran who authored this.

Number of commit authors in the curl project

Network, Web

Workshop Basel day three

2 Comments

See also: day one, day two.

There is only one thing that is better than two days of HTTP workshop, and that is of course three days of HTTP workshop. The final day of this edition of the series started out with us again shuffling around where we parked ourselves around the big table. Except Mr captain of course who once again got to herd us forward through another day from the same seat.

Why MOQ is going to replace HTTP live streaming

MOQ (Media over QUIC transport) is not HTTP, but it uses QUIC so it is at least tangentially interesting and it involves a lot of the same people so this status update still felt welcome and suitable. Compared to existing HTTP based solutions, MOQ is supposed to offer less complexity and lower latency. The moon landing was broadcasted with less latency than current live-streamed TV and maybe MOQ can make us come close to those numbers again. In MOQ clients subscribe to a track that then contains a lot of objects that are delivered. It’s not the request + response approach of HTTP. The fact that this is not HTTP of course brings a lot of questions and well, doubts, and we lingered on various aspects of this topic for quite a while.

Reverse HTTP

My prize for the best slides of the HTTP workshop 2026 goes to [redacted] for the excellent use of potato images in their presentation.

PTTH is HTTP spelled backwards, commonly pronounced as PoTaToH. A client sets up the connection but the actual HTTP request is sent from the server to the client. One of the intended use cases for this, is to allow an origin server to connect to the CDN proxy and then be able to deliver traffic to the world, rather than to have the CDN connect to the origin the way they usually do. Apparently most CDNs already have custom and proprietary solutions for exactly this kind of feature, so maybe doing it in a standard way instead makes sense?

Resumable uploads

The draft explains the new proposed way to continue a previously interrupted upload over HTTP. The upload request gets a Location: header back for the resource being uploaded, and if it gets stopped prematurely, a client can then HEAD that resource, figure out the size and then do a second upload (using the PATCH method) request that tells the server that this transfer should start at offset X.

Exactly how this should be supported in browser’ upload forms seemed a little bit uncertain. For my own sake I can see a challenge to implement this nicely for curl in particular when the upload is using formpost upload (curl’s -F flag) which after all still is a very common way to do uploads on the current web. I’ll return to this topic at a later time when I written an implementation to test…

io_uring vs. multithreaded server runtimes vs HTTP mismatch

io_uring is a Linux asynchronous I/O framework that avoids the overhead of traditional system calls. It uses two shared ring buffers between user space and the kernel, allowing applications to batch I/O operations with zero-copy efficiency.

The feature is disabled by Google in ChromeOS, Android and in production Google servers which certainly holds back some use of it.

io_uring can be helpful to speed up things, but might be complicated to use in existing software architectures and the presentation went into some details on why this is so.

Modern UDP I/O for Firefox in Rust

A walk-through of some of the recent developments and improvements in Firefox’s UDP networking stack. Going from single datagrams to the modern ways to ship large chunks of data offloaded to the kernel to speed things up. Upload throughput in Firefox is up 60-90% over the last 11 releases. Lots of fun graphs and metrics were shown. This work is based on the quinn-udp stack.

Rollout of Happy Eyeballs v3 in Firefox

Happy Eyeballs v3 is coming and Firefox is implementing it. It now takes into account many more data sources than before, including alt-svc and HTTPS-RR and races connections against each other to use the one that connects first. There are some recommended timers in the specification and parts of the discussion was around how maybe the timers could instead be tightened a bit, and maybe the delay between the subsequent attempts could then use an exponential backoff instead sticking to a fixed interval?

(I know I’ll discuss some of these details with my curl hacker friends and see what we should adjust… curl already supports most of the Happy Eyeballs v3 specification.)

Shorter ones

As we approached the end of the day a few shorter topics were ventilated to give us a little more to consider before going home:

  • Why is there no UTF8 in URIs? “If we would do it again, we would have allowed UTF8 in there” was said by someone who was there in the mid 1990s…
  • Optimistic DNS is a draft. Use stale DNS cache data while getting the new. Connection remains alive for 120 seconds while DNS data is often not cached for even 30 seconds. No one in the room seemed to hate it. Let’s do this!
  • The journey to QUERY. One of the primary authors of the RFC took us through what it took to make it happen. It was sixteen years since the most previous registered HTTP method and maybe this was the last one ever?

The end for this time

With this, the seventh HTTP workshop had ended. Again a very fine event. This time graciously sponsored and arranged by Adobe. Thank you everyone!

The general idea is to continue with these events roughly every second year and I support this. The HTTP workshops are definitely one of my favorite events.

Credits

The top image on this post was used in the final presentation and the author told me he is aware of the AI errors in there, “of which there are at least two”.

Network, Web

Workshop Basel day two

If you missed it. I already described day one.

Caffeinated and ready, we all gathered in the same spacious room as yesterday, but seated in new places as “suggested” by our captain. Some of us even remembered to move over the name tags we wrote yesterday to our new seats.

No time was wasted on introductions today. We dove straight in at the deep end.

How AI is changing how HTTP is implemented.

Is the future of software that we check-in the AI prompts in the git repository and trust it to generate the correct code? Are specifications the new level o

f abstraction for source code? These questions triggered long discussions with a huge mix of opinions and experiences getting shared about how AI is used, should be used and could be used now and in the future. 

Observations and Measurements of HTTP/2 During Large-Scale Web Crawls

The Common Crawl spidering upgraded to using HTTP/2 for their scan and as an end result, I believe 61% of the responses used HTTP/2 and the entire round ended a few percent faster than before, which when you traverse a few billion URLs really makes a difference. They apparently use a locally patched version of Apache Nutch for this.

HTTP/1.1 behavior divergence

The HTTP probe project runs a lot of tests on HTTP/1 servers and compares how they behave in a lot of different aspects and then generates these awesome tables. Looks like something for every server implementer team to have a look at and decide what of these red boxes that should rather be converted into green alternatives.

Request smuggling test suite 

HTTP Zoll is a new test suite for intermediaries that tests intermediaries (what we often call proxies) for a large amount of request and response smuggling issues. Some real world problems found were discussed and as this project aims at going Open Source words were expressed on what kind of precautions and checks that maybe should be done first. I hope we get to hear more about this project soon.

Server performance & measurement

The HTTP Arena is another project that does performance and measurements. They test HTTP server frameworks and present the results in various ways on their site.

Increase and evolve HTTP/3 & QUIC

In this presentation, we were presented with different HTTP/3 deployment numbers from different sources and the associated reasoning around why they differ but then more importantly. what can and should be done to increase HTTP/3 usage. 

Anti-virus interceptions, enterprise blocks and server-side performance not yet on par with TCP were mentioned as reasons for holding back the numbers.

Reasons for using HTTP/3 include use cases that encourage QUIC adoption: WebTransport, Media over QUIC and MASQUE (HTTP/3 proxies and HTTP/3 proxies over older HTTP proxies). 

Using HTTPS-RR for upgrade was promoted, as every alt-svc response that is returned with an ALPN using h3 should perhaps also offer h3 over DNS. Why doesn’t your server announce its h3 support over HTTPS-RR?

QUIC v2 is deployed on an amazing 0.003% of all QUIC v1 domains and there was a discussion why this is so and the common sentiment in the room seemed to be that very few saw a reason for deploying v2 and several expressed a concern that doing so might in fact introduce issues. Someone (you can probably guess who) in the room increased that number a lot by quietly mentioning that haxproxy.org certainly supports it.

QMUX

QUIC multiplexing over bi-directional streams is a proposal on how to do QUIC-style multiplexing over TLS (or anything else really). It has been adopted by the IETF QUIC working group and there was a somewhat extended discussion about what the HTTPbis group should or should not do with it. The biggest interest might be for data center use, but is that then something IETF should bother about? This is not the first time I blog about this, and even if there did not seem to be a strong demand or need for this, it also did not seem to be completely dead. I bet we will hear more about this later.

Multiplexed proxying: challenges in H2 and H3

Doing a TLS terminating MITM proxy has its challenges and we were given some insights and experiences on the challenges of doing HTTP/2 and HTTP/3 to the server.

The browsers refuse to do HTTP/3 when they detect custom CA certs installed, which apparently is mostly because of lots of past bad experiences with anti-virus software that in particular seems to break QUIC and for users it is not obvious where the blame should go. This then makes browsers not do HTTP/3 over any MITM proxy.

Some time was spent on how allowing different clients to the proxy uses a shared h2 connection to the target server is complicated and not used, even though in theory it should be possible. An argument was made that it could even lead to worse performance than when using HTTP/1 but I could not quite follow that reasoning. I’m sure I missed some subtle detail in that explanation.

Making the Web QUICer with Rapid Start

When the afternoon is running late and we have been promised beer and snacks after the final talk, what is better than a hard core technical presentation with lots of graphs and numbers showing how QUIC performance can be improved by tweaking the congestion control algorithm and send more data in the startup phase of a new QUIC connections? This new approach is called Rapid Start and it looks like a promising and yet simple improvement. According to experiments done on real world traffic, the time to last byte was reduced by 14.7% on average. Not bad at all.

Drinks and food

Our meeting sponsor Adobe graciously sponsored drinks and food so we got to linger around for a few extra hours and talk even more HTTP and networking until the personal firmly insistent they needed us to leave the room and we instead continued solving world problems elsewhere. Topics around the table included the famous HTTP/2 spec coin flip, the QUIC spin bit, the SCONE situation for QUIC, the timeline behind the QUERY method and many more great stories.

Thanks for the beer!

Now we can’t wait for day three.

Network, Web

Workshop Basel day one

4 Comments

On this hot summer’s day in Basel, Switzerland, the seventh HTTP workshop started. These events tend to work roughly the same way and the people in the room are also to large extent familiar and known since previous editions. Forty people in a meeting room, where we take turns in doing short talks on HTTP and networking topics, with the following question and discussion session. The rules for the meetings are explicitly Chatham rules, which means that everything I write about the meeting will be sufficiently fuzzy and without many company or personal names. This is not the kind of meeting that can be easily summed up in a short blog post anyway. You really should be here.

Present in the room were representatives from all the world’s most prominent and used HTTP deployments: clients, browsers, CDNs, proxies and servers. I’m happy to say that there were also several first-timers. We like fresh blood.

(If you think I’m being overly brief or vague about specifics in this post; that is partially on purpose but primarily because I’m a lousy note-taker and mostly write this up after a busy day that also may have involved beer.)

After a round of introductions, we started.

Extending REST for State synchronization

REST is a set of constraints, and in this presentation it was argued that it can or maybe even should be extended to do more. A number of recent applications like Mastodon/ActivityPub, Bluesky/AT, Matrix, Nostr, IndieWeb, all currently use HTTP to do state synchronization but they all do it differently in their own unique ways. Can REST and maybe HTTP be adjusted to help this for improved interoperability?

Last-Modified header use over time

Looking at the Common Crawl data and comparing data over time, it was observed that responses use the Last-Modified header field more now than they did in the past, and there were great follow-up speculations on why this is so. Data also shows that a large share of these headers present dates that are almost identical to the time the requests were issued.

How is HTTP used in the world?

With the cc-lint tool, data was gathered on how HTTP is actually used today, proving that there is work to be done: deprecated headers are used, some headers are done wrong, and many are overly big. This indicates that there are well used both servers and clients out there that would benefit from cleanup. It probably also shows that doing HTTP correctly and all the correct headers is far from an easy task.

AI-bots’ use of HTTP

Another presentation showed data, this time from a well-known CDN, on the impact the existing AI scraper bots have on the Internet from their point of view. It showed that roughly half of the requests and half of the bandwidth are spent by scraper bots. A long discussion followed where the numbers were questioned as maybe the numbers look like this because a sufficiently large number of the “bad AI scrapers” appear as regular users to the classifiers. Speculations of different kinds were made. 

The Apple HTTP stack two years later

As a follow-up from a presentation from a previous HTTP workshop we got to learn how the journey on developing their new HTTP stack has progressed and several fun adventures and lessons from that were shared with the audience.

Why new HTTP APIs?

A look into new HTTP API development at Apple. Some discussions and lessons learned from creating new APIs for both servers and clients.

Android Networking

We got an excellent walk-through of some details and internals of the Android networking stack. Emphasis was perhaps especially put on ECH and QUIC connection migration, and the final “don’t tell us when your connection closed” led to a long new discussion on how we really should fix the problem: when connection has been left idle for a long time and it is closed by the server, the client (mobile phones) don’t want to be told. This, because getting that RST and more, just wakes up the radio and more on the phone only to tell it to go back to sleep. It was theorized that if we could get rid of this unnecessary battery waste, the accumulated gain across billions of devices would make a serious dent.

Day one world problem solving

Several additional HTTP related problems were of course also subsequently solved as we then wandered into the city for dinner and maybe a beer. Of course yours truly returned back to his hotel room in good time to be able to write up this blog post.

The best part of these workshops might be the (no pun intended) networking and discussions had completely outside of the agenda.

End of day one. Two more to come,

cURL and libcurl, Open Source, Security

Do excellent vulnerability reports

2 Comments

Over the years, we have received, read and handled way over one thousand vulnerability reports filed against curl. We have seen most kinds.

It is time for me to try to help future reporters by providing a short guide on how to submit a truly excellent vulnerability report to an Open Source project.

Researchers

We tend to call everyone who reports a security problem a security researcher, because by the act of the submission itself they fulfill the definition. There are however many different kinds of people who submit reports; from the most rookie youngster with limited experience, to the multi-decade experienced senior in the field.

Most reports submitted to a project like curl come from reporters who never submitted anything to the project before and are completely previously unknown. Many reporters use hacker handles or pseudonyms, so there is not a lot to learn about the person behind the report either. We don’t know the reporters’ age, experience level, employer, sex or on which continent they live. But also: none of those things matter.

When you submit a vulnerability report, consider telling the project how you want to get credited, should they consider your report real.

There is a potentially almost unlimited amount of security researchers that can find problems in a project. The project receiving your report only has a limited small number of overloaded maintainers that take care of the reports. Consider this imbalance. Make your report as easy as possible for the team to manage.

Finding

To us maintainers who receive a steady stream of vulnerability reports, it rarely matters exactly how the problem was detected. Whether you fell over it by accident, you found it by reading every single line of source code or if an AI pointed it out to you, it has little relevance to the security team. The team primarily cares about if the problem is real and if it is, how serious the impact is.

Really?

If the problem is documented, then it likely isn’t a vulnerability. This is a common theme in curl: people report that they can find something strange or peculiar to happen when they do something, only to have one of us point out that the action is either documented to have that side-effect, or the action was done in spite of clear warnings in the documentation.

To make a good vulnerability report, you should make sure you understand what the software is supposed to do – and what the documentation says its limitations and conditions are. A good Open Source project has those things documented.

Where

Figure out where and how to submit your report. If you found several problems, it is considered polite to ask the team how they want to receive the rest. As separate individual submissions or maybe as a curated list. Perhaps paced at a slow rate to avoid overflow.

Never circumvent the submission method suggested by the project. That is impolite.

Consider the initial submitting of the issue to be the first step in a multi-step communication process with the project that will continue for as long as at least one of your reported issues has not been resolved or dismissed. This can be days, weeks or in some cases even months.

Expect responses and follow-up questions. Be prepared to clarify, expand and maybe provide more code and reasoning. Remember that you submit vulnerability reports in order to help and improve the project.

Report

These days people like to create enormously long and detailed reports that have all the details, often explained three times and with several embedded lists using bullet points describing impact and providing more or less good analysis attempts.

Your first paragraph of the report should be a human-written, brief explainer of what the problem is and what badness it leads to. You should be able to explain that in just a few sentences. It is a reality-check, because if you can’t do this, if you don’t understand the flaw enough yourself to write such a paragraph, then you have homework to do. Figure it out, then come back and write the intro paragraph.

Having a quality intro saves a lot of time for the security team receiving your report.

Be aware that the Open Source project you contact may be overloaded, on vacation or seeing your report as yet another duplicate they already saw reported seven times.

Be helpful and respect that you add a load to a small team that probably consists of volunteers working on this in their spare time.

Even if you have used a lot of or just a little AI when finding the issue and writing up the report, you must make sure that you communicate as a human. With your human communication skills.

Reproducer

Your report should contain a reproducer. Ideally a fully contained and stand-alone script or source code that the security team can build and run to see the vulnerability trigger.

A reproducer helps prove to the team that the problem is real or maybe already an accepted risk or behavior. It is also convenient for the developers to first understand and reproduce the issue, and then they can convert the reproducer into a project test case for the pending fix.

Without providing a reproducer in your report, you instead push that work to the receiving end. We still need the reproducer. We still need a test case.

Patch

Provide a patch for the problem.

If you can figure out a way to fix the code to make your finding no longer trigger, that is great information for the security team and such a patch usually helps them understand the issue better and get a speedier result. It reduces the load.

Sure, such a patch is often perhaps not perfect and it can usually be improved and expanded as the developers have a different view and a more nuanced understanding of the problem and the software architecture involved. It still helps. Getting 80% towards the target is still valuable.

Versions

Usually you should look for vulnerabilities in the latest version of the software, often even using an up-to-date git repository. Whatever version you used to find it, you need to specify that in your report.

If the problem turns out to be real, which your report claims and you should never report anything if you don’t think so, it is then also immediately interesting to know when this problem first appeared. Which is the earliest version of the software that you can trigger this problem with?

The project will want to know this to write up a proper advisory for the issue. You can help figuring this out by bisecting etc.

Collaborate

Remain available after your initial submission.

In the curl project at least, we want to work with the reporter to make sure we get every angle and detail right. First, when trying to understand and assess the initial report and agreeing on a severity for it.

Then, we jointly produce and agree to a remedy (patch) for the problem, which ideally means taking the reporter’s version and massaging it into perfection.

If the problem is serious enough, there could be reasons to discuss a rushed patch release at an earlier date than the pending release would otherwise happen on. To reduce the time users in the wild remain vulnerable.

Finally, we collaborate on the description and explainer for the problem that goes into the security advisory.

Advisory

For every CVE that is registered and assigned to a particular vulnerability, there needs to be a detailed security advisory written. It should ideally describe the issue, how it triggers, what it means, the impact, the affected version ranges and more. Everything related to the vulnerability that we can think might help users.

Your job as a security researcher is to make sure the description in the advisory matches your finding, your understanding of the problem and that the description is understandable.

Learn

For every confirmed security report, the receiving project will try to learn from it and fix code and practices to avoid making the same mistake again.

As a reporter, your job is to learn from the submission experience and try to improve your reporting procedure and approach for the next time.

Then submit your next report!

cURL and libcurl

A curl mountain movie

3 Comments

One of my favorite visuals for known vulnerabilities in curl is the mountain. It shows how many currently known vulnerabilities were present in the code through-out curl’s history.

In the end of June 2026 it looks like this:

Over time we get more vulnerabilities reported. Since every flaw has a version range during which the problem existed and with more issues that have overlapping version ranges, the mountain grows. It changes shape every time we do a release or we publish a new vulnerability.

At this moment in time, curl version 7.34.0 is the release that contains the most number of known vulnerabilities: 101. The worst one ever if you will. Out of a total of 206.

The mountain uses different colors for different severity levels of the published vulnerabilities, as the legend in the top-left of the image explains.

To illustrate the ever-changing nature of the shape and size, I wrote a script that renders the mountain the way it looked at specific dates in the past up until today. More specifically, the script renders one image for every month since curl started (March 1998). I then turned these 340 individual images into a little movie that shows how it grew into today’s shape. At four months/second.

The data for this come from vuln.pm and the curl git repository. The graph rendering is based on the dashboard scripts. All images put into a movie with ffmpeg of course.

The 2016 drop

Several people have asked what happened in 2016 that caused the notable drop. A slope if you will.

If we zoom in on that, we can spot that curl 7.51.0 has eleven fewer vulnerabilities than the version before that. This release was the first one after the 2016 Cure53 code audit, but other than that there is no clear distinct process or obvious code changes that explain this trend shift.

Lots of other graphs show just the ordinary pace and growth in various project areas. It was still fairly early days CI-wise but had been running at least a few CI jobs per commit for a few years already by then.

curl was adopted into the OSS-Fuzz project in July 2017, which since then makes us find some issues better, but the drop looks like it happened before then.

We had already been analyzing the code regularly on Coverity since a few years.

Better tooling? New compiler options? We simply don’t know.

Future

As we keep announcing more vulnerabilities going forward, things will continue to change. Maybe I will come back and make another movie in five years?

cURL and libcurl

Trailing dots are the worst

4 Comments

Trailing dots after hostnames in URLs remain my worst enemies. I wrote about several problems with them in the past that involved those nasty things. They are still painful. When we shipped curl 8.21.0 on June 24 2026 we fixed at least three brand new problems that involved trailing dots. C’mon, follow me down the trailing dot rabbit hole, episode two. I can just feel that there will be a third episode as well in a future…

IPv4 numerical address

Let’s for a second imagine that you create a URL that uses a numerical IPv4 address. Not entirely uncommon. For example lots of people use 127.0.0.1 in local tests etc. Used everywhere since the dawn of time.

Now imagine that you add a trailing dot to this hostname, like “192.168.0.1.”. What does the trailing dot even mean here?

This particular trailing dot caused a problem in curl. To figure out if curl should allow wildcard certificates when connecting to a TLS server, it needs to know if the given hostname is a numerical IP or a hostname. The check uses inet_pton() on the provided hostname extracted from the URL – which incidentally returns false for an IPv4 address that ends with dot! So if it isn’t a numerical address it is a hostname and then we allow wildcards… Argh.

I decided to solve this particular problem like this: if the address is a valid IPv4 address and there is only a single dot afterwards, that dot is “swallowed” as part of the regular IPv4 normalization process that curl always does for IPv4 addresses when parsing URLs. This way, a numerical IPv4 address with a trailing dot will never be passed on to curl internals anymore. And the meaning of the trailing dot for this use case is clear: it is a mistake so we get rid of it. (This also seems to be what browsers do.) Shipping in curl 8.21.0.

This choice has already been reported problematic by at least one user who expected a transfer for a URL like this to return error… I suppose this means that the jury is still out on what the best approach for this trailing dot is.

Double trailing dots HSTS

What could be more fun than trailing dots if not two trailing dots!

Two trailing dots is not possible to use as a hostname when resolving hostnames using DNS. It is an illegal name and causes an error. But as curl provides other ways to populate the DNS cache with a provided name, and you can provide names in /etc/hosts etc you can make curl work with URLs where the hostname has two trailing dots. Or rather, you could up until recently until I made sure it is properly banned always because of the trouble they cause internally.

A double-dot is correctly treated as a host with a trailing dot, but it turns out that in for example the HSTS logic that became problematic as removing the trailing dot for some functions would still have a trailing dot there when there were two of them to begin with… and it would get confused and act up.

No more double trailing dots. One is annoying enough. Shipping in curl 8.21.0.

Cookie domain

HTTP cookies are basically name/value pairs set by the server and held by the client to get sent back to the server again in later communications. The server can specify for which domain a cookie should apply to, so that it can be used across multiple domains. (Yes, it is a little crazy,)

To prevent the server from being able to set the cookie on a too wide domain cookie clients check if the specified domain is Public Suffic Domain (PSL) or not. A server is not allowed to set cookies for PSL domains, as that allows it to create “super cookies” that work across domains in ways that are not allowed. Cookies attempted to get set for such a name should be rejected.

In libcurl we check domains against the PSL using the libpsl library.

Turns out this too could be tricked by trailing dots. If you communicate with the URL “example.co.uk.” (with a trailing dot) and it sets a cookie for for “co.uk.” (with a trailing dot), the internal check would ask libpsl about the PSL status and… it did not work with trailing dots. The exact same process without trailing dots correctly says it is a PSL and the cookie is refused. But with the trailing dots present it was fooled and curl would allow the cookie to get stored and later sent back to such a host…

This particular issue ended up considered a vulnerability known as CVE-2026-8924. Fix shipped in curl 8.21.0.

We should consider these things

Yes, you can of course quite correctly argue that none of these things are actually new or sudden changes. Trailing dots are there, they have always been there and people will continue to use them in the future. I’m not blaming anyone else. I’m just expressing my frustration.

Trailing dots are the worst.