Monthly Archives: March 2025

cURL and libcurl

HTTPS RR in curl

1 Comment

RFC 9460 describes a DNS Resource Record (RR) named HTTPS. To highlight that it is exactly this DNS record called HTTPS we speak of, we try to always call it HTTPS RR using both words next to each other.

curl currently offers experimental support for HTTPS RR in git. Experimental means you need to enable it explicitly in the build to have it present.

cmake -DUSE_HTTPSRR=ON ...

or

configure --enable-httpsrr ...

What is HTTPS RR for?

It is a DNS field that provides service meta-data about a target hostname. In many ways this is an alternative record to SRV and URI that were never really used for the web and HTTP. It also kind of supersedes the HSTS and alt-svc headers.

Here is some data it offers:

ECH config

ECH is short for Encrypted Client Hello and is the established system for encrypting the SNI field in TLS handshakes. As you may recall, the SNI field is one of the last remaining pieces in a HTTPS connection that is sent in the clear and thus reveals to active listeners with which site the client intends to communicate. ECH hides this by encrypting the name.

For ECH to be able to work, the client needs information prior to actually doing the handshake and this field is provides this data. curl features experimental ECH support.

ECH has been in the works for several years, but has still not been published in an RFC.

ALPN list

A list of ALPN identifiers is provided. These identifiers basically tell the client which HTTP versions this server wants to use (over HTTPS). If this list contains HTTP/1.1 and we were asked to do a HTTP:// transfer, it implies the client should upgrade to HTTPS. Thus sort of replacing HSTS.

I think the key ALPN id provided here is ‘h3’ which tells the client this server supports HTTP/3 and we can try that already in the first flight. The previous option to properly upgrade to HTTP/3 would be to wait for a alt-svc response header that could instruct a subsequent connect attempt. That then delays the upgrade significantly since reusing the by then already existing HTTP/2 or HTTP/1 connection is typically preferred by clients.

Target

The service is offered on another hostname. Alt-svc style. It can also provide information about how to access this site for different “services” so that TCP/TLS connections go one way and QUIC connections another.

Port

It can tell the client that the service is hosted on a different port number.

IP addresses

It can provide a list of IPv4 and IPv6 addresses for the name. Supposedly to be used in case there are no A or AAAA fields retrieved at the same time.

Getting the DNS record

Adding support for HTTPS RR into an existing TCP client such as curl is unfortunately not as straight-forward as we would like.

A primary obstacle is that regular hostname resolves are done with the POSIX function getaddrinfo() and this function has no knowledge of or support for HTTPS RR and such support cannot expected to get added in a future either. The API does not really allow that. Clients like curl simply need to get the additional records using other means.

How curl does it

curl, or rather libcurl, has three different ways in source code to resolve hostnames out of which most builds feature two options:

  1. DoH (DNS-over-HTTPS) – curl sends DNS requests using HTTPS to a specified server. Users need to explicitly ask for this and point out the DoH URL.
  2. Using the local resolver. A libcurl build can then use either c-ares or getaddrinfo() to resolve host names, for when DoH or a proxy etc are not used.

DoH

The DoH code in libcurl is a native implementation (it does not use any third party libraries) and libcurl features code for both sending HTTPS RR requests and parsing the responses.

getaddrinfo

When using this API for name resolving, libcurl still needs to get built with c-ares as well to provide the service of asking for the HTTPS RR. libcurl then resolves the hostname “normally” with getaddrinfo() (fired off in a separate helper thread typically) and asks for the HTTPS RR using c-ares in parallel.

c-ares

When c-ares is used for name resolving, it also asks for the HTTPS RR at the same time. It means c-ares asks for A, AAAA and HTTPS records.

Existing shortcomings

We do not offer a runtime option to disable using HTTPS-RR yet, so if you build with it enabled it will always be attempted. I think we need to provide such an option for the times the HTTPS record is either just plain wrong or the user wants to debug or just play around differently.

curl also does not yet currently handle every aspect of the quite elaborate HTTPS RR. We have decided to go forward slowly and focus our implementation on the parts of the resource field that seem to be deployed and in actual use.

Deployed

There are sites using these records already today. Cloudflare seems to provide it for a lot of its hosted sites for example. If you enable HTTPS-RR today, there are plenty of opportunities to try it out. Both for regular service and protocol switching and also for ECH.

Please, go ahead and enable it in a build, try it out and tell us what works and what does not work. We would love to move forward on this and if we get confirmation from users that it actually works, we might be able transition it out from experimental status soon.

cURL and libcurl

curl up 2025

Leave a comment

Soon, in the first weekend of May 2025, we are gathering curl enthusiasts in a room in the wonderful city Prague and we talk curl related topics over two full days. We call it curl up 2025.

curl up is our annual curl event and physical meetup. It is a low key and casual event that usually attract somewhere between twenty and thirty people.

We sit down in a room and talk about curl related topics. Share experiences, knowledge, ideas and insights.

Get all your questions answered. Get all your curiosity satisfied. Tell the others what you do with curl and what you think the curl project should know.

Attend!

We want broad attendance. The curl developers sure, but we also love to see newcomers and curl users show up and share their stories, join the conversations and get the opportunity to meet the team and maybe learn new stuff from the wonderful world of Internet transfers.

The event is free to attend, you just need to register ahead of time. You find the registration link on the curl up 2025 information page.

Sponsor!

curl is a small and independent Open Source project. Your sponsorship could help us pay to get core developers to this conference and help cover other costs we have to run it. And your company would get cool exposure while you help us improve a little piece of internet infrastructure. Win win!

Physical meetings have merit

This is the only meetup in the real world we organize in the curl project. In my experience it serves as an incredible energy and motivation boost and it helps us understand each other better. We all communicate a little smoother once we have gotten to know and “the persons” behind the online names or aliases.

After having met physically, we cooperate better.

cURL and libcurl

19000 curl commits

Leave a comment

It has been 387 days since I announced having done 18k commits. This is commit number 19,000.

Looking at the graph below that is showing my curl commits and everyone else’s over time, it seems to imply that while I have kept my pace pretty consistently over the last few years, others in the curl project have stepped up their game and commit more.

Commits done in curl over time, by Daniel and by “everyone else”

This means that my share of the total number of commits keep shrinking and also that I personally am nowadays often not the developer doing the most curl commits per month.

This is all good. We need even more contributors and people involved. We see on average about eight new commit authors every month.

Number of new commit authors per month in the curl project, over time.

The previous one thousand commits I did took me 422 days. These latest one thousand commits took me 387 days. If I manage to maintain this exact frequency, I will reach 20K commits on April 5 2026.

cURL and libcurl

My cookie spec problem

4 Comments

Before RFC 6265 was published in 2011, the world of cookies was a combination of anarchy and guesswork because the only “spec” there was was not actually a spec but just a brief text lacking a lot of details.

RFC 6265 brought order to a world of chaos. It was good. It made things a lot better. With this spec, it was suddenly much easier to write compliant and interoperable cookie implementations.

I think these are plain facts. I have written cookie code since 1998 and I thus know this from my own personal experience. Since I believe in open protocols and doing things right, I participated in the making of that first cookie spec. As a non-browser implementer I think I bring a slightly different perspective and different angle to what many of the other involved people have.

Consensus

The cookie spec was published by the IETF and it was created and written in a typical IETF process. Lots of the statements and paragraphs were discussed and even debated. Since there were many people and many opinions involved, of course not everything I think should have been included and stated in the spec ended up the way I wanted them to, but in a way that the consensus seemed to favor. That is just natural and the way of the process. Everyone involved accept this.

I have admitted defeat, twice

The primary detail in the spec, or should I say one of the important style decisions, is one that I disagree with. A detail that I have tried to bring up again when the cookie spec was up for a revision and a new draft was made (still known as 6265bis since it has not become an official RFC yet). A detail that I have failed the get others to agree with me about to an enough degree to have it altered. I have failed twice. The update will ship with this feature as well.

Cookie basics

Cookies are part of (all versions of ) HTTP but are documented and specified in a separate spec. It is a classic client-server setup where Set-Cookie: response headers are sent from a server to a client, the client stores the received cookies and sends them back to servers according to a set of rules and matching criteria using the Cookie: request header.

Set-Cookie

This is the key header involved here. So how does it work? What is the syntax for this header we need to learn so that we all can figure out how servers and clients should be implemented to do cookies interoperable?

As with most client-server protocols, one side generates this header, the other side consumes it. They need to agree on how this header works.

My problem is two

The problem is that this header’s syntax is defined twice in the spec. Differently.

Section 4.1 describes the header from server perspective while section 5.2 does it from a client perspective.

If you like me have implemented HTTP for almost thirty years you are used to reading protocol specifications and in particular HTTP related specification. HTTP has numerous headers described and documented. No other HTTP documents describe the syntax for header fields differently in separate places. Why would they? They are just single headers.

This double-syntax approach comes as a surprise to many readers, and I have many times discussed cookie syntax with people who have read the 6265 document but only stumbled over and read one of the places and then walked away with only a partial understanding of the syntax. I don’t blame them.

The spec insists that servers should send a rather conservative Set-Cookie header but knowing what the world looks like, it simultaneously recommends the client side for the same header to be much more liberal because servers might not be as conservative as this spec tells the server to be. Two different syntax.

The spec tries to be prescriptive for servers: thou shall do it like this, but we all know that cookies were wilder than this at the time 6265 was published and because we know servers won’t stick to these “new” rules, a client can’t trust that servers are that nice but instead need to accept a broader set of data. So clients are told to accept much more. A different syntax.

Servers do what works

As the spec tells clients to accept a certain syntax and widely deployed clients and cookie engines gladly accept this syntax, there is no incitement or motive for servers to change. The do this if you are a good server instruction serves as an ideal, but there is no particularly good way to push anyone in that direction because it works perfectly well to use the extended syntax that the spec says that the clients need to accept.

A spec explaining what is used

What I want? I want the Set-Cookie header to be described in a single place in the spec with a single unified syntax. The syntax that is used and that is interoperable on the web today.

It would probably even make the spec shorter, it would remove confusion and it would certainly remove the risk that people think just one of the places is the canonical syntax.

Will I bring this up again when the cookie spec is due for refresh again soon? Yes I will. Because I think it would make it a much better spec.

Do I accept defeat and accept that I am on the losing side in an argument when nobody else seems to agree with me? Yes to that as well. Just because I think like this does in no way mean that this is objectively right or that this is the way to go in a future cookie spec.