Tag Archives: HTTP

Network, Web

Workshop Basel day three

2 Comments

See also: day one, day two.

There is only one thing that is better than two days of HTTP workshop, and that is of course three days of HTTP workshop. The final day of this edition of the series started out with us again shuffling around where we parked ourselves around the big table. Except Mr captain of course who once again got to herd us forward through another day from the same seat.

Why MOQ is going to replace HTTP live streaming

MOQ (Media over QUIC transport) is not HTTP, but it uses QUIC so it is at least tangentially interesting and it involves a lot of the same people so this status update still felt welcome and suitable. Compared to existing HTTP based solutions, MOQ is supposed to offer less complexity and lower latency. The moon landing was broadcasted with less latency than current live-streamed TV and maybe MOQ can make us come close to those numbers again. In MOQ clients subscribe to a track that then contains a lot of objects that are delivered. It’s not the request + response approach of HTTP. The fact that this is not HTTP of course brings a lot of questions and well, doubts, and we lingered on various aspects of this topic for quite a while.

Reverse HTTP

My prize for the best slides of the HTTP workshop 2026 goes to [redacted] for the excellent use of potato images in their presentation.

PTTH is HTTP spelled backwards, commonly pronounced as PoTaToH. A client sets up the connection but the actual HTTP request is sent from the server to the client. One of the intended use cases for this, is to allow an origin server to connect to the CDN proxy and then be able to deliver traffic to the world, rather than to have the CDN connect to the origin the way they usually do. Apparently most CDNs already have custom and proprietary solutions for exactly this kind of feature, so maybe doing it in a standard way instead makes sense?

Resumable uploads

The draft explains the new proposed way to continue a previously interrupted upload over HTTP. The upload request gets a Location: header back for the resource being uploaded, and if it gets stopped prematurely, a client can then HEAD that resource, figure out the size and then do a second upload (using the PATCH method) request that tells the server that this transfer should start at offset X.

Exactly how this should be supported in browser’ upload forms seemed a little bit uncertain. For my own sake I can see a challenge to implement this nicely for curl in particular when the upload is using formpost upload (curl’s -F flag) which after all still is a very common way to do uploads on the current web. I’ll return to this topic at a later time when I written an implementation to test…

io_uring vs. multithreaded server runtimes vs HTTP mismatch

io_uring is a Linux asynchronous I/O framework that avoids the overhead of traditional system calls. It uses two shared ring buffers between user space and the kernel, allowing applications to batch I/O operations with zero-copy efficiency.

The feature is disabled by Google in ChromeOS, Android and in production Google servers which certainly holds back some use of it.

io_uring can be helpful to speed up things, but might be complicated to use in existing software architectures and the presentation went into some details on why this is so.

Modern UDP I/O for Firefox in Rust

A walk-through of some of the recent developments and improvements in Firefox’s UDP networking stack. Going from single datagrams to the modern ways to ship large chunks of data offloaded to the kernel to speed things up. Upload throughput in Firefox is up 60-90% over the last 11 releases. Lots of fun graphs and metrics were shown. This work is based on the quinn-udp stack.

Rollout of Happy Eyeballs v3 in Firefox

Happy Eyeballs v3 is coming and Firefox is implementing it. It now takes into account many more data sources than before, including alt-svc and HTTPS-RR and races connections against each other to use the one that connects first. There are some recommended timers in the specification and parts of the discussion was around how maybe the timers could instead be tightened a bit, and maybe the delay between the subsequent attempts could then use an exponential backoff instead sticking to a fixed interval?

(I know I’ll discuss some of these details with my curl hacker friends and see what we should adjust… curl already supports most of the Happy Eyeballs v3 specification.)

Shorter ones

As we approached the end of the day a few shorter topics were ventilated to give us a little more to consider before going home:

  • Why is there no UTF8 in URIs? “If we would do it again, we would have allowed UTF8 in there” was said by someone who was there in the mid 1990s…
  • Optimistic DNS is a draft. Use stale DNS cache data while getting the new. Connection remains alive for 120 seconds while DNS data is often not cached for even 30 seconds. No one in the room seemed to hate it. Let’s do this!
  • The journey to QUERY. One of the primary authors of the RFC took us through what it took to make it happen. It was sixteen years since the most previous registered HTTP method and maybe this was the last one ever?

The end for this time

With this, the seventh HTTP workshop had ended. Again a very fine event. This time graciously sponsored and arranged by Adobe. Thank you everyone!

The general idea is to continue with these events roughly every second year and I support this. The HTTP workshops are definitely one of my favorite events.

Credits

The top image on this post was used in the final presentation and the author told me he is aware of the AI errors in there, “of which there are at least two”.

Network, Web

Workshop Basel day two

Leave a comment

If you missed it. I already described day one.

Caffeinated and ready, we all gathered in the same spacious room as yesterday, but seated in new places as “suggested” by our captain. Some of us even remembered to move over the name tags we wrote yesterday to our new seats.

No time was wasted on introductions today. We dove straight in at the deep end.

How AI is changing how HTTP is implemented.

Is the future of software that we check-in the AI prompts in the git repository and trust it to generate the correct code? Are specifications the new level o

f abstraction for source code? These questions triggered long discussions with a huge mix of opinions and experiences getting shared about how AI is used, should be used and could be used now and in the future. 

Observations and Measurements of HTTP/2 During Large-Scale Web Crawls

The Common Crawl spidering upgraded to using HTTP/2 for their scan and as an end result, I believe 61% of the responses used HTTP/2 and the entire round ended a few percent faster than before, which when you traverse a few billion URLs really makes a difference. They apparently use a locally patched version of Apache Nutch for this.

HTTP/1.1 behavior divergence

The HTTP probe project runs a lot of tests on HTTP/1 servers and compares how they behave in a lot of different aspects and then generates these awesome tables. Looks like something for every server implementer team to have a look at and decide what of these red boxes that should rather be converted into green alternatives.

Request smuggling test suite 

HTTP Zoll is a new test suite for intermediaries that tests intermediaries (what we often call proxies) for a large amount of request and response smuggling issues. Some real world problems found were discussed and as this project aims at going Open Source words were expressed on what kind of precautions and checks that maybe should be done first. I hope we get to hear more about this project soon.

Server performance & measurement

The HTTP Arena is another project that does performance and measurements. They test HTTP server frameworks and present the results in various ways on their site.

Increase and evolve HTTP/3 & QUIC

In this presentation, we were presented with different HTTP/3 deployment numbers from different sources and the associated reasoning around why they differ but then more importantly. what can and should be done to increase HTTP/3 usage. 

Anti-virus interceptions, enterprise blocks and server-side performance not yet on par with TCP were mentioned as reasons for holding back the numbers.

Reasons for using HTTP/3 include use cases that encourage QUIC adoption: WebTransport, Media over QUIC and MASQUE (HTTP/3 proxies and HTTP/3 proxies over older HTTP proxies). 

Using HTTPS-RR for upgrade was promoted, as every alt-svc response that is returned with an ALPN using h3 should perhaps also offer h3 over DNS. Why doesn’t your server announce its h3 support over HTTPS-RR?

QUIC v2 is deployed on an amazing 0.003% of all QUIC v1 domains and there was a discussion why this is so and the common sentiment in the room seemed to be that very few saw a reason for deploying v2 and several expressed a concern that doing so might in fact introduce issues. Someone (you can probably guess who) in the room increased that number a lot by quietly mentioning that haxproxy.org certainly supports it.

QMUX

QUIC multiplexing over bi-directional streams is a proposal on how to do QUIC-style multiplexing over TLS (or anything else really). It has been adopted by the IETF QUIC working group and there was a somewhat extended discussion about what the HTTPbis group should or should not do with it. The biggest interest might be for data center use, but is that then something IETF should bother about? This is not the first time I blog about this, and even if there did not seem to be a strong demand or need for this, it also did not seem to be completely dead. I bet we will hear more about this later.

Multiplexed proxying: challenges in H2 and H3

Doing a TLS terminating MITM proxy has its challenges and we were given some insights and experiences on the challenges of doing HTTP/2 and HTTP/3 to the server.

The browsers refuse to do HTTP/3 when they detect custom CA certs installed, which apparently is mostly because of lots of past bad experiences with anti-virus software that in particular seems to break QUIC and for users it is not obvious where the blame should go. This then makes browsers not do HTTP/3 over any MITM proxy.

Some time was spent on how allowing different clients to the proxy uses a shared h2 connection to the target server is complicated and not used, even though in theory it should be possible. An argument was made that it could even lead to worse performance than when using HTTP/1 but I could not quite follow that reasoning. I’m sure I missed some subtle detail in that explanation.

Making the Web QUICer with Rapid Start

When the afternoon is running late and we have been promised beer and snacks after the final talk, what is better than a hard core technical presentation with lots of graphs and numbers showing how QUIC performance can be improved by tweaking the congestion control algorithm and send more data in the startup phase of a new QUIC connections? This new approach is called Rapid Start and it looks like a promising and yet simple improvement. According to experiments done on real world traffic, the time to last byte was reduced by 14.7% on average. Not bad at all.

Drinks and food

Our meeting sponsor Adobe graciously sponsored drinks and food so we got to linger around for a few extra hours and talk even more HTTP and networking until the personal firmly insistent they needed us to leave the room and we instead continued solving world problems elsewhere. Topics around the table included the famous HTTP/2 spec coin flip, the QUIC spin bit, the SCONE situation for QUIC, the timeline behind the QUERY method and many more great stories.

Thanks for the beer!

Now we can’t wait for day three.

Network, Web

Workshop Basel day one

4 Comments

On this hot summer’s day in Basel, Switzerland, the seventh HTTP workshop started. These events tend to work roughly the same way and the people in the room are also to large extent familiar and known since previous editions. Forty people in a meeting room, where we take turns in doing short talks on HTTP and networking topics, with the following question and discussion session. The rules for the meetings are explicitly Chatham rules, which means that everything I write about the meeting will be sufficiently fuzzy and without many company or personal names. This is not the kind of meeting that can be easily summed up in a short blog post anyway. You really should be here.

Present in the room were representatives from all the world’s most prominent and used HTTP deployments: clients, browsers, CDNs, proxies and servers. I’m happy to say that there were also several first-timers. We like fresh blood.

(If you think I’m being overly brief or vague about specifics in this post; that is partially on purpose but primarily because I’m a lousy note-taker and mostly write this up after a busy day that also may have involved beer.)

After a round of introductions, we started.

Extending REST for State synchronization

REST is a set of constraints, and in this presentation it was argued that it can or maybe even should be extended to do more. A number of recent applications like Mastodon/ActivityPub, Bluesky/AT, Matrix, Nostr, IndieWeb, all currently use HTTP to do state synchronization but they all do it differently in their own unique ways. Can REST and maybe HTTP be adjusted to help this for improved interoperability?

Last-Modified header use over time

Looking at the Common Crawl data and comparing data over time, it was observed that responses use the Last-Modified header field more now than they did in the past, and there were great follow-up speculations on why this is so. Data also shows that a large share of these headers present dates that are almost identical to the time the requests were issued.

How is HTTP used in the world?

With the cc-lint tool, data was gathered on how HTTP is actually used today, proving that there is work to be done: deprecated headers are used, some headers are done wrong, and many are overly big. This indicates that there are well used both servers and clients out there that would benefit from cleanup. It probably also shows that doing HTTP correctly and all the correct headers is far from an easy task.

AI-bots’ use of HTTP

Another presentation showed data, this time from a well-known CDN, on the impact the existing AI scraper bots have on the Internet from their point of view. It showed that roughly half of the requests and half of the bandwidth are spent by scraper bots. A long discussion followed where the numbers were questioned as maybe the numbers look like this because a sufficiently large number of the “bad AI scrapers” appear as regular users to the classifiers. Speculations of different kinds were made. 

The Apple HTTP stack two years later

As a follow-up from a presentation from a previous HTTP workshop we got to learn how the journey on developing their new HTTP stack has progressed and several fun adventures and lessons from that were shared with the audience.

Why new HTTP APIs?

A look into new HTTP API development at Apple. Some discussions and lessons learned from creating new APIs for both servers and clients.

Android Networking

We got an excellent walk-through of some details and internals of the Android networking stack. Emphasis was perhaps especially put on ECH and QUIC connection migration, and the final “don’t tell us when your connection closed” led to a long new discussion on how we really should fix the problem: when connection has been left idle for a long time and it is closed by the server, the client (mobile phones) don’t want to be told. This, because getting that RST and more, just wakes up the radio and more on the phone only to tell it to go back to sleep. It was theorized that if we could get rid of this unnecessary battery waste, the accumulated gain across billions of devices would make a serious dent.

Day one world problem solving

Several additional HTTP related problems were of course also subsequently solved as we then wandered into the city for dinner and maybe a beer. Of course yours truly returned back to his hotel room in good time to be able to write up this blog post.

The best part of these workshops might be the (no pun intended) networking and discussions had completely outside of the agenda.

End of day one. Two more to come,

cURL and libcurl, Web

QUERY with curl

2 Comments

RFC 10008 is brand new a specification detailing the new HTTP method called QUERY:

This specification defines the QUERY method for HTTP. A QUERY requests that the request target process the enclosed content in a safe and idempotent manner and then respond with the result of that processing. This is similar to POST requests but can be automatically repeated or restarted without concern for partial state changes

A GET with body

For all practical purposes you can think of QUERY as a way to send a GET with a body. It looks exactly like POST, but done with another verb.

Contrary to POST, QUERY requests are idempotent – they can be retried or repeated when needed, for instance after a connection failure.

curl it

You can use curl to do HTTP requests with QUERY just fine. curl offers the --request option (also known as -X in the short form) that you can use like this:

curl -d "data to send" -X QUERY https://example.com/

But redirects!

There is one little caveat to remember with this curl option that changes the method. When also asking curl to follow any possible redirects, it is important that you use a new enough curl version because you want the --follow option. Not the old --location/-L one.

Why? Because the old option changes the HTTP method on all subsequent requests independently of what the server responds, which in many cases is not what you want.

The newer --follow option instead acts according to what the HTTP response code suggests in should do. Stick to the same method again, or maybe switch to GET in the following request.

Why?

Why or when would you use this? First of course you only want to use this if the server supports it, but the spec offers some reasons why this might be a good choice:

  • avoid or circumvent URL size limits. Somewhere around 8000 bytes they start to no longer work reliably because servers and intermediaries set limits.
  • expressing certain kinds of data in the URL is inefficient because encoding overhead
  • URLs are more likely to be logged than request content

cURL and libcurl

Improving curl -J

2 Comments

We introduced curl’s -J option, also known as --remote-header-name back in February 2010. A decent amount of years ago.

The option is used in combination with -O (--remote-name) when downloading data from a HTTP(S) server and instructs curl to use the filename in the incoming Content-Disposition: header when saving the content, instead of the filename of the URL passed on the command line (if provided). That header would later be explained further in RFC 6266.

The idea is that for some URLs the server can provide a more suitable target filename than what the URL contains from the beginning. Like when you do a command similar to:

curl -O -J https://example.com/download?id=6347d

Without -J, the content would be save in the target output filename called ‘download’ – since curl strips off the query part.

With -J, curl parses the server’s response header that contains a better filename; in the example below fun.jpg.

Content-Disposition: filename="fun.jpg";

But redirects?

The above approach mentioned works pretty well, but has several limitations. One of them being that the obvious that if the site instead of providing a Content-Disposition header perhaps only redirects the client to a new URL to the download from, curl does not pick up the new name but instead keeps using the one from the originally provided URL.

This is not what most users want and not what they expect. As a consequence, we have had this potential improvement mentioned in the TODO file for many years. Until today.

We have now merged a change that makes curl with -J pick up the filename from Location: headers and it uses that filename if no Content-Disposition.

This means that if you now rerun a similar command line as mentioned above, but this one is allowed to follow redirects:

curl -L -O -J https://example.com/download?id=6347d

And that site redirects curl to the actual download URL for the tarball you want to download:

HTTP/1 301 redirect

Location: https://example.org/release.tar.gz

… curl now saves the contents of that transfer in a local file called release.tar.gz.

If there is both a redirect and a Content-Disposition header, the latter takes precedence.

The filename is set remotely

Since this gets the filename from the server’s response, you give up control of the name to someone else. This can of course potentially mess things up for you. curl ignores all provided directory names and only uses the filename part.

If you want to save the download in a dedicated directory other than the current one, use –output-dir.

As an additional precaution, using -J implies that curl avoids to clobber, overwrite, any existing files already present using the same filename unless you also use –clobber.

What name did it use?

Since the selected final name used for storing the data is selected based on contents of a header passed from the server, using this option in a scripting scenario introduces the challenge: what filename did curl actually use?

A user can easily extract this information with curl’s -w option. Like this:

curl -w '%{filename_effective}' -O -J -L https://example.com/download?id=6347d

This command line outputs the used filename to stdout.

Tweak the command line further to instead direct that name to stderr or to a specific file etc. Whatever you think works.

Remaining restrictions

The content-disposition RFC mentioned above details a way to provide a filename encoded as UTF-8 using something like the below, which includes a U+20AC Euro sign:

Content-Disposition: filename*= UTF-8''%e2%82%ac%20rates

curl still does not support this filename* style of providing names. This limitation remains because curl cannot currently convert that provided name into a local filename using the provided characters – with certainty.

Room for future improvement!

Ships

This -J improvement ships in curl 8.19.0, coming in March 2026.

cURL and libcurl

chart: which host, which protocol

1 Comment

A flow chart describing some steps and decisions done within curl when a HTTP URL is provided. For hostnames, protocol and port numbers.

This flow chart ignores proxies, authentication considerations and use of unix domain sockets to keep things simpler.

URL

An initial step is of course to extract the hostname part from the URL. The hostname in a URL can be provided as a plain IP address or as a name. If a numerical IPv4 or IPv6 address are not provided in the URL, curl checks if the hostname is provided using IDN (International Domain Names) and if so, it converts the name into punycode that it then can continue with.

Existing connection

Given the protocol, the hostname and port number curl checks if it has an existing connection alive suitable for use. Reusing an existing connection is preferred as it is the fastest way to start the new transfer. Connection reuse is done based on the provided name and not the IP address so that curl can skip figuring that out if there already is a connection available.

–connect-to

When trying to connect to a host, curl first checks if there are any tricks selected, like this option that makes curl actually resolve hostname B even when asked to connect to host A.

alt-svc

curl might have a populated alt-svc cache from previous transfers. It is basically a mapping for specific HTTP versions and hostnames over to another HTTP version and hostname for a certain amount of time. This can change hostname A into hostname B.

–resolve

This is an option that populates the DNS cache with one or more user provided IP addresses for a given hostname.

DNS cache

Before curl resolves a hostname into a set of IP addresses, it checks if it already has the information in its DNS cache, as that is usually much faster than having to ask for that data again. Entries are typically only kept in this cache for a minute until evicted.

Resolving

When curl resolves a hostname, it wants the A, AAAA and HTTPS DNS records data. A and AAAA provides a list of IP addresses to try to connect to, and the HTTPS field provides HTTP version information, port number, ECH config and possibly more.

HSTS

curl might also have an HSTS cache, which is another map for when plain HTTP accesses should rather be internally upgraded to instead use HTTPS. This changes protocol to use and default port number.

Racing

Depending on what IP versions and HTTP versions the above steps have determined curl should try to use, curl starts a connection race with potentially quite a few parallel connection attempts, each started a little delayed after the previous.

  1. QUIC connect attempt over IPv6 starts first
  2. QUIC connect attempt over IPv4 runs as number two
  3. TCP connect attempt over IPv6 is third in line
  4. TCP connect attempt over IPv4 is the fourth

Of course, if any of them can’t be done or fails, they are immediately skipped and the next one in line starts. Each of them also possibly start a new one if the previous one has not connected with a certain time.

The first contender to successfully connect to the host wins and the other attempts are quickly discarded.

curl connection races to curl.se

TLS handshake

If the protocol is HTTPS (which it always is if HTTP/3 is selected), the TLS handshake is performed after the TCP connection is established. For HTTP/3, the TLS handshake is integrated into the QUIC connection setup.

The TLS handshake can make curl reuse an existing session, decide ALPN, use ECH and send early data.

The session id/ticket handling is also a cache curl holds that allows for faster reconnects to hosts it has connected to before.

Connection

Once curl has an established connection to use, it starts with sending off the HTTP request, which begins the transfer.

The chart

cURL and libcurl

HTTP is not simple

12 Comments

I often hear or see people claim that HTTP is a simple protocol. Primarily of course from people without much experience or familiarity with actual implementations. I think I personally also had thoughts in that style back when I started working with the protocol.

After personally having devoted soon three decades on writing client-side code doing HTTP and having been involved in the IETF on all the HTTP specs produced since 2008 or so, I think I am in a decent position to give a more expanded view on it. HTTP is not a simple protocol. Far from it. Even if we presume that people actually mean HTTP/1 when they say that.

HTTP/1 may appear simple because of several reasons: it is readable text, the most simple use case is not overly complicated and existing tools like curl and browsers help making HTTP easy to play with.

The HTTP idea and concept can perhaps still be considered simple and even somewhat ingenious, but the actual machinery is not.

But yes, you can telnet to a HTTP/1 server and enter a GET / command manually and see a response. However I don’t think that is enough to qualify the entire thing as simple.

I don’t believe anyone has tried to claim that HTTP/2 or HTTP/3 are simple. In order to properly implement version two or three, you pretty much have to also implement version one so in that regard they are accumulating complexity and bring quite a lot of extra challenges in their own respective specifications.

Let me elaborate on some aspects of the HTTP/1 protocol that make me say it is not simple.

newlines

HTTP is not only text-based, it is also line-based; the header parts of the protocol that is. A line can be arbitrarily long as there is no limit in the specs – but they need to have a limit in implementations to prevent DoS etc. How long can they be before a server rejects them? Each line ends with a carriage-return and linefeed. But in some circumstances only a linefeed is enough.

Also, headers are not UTF-8, they are octets and you must not assume that you can just arbitrarily pass through anything you like.

whitespace

Text based protocols easily gets this problem. Between fields there can be one or more whitespace characters. Some of these are mandatory, some are optional. In many cases HTTP also does tokens that can either be a sequence of characters without any whitespace, or it can be text within double quotes (“). In some cases they are always within quotes.

end of body

There is not one single way to determine the end of a HTTP/1 download – the “body” as we say in protocol lingo. In fact, there are not even two. There are at least three (Content-Length, chunked encoding and Connection: close). Two of them require that the HTTP client parses content size provided in text format. These many end-of-body options have resulted in countless security related problems involving HTTP/1 over the years.

parsing numbers

Numbers provided as text are slow to parse and sometimes error-prone. Special care needs to be taken to avoid integer overflows, handle whitespace, +/- prefixes, leading zeroes and more. While easy to read for humans, less ideal for machines.

folding headers

As if the arbitrary length headers with unclear line endings are not enough, they can also be “folded” – in two ways. First: a proxy can merge multiple headers into a single one, comma-separated – except some headers (like cookies) that cannot. Then, a server can send a header as a continuation of the previous header by adding leading whitespace. This is rarely used (and discouraged in recent spec versions), but a protocol detail that an implementation needs to care about because it is used.

never-implemented

HTTP/1.1 ambitiously added features that at the time were not used or deployed onto the wide Internet so while the spec describes how for example HTTP Pipelining works, trying to use it in the wild is asking for a series of problems and is nothing but a road to sadness.

Later HTTP versions added features that better fulfilled the criteria that Pipelining failed to: mostly in the way of multiplexing.

The 100 response code is in similar territory: specified, but rarely actually used. It complicates life for new implementations. The fact that there is a discussion this week about particulars in the 100 response state handling, twenty-eight years since it was first published in a spec I think tells something.

so many headers

The HTTP/1 spec details a lot of headers and their functionality, but that is not enough for a normal current HTTP implementation to support. This, because things like cookies, authentication, new response codes and much more that an implementation may want to support today are features outside of the main spec and are described in additional separate documents. Some details, like NTLM, are not even found in RFC documents.

Thus, a modern HTTP/1 client needs to implement and support and a whole range of additional things and headers to work fine across the web. “HTTP/1.1” is mentioned in at least 40 separate RFC documents. Some of them quite complex by themselves.

not all methods are alike

While the syntax should ideally be possible to work exactly the same independently of which method that is used (sometimes referred to as verb), that is not how the reality works.

For example, if the method is GET we can also indeed send a body in the request similar to how we typically do with POST and PUT, but due to how it was never properly spelled out in the past, that is not interoperable today to the extent that doing it is just recipe for failure in a high enough share of attempts across the web.

This is one of the reasons why there is now work on a new HTTP method called QUERY which is basically what GET + request body should have been. But that does not simplify the protocol.

not all headers are alike

Because of the organic way several headers were created, deployed and evolved, a proxy for example cannot blindly just combine two headers into one, as the generic rules say it could. Because there are headers that specifically don’t follow there rules and need to be treated differently. Like for example cookies.

spineless browsers

Remember how browser implementations of protocols always tend to prefer to show the user something and guess the intention rather than showing an error because if they would be stringent and strict they risk that users would switch to another browsers that is not.

This impacts how the rest of the world gets to deal with HTTP, as users then come to expect that what works with the browsers should surely also work with non-browsers and their HTTP implementations.

This makes interpreting and understanding the spec secondary compared to just following what the major browsers have decided to do in particular circumstances. They may even change their stances over time and they may at times contradict explicit guidance in the specs.

size of the specs

The first HTTP/1.1 RFC 2068 from January 1997 was 52,165 words in its plain text version – which almost tripled the size from the HTTP/1.0 document RFC1945 at merely 18,615. A clear indication how the perhaps simple HTTP 1.0 was no longer simple anymore in 1.1.

In June 1999, the updated RFC 2616 added several hundred lines and clocked in at 57,897 words. Almost 6K more words.

A huge work was then undertaken within the IETF and in the fifteen years following the single document HTTP/1.1 spec was instead converted into six separate documents.

RFC7230 to RFC7235 were published in June 2014 and they hold a total of 90,358 words. It had grown another 56%. It is comparable to an average sized novel in number of words.

The whole spec was subsequently rearranged and reorganized again to better cater for the new HTTP versions, and the latest update was published in June 2022. The HTTP/1.1 parts had then been compacted into three documents RFC 9110 to RFC9112, with a total of 95,740 words.

For the argument sake, let’s say we can read two hundred words per minute when plowing this. It is probably a little slower than average reading speed, but I imagine we read standard specs a little slower than we read novels for example. Let’s say that 10% of the words are cruft we don’t need to read.

If we read only the three latest HTTP/1.1 related RFC documents non-stop, it would still take more than seven hours.

Must die?

In a recent conference talk with this click bait title, it was suggested that HTTP/1 is so hard to get implemented right that we should all stop using it.

Necessarily so?

All this, and yet there are few other Internet protocols that can compete with HTTP/1 in terms of use, adoption and popularity. HTTP is a big shot on the internet. Maybe this level of complication has been necessary to reach this success?

Comparing with other popular protocols still in use like DNS or SMTP I think we can see similar patterns: started out as something simple a long time ago. Decades later: not so simple anymore.

Perhaps this is just life happening?

Conclusion

HTTP is not a simple protocol.

The future is likely just going to be even more complicated as more things are added to HTTP over time – for all versions.

Network, Web

Workshop season six, episode three

One positive thing among many others at this version of the HTTP Workshop (day one, day two) is the fact that there have been several new faces showing up here. People who have not previously attended any HTTP Workshops. Getting fresh blood into the mix is great. A chance to maybe lower the average age of the attendees also feels welcome.

This half day was the final session for this time. Three topics were dealt with.

Do you speak HTTP? Getting your HTTP implementation to do right according to the specification can be a challenge. There is a whole range of existing tests for various areas of HTTP but there might still be a place to add HTTP semantic tests in particular for servers. Discussions brought about reflections around testing, doing tests, test formats, other tests, test infrastructure and more. I think the general sense was that yes it would be great. At least if someone else makes it happen…

Every HTTP stack is an intermediary – HTTP semantics is the (requirement for low-level) API. Yes. Lots of nodding around the huge table.

Workshop feedback and thoughts. What is a good cadence for future events, how long should the events be etc. This is probably the maximum amount of attendees we can handle using the same setup. This event was clearly better than several of the past ones in terms of diversity, but I will second our “workshop maestro” in that it could improve further still. We also discussed whether do-arranging together with IETF is good or bad, should it then be before or after IETF?

I think the consensus said that making it biannual event is good. The reasoning for keeping the event in Europe has been because a larger share of the European attendees come from smaller companies compared to the non-Europeans which to a larger degree come from larger companies that might have it easier to pay for longer trips.

My personal take

The HTTP Workshop is a one-of-a-kind event. At these events everything is about and around HTTP with an information density level that is super high. We get to learn how things actually work for people or that do not work. And that we are not alone in whatever struggles or HTTP challenges we have.

Networking with other doers here and absorbing every protocol detail being expressed, gives food for thoughts and lessons to take advantage from in years to come when we for sure are going to take HTTP transfers further. This is in many ways a kind of brain fertilizer event.

Did I mention I enjoyed it? I will certainly try to attend the next one.

Network, Web

The 2024 Workshop, day two

The fun continues. See day one.

In an office building close to the Waterloo station in London, around 40 persons again sat down at this giant table forming a big square that made it possible for us all to see each other. One by one there were brief presentations done with follow-up discussions. The discussions often reiterated old truths, brought up related topics and sometimes went deep down into the weeds about teeny weeny details of the involved protocol specs. The way we love it.

The people around the table represent Ericsson, Google, Microsoft, Apple, Meta, Akamai, Cloudflare, Fastly, Mozilla, Varnish. Caddy, Nginx, Haproxy, Tomcat, Adobe and curl and probably a few more I forget now. One could say with some level of certainty that a large portion of every day HTTP traffic in the world is managed by things managed by people present here.

This morning we all actually understood that the south entrance is actually the east one (yeah, that’s a so called internal joke) and most of us were sitting down, eager and prepared when the day started at 9:30 am.

Capsule. The capsule protocol (RFC 9297) is a way to, put simply, send UDP packets/datagrams over old style HTTP/1 or HTTP/2 proxies.

Cookies. With the 6265bis effort well on its way to ship as an updated RFC, there is an effort and intent to take yet another stab at improving and refreshing the cookie spec situation. In particular to better split off browser management and API related stuff from the more network-oriented over-the-wire details. You know yours truly never ceases an opportunity to voice his opinion on cookies… I approve of this attempt as well, as I think increasing clarity and improving the specification situation can’t but to help improve things.

Declarative web push. There’s an ongoing effort to improve web push – not to be confused with server push, so that it can be done easier and without needing JavaScript to manage it in the client side.

Reverse HTTP. There are origins who want to contact their CDNs without having to listening on any ports/sockets and still be able to provide content. That’s one of the use cases for Reverse HTTP and we got to learn details from internet drafts done on the topics for the last fifteen years and why it might still be a worthwhile effort and why the use cases still exist. But is there an enough demand to put it into HTTP?

Server Stack Detection. A discussion around how someone can detect the origin of the server stack of any given HTTP server implementation. Should there be a better way? What is the downside of introducing what would basically be the server version of the user-agent header field? Lots of productive discussions on how to avoid recreating problems of the past but in a reversed way.

MoQ: What is it and why is it not just HTTP/3? Was an educational session about the ongoing work done in this working group that is wrongly named and would appreciate more input from the general protocol community.

New HTTP stack. A description of the journey of a full HTTP stack rewrite: how components can be chained together and in which order and a follow-up discussion about if this should be included in documentation and if so in which way etc. Lessons included that the spec is one thing, the Internet is another. Maybe not an entirely new revelation.

Multiplexing in the year 2024. There are details in HTTP/2 multiplexing that does not really work, there are assumptions that are now hard to change. To introduce new protocols and features in the modern HTTP stack, things need to be done for both HTTP/2 and HTTP/3 that are similar but still different and it forces additional work and pain.

What if we create a way to do multiplexing over TCP, so called “over streams”, so that the HTTP/3 fallback over TCP could still be done using HTTP/3 framing. This would allow future new protocols to remain HTTP/3-only and just make the transport be either QUIC+UDP or Streams-over TCP+TLS. This triggered a lot of discussions, mostly positive and forward-looking but also a lot of concerns raised about additional work and yet another protocols component to write and implement that then needs to be supported until the end of times because things never truly go away completely.

I think this sounds like a fun challenge! Count me in.

End of day two. I need a beer or two to digest this.

Network, Web

The 2024 HTTP Workshop

2 Comments

Day one.

For the sixth time, this informal group of HTTP implementers and related “interested parties” unite in a room over a couple of days doing a HTTP Workshop. Nine years since that first event in Münster, Germany.

If you are someone like me, obsessed with networking and HTTP in particular this is certainly the place to be. Talking and discussing past lessons, coming changes and protocol dreams in several days with like-minded people is a blast. The people on these events are friends that I don’t always get to hang out with too often. Many of the attendees here have been involved in this community for a long time and have attended all or most of the past workshops as well. I have fortunately been able to attend all of them so far.

This time we are in London.

Let me tell you a little about the topics of day one – without spilling the beans about exactly who said what or what the company they came from. (I will add links to the presentations later once I know where to link to.)

Make Cleartext HTTP harder. The first discussion point of the day. While we have made HTTPS easier over recent years it can only take us so far. What if we considered means to make HTTP harder to use as the next level efforts to further reduce its use over the internet. The HSTS preload list is only growing. Should we instead convert it into a HSTS exclude list that can shrink over time? This triggered a looong a discussion in the group which brought back a lot of old arguments and reasoning from days I thought we had left behind long ago.

HTTP 2/3 abuses. A prominent implementer of a HTTP proxy/load balancer walked us through a whole series of different HTTP/2 and HTTP/3 protocols details that attackers can have been exploiting in recent years, with details about what can be done to mitigate such attacks. It made several other implementers mention how they take similar precautions and some other general discussions around the topics of what can be considered normal use of the protocol and what is not.

Idle connections & mobile: beneficial or harmful. 0-RTT instead of idle connections. There is a non-negligible cost associate with keeping connections alive for clients running on mobile phones. Would it be possible to instead move forward into a world when they are not kept alive but instead closed and 0-RTT opened again next time they are needed? Again the room woke up to a long discussion about the benefits and problems with doing this – which if it would be possible probably would save a lot of battery time on the average mobile phones.

QUIC pacing. We learned that it is very important for servers to implement decent QUIC pacing as it can increase performance up to 20 times compared to no pacing at all. What about using the flow control properly? What about changing the default buffer sizes for UDP sockets in the Linux kernel to something similar to TCP sockets in order to help the default case to perform better?

HTTP prioritization for product performance. The HTTP/2 way of doing prioritization was deemed a failure and too complicated a long time ago but in this presentation we were taught that there are definitely use cases and scenarios where the regular HTTP/3 priority setup is helpful and improves performs. Examples and descriptions for a popular and well used client were shown.

Allowing HTTP clients to use stale DNS data. What if HTTP clients would use stale data instead of having to wait for the DNS resolver response as a means to avoid having to wait a whole RTT to get the date that in a fair amount of the cases is the same as the stale data. Again a long discussion around TTLs for DNS queries and the fact that some clients are already doing this, in more or less explicit ways.

QUIC cache DSR. As the last talk of the the afternoon we got in the details of Direct Server Response for QUIC and how this can improve performance and problems and challenges involved with this. It then indirectly took us into a long sub-thread talking about HTTP caching, Vary headers and what could and should be done to improve things going forward. There seems to be an understanding that it would be good to improve the current situation but it is not entirely clear to this author exactly what that would entail.

When we then took a walk through the streets of London, only to have an awesome dinner during which we could all conclude that HTTP still is not ready. There is still work to be done. There are challenges left to overcome.

See you tomorrow for day two.