Monthly Archives: May 2025

cURL and libcurl

Leeks and leaks

Leave a comment

On the completely impossible situation of blocking the Tor .onion TLD to avoid leaks, but at the same time not block it to make users able to do what they want.

dot-onion leaks

The onion TLD is a Tor specific domain that does not mean much to the world outside of Tor. If you try to resolve one of the domains in there with a normal resolver, you will get told that the domain does not exist.

If you ask your ISP’s resolver for such a hostname, you also advertise your intentions to speak with that domain to a sizeable portion of the Internet. DNS is inherently commonly insecure and has a habit of getting almost broadcast to lots of different actors. A user on this IP address intends to interact with this .onion site.

It is thus of utter importance for Tor users to never use the normal DNS system for resolving .onion names.

Remedy: refuse them

To help preventing DNS leaks as mentioned above, Tor people engaged with the IETF and the cooperation ultimately ended up with the publication of RFC 7686: The “.onion” Special-Use Domain Name. Published in 2015.

This document details how libraries and software should approach handling of this special domain. It basically says that software should to a large degree refuse to resolve such hostnames.

curl

In November 2015 we were made aware of the onion RFC and how it says we should filter these domains. At the time nobody seemed keen to work on this, and the problem was added to the known bugs document.

Eight years later the issue was still lingering in that document and curl had still not done anything about it, when Matt Jolly emerged from the shadows and brought a PR that finally implemented the filtering the RFC says we should do. Merged into curl on March 30, 2023. Shipped in the curl 8.1.0 release that shipped in May 17, 2023. Two years ago.

Since that release, curl users would not accidentally leak their .onion use.

A curl user that uses Tor would obviously use a SOCKS proxy to access the Tor network like they always have and then curl would let the Tor server do the name resolving as that is the entity here that knows about Tor and the dot onion domain etc.

That’s the thing with using a proxy like this. A network client like curl can hand the full host name to the proxy server and let that do the resolving magic instead of it being done by the client. It avoids leaking names to local name resolvers.

Controversy

It did not take long after curl started support the RFC that Tor themselves had pushed for, when Tor users with creative network setups realized and had opinions.

A proposal appeared to provide an override of the filter based on an environment variable for users who have a setup where the normal name resolver is already protected somehow and is known to be okay. Environment variables make for horrible APIs and the discussion did not end up in any particular consensus for other solutions so this suggestion did not make it through into code.

This issue has subsequently popped up a few more times when users have had some issues, but no fix or solution has been merged. curl remains blocking this domain. Usually people realize that when using the SOCKS proxy correctly, the domain name works as expected and that has then been the end of the discussions.

oniux

This week the Tor project announced a new product of their: oniux: a command-line utility providing Tor network isolation for third-party applications using Linux namespaces.

On their introductory web page explaining this new tool they even show it off using a curl command line:

$ oniux curl http://2gzyxa5ihm7wid.onion/index.html

(I decided to shorten the hostname here for illustration purposes.)

Wait a second, isn’t that a .onion hostname in that URL? Now what does curl do with .onion host names since that release two years ago?

Correct: that illustrated command line only works with old curl versions from before we implemented support for RFC 7686. From before we tried to do in curl what Tor indirectly suggested we should do. So now, when we try to do the right thing, curl does not work with this new tool Tor themselves launched!

At least we can’t say we get do live a dull life.

Hey this doesn’t work!

No really? Tell us all about it. Of course there was immediately an issue submitted against curl for this, quite rightfully. That tool was quite clearly made for a use case like this.

So how do we fix this?

cURL and libcurl

Detecting malicious Unicode

2 Comments

In a recent educational trick, curl contributor James Fuller submitted a pull-request to the project in which he suggested a larger cleanup of a set of scripts.

In a later presentation, he could show us how not a single human reviewer in the team nor any CI job had spotted or remarked on one of the changes he included: he replaced an ASCII letter with a Unicode alternative in a URL.

This was an eye-opener to several of us and we decided we needed to up our game. We are the curl project. We can do better.

GitHub

The replacement symbol looked identical to the ASCII version so it was not possible to visually spot this, but the diff viewer knows there is a difference.

In this GitHub website screenshot below I reproduced a similar case. The right-side version has the Latin letter ‘g’ replaced with the Armenian letter co. They appear to be the same.

GitHub shows a diff. But what is actually the difference?

The diff viewer says there is a difference but as a human it isn’t possible to detect what it is. Is it a flaw? Does it matter? If done “correctly”, it would be done together with a real and expected fix.

The impact of changing one or more letters in a URL can of course be devastating depending on conditions.

When I flagged about this rather big omission to GitHub people, I got barely no responses at all and I get the feeling the impact of this flaw is not understood and acknowledged. Or perhaps they are all just too busy implementing the next AI feature we don’t want.

Warnings

When we discussed this problem on Mastodon earlier this week, Viktor Szakats provided me with an example screenshot of doing a similar stunt with Gitea which quite helpfully highlights that there is something special about the replacement:

Gitea warns that the replacement is using “ambiguous Unicode characters”

I have been told that some of the other source code hosting services also show similar warnings.

As a user, I would actually like to know even more than this, but at least this warns about the proposed change clearly enough so that if this happens I would get the code manually and investigate before accepting such a change.

Detect

While we wait for GitHub to wake up and react (which I have no expectation will actually happen anytime soon), we have implemented checks to help us poor humans spot things like this. To detect malicious Unicode.

We have added a CI job that scans all files and validates every UTF-8 sequence in the git repository.

In the curl git repository most files and most content are plain old ASCII so we can “easily” whitelist a small set of UTF-8 sequences and some specific files, the rest of the files are simply not allowed to use UTF-8 at all as they will then fail the CI job and turn up red.

In order to drive this change home, we went through all the test files in the curl repository and made sure that all the UTF-8 occurrences were instead replaced by other kind of escape sequences and similar. Some of them were also used more or less by mistake and could easily be replaced by their ASCII counterparts.

The next time someone tries this stunt on us it could be someone with less good intentions, but now ideally our CI will tell us.

Confusables

There are plenty of tools to find similar-looking characters in different Unicode sets. One of them is provided by the Unicode consortium themselves:

https://util.unicode.org/UnicodeJsps/confusables.jsp

Reactive

This was yet another security-related fix reacting on a demonstrated problem. I am sure there are plenty more problems which we have not yet thought about nor been shown and therefore we do not have adequate means to detect and act on automatically.

We want and strive to be proactive and tighten everything before malicious people exploit some weakness somewhere but security remains this never-ending race where we can only do the best we can and while the other side is working in silence and might at some future point attack us in new creative ways we had not anticipated.

That future unknown attack is a tricky thing.

Update

(At 17:30 on the same day of the original post) GitHub has told me they have raised this as a security issue internally and they are working on a fix.

cURL and libcurl

Supported curl versions and end of life

Leave a comment

The other week we shipped the 266th curl release. This counter is perhaps a little inflated since it also includes the versions we did before we renamed it to curl, but still, there are hundreds of them. We keep cranking them out at least once every eight weeks; more often than so when we need to do patch releases. There is no planned end or expected change to this system for the foreseeable future. We can assume around ten new curl releases per year for a long time to come.

curl releases over time since late 1996 to April 2025

Release versions

We have the simplest possible release branching model: there is only one long term development branch. We create releases from master when the time comes. This means that we only have one version that is the latest and that we never fix or patch old releases. No other long-living release branches.

You can still find older curl versions in the wild getting patched, but those are not done by the curl project. Just about every Linux distribution, for example, maintains several old curl versions to which they back-port security fixes etc.

As we work crazy hard at not breaking users and to maintain behaviors, users should always be able to upgrade to the latest version without risking to break their use cases. Even when that upgrade jump is enormous. (We offer commercial alternatives for those who want even stronger guarantees, but they are provided slightly separate from the Open Source project.)

Supported versions

The curl support we provide for no money in the Open Source curl project is of course always a best-effort with no promises. We offer paid support for those that need promises, guaranteed response times or just want more and dedicated attention.

We support users with their curl issues, independent of their version – if we can. It is however likely that we ask the reporters using old versions to first try their cases using a modern curl version to see if the problem is not already fixed, so that we do not have to waste time researching something that might not need any work.

If the user’s reported problem cannot be reproduced with the latest curl version, then we are done. Otherwise, again, the paid support option exists.

So, while this is not quite a supported versions concept, we focus our free-support efforts on recent releases – bugs that are reported on old versions that cannot be reproduced with a modern version are considered outdated.

Not really End of life

Because of this concept, we don’t really have end of life dates for our products. They are all just in varying degrees of aging. We still happily answer questions about versions shipped twenty years ago if we can, but we do not particularly care about bugs in them if they don’t seem to exist anymore.

We urge and push users into using the most recent curl versions at any time so that they get the best features, the most solid functionality and the least amount of security problems.

Or that they pay for support to go beyond this.

In reality, of course, users are regularly stuck with old curl versions. Often because they use an (outdated) Linux distribution which does not upgrade its curl package.

They all “work”

We regularly have users ask questions about curl versions we shipped ten, twelve or even fifteen years ago so we know old releases are used widely. All those old versions still mostly work and as long as they do what the users want and ask curl to do, then things are fine. At least if they use versions from distributions that back-port security fixes.

In reality of course, the users who still use the most ancient curl versions also do this on abandoned or end-of-lived distros, which means that they run insecure versions of curl – and probably basically every other tool and library they use are also insecure by then. In the best of worlds the users have perfect control and awareness of those details.

Feature window

Since we do all releases from the single master branch we have the feature window/freeze concept. We only allow merging features/changes during a few weeks in the release cycle, and all the other time we only merge bugfixes and non-feature changes. This, to make sure that the branch is as stable as possible by the time the release is to be shipped.

Network, Technology

1k-0036 means sad eyeballs on my LG

Leave a comment

For a to me unknown reason IPv6 connectivity has been failing to my home the last few days. When I try to curl curl.se I get to see a lot of IPv6 related failures and instead it connects to and uses one of the IPv4 addresses.

IPv6 has been working fine for me non-stop for the last few years before this. I suspect there is something on the ISP side and they are doing some planned maintenance in a few days that might change things. It’s not a big deal, everything I do on my machine just magically and transparently adapts.

Enter the TV

In my living room my family has an LG TV from a few years back. I find it pretty neat. It runs WebOS and has a bunch of streaming apps installed. Our household currently streams shows from Netflix, Disney, Max and SVT Play (The Swedish national broadcasting) on it.

What do you think happens to the TV and its apps when IPv6 does not work although hosts still resolve to a bunch of IPv6 + IPv4 addresses?

The TV OS itself, installing apps and everything works exactly as always.

Netflix: no difference. Streams as nicely and cleanly as always. SVT Play: runs perfectly.

Disney’s app gets stuck showing a rotating progress bar that never ends. Horribly user hostile.

The Max app fires up and allows me to select a media to watch, and then after I press play it sits showing the progress bar for a while until it fails with this 1k-0036 error.

The original Swedish error. My TV is set to Swedish.
The translated to English version just to keep everyone in the loop.

On a computer

Trying their services using the same account on the same network but from a computer in a browser showed no problems at all.

Tracking down the problem

The Max customer service advice on how to fix this of course started out with the standard most predictable actions:

  1. Unplug your device, keep it off for ten seconds and then start it again.
  2. The exact same procedure with your router.

Not a word or attempt to explain what the error code actually means. But then when I told the support person that these tricks did not help, they instead asked me to disable IPv6 in the TV’s network settings.

Even though I already knew I had this glitch for the moment with IPv6, it was first when I read his advise that I actually connected the two issues. To me, the problems were so unlikely to be related that I had not even considered it!

So now we know what 1k-0036 means.

Bye bye IPv6 TV

And yeps it was quickly confirmed: disabling IPv6 in the network settings for the TV now made streaming with the Max app work again. And yes, with the Disney app as well.

I was mildly negatively surprised that these two highly popular streaming apps actually do not handle happy eyeballs and selecting between IP address families better. Rather lame.

While we know curl is part of WebOS this clearly hints that it is not used for streaming using these services at least. (Because curl has supported happy eyeballs for decades already and clearly has no problem to connect to a host just because IPv6 glitches.) Not that surprising really. We already know that Netflix for example also use curl in their app but only for most things around and not the actual media stream.

Disabling IPv6 on the TV config comes with basically no downside so I will probably just leave it off now.

cURL and libcurl

curl up 2025 is over

Leave a comment

James Fuller was our man on the ground in Prague. He found the venue, organized most of the event and made it another curl up success. Thank you! The curl up series started in 2017 and is our annual curl developers meetup, but since it was converted to online only for a few years through Covid, this was only our fifth physical event.

We managed to cram in no less than sixteen curl related presentations over the weekend. Some fifteen eager hackers had gathered for this years event, as per usual a whole bunch of registered attendees decided to not show up. We did not let that stop us or bring us down.

The curl up 2025 venue, as we were closing up.

In person

I think we all feel that there is value in meeting up in person every once in a while, as it helps us cooperate and work better together the rest of the time.

This time, a handful of the regular old-timers attended, and a number of less regular contributors and even a set of local interested hackers decided to take the opportunity – which is so lovely to see. We love to share our knowledge with others, and getting fresh eyes and feedback from people with different views and experiences helps us!

We talked. At the conference itself of course but also over lunches, dinners, coffee breaks and beers.

Recordings

We streamed the entire event live on Twitch and recorded all the sessions. Because we are a tiny team without any dedicated video enthusiasts etc, we of course ran into some technical snags that made us mostly botch the two initial talks (by me and Stefan Eissing) and then we gradually improved the recordings through the event. We still have fourteen talks on video now available on YouTube.

You can also head over to the curl up 2025 agenda page, and find links to the recorded talks and their corresponding slide-sets.

I have a fair idea now how we should go about this next year! And yeah, it probably involves me getting a new laptop that can actually handle the task a little gentler. At these times, my ten year old work horse certainly shows its age. It was not even considered powerful when it was new.

Sponsors

We reimburse travel and hotel expenses for top contributors to attend curl up and we make the event a free-to-attend one to remove friction and really not make it about the cost. We can only do this of course thanks to our awesome project sponsors.

Next time

We intend to run curl up again in 2026, but we do not yet know where. Ideally in another European capital that is easy reachable by plane. If you want to help us make it happen in your city, please reach out!