You wanted WebSockets?

WebSockets has been one of the most requested features and protocol to add to curl and libcurl in the annual user survey. Repeatedly, over the last few years.

WebSockets is not perfectly suitable to be done by libcurl since it’s not really an upload or download transfer protocol, but is more something like “a TCP for JavaScript”. It provides a bidirectional data stream over HTTP. (I was there when it was created, first mentioned on my blog here.)

Ignoring that technicality, WebSockets is often used more or less for a one-directional data stream. Commonly together with the use of other protocols that curl already supports. If libcurl would support it, there will be plenty of applications out there that could simplify their code.

Today, users use a mix of libcurl, custom code on top or “over” libcurl and other WebSockets libraries. There’s no single de-facto way or practice to do WebSockets with libcurl.

WebSockets for libcurl?

I took the topic of drafting a WebSockets API for libcurl to the libcurl mailing list a while ago and after a lot of back and forths and feedback from multiple people, we have a decent beginning of a WebSockets API that might work jotted down.

This is just a potential API described in a document. How it could be made to work. Nobody has actually implemented any of it.

Implement?

We know users ask for WebSockets, repeatedly and several people helped contributing to the tentative API design.

It’s just that this time I decided to pause and see if I couldn’t get some help in implementing this. To create a team of implementers willing to work before I dive in, alternatively to find someone who’d sponsor this work to allow me to spend more and dedicated time on it. I decided to do this, because I already have a lot of other things on my plate and I have to focus on my paying (curl) customers. I estimate that implementing WebSockets support is quite a lot of work.

If nobody is willing to put in the work or money to make it happen, then maybe that’s rather clear message that this is not a feature that is meant to be provided by curl. At least not now.

WebSockets future

WebSockets was created in the HTTP/1.1 era, and is probably still mostly done using that protocol as bootstrap. There are indications hinting that the future might hold less WebSockets.

It took a long time but eventually a way to do WebSockets over HTTP/2 was provided via RFC 8441, “Bootstrapping WebSockets with HTTP/2”, published in 2018. This allows a WebSockets connection to be done over a single HTTP/2 stream.

The next evolutionary step seems to rather be WebTransport. It is a new take and protocol and is meant to be used over HTTP/3 and QUIC. It is described to “send data to and receive data from servers. It can be used like WebSockets but with support for multiple streams, unidirectional streams, out-of-order delivery, and reliable as well as unreliable transport.”

Credits

Image by pisauikan from Pixabay

curl 7.79.0 – secure local cookies

The curl factory has once again cranked out a new curl release.

Release presentation

Numbers

the 202nd release
3 changes
56 days (total: 8,580)

128 bug-fixes (total: 7,270)
186 commits (total: 27,651)
0 new public libcurl function (total: 85)
0 new curl_easy_setopt() option (total: 290)

0 new curl command line option (total: 242)
62 contributors, 25 new (total: 2,484)
41 authors, 16 new (total: 948)
3 security fixes (total: 111)
3,500 USD paid in Bug Bounties (total: 16,900 USD)

Security

This time we once again announce security advisories in association with the release.

CVE-2021-22945 is a double-free flaw in the MQTT code. Patch your old curl or upgrade to this version if you use it to send MQTT. The reporter of this flaw was awarded 1,000 USD from the curl bug-bounty program.

CVE-2021-22946 is a bug in response handling for several protocols (IMAP, POP3 and FTP) that bypasses the enforced TLS check so that even transfers that are explicitly told to require TLS can accidentally silently be performed in clear text! Rewarded 1,000 USD.

CVE-2021-22947 allows a mitm attacker to inject data into the protocol stream for FTP, IMAP, POP3 or SMTP in a way before the TLS upgrade so that curl accepts that data and uses it after after having upgraded to TLS. The untrusted data slips in and gets treated as trusted! Rewarded 1,500 USD.

These two latter ones came as an indirect result/inspiration from the NO STARTTLS research.

Changes

This release comes with three changes to take note of…

Users of the bearssl TLS backend will appreciate that it too now supports the CURLOPT_CAINFO_BLOB option so that the CA certificate easily can be provided in-memory by applications.

The cookie engine in curl now considers http://localhost to be secure and thus cookies that are marked “secure” will be sent over it – even when not using HTTPS. This is done because curl now since a while back makes sure that localhost is always truly local.

Users of the Secure Transport TLS backend can now use CURLINFO_CERTINFO to extract information about the server’s certificate chain.

Bug-fixes

Some of the most interesting bug-fixes we did this round.

use ares_getaddrinfo()

When you build curl to use the c-ares name resolver backend, curl will now use this function to get improved handling for IPv4+ IPv6. This also ups our requirement on c-ares to 1.16.0.

hyper works better

1xx responses, Transfer-Encoding and more have been fixed. The number of tests that are disabled for hyper builds are even fewer than before, but there’s still plenty of work to do before it can be considered not experimental.

cmake builds: avoid poll() on macOS

We have deliberately not used poll() in macOS builds for a long time when building with configure, and now we realized that cmake builds inadvertently had poll() use enabled, which caused curl to misbehave when for example connecting to a host while that connection got closed by the peer. poll() is now disabled on macOS even when cmake is used.

configure: also check lib64 for the OpenSSL pkg-config file

OpenSSL did a very late change just before they shipped version 3.0.0: they modified the default installation path for the library for 64 bit systems from $prefix/lib to $prefix/lib64, and subsequently we had to update our configure script detection logic accordingly. This helps configure to find OpenSSL v3 installs.

curl.1: provide examples for each option

The documentation now must provide at least one example command line for each command line option curl provides. This is verified in the build and will cause build errors if a file doesn’t comply! Feel free to suggest new, more or better examples when you start to see them in the man page.

HTTP 1.1: disallow >3-digit response codes

The HTTP protocol is defined to only allow three-digit numbers and now curl enforces that check stricter. This was in part made to align behavior when curl is built to use hyper.

HTTP 1.1: ignore content-length if any transfer-encoding is used

Non-chunked transfer-encoded content that also sends Content-Length headers is rare but was incorrectly handled by curl. Found when aligning behavior with hyper builds.

http_proxy: only wait for writable socket while sending request

Due to a mistake in the handling of what socket activity to wait for, curl could accidentally be made to busy-loop from the CONNECT request was sent to the proxy until the first data arrived.

Support mbedTLS 3.0.0

When mbedTLS released a new version with support for TLS 1.3 etc, they also modified the API a bit.

Ban strerror

We’ve had our own internal strerror replacement function for a long time (primarily due to it not being thread-safe), but a recent code review revealed that a lot of uses of this function had still crept in. Starting now, our code check tool (checksrc) will error if strerror is used in libcurl code.

The mailing lists move from cool.haxx.se to lists.haxx.se

Our old decommissioned server hosted 29 mailing lists. We moved most of them and killed off a few. All our mailing lists are now hosted on lists.haxx.se, including all the curl related ones of course! The old server name will simply redirect to the new one if you go there with a browser.

Heading towards curl eight

There’s plan for version 8 being forged! Let me just take you back a bit in time first..

The early days

When we first created libcurl, we bumped the major version number of the project from the previous version 6 to version 7. In late summer of 2000 we shipped curl and libcurl 7.1 as the first ever release that featured a separate library for Internet transfer powers. Everything before version 7 was just the command line tool, curl. That was the moment in time we decided we should leave kindergarten and we were ready to take on some tougher loads.

I had the main approach to the API worked out already from the start. It would be transfer-oriented, it would be built up around URLs and shouldn’t necessarily require that the users are themselves protocol experts to use it. I used inspiration from ioctl and fcntl when I made curl_easy_setopt and curl_easy_getinfo. They’re fixed functions with flexible arguments. The idea was that by doing that, we wouldn’t have to add new functions for new features but we would “just” add options and new options would simply just not work with older libcurls.

The first API we shipped also only provided synchronous single transfers. The “easy” interface.

I had no anticipations or particular hopes on the library then. It would be cool if someone found good use for it and it would be even cooler if someone would help out to improve it further.

It grew, I learned

I had never before developed and shipped a library for the world to use. I hadn’t really fully grasped and considered the impact of APIs and ABI stability etc.

We gradually improved the library over time. We bumped the SONAME several times in the first few years as we modified internals. In the same time the library caught on a bigger and bigger audience and in September 2006 as I ripped out code for what is commonly referred to as “FTP third party transfers” I once again bumped the ABI number (to 4) since the older libcurl was no longer compatible with the new release.

People don’t like SONAME bumps

That bump was met with quite a lot of resistance and objections among users. Changing SONAME of a widely used library it turns out causes a lot of pain, agony and squeaking. Possibly this was one of the earlier signs that libcurl had grown up and I decided that we should try to avoid going through this again. We shall not break ABI compatibility again. Ever.

No bumps, no worries

In this world with no SONAME bumps I wanted to keep that solidity visible in the major number of the project so even though the version number of the releases aren’t strictly related to the SONAME we kept shipping curl version seven. In September 2021, we reach curl 7.79.0.

We’ve managed to stick to our goals and a binary libcurl using application built after September 2006 can run with the latest libcurl with no modification needed! This is one of the biggest edges and “selling points” we have in libcurl: We take compatibility and unmodified behavior very seriously.

In 2013 I even wrote blog post emphasizing this and in there I said there won’t be a curl version 8 “in a long time”. But read on, things have changed a little!

An ever-growing minor number

We bump the minor version number every time we “change” something in the project, or add features. If we only do bug-fixes we only bump the patch number. You know, in classic “major,minor.patch” style.

As we do curl releases at least every 8 weeks and most releases have changes added, we bump the minor number very frequently, up to 6 times a year or so.

We provide version number information for libcurl provided as a 24 bit number, using 8 bits for each field. This implies that none of the numbers can ever go above 255. We can’t ship a 7.256.0 for this practical reason.

But also, from what I’ve seen people do with and think about version numbers before, I’m concerned that increasing the minor number beyond 99 will cause confusions. Version 7.100.0 risks gettinged confused and mixed up with 7.1 or 7.10.0, two versions that are terribly old. And frankly that’s a very large minor number and it starts becoming many digits in that release version.

There exists a solution to this!

Reset minor, bump major, keep SONAME

The idea is simply to do “a Linux kernel”. We change to version 8.0.0 at a given point in time, but we stick to the same SONAME as before.

We don’t break any compatibility, there will no no API or functionality cleanups. There will just be a version number bump to lower the minor number and let us start over that journey. Reset the counter so to speak.

When do we do this? We have roughly 20 releases left before the minor counter can reach 100, 20 releases take at least 6*20 = 120 weeks. 120 weeks is 2.3 years.

Another event within this period

On March 20, 2023. About 18 months into the future, the curl project turns 25 years old. Here’s a golden opportunity! Let’s top off the 25 year celebrations with a major version number bump!

The plan

Independently of what version number we have reached to at that point, independently of if we add features or not in that release, independently of exactly how that date fits within the pre-determined release cycle and without changing any APIs, we ship curl version 8.0.0 on that day.

Turning 25 and bumping the major version number on the same day should be fun.

Version 8

I hope that a small side-effect of bumping the major number will make users still left on version 7 to slightly faster feel outdated and push for getting up to 8. It could work as a minor push to get users to catch up a bit.

The minor number will of course immediately start “climbing” again and in a worst-case scenario, we risk reaching minor number 100 again within another 17 years. Maybe we can plan another bump for the 40th birthday?

Making world-class docs takes effort

Here are six requirements that I have on a project for it to reach my gold approval for stellar docs. Then something about what I’ve done recently to further improve the docs for curl.

Your docs belong in the code repository

It needs to be next to the code so that authors and contributors can update/read the docs while working on the code or docs. Providing it in a separate repository or otherwise separated will undoubtedly lead to discrepancies sooner or later. Similar to how all wikis are always wrong.

Your docs is not extracted from code

Admit it. If you browse around you will realize that the best documented projects you find never provide that docs generated directly from code. Such generated docs can still provide value, but you will not reach gold level without more effort.

Separated docs encourage more writing and writing by people who are otherwise scared of touching the code or thinking that fixing spelling errors in the docs isn’t worth patching the code for. It also allows for other and better formatting on the docs.

Your docs features examples

Users always ask for (more) examples. You can never go wrong by providing examples. If your docs don’t have enough examples, you’re not doing the docs good enough yet.

You document every API call you provide

Fewer things make me sigh more than when I have to dig up the source code and from that try to figure out exactly how an external and publicly provided API works. Yet this still happens regularly even for libraries that have been released and maintained for decades.

In one fairly recent instance I reported such an omission in a popular library. It took them over two years to add it.

Long-living libraries should also provide information about from which versions certain functionality or options exist or can be expected to work or not work.

Your docs is easily accessible and browsed

Good docs means that we can find what we’re looking for and that the documentation flows and is easily read and understood. Ideally, even simple google searches for API details in your library should lead us to suitable entry points.

Preferably, the documentation should also be provided for proper off-line reading, meaning man pages or something similar that can be browsed when disconnected from the Internet.

Your docs should be easy to contribute to

The docs should be easy for contributors to help out with (independently from the code if desired). That also includes that they should be easy for contributors to build and render locally so that they can test and view their updates while working on them.

Documentation in the curl project

I want the documentation for curl and libcurl to be known, recognized and widely admitted to be world-class.

I want the curl documentation to be of a quality and content to make users not able to find competitors or similar projects with better docs.

Documentation in curl is not an after-thought. It is not a second-tier component. It is a crucial and important foundation that allows users to use, trust and rely on our products. We require that new changes or improved functionality are provided with the corresponding updated and accurate documentation.

We also try to verify and check the docs as much as possible with scanners , tests and tools.

Non-stop iterating is key

I maintain that our documentation is as good as it is today a lot thanks to us very rigidly sticking to our guiding principles: compatibility and not breaking existing behavior. Documentation we wrote decades ago is still valid. It gives us plenty of time to keep refining and polishing the documentation of a feature that doesn’t change. No documentation was perfect already at the first attempt, but after numerous iterations and improvements chances are it is better. Time is on our side. And we are never done, documentation can always be improved.

I’m putting in the work

A few days ago I talked documentation with someone and when doing so I thought about what guiding principles I think we should put on project documentation. What I’ve listed above basically.

In then dawned on me that the current man curl.1 man page is actually not featuring that many examples, in spite of it being 3535 lines long. I pondered a little bit on how best add some, and then dove in and extended our system that generates it from the hundreds of individual files that each describe a single command line option. They should of course all offer at least one example!

Having 242 command line options (as of right now, it will be more soon), that sudden idea that seemed simple enough turned into a quite gruesome work and I spent many hours walking over the options to make up examples. I also made sure that our build system now returns an error if there’s a command option without an example in the documentation! This way, we can be sure that also all future command line options will have examples in the man page.

This made the curl.1 man page grew with over 1200 lines!

libcurl options too

A few years ago I did a manual effort and made sure most man pages for libcurl options include examples, but I never made that into a test or anything so there’s nothing that forces us to stick to this.

Having started this journey, I decided now was the time to add that requirement to the scripts. I extended test case 1173, which already scans all man pages to verify some basic syntax, to also check that man pages for options feature an EXAMPLE section.

There are at this moment 374 stand-alone man pages for libcurl options. Only ten of them were detected to not feature good enough examples and it wasn’t very hard to fix this.

Consistency is good

Having just extended the man page checker, another idea came up.

I made the script also check that each man page features the correct set of sections, in the required order! I’m a true believer in consistency and that using the same set in the same order will make the docs easier to read and find information in, and checking for these things will make sure that all future additions will be forced stick to the same.

The mandatory sections for libcurl option man pages are right now, in this order:

NAME
SYNOPSIS
DESCRIPTION
PROTOCOLS
EXAMPLE
AVAILABILITY
RETURN VALUE
SEE ALSO

These man pages are allowed to have other sections as well, and they can be placed anywhere among the mandatory ones, but the eight section headers that has to be there has to be in that order.

Cross-references

While at it, I also extended the man page scanner to check that all references in all curl man pages to libcurl options are verified to actually refer to existing options, to find typos. Ironically this extra check turned out finding exactly no such typos in the current 463 man pages!

Future

The outcome of the work I write about here will of course be merged asap and will be part of future releases and on the website.

We should keep thinking of more ways to improve the documentation and for more ways to verify and cross-reference things mentioned in the docs to increase its accuracy and detect typos.

If you find a problem, an inferior wording or just something you think we should improve in any curl documentation, file a bug or a PR! We also try to make this as easy as possible for users to do directly from the curl website by providing bug-reporting direct links from the documentation pages.

Updates

I added the sixth “rule” days after the initial post after Willy Tarreau’s feedback.

Early me caught on TV

SVT is the Swedish national public television. They recently announced their new series Det svenska spelundret, which translated to “The Swedish game wonder” or something. It is a five episode documentary series about the history behind and the creation of the flourishing Swedish computer game industry. In Swedish and the only subtitles available are also in Swedish.

In episode two, there’s a look into the home computer demo scene of the late 80s and early 90s that more or less kick-started some of the early game studios.

You know who were very present in the Swedish demo scene in the late 1980s? Me. I even dabbled a little into games and I wrote a few, at least beginnings of games on the Commodore 64, but I didn’t have the persistence and discipline to finish those projects so they never actually ended up anywhere further than playable prototypes.

I grabbed a frame from the show. At 5:06 there’s this photo, taken at some copy party in the late 80s (in the show they say early 90s but many photos were older).

The yellow circle was added by me to better highlight yours truly. Behind me in the blue-white striped shirt is my brother Björn.

This specific picture was from the copy party in Eskilstuna Sweden 12 – 15 May 1989 that I helped organize. A few hundred teenage nerds in a school over the weekend. I was eighteen years old at the time.

We, in the form of the demo group Horizon, won the demo competition there with our demo Dolly Party (in which I contributed code using my nick name Bagder). Here’s a YouTube recording of it.

Of course a lot of friends and acquaintances from those days also flickered past on the many old golden photos that were shown. Most of them very shown very quickly. A little disappointing was that Kjell Ericson who shot several of those photos didn’t get any credit at all in the show. Thanks Kjell!

The TV show says we called those events “demo parties” when they were in fact called “copy parties” to all of us (probably due to how they once started as a chance to meet up and exchange copies of games and other software) far into the 90s when I believe they rather became LAN parties.

More?

Demos and pictures from my C64 days as member of the demo group Horizon have been collected by Kjell Ericson.

The demo days laid the solid foundation of my computer life and entire career. Slightly more about that can be found on my about me page.

A GitHub star

“The GitHub Stars program thanks GitHub’s most influential developers and gives them a platform to showcase their work, reach more people, and shape the future of GitHub.”

That’s a quote from stars.github.com. In the beginning of June 2021 I was invited into the program. I consider it an honor to be recognized. See my featured profile.

The stars program provides insights into and early access to members about what GitHub is working on next and allows me to channel back feedback on such things.

As someone who basically lives on GitHub I believe this could be useful and productive. GitHub is the first site I visit in the morning and the last one I view before I go to bed at night.

Previous GitHub presents

I got coasters and a pint glass saying “100 million repositories” some years back, I got my 3D-printed contribution graph in steel and I got a GitHub notebook at a conference once.

A GitHub Star

Today a delivery guy arrived at my door and I unpacked this 20x30x5 cm dark wooden box with a transparent plastic front showing a very shiny GitHub star and a similar shiny plaque saying

Daniel Stenberg
@bagder
Presented with <3 by GitHub in 2021

It’s hard to photograph due to all the glare!

The thing is beautiful and will get an honorary placement in my house.

Motivation

On this thick paper that came with the “starbox”, the following text was printed

Congratulations Daniel Stenberg!

We are pleased to present you with your 2021 GitHub Stars award!

The document

Thank you for the tremendous work that you do in the community by inspiring, educating and influencing all those around you. You are a true star in our eyes, which is why we wanted to say ‘Thank you’ and recognize you as part of a select band of volunteer GitHub Stars from across the world. Together we are supporting communities where more than 60 million people learn, share, and work together to build software. We’re helping make a welcome and inclusive home for all developers and helping others to join us as the next generation.

So thank you for your passion, your love for sharing your knowledge, for your support of open source communities, the amazing things that you’ve done, and the exciting things to follow!

Again, congratulations on your GitHub Stars Award!

With <3 from GitHub

Swag

The day after, this second package arrived that was shock full of GitHub swag,

Nocai’s apology

Back in February of this year, I received one of the most chilling emails in my life when someone threatened my life: “I will slaughter you“.

That email penetrated something deep into my soul and heart and I was not quite myself for a few days.

Almost six month later (a few days ago), I received another email from a person who claims to be this “Al Nocai” from that first threat. I read the beginning of this email several times before I actually could make myself continue past the opening paragraphs. Slightly worried what to find, with the memories from the last time coming back.

This email however, says “an apology” in the subject. I believe this email is written by the same guy.

The long and full apology is inserted below. The gist of it is: he claims to have been the victim of all sorts of bad stuff by several people named “Dan” and I got lumped in there because of my first name. He (?) also says he suffers from schizophrenia.

I’m publishing this partly because I got a lot of attention when I made the initial threat public so I figure it could be interesting for some to learn about this development. I can of course not verify that this is in fact the same person nor will I even attempt to verify any of his many claims of wrongdoings against him.

I’m happy for “Al” that he’s getting help and tries to move on. For me, this apology at least finally proves that this threat is over and in fact never was intended literally. I hope I will never receive anything close to that again.

The apology in full

I am Al Nocai. When I contacted you initially, I believed you to be a Dan E., from texas, or a Dan S from delaware or a Dan from Minneapolis. I didn’t do my research, and when I found it was actually you and you had nothing to do with my situation, I became indignant and even more of an asshole. You had every right to be mad, and publish as you did. I’m not trying to justify what I did, there is none, I should have been a lot more cordial. I just want to provide context around what was happening, I believe I at least owe you why.

I had to retire from my career do to schizophrenia. Again, I should have not let my delusions go to the point they did nor should I have acted the way it does. My illness doesn’t detract from the rashness of my actions.

I, at the time, lost my defense project after getting hacked through the Department of Veterans Affairs in the US. It was a few years into development, and it was meant to be a pathway to get homeless veterans off the streets. I was trying to develop a “trade-route” in tech. At the time I was very lucid. I was volunteering regularly in my community and I had a great family life. In October 2020, there was an attempt on my life which left me bludgeoned and near death with 2 orbital fractures. These event led me to uncover a massive money laundering operation and then had ties elsewhere. I then got hacked. When I say hacked, I lost every device. They rooted my charge arbitrator, I was bios bonded and I basically lost every document and all my software.

The people who did it gloated to me about it to me through linkedin. Trying to take my computers back, I ended up QAing a lot of their malware. This led to be being whaled for months. People attempted to blackmail me, they stole my identity and I was lied to about what I was doing as the people approaching me hid their relations to the people who initially hacked me.

These stressors where the predicates to my psychotic break. My federal correspondence about the hacking was re-routed. I had people impersonating microsoft employees. I even had a $50 billion dollar mortgage servicer trying to sue me over tweets. I also had firms like outpost 24 doing MiTM attacks on me, which is also a factor as I didnt realize at first where all my EU Certs were coming from in my Active directory.

Last, just to give you a clearer picture at how much I was monitored: I ended up not being able to take anything back in Windows. They were in my Windows Registry directly from install. Finding what I did, I believe I misattributed a good amount of sloppy programming to malicious behavior due to stress and paranoia.

So why did I believe you to be someone else? All the people responsible for these actions, well most of them were named Dan. They were mostly ex-google employees. Which is why my results changed and I ended up with about 19k various porn accounts in my name. This, they were also posing as random people named Dan (i.e. when IO tried telling Convercent, Microsoft’s auditor) about the issues. They were named Dan, and they were fraudulent.

So when I found what I did, and as well maintained as I did, I jumped the gun and assumed you were, again, one of the individuals I had whaling me. And at that point, after going to every agency I could, state and local, about the issue, I was a vicious dog backed into a corner. And I also at the time didn’t mean I’d physically harm you. I meant I was going to keep taking out your site.

Again, this doesn’t excuse my behavior. And in the end after all I tried to do to report everything that was happening. I ended up making the people who whaled me a lot of money. And I ended up still losing everything, but I also hurt and was a real asshole to a lot of people that had nothing to do with anything.

In the end, I had a hard time realizing what I did and the curl reason was a piss poor one. Again, I was just being an indignant ass. Especially to someone who actually reported security issues.

Sorry it took me a long time to write this. I should have apologized right after I knew I was wrong. I apologize for that mistake as well.

This amends is also a part of my getting better. So I apologize if this angered you. I just needed to make sure I tell you that I was wrong and I should have had better judgement.

I hope this message finds you well,
nocai

curl 7.78.0 five in one

Welcome to another release! We did more bug-fixes than in any previous release (176). We paid more in bug-bounties than during any previous release cycle (4,200 USD) and we thank more contributors in the RELEASE-NOTES than ever before (83).

Release presentation

Numbers

the 201st release
6 changes
56 days (total: 8,524)

176 bug-fixes (total: 7,142)
263 commits (total: 27,465)
0 new public libcurl function (total: 85)
0 new curl_easy_setopt() option (total: 290)

0 new curl command line option (total: 242)
83 contributors, 49 new (total: 2,459)
56 authors, 32 new (total: 933)
5 security fixes (total: 108)
4,200 USD paid in Bug Bounties (total: 13,200 USD)

Security

This time we announce no less than 5 separate security advisories and we are once again setting a new bug-bounty record. This release cycle we spent 4,200 USD on rewarding security researchers.

Let’s do them in numerical order. Click the CVE links to get to the full and much more detailed advisories.

CVE-2021-22922: Wrong content via metalink not discarded

This was one of the problems we found that that all together made us take the drastic decision to completely remove metalink support.

The metalink format has a hash for the content so that a client can detect faulty contents. curl didn’t act properly if the has mismatched and it could easily make users not realize the bad content.

CVE-2021-22923: Metalink download sends credentials

If you download the metalink file using credentials, the subsequent download(s) of the file mentioned in that XML file will also get the same credentials passed to those servers, unexpectedly, thus potentially leaking sensitive information to other parties!

CVE-2021-22924: Bad connection reuse due to flawed path name checks

libcurl keeps previously used connections in a connection pool for subsequent transfers to reuse, if one of them matches the setup.

Due to errors in the logic, the config matching function did not take ‘issuer cert’ into account and it compared the involved paths case insensitively, which could lead to libcurl reusing wrong connections!

CVE-2021-22925: TELNET stack contents disclosure again

Possibly the most embarrassing security flaw in a long time.

When we shipped 7.77.0 we announced CVE-2021-22898, which was a flaw in the telnet code and an associated fix. Know what? The fix was incomplete and plain wrong so the original problem actually remained for a certain set of input.

This is thus the second advisory for the same problem and now we fix this again. Hopefully for real and for good this time…

CVE-2021-22926: CURLOPT_SSLCERT mixup with Secure Transport

When libcurl is built to use the macOS native TLS library Secure Transport, an application can ask for the client certificate by name or with a file name – using the same option. If the name exists as a file, it will be used instead of by name. This could be exploited in rare circumstances.

Changes

The six big changes this time around are:

curl_url_set now rejects spaces in the URL unless specifically asked to allow them.

CURLE_SETOPT_OPTION_SYNTAX is a brand new return code (name) for when libcurl detects an illegally formatted input passed to a setopt(), when it is detected later in the transfer.

localhost is now always local!

The mbedTLS backend now supports client certificate and key provided as “blob options” in memory instead of as files.

Metalink support was dropped.

Now username and password can be used for MQTT transfers.

Bug-fixes

I’m doing this release in the midst of my vacation so I’m doing this section a little shorter than usual. Here are some bug-fixes to highlight:

Lots of tiny fixes when built to use hyper for HTTP. Now curl built to use hyper can run many more test cases. There’s more to do and more will be done going forward.

Travis CI is gone. Zuul and Circle CI are in.

GnuTLS: set the preferred TLS versions in correct order. Previously the occasional TLS connection would fail because of the wrong way the code would instruct GnuTLS…

on macOs: free returned memory of SCDynamicStoreCopyProxies. A small memory leak on Apple operating systems, possibly as many as one per name resolve?

HSTS: not experimental anymore. It is now built and provided by default.

netrc: skip ‘macdef’ definitions. The netrc parser is ancient but it turned out this kind of macro use could threw it off.

OpenSSL: don’t remove session id entry in disassociate. We had a regression that basically killed session-id use and made subsequent TLS handshakes to the same host much slower.

Next

The plans says we ship the next release on September 15th 2021. See you then!

curl reaches 100K raised

I’m proud and happy to mention that curl just passed the magic limit of 100,000 USD in raised sponsorship money. Or call it donations if you want. Since April 2018. That’s about 40 months.

Screenshot 06:50 UTC 2021-07-06

Donate?

Do it here!

Donations over time

A grand total of 440 awesome organizations and individuals have donated money to the curl project since we started our Open Collective fund, at almost 1300 separate occasions. It makes the averages to be about 77 USD per donation and 230 USD per sponsor. As usual, there’s a very long tail of single sponsors that donated a small amount and there’s a small set of sponsors who have donated lots of money many times.

We use donated money primarily for the bug-bounty, but recently we also spread sticker love across the world with the help of donated funds. The fund will also be used to pay for our annual developer meetups (that have been paused during covid) and potentially for some hardware and other infrastructure to aid the project and it’s core contributors.

Note: that we also have a set of sponsors who fund services and infrastructure directly for us without funneling the money through us. The shear value of those services are in several instances even greater in total than what the largest monetary contributors have given us.

Net vs gross details

This counts the 100K USD net amount that ended up in our fund. That is with the fees involved already deducted. Gross, that means we were given more than 100K already.

Before Open Collective you ask?

We never saw any serious donations to speak of before we started this collective. Before then we received the occasional donations to my PayPal account but they were very spurious and very far apart and never amounted to any “real money”.

Independent

I want to take this opportunity and remind readers that curl is a totally independent and stand-alone project. We’re not part of any larger/umbrella organization and we’re not run or owned by any company. It gives us total freedom to do whatever we want but it also means we need to fund things ourselves and find our own benefactors. Fortunately, we have many friends!

Top donors

  1. Backblaze
  2. Indeed

curl user survey 2021

It is time to once again tell you that people responded very similarly to how they did last year…

curl user survey 2021 analysys

Not a lot changed this year compared to last year. Perhaps the biggest three changes this year were that

1. HTTP/3, Unix domain sockets and DNS-over-HTTPS increased significantly among “used features”

2. NSS and GnuTLS both had their usage shares among used TLS libraries fall significantly.

3. My twitter account and this blog are now top-voted as the two channels people follow mostly for participation in curl related topics.

The most used protocols are of course still HTTPS and HTTP, and the newest supported protocol (GOPHERS) checks in as the least used protocol this time around.

Much more details can be found in the linked PDF. Enjoy.

tech, open source and networking