Monthly Archives: April 2022

Open Source

Uncurled

– Everything I know and learned about running and maintaining Open Source projects for three decades.

For several years now, I have had a blog post series in mind to describe something about what people could expect to happen in Open Source projects. I had a few already half-started blog post drafts for some sub topics.

I couldn’t really make up my mind how to craft a series of blog posts about this wide topic in a sensible way so I kept postponing it for later. I did this for years.

A book, it has to be a book

It just dawned on my one day: the only way to get all this into a comprehensible way that also can hold all the thoughts I would like it to have, is to put it into a book. By book, I mean a document. An essay. A collection of pages. A booklet maybe. I don’t know how many words it might end up to become and I have no illusions of it ever ending up in print.

I mean to write the document in the open and provide it for free, online. Open Source style.

Day one

I grabbed my original draft for my blog series “You can expect this in your Open Source project”. I had worked on that document in the background for a long time, adding some little thing here and there over years – and it now had maybe twenty-five “lessons” listed with a short paragraph of text next to each.

I also had started three blog posts based on such lessons that were in pending state here on daniel.haxx.se in my queue of drafts.

I first copied the blog post content back into the text file from those potential blog posts, before I deleted them, and converted the entire file to markdown.

I then grouped the “lessons” I had listed in the markdown file and moved them into a few different sections. Like what to expect, code, money, people and project. I put subtitles into separate files for those five main areas.

How hard can it be?

I didn’t want to do a lot of work before I put the thing into git, and I didn’t want to run any private git repository so I had to make a new repo with a name. I went with “How hard can it be” as a working title and created the repo on GitHub. On April 6 I made the first git push with initial contents to that repository.

The first external contributor appeared after just a few minutes with the first pull-request fixing typos. Clearly people are following me on GitHub and spotted the creating of the repository and checked out what it was. I hadn’t told anyone or given any pointers.

I started expanding on subjects in the book.

Let’s get a real title

In the evening of April 7 I posted this question on Twitter:

"If I write a booklet collecting everything I know and learned about running and maintaining Open Source projects for three decades, what should I call it?"

I got a flood of replies. Lots of good ones and also lots of fun and sarcastic ones. The one that I think really talked to me the best was also the shortest: Uncurled.

  • It’s short and sweet
  • It includes a reference to curl without saying it is “a curl book” (it isn’t)
  • The topic is a bit about “untangling” and curl is a project that probably has taught me the most of what I include here
  • It sounds a little like “debriefed” from the curl project, and it is…
  • I can put it up on the domain name un.curl.dev

I figured I could possibly go with a longer subtitle that could explain the book more: “Everything I know and learned about running and maintaining Open Source projects”.

A name

I renamed the GitHub repository and added a description there. I created the URL (by adding the “un” CNAME entry in the “curl.dev” domain) and I setup gitbook.com to render the content to appear on un.curl.dev.

With a little more thoughts and then spilling some beans about my plans in my weekly report on April 8 (but not leaking the URL or repo to anyone yet) that made people provide some more ideas, I added more content.

10,000 words

By the evening of April 9, I surpassed 10,000 words of contents. Still having the contents and the order of everything pretty much in flux and not yet sorted out.

20,000 words

On April 25, I surpassed 20,000 words. It starts to look like something I can announce soon.

Getting there, but not done

The uncurled book is now in a state I think I can show off without feeling embarrassed. I believe I will still need to work on it more going forward to add and polish content and make it more coherent and less of a collection of snippets. I hope that I over time can settle down and gradually slow down the change pace. It will of course also depend a lot on the feedback I get.

Cover

Since it doesn’t exist physically and probably never will, I don’t think it actually needs a cover image, but it would probably be cool to still have one to use as an image and symbol for the book. If someone has a good idea or feels artistically inclined to make one, let me know!

cURL and libcurl

curl 7.83.0 headers bonanza

Welcome to the third curl release of the year.

Release presentation

curl 7.83.0 release presentation

Numbers

the 207th release
6 changes
53 days (total: 8,804)
125 bug-fixes (total: 7,816)
185 commits (total: 28,507)
2 new public libcurl function (total: 88)
0 new curl_easy_setopt() option (total: 295)
2 new curl command line option (total: 247)
60 contributors, 29 new (total: 2,626)
35 authors, 13 new (total: 1,027)
4 security fixes (total: 115)
0 USD paid in Bug Bounties (total: 16,900 USD)

Security fixes

The reason the Bug Bounty amount above is still at zero dollars for this cycle is that the rewards have not been set yet. There will be money handed out for all of them.

CVE-2022-22576– OAUTH2 bearer bypass in connection re-use

curl might reuse wrong connections when OAUTH2 bearer tokens are used.

CVE-2022-27774 – Credential leak on redirect

When curl follows a redirect to another protocol or to another port number, it could keep sending the credentials over the new connection and thus leak sensible information to the wrong party.

CVE-2022-27775 – Bad local IPv6 connection reuse

curl could reuse the wrong connection when asking to connect to an IPv6 address using zone id, as the zone id was not correctly checked when picking connection from the pool.

CVE-2022-27776 – Auth/cookie leak on redirect

curl’s system to avoid sending custom auth and cookies to other hosts after redirects did not take port number or protocol into account, and could leak sensible information to the wrong party.

Changes

While the number of changes can be counted to six, I will group them under four subtitles.

Cherry-pick headers

(These features are all landed as experimental to start with so you need to make sure to enable these in the build if you want to play with them.)

Two new functions have been introduced, curl_easy_header() and curl_easy_nextheader(). They allow applications to get the contents of specific HTTP headers or iterate over all of them after a transfer has been done. Applications have been able to get access to headers already before, but these functions bring a new level of ease and flexibility.

The command line tool was also extended to use these functions to allow easy header output to the --write-out option, both individual headers and also all headers as a JSON object. Read further.

--no-clobber

Long time TODO listing was now made into reality. Using this option, you can ask curl to not overwrite a local file even if you have specified it as an output file name in curl a command line.

--remove-on-error

The second of the new command line options: tell curl to remove the possibly partial file that might have been downloaded when it detects and returns an error.

msh3

This is the third supported HTTP/3 backend.

Bug-fixes

curl: error out if -T and -d are used for the same URL

One of them implies PUT and the other implies POST, they cannot both be used for the same target URL and starting now curl will error out properly with a message saying so.

system.h: ifdefs for MCST-LCC compiler

Yet another compiler is now supported by default when you build curl.

curl: fix segmentation fault for empty output file names

Also now generally behave better as in telling the user why it errors out because of this situation.

http2: RST the stream if we stop it on our own will

When an application stops a transfer that is being done over HTTP/2, it was not properly shut down from curl’s side and therefore could end up wasting data that the server kept sending but that the client wouldn’t receive anymore!

http: close the stream (not connection) on time condition abort

For a special kind of transfer abort due to a failed time condition, curl would always close the connection to stop the transfer, instead of just closing the stream. This of course made no different on HTTP/1 but for later HTTP versions the connection should be kept alive even for this condition.

http: streamclose “already downloaded”

Another case of curl deciding the connection shouldn’t continue when it for in fact should be kept alive for HTTP/2 and HTTP/3.

http: reject header contents with nul bytes

HTTP headers cannot legally contain these bytes as per the protocol specification and as hyper already rejects these response it made sense to unify the implementation and refuse them in native code as well. It might also save us from future badness.

http: return error on colon-less HTTP headers

Similar to the change above, HTTP/1 headers must have colons so curl now will consider it a broken transfer if a header arrives without. This makes curl much pickier of course, but should not affect any “real” HTTP transfers.

mqtt: better handling of TCP disconnect mid-message

A nasty busy-loop occurred if the connection was cut off at the wrong time for an MQTT transfer.

ngtcp2: numerous improvements

HTTP/3 with ngtcp2 was greatly enhanced during this cycle in several ways. Check out the changelog for the specific details and do try it out!

tls: make mbedtls and NSS check for h2, not nghttp2

In leftovers from the past we still checked if HTTP/2 support is present by the wrong #ifdef in a few places in the code. nghttp2 is no longer the only HTTP/2 library we can use.

curl: escape ‘?’ in code generated with --libcurl

It turns out you could sneakily insert and get fooled by trigraphs otherwise:

curl --libcurl client.c --user-agent "??/\");char c[]={'i','d',' ','>','x',0},m[]={'r',0};fclose(popen(c,m));//" http://example.invalid
cURL and libcurl

curl up 2022 San Francisco

1 Comment

On June 6 2022, we will gather a bunch of curl aficionados in the Firehouse at the Fort Mason Centre in San Francisco, USA.

All details can be found here. We will add more info and details as we get closer to the event.

curl up is the annual curl developers and users “conference” where we meet up over a day and talk curl, curl related topics and share ideas about curl, its present and and its future. It is also really the only time of the year where we actually get to meet fellow curl hackers in person. The only day of the year that is completely devoted to curl. The best kind of day!

The last two years we have not run the conference for covid reasons but now we are back. The first time we arrange the event outside Europe.

I fully realize this geographic choice will prevent some of our European friends and contributors from attending, it will also allow North Americans to join the fun for the first time.

We help contributors attend

To better allow and encourage top curl contributors to attend this event, no matter where you live, we will help cover travel and lodging expenses for all and any top-100 curl committers who wants to come.

Sign up

Head over to the curl up 2022 page to find the link and details.

Agenda

Over the coming month I hope we can create an agenda with curl talks from several people. I need your ideas and your talks. We have started to collect some ideas for the 2022 agenda.

Tell us what you want to hear and what you want to share with us!

Who will be there?

I will of course be there and I hope we can attract a decent set of additional contributors, but also curl users and fans of all kinds and types.

Yes I can enter the country

Lots of you remember my struggles in the past to get permission to enter the US, but that was resolved a while ago. No problems remain.

Credits

Image by David Mark from Pixabay

cURL and libcurl

msh3 as the third h3 backend

With the brand new merged support for the msh3 library, curl now supports no less than three different HTTP/3 backends. It was merged into curl’s git repository on April 10.

When you build curl, you have the option to build it with HTTP/3 support enabled. The HTTP/3 support in curl is still considered experimental so it is still not enabled by default.

The HTTP/3 support in curl depends on the presence and support from third party libraries. You need to select and enable a specific HTTP/3 backend when you build curl. It has previously been doing HTTP/3 using either quiche or ngtcp2 + nghttp3. Starting now, there is yet another option to consider: the msh3 library.

The msh3 library itself uses msquic for doing QUIC. This is a multi platform library that uses Schannel for TLS when on Windows and OpenSSL/quictls for other platforms. The Schannel part probably makes solution this particularly interesting for curl users on Windows.

Open Source

More steel

I didn’t expect this, and this year I wasn’t asked ahead of time if I wanted to receive this gift. It is however something of a collector’s item that I find very enjoyable.

I received my GitHub contribution matrix printed in steel. This is my 2021 contribution skyline. (Click the images for higher resolution.)

The thing is surprisingly heavy and sturdy. If I had papers lying around, this would an awesome paper press.

You might remember that I got a similar gift last year, so it felt natural to do a comparison shot of 2021 and 2020.

For 2020, GitHub counted 2,466 contributions, while I reached 2,543 in 2021. Very similar numbers, but clearly distributed very differently. The two matrix images look like this.

2020

2021

Letter

Enclosed with this gift was also a friendly and encouraging letter.

Oh, and you can of course also see a rendered version in your browsers or download it in STL format so that you can print using your own 3d-printer.

Thank you, all my friends at GitHub!