We have just opened up the registration site for curl up 2018, the annual curl developers meeting that this year takes place in Stockholm, Sweden, over the weekend April 14-15. There’s a limited number of seats available, so if you want to join in the fun it might be a good idea to decide early on.
Also, to get into the proper curl up spirit, here’s the curl quiz we ran last year. I hope to run something similar again this year, but of course with a different set of questions.
Here’s to another curl release!
curl 7.58.0 is the 172nd curl release and it contains, among other things, 82 bug fixes thanks to 54 contributors (22 new). All this done with 131 commits in 56 days.
The bug fix rate is slightly lower than in the last few releases, which I tribute mostly to me having been away on vacation for a month during this release cycle. I retain my position as “committer of the Month” and January 2018 is my 29th consecutive month where I’ve done most commits in the curl source code repository. In total, almost 58% of the commits have been done by me (if we limit the count to all commits done since 2014, I’m at 43%). We now count a total of 545 unique commit authors and 1,685 contributors.
So what’s new this time? (full changelog here)
Introducing the pluggable SSH backend, and libssh is now the new alternative SSH backend to libssh2 that has been supported since late 2006. This change alone brought thousands of new lines of code.
Tell configure to use it with –with-libssh and you’re all set!
The libssh backend work was done by Nikos Mavrogiannopoulos, Tomas Mraz, Stanislav Zidek, Robert Kolcun and Andreas Schneider.
Yet again we announce security issues that we’ve found and fixed. Two of them to be exact:
A minor thing, but we refreshed the progress bar layout for when no total size is known.
March 21 is the date set for next release. Unless of course we find an urgent reason to fix and release something before then…
curl features an alternative progress bar. When you invoke it with -# or the longer version –progress-bar, curl will show the transfer progress using a single “bar” on the screen instead of the default meter that shows a lot of data like amount of data, transfer speeds and times.
$ curl -# -O https://example.com/coolfile.tar.gz ############################################ 100.0%
The alternative progress bar works great when the amount of data to transfer is known since then it can actually know how large part of the transfer that is done etc. If the amount of data is unknown – which is not a super rare situation – the progress bar output instead used to output one ‘#’ per kilobyte of data so that it would still show something. That could then end up filling up the screen and more if you did a large transfer.
$ curl -# -O https://example.com/nosize.html ###########################################################################################################
Starting in curl 7.58.0 (to be released on January 24, 2018), this latter progress bar layout is modified. If the total size is unknown, it will now instead display a small space ship flying across the line, back and forth – and it will only move as long as there is data being transferred. If it stalls, the little ship stops.
“Over” the space ship there are four nonsensical flying hashes (‘#’) that are simply moving across the line on a sine wave, following each other. They move independently of there being data transferred or not.
It can then end up looking similar to this:
There’s no real “meaning” behind this new progress bar output mode. I wanted it to
Since this new bar has just landed and this is the first time we ship a release with it, I wouldn’t be surprised if we end up polishing it further later on.
Can you tell I started out my programming life as a demo programmer on the Commodore 64? 🙂
Since a long time back, the venerable network analyzer tool Wireshark (screenshot above) has provided a way to decrypt and inspect TLS traffic when sent and received by Firefox and Chrome.
You do this by making the browser tell Wireshark the SSL secrets:
Having done this simple operation, you can now inspect your browser’s HTTPS traffic in Wireshark. Just super handy and awesome.
Just remember that if you record TLS traffic and want to save it for analyzing later, you need to also save the file with the secrets so that you can decrypt that traffic capture at a later time as well.
Adding curl to the mix. curl can be built using a dozen different TLS libraries and not just a single one as the browsers do. It complicates matters a bit.
In the NSS library for example, which is the TLS library curl is typically built with on Redhat and Centos, handles the SSLKEYLOGFILE magic all by itself so by extension you have been able to do this trick with curl for a long time – as long as you use curl built with NSS. A pretty good argument to use that build really.
Since curl version 7.57.0 the SSLKEYLOGFILE feature can also be enabled when built with GnuTLS, BoringSSL or OpenSSL. In the latter two libs, the feature is powered by new APIs in those libraries and in GnuTLS the library’s own logic similar to how NSS does it. Since OpenSSL is the by far most popular TLS backend for curl, this feature is now brought to users more widely.
In curl 7.58.0 (due to ship on Janurary 24, 2018), this feature is built by default also for curl with OpenSSL and in 7.57.0 you need to define ENABLE_SSLKEYLOGFILE to enable it for OpenSSL and BoringSSL.
And what’s even cooler? This feature is at the same time also brought to every single application out there that is built against this or later versions of libcurl. In one single blow. now suddenly a whole world opens to make it easier for you to debug, diagnose and analyze your applications’ TLS traffic when powered by libcurl!
Like the description above for browsers, you
Of course, as a light weight alternative: you may opt to use the –trace or –trace-ascii options with the curl tool and be fully satisfied with that. Using those command line options, curl will log everything sent and received in the protocol layer without the TLS applied. With HTTPS you’ll see all the HTTP traffic for example.
Most of the curl work to enable this feature was done by Peter Wu and Ray Satiro.
I’m about to take an extended vacation for the rest of the year and into the beginning of the next, so I decided I’d sum up the year from a curl angle already now, a few weeks early. (So some numbers will grow a bit more after this post.)
So what did we do this year in the project, how did curl change?
The first curl release of the year was version 7.53.0 and the last one was 7.57.0. In the separate blog posts on 7.55.0, 7.56.0 and 7.57.0 you’ll note that we kept up adding new goodies and useful features. We produced a total of 9 releases containing 683 bug fixes. We announced twelve security problems. (Down from 24 last year.)
At least 125 different authors wrote code that was merged into curl this year, in the 1500 commits that were made. We never had this many different authors during a single year before in the project’s entire life time! (The 114 authors during 2016 was the previous all-time high.)
We added more than 160 new names to the THANKS document for their help in improving curl. The total amount of contributors is now over 1660.
This year we truly started to use travis for CI builds and grew from a mere two builds per commit and PR up to nineteen (with additional ones run on appveyor and elsewhere). The current build set is a very good verification that that most things still compile and work after a PR is merged. (see also the testing curl article).
Mozilla announced that they too will use colon-slash-slash in their logo. Of course we all know who had it that in their logo first… =)
In March 2017, we had our first ever curl get-together as we arranged curl up 2017 a weekend in Nuremberg, Germany. It was very inspiring and meeting parts of the team in real life was truly a blast. This was so good we 
intend to do it again: curl up 2018 will happen.
curl turned 19 years old in March. In May it surpassed 5,000 stars on github.
Also in May, we moved over the official curl site (and my personal site) to get hosted by Fastly. We were beginning to get problems to handle the bandwidth and load, and in one single step all our worries were graciously taken care of!
We got curl entered into the OSS-fuzz project, and Max Dymond even got a reward from Google for his curl-fuzzing integration work and thanks to that project throwing heaps of junk at libcurl’s APIs we’ve found and fixed many issues.
The source code (for the tool and library only) is now at about 143,378 lines of code. It grew around 7,057 lines during the year. The primary reasons for the code growth were:
Oh what an eventful year it has been for me personally.
The first interim meeting for QUIC took place in Japan, and I participated from remote. After all, I’m all set on having curl support QUIC and I’ll keep track of where the protocol is going! I’ve participated in more interim meetings after that, all from remote so far.
I talked curl on the main track at FOSDEM in early February (and about HTTP/2 in the Mozilla devroom). I’ve then followed that up and have also had the pleasure to talk in front of audiences in Stockholm, Budapest, Jönköping and Prague through-out the year.
I went to London and “represented curl” in the third edition of the HTTP workshop, where HTTP protocol details were discussed and disassembled, and new plans for the future of HTTP were laid out.
In late June I meant to go to San Francisco to a Mozilla “all hands” conference but instead I was denied to board the flight. That event got a crazy amount of attention and I received massive amounts of love from new and old friends. I have not yet tried to enter the US again, but my plan is to try again in 2018…
I wrote and published my h2c tool, meant to help developers convert a set of HTTP headers into a working curl command line.
The single occasion that overshadows all other events and happenings for me this year by far, was without doubt when I was awarded the Polhem Prize and got a gold medal medal from no other than his majesty the King of Sweden himself. For all my work and years spent on curl no less.
Not really curl related, but in November I was also glad to be part of the huge Firefox Quantum release. The biggest Firefox release ever, and one that has been received really well.
I’ve managed to commit over 800 changes to curl through the year, which is 54% of the totals and more commits than I’ve done in curl during a single year since 2005 (in which I did 855 commits). I explain this increase mostly on inspiration from curl up and the prize, but I think it also happened thanks to excellent feedback and motivation brought by my fellow curl hackers.
We’re running towards the end of 2017 with me being the individual who did most commits in curl every single month for the last 28 months.
More things to come!
The never-ending series of curl releases continued today when we released version 7.57.0. The 171th release since the beginning, and the release that follows 37 days after 7.56.1. Remember that 7.56.1 was an extra release that fixed a few most annoying regressions.
We bump the minor number to 57 and clear the patch number in this release due to the changes introduced. None of them very ground breaking, but fun and useful and detailed below.
41 contributors helped fix 69 bugs in these 37 days since the previous release, using 115 separate commits. 23 of those contributors were new, making the total list of contributors now contain 1649 individuals! 25 individuals authored commits since the previous release, making the total number of authors 540 persons.
The curl web site currently sends out 8GB data per hour to over 2 million HTTP requests per day.
This allows HTTP Digest authentication to use the must better SHA256 algorithm instead of the old, and deemed unsuitable, MD5. This should be a transparent improvement so curl should just be able to use this without any particular new option has to be set, but the server-side support for this version seems to still be a bit lacking.
(Side-note: I’m credited in RFC 7616 for having contributed my thoughts!)
In this modern age with multi core processors and applications using multi-threaded designs, we of course want libcurl to enable applications to be able to get the best performance out of libcurl.
libcurl is already thread-safe so you can run parallel transfers multi-threaded perfectly fine if you want to, but it doesn’t allow the application to share handles between threads. Before this specific change, this limitation has forced multi-threaded applications to be satisfied with letting libcurl has a separate “connection cache” in each thread.
The connection cache, sometimes also referred to as the connection pool, is where libcurl keeps live connections that were previously used for a transfer and still haven’t been closed, so that a subsequent request might be able to re-use one of them. Getting a re-used connection for a request is much faster than having to create a new one. Having one connection cache per thread, is ineffective.
Starting now, libcurl’s “share concept” allows an application to specify a single connection cache to be used cross-thread and cross-handles, so that connection re-use will be much improved when libcurl is used multi-threaded. This will significantly benefit the most demanding libcurl applications, but it will also allow more flexible designs as now the connection pool can be designed to survive individual handles in a way that wasn’t previously possible.
The popular browsers have supported brotli compression method for a while and it has already become widely supported by servers.
Now, curl supports it too and the command line tool’s –compressed option will ask for brotli as well as gzip, if your build supports it. Similarly, libcurl supports it with its CURLOPT_ACCEPT_ENCODING option. The server can then opt to respond using either compression format, depending on what it knows.
According to CertSimple, who ran tests on the top-1000 sites of the Internet, brotli gets contents 14-21% smaller than gzip.
As with other compression algorithms, libcurl uses a 3rd party library for brotli compression and you may find that Linux distributions and others are a bit behind in shipping packages for a brotli decompression library. Please join in and help this happen. At the moment of this writing, the Debian package is only available in experimental.
(Readers may remember my libbrotli project, but that effort isn’t really needed anymore since the brotli project itself builds a library these days.)
In spite of our hard work and best efforts, security issues keep getting reported and we fix them accordingly. This release has three new ones and I’ll describe them below. None of them are alarmingly serious and they will probably not hurt anyone badly.
Two things can be said about the security issues this time:
1. You’ll note that we’ve changed naming convention for the advisory URLs, so that they now have a random component. This is to reduce potential information leaks based on the name when we pass these around before releases.
2. Two of the flaws happen only on 32 bit systems, which reveals a weakness in our testing. Most of our CI tests, torture tests and fuzzing are made on 64 bit architectures. We have no immediate and good fix for this, but this is something we must work harder on.
(CVE-2017-8816) Limited to 32 bit systems, this is a flaw where curl takes the combined length of the user name and password, doubles it, and allocates a memory area that big. If that doubling ends up larger than 4GB, an integer overflow makes a very small buffer be allocated instead and then curl will overwrite that.
Yes, having user name plus password be longer than two gigabytes is rather excessive and I hope very few applications would allow this.
(CVE-2017-8817) curl’s wildcard functionality for FTP transfers is not a not very widely used feature, but it was discovered that the default pattern matching function could erroneously read beyond the URL buffer if the match pattern ends with an open bracket ‘[‘ !
This problem was detected by the OSS-Fuzz project! This flaw has existed in the code since this feature was added, over seven years ago.
(CVE-2017-8818) In July this year we introduced multissl support in libcurl. This allows an application to select which TLS backend libcurl should use, if it was built to support more than one. It was a fairly large overhaul to the TLS code in curl and unfortunately it also brought this bug.
Also, only happening on 32 bit systems, libcurl would allocate a buffer that was 4 bytes too small for the TLS backend’s data which would lead to the TLS library accessing and using data outside of the heap allocated buffer.
The next release will ship no later than January 24th 2018. I think that one will as well add changes and warrant the minor number to bump. We have fun pending stuff such as: a new SSH backend, modifiable happy eyeballs timeout and more. Get involved and help us do even more good!
We’ve had volunteers donating bandwidth to the curl project basically since its inception. They mirror our download archives so that you can download them directly from their server farms instead of hitting the main curl site.
On the main site we check the mirrors daily and offers convenient download links from the download page. It has historically been especially useful for the rare occasions when our site has been down for administrative purpose or others.
Since May 2017 the curl site is fronted by Fastly which then has reduced the bandwidth issue as well as the downtime problem. The mirrors are still there though.
Starting now, we will only link to download mirrors that offer the curl downloads over HTTPS in our continued efforts to help our users to stay secure and avoid malicious manipulation of data. I’ve contacted the mirror admins and asked if they can offer HTTPS instead.
The curl download page still contains links to HTTP-only packages and pages, and we would really like to fix them as well. But at the same time we’ve reasoned that it is better to still help users to find packages than not, so for the packages where there are no HTTPS linkable alternatives we still link to HTTP-only pages. For now.
If you host curl packages anywhere, for anyone, please consider hosting them over HTTPS for all the users’ sake.
Welcome all curl hackers and fans to Stockholm 2018! We are hosting the curl up developers conference April 14-15 2018 in my home city in what now might really become an annual tradition. Remember Nuremberg 2017?
All details are collected and updated on the curl up 2018 wiki page:
curl up 2018Â will be another two full days event over a weekend and we will make an effort to keep the attendance fee at a minimum.
Presentations by core curl contributors on how things work, what we’ve done lately, what we’re working on how, what we should work on in the future and blue-sky visions about what curl should become when it grows up. Everyone attending are encouraged to present something. All this, mixed with lots of discussions, Q&As and socializing.
This time, I hope you will also have the chance to arrive early and spend the Friday, or a part of it, working hands-on with other curl developers on actual programming/debugging curl features or bugs. The curl-hacking-Friday? While not firmly organized yet, I’d like to see this become reality. Let me know if that’s something you’d be interested to participate on.
Anyone interested in curl, curl development or how to best use curl in application or solutions. Here’s your chance to learn a lot and an excellent opportunity to influence the future of curl. And to get some short glimpses of a spring-time Stockholm when going back and forth to the curl up venue!
We will open up the “ticket booth” in January/February 2018, so just keep your eyes and ears open and you won’t miss it! The general idea is to keep the fee at a minimum, ideally zero. Currently the exact price depends on how we manage to cover the remaining costs with friendly sponsors.
We are looking for sponsors. If your company is interested and willing to help us out, in any capacity, please contact me! Remember that our project has no funds of its own and we have no particular company backing.
We will spend the curl up 2018 weekend at Goto 10, which is near both the Skanstull and Gullmarsplan subway stations.
Action photo from curl up 2017 when Kamil Dudka spoke about Redhat’s use of curl built with NSS.
Thursday October 19, 2017,
I arrived at the Technical Museum in Stockholm together with my two kids just a short while before 17:30. A fresh, cool and clear autumn evening. For this occasion I had purchased myself a brand new suit as I hadn’t gotten one since almost twenty years before this and it had been almost that long since I last wore it. I went for a slightly less conservative purple colored shirt with the dark suit.
Apart from my kids, my wife was of course also present and so was my brother Björn and my parents in law. Plus a few hundred other visitors, most of them of course unknown to me.
My eleven year old son truly appreciates this museum so we took the opportunity to quickly check out parts of the exhibitions while the pre-event mingling went on and drinks were served. Not too long though as we were soon asked to proceed to the restaurant part and take our assigned seats. I was seated at table #6.
The whole evening was not entirely “mine”, but as I am the winner of this year’s Polhem Prize it was setup to eventually lead to the hand over of the award to me. An evening for me. Lots of attention on me and references to my work through-out the evening, that otherwise had the theme of traffic safety (my guess is that’s partly due to last year’s Prize winner who was a lead person in the invention of seat belts in cars).
A three-course dinner, with some entertainment intermixed. At my table I sat next to some brilliant and interesting people and I had a great time and good conversations. Sitting across the table from the His Majesty the king of Sweden was an unexpected and awesome honor.
Somewhere mid-through the evening, a short movie was presented on the big screens. A (Swedish-speaking) movie with me trying to explain what curl is, what it does and why I’ve made it. I think the movie was really great and I think it helps explaining curl to non-techies (including my own family). The movie is the result of a perhaps 40 minutes interview/talk we did on camera and then a fair amount of skilled editing by the production company. (Available here.)
At around 21:30 I was called on stage. I received a gold medal from the king and shook his hand. I also received a diploma and a paper with the award committee’s motivation for me getting the prize. And huge bouquet of lovely flowers. A bit more than what I could hold in my arms really.
(me, and Carl XVI Gustaf, king of Sweden)
As the king graciously offered to hold my diploma and medal, I took the microphone and expressed a few words of thanks. I was and I still am genuinely and deeply moved by receiving this prize. I’m happy and proud. I said my piece in which I explicitly mentioned my family members by name: Anja, Agnes and Rex for bearing with me.
(me, H.M the king and Cecilia Schelin Seidegård)
Afterwards I received several appraisals for my short speech which made me even happier. Who would’ve thought that was even possible?
I posed for pictures, shook many hands, received many congratulations and I even participated in a few selfies until the time came when it was time for me and my family to escape into a taxi and go home.
What a night. In the cab home we scanned social media and awed over pictures and mentions. I hadn’t checked my phone even once during the event so it had piled up a bit. It’s great to have so many friends and acquaintances who shared this award and moment with us!
I also experienced a strong “post award emptiness” sort of feeling. Okay, that was it. That was great. Now it’s over. Back to reality again. Back to fixing bugs and responding to emails.
Thank you everyone who contributed to this! In whatever capacity.
The Swedish motivation (shown in a picture above) goes like this, translated to English with google and edited by me:
Motivation for the Polhem Prize 2017
Our modern era consists of more and more ones and zeroes. Each individual programming tool that instructs technical machines to do
what we want has its own important function.
Everything that is connected needs to exchange information. Twenty years ago, Daniel Stenberg started working on what we now call cURL. Since then he has spent late evenings and weekends, doing unpaid work to refine his digital tool. It consists of open source code and allows you to retrieve data from home page URLs. The English letter c, see, makes it “see URL”.
In practice, its wide spread use means that millions, up to billions of people, worldwide, every day benefit from cURL in their mobile phones, computers, cars and a lot more. The economic value created with this can not be overestimated.
Daniel Stenberg initiated, keeps it together and leads the continuous development work with the tool. Completely voluntary. In total, nearly 1400 individuals have contributed. It is a solid engineering work and an expression of dedicated governance that has benefited many companies and the entire society. For this, Daniel Stenberg is awarded the Polhem Prize 2017.
I’m awarded the Swedish Polhem Prize 2017. (Link to a Swedish-speaking site.)
The Polhem Prize (Polhemspriset in Swedish), is awarded “for a high-level technological innovation or an ingenious solution to a technical problem.” The Swedish innovation must be available and shown competitive on the open market.
This award has been handed out in the name of the scientist and inventor Christopher Polhem, sometimes called the father of Swedish engineering, since 1878. It is Sweden’s oldest and most prestigious award for technological innovation.
I first got the news on the afternoon on September 24th and I don’t think I exaggerate much if I say that I got a mild shock. Me? A prize? How did they even find me or figure out what I’ve done?
I get this award for having worked on curl for a very long time, and by doing this having provided an Internet infrastructure of significant value to the world. I’ve never sold it nor earned much of commercial income from this hobby of mine, but my code now helps to power an almost unimaginable amount of devices, machines and other connected things in the world.
I’m not used to getting noticed or getting awards. I’m used to sitting by myself working on bugs, merging patches and responding to user emails. I don’t expect outsiders to notice what I do much and I always have a hard time to explain to friends and “mortals” what it is I actually do.
I accept this prize, not as a single inventor or brilliant mind of anything, but like the captain of a boat with a large and varying crew without whom I would never have reached this far. I’m excited that the nominee board found me and our merry project and that they were open-minded enough to see and realize the value and position of an open source project that is used literally everywhere. I feel deeply honored.
I’m fascinated the award nominee group found me and I think it is super cool that an open source project gets this attention and acknowledgement.
Apart from the honor, the prize comes in form of a monetary part (250K SEK, about 31,000 USD) and a gold medal with Polhem’s image on. See this blog post’s featured image. The official award ceremony will take place in a few days at the Technical Museum in Stockholm. I’m then supposed to get the medal handed to me by his royal highness Carl XVI Gustav , the king of Sweden. An honor very few people get to experience. Especially very few open source hackers.
While I have so many people to thank for having contributed to my (and curl’s) success, there are some that have been fundamental.
I’d like to specifically highlight my wife Anja and my kids Agnes and Rex who are the ones I routinely steal time away from to instead spend on curl. They’re the ones who I drift away from when I respond to issues on the phones or run off to the computer to “just respond to something quickly”. They’re the best.
I’d like to thank Björn, my brother, who chipped in half the amount of money for that first Commodore 64 we purchased back in 1985 and which was the first stepping stone to me being here.
I’d like to thank all my friends and team mates in the curl project without whom curl would’ve died as an infant already in the 1990s. It is with honest communication, hard work and good will that good software is crafted. (Well, there might be some more components necessary too, but let’s keep it simple here.)
I’d like to thank everyone who ever said thanks to me for curl and told me that what I did or brought to the world actually made a difference or served a purpose. Positive feedback is what drives me. It is the fuel that keeps me going.
I hope the award will strengthen my spine even more in knowing that we’re going down the right path here. Not necessarily with every single decision or choice we do, but the general one: we do things open source, we do things together and we work long-term.
I hope the award puts a little more light and attention on the world of open source and how this development model can produce the most stellar and robust software components you can think of – without a “best before” stamp.
I would like the award to make one or two more people find and take a closer look at the curl project. To dive in and contribute, in one way or another. We always need more eyes and hands!
Further, I realize that this award might bring some additional eyes on me who will watch how I act and behave. I intend to keep trying to do the right thing and act properly in every situation and I know my friends and community will help me stand straight – no matter how the winds blow.
I intend to take my family with me on an extended vacation trip to New Zealand!
Hopefully there will be some money left afterward, that I hope to at least in part spend on curl related activities such as birthday cakes on the pending curl 20th birthday celebrations in spring 2018…
Virtually every smart phone has one or more curl installs. Most modern cars and television sets do as well. Probably just about all Linux servers on the Internet run it. Almost all PHP sites on the Internet do. Portable devices and internet-connected machines use it extensively. curl sends crash-reports when your Chrome or Firefox browser fail. It is the underlying data transfer engine for countless systems, languages, programs, games and environments.
Every single human in the connected world use something that runs curl every day. Probably more than once per day. Most have it installed in devices they carry around with them.
It is installed and runs in tens of billions of instances, as most modern-life rich people have numerous installations in their phones, with their web browsers, in their tablets, their cars, their TVs, their kitchen appliances etc.
Most humans, of course, don’t know this. They use devices and apps that just work and are fine with that. curl is just a little piece in the engines of those systems.
