Category Archives: Open Source

Open Source, Free Software, and similar

cURL and libcurl, Open Source, Windows

Microsoft curls too

12 Comments

On December 19 2017, Microsoft announced that since insider build 17063 of Windows 10, curl is now a default component. I’ve been away from home since then so I haven’t really had time to sit down and write and explain to you all what this means, so while I’m a bit late, here it comes!

I see this as a pretty huge step in curl’s road to conquer the world.

curl was already existing on Windows

Ever since we started shipping curl, it has been possible to build curl for Windows and run it on Windows. It has been working fine on all Windows versions since at least Windows 95. Running curl on Windows is not new to us. Users with a little bit of interest and knowledge have been able to run curl on Windows for almost 20 years already.

Then we had the known debacle with Microsoft introducing a curl alias to PowerShell that has put some obstacles in the way for users of curl.

Default makes a huge difference

Having curl shipped by default by the manufacturer of an operating system of course makes a huge difference. Once this goes out to the general public, all of a sudden several hundred million users will get a curl command line tool install for them without having to do anything. Installing curl yourself on Windows still requires some skill and knowledge and on places like stackoverflow, there are many questions and users showing how it can be problematic.

I expect this to accelerate the curl command line use in the world. I expect this to increase the number of questions on how to do things with curl.

Lots of people mentioned how curl is a “good” new tool to use for malicious downloads of files to windows machines if you manage to run code on someone’s Windows computer. curl is quite a capable thing that you truly do not want to get invoked involuntarily. But sure, any powerful and capable tool can of course be abused.

About the installed curl

This is what it looks when you check out the curl version on this windows build:

(screenshot from Steve Holme)

I don’t think this means that this is necessarily exactly what curl will look like once this reaches the general windows 10 installation, and I also expect Microsoft to update and upgrade curl as we go along.

Some observations from this simple screenshot, and if you work for Microsoft you may feel free to see this as some subtle hints on what you could work on improving in future builds:

  1. They ship 7.55.1, while 7.57.0 was the latest version at the time. That’s just three releases away so I consider that pretty good. Lots of distros and others ship (much) older releases. It’ll be interesting to see how they will keep this up in the future.
  2. Unsurprisingly, they use a build that uses the WinSSL backend for TLS.
  3. They did not build it with IDN support.
  4. They’ve explicitly disabled support a whole range of protocols that curl supports natively by default (gopher, smb, rtsp etc), but they still have a few rare protocols enabled (like dict).
  5. curl supports LDAP using the windows native API, but that’s not used.
  6. The Release-Date line shows they built curl from unreleased sources (most likely directly from a git clone).
  7. No HTTP/2 support is provided.
  8. There’s no automatic decompression support for gzip or brotli content.
  9. The build doesn’t support metalink and no PSL (public suffix list).

(curl gif from the original Microsoft curl announcement blog post)

Independent

Finally, I’d like to add that like all operating system distributions that ship curl (macOS, Linux distros, the BSDs, AIX, etc) Microsoft builds, packages and ships the curl binary completely independently from the actual curl project.

Sure I’ve been in contact with the good people working on this from their end, but they are working totally independently of us in the curl project. They mostly get our code, build it and ship it.

I of course hope that we will get bug fixes and improvement from their end going forward when they find problems or things to polish.

The future looks as great as ever before!

Update: in March 2018, they mentioned that curl comes in Windows 10 version 1803.

cURL and libcurl, Open Source

The curl year 2017

I’m about to take an extended vacation for the rest of the year and into the beginning of the next, so I decided I’d sum up the year from a curl angle already now, a few weeks early. (So some numbers will grow a bit more after this post.)

2017

So what did we do this year in the project, how did curl change?

The first curl release of the year was version 7.53.0 and the last one was 7.57.0. In the separate blog posts on 7.55.0, 7.56.0 and 7.57.0 you’ll note that we kept up adding new goodies and useful features. We produced a total of 9 releases containing 683 bug fixes. We announced twelve security problems. (Down from 24 last year.)

At least 125 different authors wrote code that was merged into curl this year, in the 1500 commits that were made. We never had this many different authors during a single year before in the project’s entire life time! (The 114 authors during 2016 was the previous all-time high.)

We added more than 160 new names to the THANKS document for their help in improving curl. The total amount of contributors is now over 1660.

This year we truly started to use travis for CI builds and grew from a mere two builds per commit and PR up to nineteen (with additional ones run on appveyor and elsewhere). The current build set is a very good verification that that most things still compile and work after a PR is merged. (see also the testing curl article).

Mozilla announced that they too will use colon-slash-slash in their logo. Of course we all know who had it that in their logo first… =)

 

In March 2017, we had our first ever curl get-together as we arranged curl up 2017 a weekend in Nuremberg, Germany. It was very inspiring and meeting parts of the team in real life was truly a blast. This was so good we intend to do it again: curl up 2018 will happen.

curl turned 19 years old in March. In May it surpassed 5,000 stars on github.

Also in May, we moved over the official curl site (and my personal site) to get hosted by Fastly. We were beginning to get problems to handle the bandwidth and load, and in one single step all our worries were graciously taken care of!

We got curl entered into the OSS-fuzz project, and Max Dymond even got a reward from Google for his curl-fuzzing integration work and thanks to that project throwing heaps of junk at libcurl’s APIs we’ve found and fixed many issues.

The source code (for the tool and library only) is now at about 143,378 lines of code. It grew around 7,057 lines during the year. The primary reasons for the code growth were:

  1. the new libssh-powered SSH backend (not yet released)
  2. the new mime API (in 7.56.0) and
  3. the new multi-SSL backend support (also in 7.56.0).

Your maintainer’s view

Oh what an eventful year it has been for me personally.

The first interim meeting for QUIC took place in Japan, and I participated from remote. After all, I’m all set on having curl support QUIC and I’ll keep track of where the protocol is going! I’ve participated in more interim meetings after that, all from remote so far.

I talked curl on the main track at FOSDEM in early February (and about HTTP/2 in the Mozilla devroom). I’ve then followed that up and have also had the pleasure to talk in front of audiences in Stockholm, Budapest, Jönköping and Prague through-out the year.

 

I went to London and “represented curl” in the third edition of the HTTP workshop, where HTTP protocol details were discussed and disassembled, and new plans for the future of HTTP were laid out.

 

In late June I meant to go to San Francisco to a Mozilla “all hands” conference but instead I was denied to board the flight. That event got a crazy amount of attention and I received massive amounts of love from new and old friends. I have not yet tried to enter the US again, but my plan is to try again in 2018…

I wrote and published my h2c tool, meant to help developers convert a set of HTTP headers into a working curl command line.

The single occasion that overshadows all other events and happenings for me this year by far, was without doubt when I was awarded the Polhem Prize and got a gold medal medal from no other than his majesty the King of Sweden himself. For all my work and years spent on curl no less.

Not really curl related, but in November I was also glad to be part of the huge Firefox Quantum release. The biggest Firefox release ever, and one that has been received really well.

I’ve managed to commit over 800 changes to curl through the year, which is 54% of the totals and more commits than I’ve done in curl during a single year since 2005 (in which I did 855 commits). I explain this increase mostly on inspiration from curl up and the prize, but I think it also happened thanks to excellent feedback and motivation brought by my fellow curl hackers.

We’re running towards the end of 2017 with me being the individual who did most commits in curl every single month for the last 28 months.

2018?

More things to come!

cURL and libcurl, Security

curl 7.57.0 happiness

The never-ending series of curl releases continued today when we released version 7.57.0. The 171th release since the beginning, and the release that follows 37 days after 7.56.1. Remember that 7.56.1 was an extra release that fixed a few most annoying regressions.

We bump the minor number to 57 and clear the patch number in this release due to the changes introduced. None of them very ground breaking, but fun and useful and detailed below.

41 contributors helped fix 69 bugs in these 37 days since the previous release, using 115 separate commits. 23 of those contributors were new, making the total list of contributors now contain 1649 individuals! 25 individuals authored commits since the previous release, making the total number of authors 540 persons.

The curl web site currently sends out 8GB data per hour to over 2 million HTTP requests per day.

Support RFC7616 – HTTP Digest

This allows HTTP Digest authentication to use the must better SHA256 algorithm instead of the old, and deemed unsuitable, MD5. This should be a transparent improvement so curl should just be able to use this without any particular new option has to be set, but the server-side support for this version seems to still be a bit lacking.

(Side-note: I’m credited in RFC 7616 for having contributed my thoughts!)

Sharing the connection cache

In this modern age with multi core processors and applications using multi-threaded designs, we of course want libcurl to enable applications to be able to get the best performance out of libcurl.

libcurl is already thread-safe so you can run parallel transfers multi-threaded perfectly fine if you want to, but it doesn’t allow the application to share handles between threads. Before this specific change, this limitation has forced multi-threaded applications to be satisfied with letting libcurl has a separate “connection cache” in each thread.

The connection cache, sometimes also referred to as the connection pool, is where libcurl keeps live connections that were previously used for a transfer and still haven’t been closed, so that a subsequent request might be able to re-use one of them. Getting a re-used connection for a request is much faster than having to create a new one. Having one connection cache per thread, is ineffective.

Starting now, libcurl’s “share concept” allows an application to specify a single connection cache to be used cross-thread and cross-handles, so that connection re-use will be much improved when libcurl is used multi-threaded. This will significantly benefit the most demanding libcurl applications, but it will also allow more flexible designs as now the connection pool can be designed to survive individual handles in a way that wasn’t previously possible.

Brotli compression

The popular browsers have supported brotli compression method for a while and it has already become widely supported by servers.

Now, curl supports it too and the command line tool’s –compressed option will ask for brotli as well as gzip, if your build supports it. Similarly, libcurl supports it with its CURLOPT_ACCEPT_ENCODING option. The server can then opt to respond using either compression format, depending on what it knows.

According to CertSimple, who ran tests on the top-1000 sites of the Internet, brotli gets contents 14-21% smaller than gzip.

As with other compression algorithms, libcurl uses a 3rd party library for brotli compression and you may find that Linux distributions and others are a bit behind in shipping packages for a brotli decompression library. Please join in and help this happen. At the moment of this writing, the Debian package is only available in experimental.

(Readers may remember my libbrotli project, but that effort isn’t really needed anymore since the brotli project itself builds a library these days.)

Three security issues

In spite of our hard work and best efforts, security issues keep getting reported and we fix them accordingly. This release has three new ones and I’ll describe them below. None of them are alarmingly serious and they will probably not hurt anyone badly.

Two things can be said about the security issues this time:

1. You’ll note that we’ve changed naming convention for the advisory URLs, so that they now have a random component. This is to reduce potential information leaks based on the name when we pass these around before releases.

2. Two of the flaws happen only on 32 bit systems, which reveals a weakness in our testing. Most of our CI tests, torture tests and fuzzing are made on 64 bit architectures. We have no immediate and good fix for this, but this is something we must work harder on.

1. NTLM buffer overflow via integer overflow

(CVE-2017-8816) Limited to 32 bit systems, this is a flaw where curl takes the combined length of the user name and password, doubles it, and allocates a memory area that big. If that doubling ends up larger than 4GB, an integer overflow makes a very small buffer be allocated instead and then curl will overwrite that.

Yes, having user name plus password be longer than two gigabytes is rather excessive and I hope very few applications would allow this.

2. FTP wildcard out of bounds read

(CVE-2017-8817) curl’s wildcard functionality for FTP transfers is not a not very widely used feature, but it was discovered that the default pattern matching function could erroneously read beyond the URL buffer if the match pattern ends with an open bracket ‘[‘ !

This problem was detected by the OSS-Fuzz project! This flaw  has existed in the code since this feature was added, over seven years ago.

3. SSL out of buffer access

(CVE-2017-8818) In July this year we introduced multissl support in libcurl. This allows an application to select which TLS backend libcurl should use, if it was built to support more than one. It was a fairly large overhaul to the TLS code in curl and unfortunately it also brought this bug.

Also, only happening on 32 bit systems, libcurl would allocate a buffer that was 4 bytes too small for the TLS backend’s data which would lead to the TLS library accessing and using data outside of the heap allocated buffer.

Next?

The next release will ship no later than January 24th 2018. I think that one will as well add changes and warrant the minor number to bump. We have fun pending stuff such as: a new SSH backend, modifiable happy eyeballs timeout and more. Get involved and help us do even more good!

cURL and libcurl, Security, Web

HTTPS-only curl mirrors

We’ve had volunteers donating bandwidth to the curl project basically since its inception. They mirror our download archives so that you can download them directly from their server farms instead of hitting the main curl site.

On the main site we check the mirrors daily and offers convenient download links from the download page. It has historically been especially useful for the rare occasions when our site has been down for administrative purpose or others.

Since May 2017 the curl site is fronted by Fastly which then has reduced the bandwidth issue as well as the downtime problem. The mirrors are still there though.

Starting now, we will only link to download mirrors that offer the curl downloads over HTTPS in our continued efforts to help our users to stay secure and avoid malicious manipulation of data. I’ve contacted the mirror admins and asked if they can offer HTTPS instead.

The curl download page still contains links to HTTP-only packages and pages, and we would really like to fix them as well. But at the same time we’ve reasoned that it is better to still help users to find packages than not, so for the packages where there are no HTTPS linkable alternatives we still link to HTTP-only pages. For now.

If you host curl packages anywhere, for anyone, please consider hosting them over HTTPS for all the users’ sake.

cURL and libcurl, Open Source

curl up in Stockholm 2018

Welcome all curl hackers and fans to Stockholm 2018! We are hosting the curl up developers conference April 14-15 2018 in my home city in what now might really become an annual tradition. Remember Nuremberg 2017?

All details are collected and updated on the curl up 2018 wiki page:

curl-up-2018

curl up 2018  will be another two full days event over a weekend and we will make an effort to keep the attendance fee at a minimum.

Presentations by core curl contributors on how things work, what we’ve done lately, what we’re working on how, what we should work on in the future and blue-sky visions about what curl should become when it grows up. Everyone attending are encouraged to present something. All this, mixed with lots of discussions, Q&As and socializing.

This time, I hope you will also have the chance to arrive early and spend the Friday, or a part of it, working hands-on with other curl developers on actual programming/debugging curl features or bugs. The curl-hacking-Friday? While not firmly organized yet, I’d like to see this become reality. Let me know if that’s something you’d be interested to participate on.

Who should come?

Anyone interested in curl, curl development or how to best use curl in application or solutions. Here’s your chance to learn a lot and an excellent opportunity to influence the future of curl. And to get some short glimpses of a spring-time Stockholm when going back and forth to the curl up venue!

Sign up?

We will open up the “ticket booth” in January/February 2018, so just keep your eyes and ears open and you won’t miss it! The general idea is to keep the fee at a minimum, ideally zero. Currently the exact price depends on how we manage to cover the remaining costs with friendly sponsors.

Sponsor us!

We are looking for sponsors. If your company is interested and willing to help us out, in any capacity, please contact me! Remember that our project has no funds of its own and we have no particular company backing.

Where in Stockholm exactly?

We will spend the curl up 2018 weekend at Goto 10, which is near both the Skanstull and Gullmarsplan subway stations.

Action photo from curl up 2017 when Kamil Dudka spoke about Redhat’s use of curl built with NSS.

Firefox, Mozilla, Open Source, Web

Firefox Quantum

29 Comments

Next week, Mozilla will release Firefox 57. Also referred to as Firefox Quantum, from the project name we’ve used for all the work that has been put into making this the most awesome Firefox release ever. This is underscored by the fact that I’ve gotten mailed release-swag for the first time during my four years so far as a Mozilla employee.

Firefox 57 is the major milestone hundreds of engineers have worked really hard toward during the last year or so, and most of the efforts have been focused on performance. Or perhaps perceived end user snappiness. Early comments I’ve read and heard also hints that it is also quite notable. I think every single Mozilla engineer (and most non-engineers as well) has contributed to at least some parts of this, and of course many have done a lot. My personal contributions to 57 are not much to write home about, but are mostly a stream of minor things that combined at least move the notch forward.

[edited out some secrets I accidentally leaked here.] I’m a proud Mozillian and being part of a crowd that has put together something as grand as Firefox 57 is an honor and a privilege.

Releasing a product to hundreds of millions of end users across the world is interesting. People get accustomed to things, get emotional and don’t particularly like change very much. I’m sure Firefox 57 will also get a fair share of sour feedback and comments written in uppercase. That’s inevitable. But sometimes, in order to move forward and do good stuff, we have to make some tough decisions for the greater good that not everyone will agree with.

This is however not the end of anything. It is rather the beginning of a new Firefox. The work on future releases goes on, we will continue to improve the web experience for users all over the world. Firefox 58 will have even more goodies, and I know there are much more good stuff planned for the releases coming in 2018 too…

Onwards and upwards!

(Update: as I feared in this text, I got a lot of negativism, vitriol and criticism in the comments to this post. So much that I decided to close down comments for this entry and delete the worst entries.)

cURL and libcurl, Open Source

My night at the museum

8 Comments

Thursday October 19, 2017,

I arrived at the Technical Museum in Stockholm together with my two kids just a short while before 17:30. A fresh, cool and clear autumn evening. For this occasion I had purchased myself a brand new suit as I hadn’t gotten one since almost twenty years before this and it had been almost that long since I last wore it. I went for a slightly less conservative purple colored shirt with the dark suit.

Apart from my kids, my wife was of course also present and so was my brother Björn and my parents in law. Plus a few hundred other visitors, most of them of course unknown to me.

My eleven year old son truly appreciates this museum so we took the opportunity to quickly check out parts of the exhibitions while the pre-event mingling went on and drinks were served. Not too long though as we were soon asked to proceed to the restaurant part and take our assigned seats. I was seated at table #6.

The whole evening was not entirely “mine”, but as I am the winner of this year’s Polhem Prize it was setup to eventually lead to the hand over of the award to me. An evening for me. Lots of attention on me and references to my work through-out the evening, that otherwise had the theme of traffic safety (my guess is that’s partly due to last year’s Prize winner who was a lead person in the invention of seat belts in cars).

A three-course dinner, with some entertainment intermixed. At my table I sat next to some brilliant and interesting people and I had a great time and good conversations. Sitting across the table from the His Majesty the king of Sweden was an unexpected and awesome honor.

Somewhere mid-through the evening, a short movie was presented on the big screens. A (Swedish-speaking) movie with me trying to explain what curl is, what it does and why I’ve made it. I think the movie was really great and I think it helps explaining curl to non-techies (including my own family). The movie is the result of a perhaps 40 minutes interview/talk we did on camera and then a fair amount of skilled editing by the production company. (Available here.)

At around 21:30 I was called on stage. I received a gold medal from the king and shook his hand. I also received a diploma and a paper with the award committee’s motivation for me getting the prize. And huge bouquet of lovely flowers. A bit more than what I could hold in my arms really.

(me, and Carl XVI Gustaf, king of Sweden)

As the king graciously offered to hold my diploma and medal, I took the microphone and expressed a few words of thanks. I was and I still am genuinely and deeply moved by receiving this prize. I’m happy and proud. I said my piece in which I explicitly mentioned my family members by name: Anja, Agnes and Rex for bearing with me.

(me, H.M the king and Cecilia Schelin Seidegård)

Afterwards I received several appraisals for my short speech which made me even happier. Who would’ve thought that was even possible?

I posed for pictures, shook many hands, received many congratulations and I even participated in a few selfies until the time came when it was time for me and my family to escape into a taxi and go home.

What a night. In the cab home we scanned social media and awed over pictures and mentions. I hadn’t checked my phone even once during the event so it had piled up a bit. It’s great to have so many friends and acquaintances who shared this award and moment with us!

I also experienced a strong “post award emptiness” sort of feeling. Okay, that was it. That was great. Now it’s over. Back to reality again. Back to fixing bugs and responding to emails.

Thank you everyone who contributed to this! In whatever capacity.

The Swedish motivation (shown in a picture above) goes like this, translated to English with google and edited by me:

Motivation for the Polhem Prize 2017

Our modern era consists of more and more ones and zeroes. Each individual programming tool that instructs technical machines to do
what we want has its own important function.

Everything that is connected needs to exchange information.  Twenty years ago, Daniel Stenberg started working on what we  now call cURL. Since then he has spent late evenings and weekends, doing unpaid work to refine his digital tool. It consists of open source code and allows you to retrieve data from home page URLs. The English letter c, see, makes it “see URL”.

In practice, its wide spread use means that millions, up to billions of people, worldwide, every day benefit from cURL in their mobile phones, computers, cars and a lot more. The economic value created with this can not be overestimated.

Daniel Stenberg initiated, keeps it together and leads the continuous development work with the tool. Completely voluntary. In total, nearly 1400 individuals have contributed. It is a solid engineering work and an expression of dedicated governance that has benefited many companies and the entire society. For this, Daniel Stenberg is awarded the Polhem Prize 2017.

cURL and libcurl, Open Source

Polhemspriset 2017

55 Comments

I’m awarded the Swedish Polhem Prize 2017. (Link to a Swedish-speaking site.)

The Polhem Prize (Polhemspriset in Swedish), is awarded “for a high-level technological innovation or an ingenious solution to a technical problem.” The Swedish innovation must be available and shown competitive on the open market.

This award has been handed out in the name of the scientist and inventor Christopher Polhem, sometimes called the father of Swedish engineering, since 1878. It is Sweden’s oldest and most prestigious award for technological innovation.

I first got the news on the afternoon on September 24th and I don’t think I exaggerate much if I say that I got a mild shock. Me? A prize? How did they even find me or figure out what I’ve done?

I get this award for having worked on curl for a very long time, and by doing this having provided an Internet infrastructure of significant value to the world. I’ve never sold it nor earned much of commercial income from this hobby of mine, but my code now helps to power an almost unimaginable amount of devices, machines and other connected things in the world.

I’m not used to getting noticed or getting awards. I’m used to sitting by myself working on bugs, merging patches and responding to user emails. I don’t expect outsiders to notice what I do much and I always have a hard time to explain to friends and “mortals” what it is I actually do.

I accept this prize, not as a single inventor or brilliant mind of anything, but like the captain of a boat with a large and varying crew without whom I would never have reached this far. I’m excited that the nominee board found me and our merry project and that they were open-minded enough to see and realize the value and position of an open source project that is used literally everywhere. I feel deeply honored.

I’m fascinated the award nominee group found me and I think it is super cool that an open source project gets this attention and acknowledgement.

Apart from the honor, the prize comes in form of a monetary part (250K SEK, about 31,000 USD) and a gold medal with Polhem’s image on. See this blog post’s featured image. The official award ceremony will take place in a few days at the Technical Museum in Stockholm. I’m then supposed to get the medal handed to me by his royal highness Carl XVI Gustav , the king of Sweden. An honor very few people get to experience. Especially very few open source hackers.

Thank you

While I have so many people to thank for having contributed to my (and curl’s) success, there are some that have been fundamental.

I’d like to specifically highlight my wife Anja and my kids Agnes and Rex who are the ones I routinely steal time away from to instead spend on curl. They’re the ones who I drift away from when I respond to issues on the phones or run off to the computer to “just respond to something quickly”. They’re the best.

I’d like to thank Björn, my brother, who chipped in half the amount of money for that first Commodore 64 we purchased back in 1985 and which was the first stepping stone to me being here.

I’d like to thank all my friends and team mates in the curl project without whom curl would’ve died as an infant already in the 1990s. It is with honest communication, hard work and good will that good software is crafted. (Well, there might be some more components necessary too, but let’s keep it simple here.)

I’d like to thank everyone who ever said thanks to me for curl and told me that what I did or brought to the world actually made a difference or served a purpose. Positive feedback is what drives me. It is the fuel that keeps me going.

How will this award affect me and the curl project going forward?

I hope the award will strengthen my spine even more in knowing that we’re going down the right path here. Not necessarily with every single decision or choice we do, but the general one: we do things open source, we do things together and we work long-term.

I hope the award puts a little more light and attention on the world of open source and how this development model can produce the most stellar and robust software components you can think of – without a “best before” stamp.

I would like the award to make one or two more people find and take a closer look at the curl project. To dive in and contribute, in one way or another. We always need more eyes and hands!

Further, I realize that this award might bring some additional eyes on me who will watch how I act and behave. I intend to keep trying to do the right thing and act properly in every situation and I know my friends and community will help me stand straight – no matter how the winds blow.

What will I do with the money?

I intend to take my family with me on an extended vacation trip to New Zealand!

Hopefully there will be some money left afterward, that I hope to at least in part spend on curl related activities such as birthday cakes on the pending curl 20th birthday celebrations in spring 2018…

But really, how many use curl?

Virtually every smart phone has one or more curl installs. Most modern cars and television sets do as well. Probably just about all Linux servers on the Internet run it. Almost all PHP sites on the Internet do. Portable devices and internet-connected machines use it extensively. curl sends crash-reports when your Chrome or Firefox browser fail. It is the underlying data transfer engine for countless systems, languages, programs, games and environments.

Every single human in the connected world use something that runs curl every day. Probably more than once per day. Most have it installed in devices they carry around with them.

It is installed and runs in tens of billions of instances, as most modern-life rich people have numerous installations in their phones, with their web browsers, in their tablets, their cars, their TVs, their kitchen appliances etc.

Most humans, of course, don’t know this. They use devices and apps that just work and are fine with that. curl is just a little piece in the engines of those systems.

cURL and libcurl, Development, Open Source

Testing curl

1 Comment

In order to ship a quality product – once every eight weeks – we need lots of testing. This is what we do to test curl and libcurl.

checksrc

We have basic script that verifies that the source code adheres to our code standard. It doesn’t catch all possible mistakes, but usually it complains with enough details to help contributors to write their code to match the style we already use. Consistent code style makes the code easier to read. Easier reading makes less bugs and quicker debugging.

By doing this check with a script (that can be run automatically when building curl), it makes it easier for everyone to ship properly formatted code.

We have not (yet) managed to convince clang-format or other tools to reformat code to correctly match our style, and we don’t feel like changing it just for the sake of such a tool. I consider this a decent work-around.

make test

The test suite that we bundle with the source code in the git repository has a large number of tests that test…

  • curl – it runs the command line tool against test servers for a large range of protocols and verifies error code, the output, the protocol details and that there are no memory leaks
  • libcurl – we then build many small test programs that use the libcurl API and perform tests against test servers and verifies that they behave correctly and don’t leak memory etc.
  • unit tests – we build small test programs that use libcurl internal functions that aren’t exposed in the API and verify that they behave correctly and generate the presumed output.
  • valgrind – all the tests above can be run with and without valgrind to better detect memory issues
  • “torture” – a special mode that can run the tests above in a way that first runs the entire test, counts the number of memory related functions (malloc, strdup, fopen, etc) that are called and then runs the test again that number of times and for each run it makes one of the memory related functions fail – and makes sure that no memory is leaked in any of those situations and no crash occurs etc. It runs the test over and over until all memory related functions have been made to fail once each.

Right now, a single “make test” runs over 1100 test cases, varying a little depending on exactly what features that are enabled in the build. Without valgrind, running those tests takes about 8 minutes on a reasonably fast machine but still over 25 minutes with valgrind.

Then we of course want to run all tests with different build options…

CI

For every pull request and for every source code commit done, the curl source is built for Linux, mac and windows. With a large set of different build options and TLS libraries selected, and all the tests mentioned above are run for most of these build combinations. Running ‘checksrc’ on the pull requests is of course awesome so that humans don’t have to remark on code style mistakes much. There are around 30 different builds done and verified for each commit.

If any CI build fails, the pull request on github gets a red X to signal that something was not OK.

We also run test case coverage analyses in the CI so that we can quickly detect if we for some reason significantly decrease test coverage or similar.

We use Travis CI, Appveyor and Coveralls.io for this.

Autobuilds

Independently of the CI builds, volunteers run machines that regularly update from git, build and run the entire test suite and then finally email the results back to a central server. These setups help us cover even more platforms, architectures and build combinations. Just with a little longer turn around time.

With millions of build combinations and support for virtually every operating system and CPU architecture under the sun, we have to accept that not everything can be fully tested. But since almost all code is shared for many platforms, we can still be reasonably sure about the code even for targets we don’t test regularly.

Static code analyzing

We run the clang scan-build on the source code daily and we run Coverity scans on the code “regularly”, about once a week.

We always address defects detected by these analyzers immediately when notified.

Fuzzing

We’re happy to be part of Google’s OSS-fuzz effort, which with a little help with integration from us keeps hammering our code with fuzz to make sure we’re solid.

OSS-fuzz has so far resulted in two security advisories for curl and a range of other bug fixes. It hasn’t been going on for very long and based on the number it has detected so far, I expect it to keep finding flaws – at least for a while more into the future.

Fuzzing is really the best way to hammer out bugs. When we’re down to zero detected static analyzer detects and thousands of test cases that all do good, the fuzzers can still continue to find holes in the net.

External

Independently of what we test, there are a large amount of external testing going on, for each curl release we do.

In a presentation by Google at curl up 2017, they mentioned their use of curl in “hundreds of applications” and how each curl release they adopt gets tested more than 400,000 times. We also know a lot of other users also have curl as a core component in their systems and test their installations extensively.

We have a large set of security interested developers who run tests and fuzzers on curl at their own will.

(image from pixabay)

cURL and libcurl, Open Source, Security

The life of a curl security bug

The report

Usually, security problems in the curl project come to us out of the blue. Someone has found a bug they suspect may have a security impact and they tell us about it on the curl-security@haxx.se email address. Mails sent to this address reach a private mailing list with the curl security team members as the only subscribers.

An important first step is that we respond to the sender, acknowledging the report. Often we also include a few follow-up questions at once. It is important to us to keep the original reporter in the loop and included in all subsequent discussions about this issue – unless they prefer to opt out.

If we find the issue ourselves, we act pretty much the same way.

In the most obvious and well-reported cases there are no room for doubts or hesitation about what the bugs and the impact of them are, but very often the reports lead to discussions.

The assessment

Is it a bug in the first place, is it perhaps even documented or just plain bad use?

If it is a bug, is this a security problem that can be abused or somehow put users in some sort of risk?

Most issues we get reported as security issues are also in the end treated as such, as we tend to err on the safe side.

The time plan

Unless the issue is critical, we prefer to schedule a fix and announcement of the issue in association with the pending next release, and as we do releases every 8 weeks like clockwork, that’s never very far away.

We communicate the suggested schedule with the reporter to make sure we agree. If a sooner release is preferred, we work out a schedule for an extra release. In the past we’ve did occasional faster security releases also when the issue already had been made public, so we wanted to shorten the time window during which users could be harmed by the problem.

We really really do not want a problem to persist longer than until the next release.

The fix

The curl security team and the reporter work on fixing the issue. Ideally in part by the reporter making sure that they can’t reproduce it anymore and we add a test case or two.

We keep the fix undisclosed for the time being. It is not committed to the public git repository but kept in a private branch. We usually put it on a private URL so that we can link to it when we ask for a CVE, see below.

All security issues should make us ask ourselves – what did we do wrong that made us not discover this sooner? And ideally we should introduce processes, tests and checks to make sure we detect other similar mistakes now and in the future.

Typically we only generate a single patch from the git master master and offer that as the final solution. In the curl project we don’t maintain multiple branches. Distros and vendors who ship older or even multiple curl versions backport the patch to their systems by themselves. Sometimes we get backported patches back to offer users as well, but those are exceptions to the rule.

The advisory

In parallel to working on the fix, we write up a “security advisory” about the problem. It is a detailed description about the problem, what impact it may have if triggered or abused and if we know of any exploits of it.

What conditions need to be met for the bug to trigger. What’s the version range that is affected, what’s the remedies that can be done as a work-around if the patch is not applied etc.

We work out the advisory in cooperation with the reporter so that we get the description and the credits right.

The advisory also always contains a time line that clearly describes when we got to know about the problem etc.

The CVE

Once we have an advisory and a patch, none of which needs to be their final versions, we can proceed and ask for a CVE. A CVE is a unique “ID” that is issued for security problems to make them easy to reference. CVE stands for Common Vulnerabilities and Exposures.

Depending on where in the release cycle we are, we might have to hold off at this point. For all bugs that aren’t proprietary-operating-system specific, we pre-notify and ask for a CVE on the distros@openwall mailing list. This mailing list prohibits an embargo longer than 14 days, so we cannot ask for a CVE from them longer than 2 weeks in advance before our release.

The idea here is that the embargo time gives the distributions time and opportunity to prepare updates of their packages so they can be pretty much in sync with our release and reduce the time window their users are at risk. Of course, not all operating system vendors manage to actually ship a curl update on two weeks notice, and at least one major commercial vendor regularly informs me that this is a too short time frame for them.

For flaws that don’t affect the free operating systems at all, we ask MITRE directly for CVEs.

The last 48 hours

When there is roughly 48 hours left until the coming release and security announcement, we merge the private security fix branch into master and push it. That immediately makes the fix public and those who are alert can then take advantage of this knowledge – potentially for malicious purposes. The security advisory itself is however not made public until release day.

We use these 48 hours to get the fix tested on more systems to verify that it is not doing any major breakage. The weakest part of our security procedure is that the fix has been worked out in secret so it has not had the chance to get widely built and tested, so that is performed now.

The release

We upload the new release. We send out the release announcement email, update the web site and make the advisory for the issue public. We send out the security advisory alert on the proper email lists.

Bug Bounty?

Unfortunately we don’t have any bug bounties on our own in the curl project. We simply have no money for that. We actually don’t have money at all for anything.

Hackerone offers bounties for curl related issues. If you have reported a critical issue you can request one from them after it has been fixed in curl.