Category Archives: Open Source

Open Source, Free Software, and similar

cURL and libcurl, wolfSSL

curl 7.65.2 fixes even more

Six weeks after our previous bug-fix release, we ship a second release in a row with nothing but bug-fixes. We call it 7.65.2. We decided to go through this full release cycle with a focus on fixing bugs (and not merge any new features) since even after 7.65.1 shipped as a bug-fix only release we still seemed to get reports indicating problems we wanted fixed once and for all.

Download curl from curl.haxx.se as always!

Also, I personally had a vacation already planned to happen during this period (and I did) so it worked out pretty good to take this cycle as a slightly calmer one.

Of the numbers below, we can especially celebrate that we’ve now received code commits by more than 700 persons!

Numbers

the 183rd release
0 changes
42 days (total: 7,789)
76 bug fixes (total: 5,259)
113 commits (total: 24,500)
0 new public libcurl function (total: 80)
0 new curl_easy_setopt() option (total: 267)
0 new curl command line option (total: 221)
46 contributors, 25 new (total: 1,990)
30 authors, 19 new (total: 706)
1 security fix (total: 90)
200 USD paid in Bug Bounties

Security

Since the previous release we’ve shipped a security fix. It was special in the way that it wasn’t actually a bug in the curl source code, but in the build procedure for how we made curl builds for Windows. For this report, we paid out a 200 USD bug bounty!

Bug-fixes of interest

As usual I’ve carved out a list with some of the bugs since the previous release that I find interesting and that could warrant a little extra highlighting. Check the full changelog on the curl site.

bindlocal: detect and avoid IP version mismatches in bind

It turned out you could ask curl to connect to a IPv4 site and if you then asked it to bind to an interface in the local end, it could actually bind to an ipv6 address (or vice versa) and then cause havok and fail. Now we make sure to stick to the same IP version for both!

configure: more –disable switches to toggle off individual features

As part of the recent tiny-curl effort, more parts of curl can be disabled in the build and now all of them can be controlled by options to the configure script. We also now have a test that verifies that all the disabled-defines are indeed possible to set with configure!

(A future version could certainly get a better UI/way to configure which parts to enable/disable!)

http2: call done_sending on end of upload

Turned out a very small upload over HTTP/2 could sometimes end up not getting the “upload done” flag set and it would then just linger around or eventually cause a time-out…

libcurl: Restrict redirect schemes to HTTP(S), and FTP(S)

As a stronger safety-precaution, we’ve now made the default set of protocols that are accepted to redirect to much smaller than before. The set of protocols are still settable by applications using the CURLOPT_REDIR_PROTOCOLS option.

multi: enable multiplexing by default (again)

Embarrassingly enough this default was accidentally switched off in 7.65.0 but now we’re back to enabling multiplexing by default for multi interface uses.

multi: fix the transfer hashes in the socket hash entries

The handling of multiple transfers on the same socket was flawed and previous attempts to fix them were incorrect or simply partial. Now we have an improved system and in fact we now store a separate connection hash table for each internal separate socket object.

openssl: fix pubkey/signature algorithm detection in certinfo

The CURLINFO_CERTINFO option broke with OpenSSL 1.1.0+, but now we have finally caught up with the necessary API changes and it should now work again just as well independent of which version you build curl to use!

runtests: keep logfiles around by default

Previously, when you run curl’s test suite, it automatically deleted the log files on success and you had to use runtests.pl -k to prevent it from doing this. Starting now, it will erase the log files on start and not on exit so they will now always be kept on exit no matter how the tests run. Just a convenience thing.

runtests: report single test time + total duration

The output from runtests.pl when it runs each test, one by one, will now include timing information about each individual test. How long each test took and how long time it has spent on the tests so far. This will help us detect if specific tests suddenly takes a very long time and helps us see how they perform in the remote CI build farms etc.

Next?

I truly think we’ve now caught up with the worst problems and can now allow features to get merged again. We have some fun ones in the pipe that I can’t wait to put in the hands of users out there…

cURL and libcurl, Open Source

openssl engine code injection in curl

This flaw is known as CVE-2019-5443.

If you downloaded and installed a curl executable for Windows from the curl project before June 21st 2019, go get an updated one. Now.

On Windows, using OpenSSL

The official curl builds for Windows – that the curl project offers – are built cross-compiled on Linux. They’re made to use OpenSSL by default as the TLS backend, the by far most popular TLS backend by curl users.

The curl project has provided official curl builds for Windows on and off through history, but most recently this has been going on since August 2018.

OpenSSL engines

These builds use OpenSSL. OpenSSL has a feature called “engines”. Described by the project itself like this:

“a component to support alternative cryptography implementations, most commonly for interfacing with external crypto devices (eg. accelerator cards). This component is called ENGINE”

More simply put, an “engine” is a plugin for OpenSSL that can be loaded and run dynamically. The particular engine is activated either built-in or by loading a config file that specifies what to do.

curl and OpenSSL engines

When using curl built with OpenSSL, you can specify an “engine” to use, which in turn allows users to use their dedicated hardware when doing TLS related communications with curl.

By default, the curl tool allows OpenSSL to load a config file and figure out what engines to load at run-time but it also provides a build option to make it possible to build curl/libcurl without the ability to load that config file at run time – which some users want, primarily for security reasons.

The mistakes

The primary mistake in the curl build for Windows that we offered, was that the disabling of the config file loading had a typo which actually made it not disable it (because the commit message had it wrong). The feature was therefore still present and would load the config file if present when curl was invoked, contrary to the intention.

The second mistake comes a little more from the OpenSSL side: by default if you build OpenSSL cross-compiled like we do, the default paths where it looks for the above mentioned config file is under the c:\usr\local tree. It is in fact even complicated and impossible to fix this path in the build without a patch.

What the mistakes enable

A non-privileged user or program (the attacker) with access to the host to put a config file in the directory where curl would look for a config file (and create the directory first as it probably didn’t already exist) and the suitable associated engine code.

Then, when an privileged user subsequently executes curl, it will run with more power and run the code, the engine, the attacker had put there. An engine is a piece of compiled code, it can do virtually anything on the machine.

The fix

Already three days ago, on June 21st, a fixed version of the curl executable for Windows was uploaded to the curl web site (“curl 7.65.1_2”). All older versions that had been provided in the past were removed to reduce the risk of someone still using an old lingering download link.

The fix now makes the curl build switch off the loading of the config file, as was already intended. But also, the OpenSSL build that is used for the build is now modified to only load the config file from a privileged path that isn’t world writable (C:/Windows/System32/OpenSSL/).

Widespread mistake

This problem is very widespread among projects on Windows that use OpenSSL. The curl project coordinated this publication with the postgres project and have worked with OpenSSL to make them improve their default paths. We have also found a few other openssl-using projects that already have fixed their builds for this flaw (like stunnel) but I think we have reason to suspect that there are more vulnerable projects out there still not fixed.

If you know of a project that uses OpenSSL and ships binaries for Windows, give them a closer look and make sure they’re not vulnerable to this.

The cat is already out of the bag

When we got this problem reported, we soon realized it had already been publicly discussed and published for other projects even before we got to know about it. Due to this, we took it to publication as quick as possible to minimize user impact as much as we can.

Only on Windows and only with OpenSSL

This flaw only exists on curl for Windows and only if curl was built to use OpenSSL with this bad path and behavior.

Microsoft ships curl as part of Windows 10, but it does not use OpenSSL and is not vulnerable.

Credits

This flaw was reported to us by Rich Mirch.

The build was fixed by Viktor Szakats.

The image on the blog post comes from pixabay.

cURL and libcurl, Open Source

Google to reimplement curl in libcrurl

10 Comments

Not the entire thing, just “a subset”. It’s not stated very clearly exactly what that subset is but the easy interface is mentioned in the Chrome bug about this project.

What?

The Chromium bug states that they will create a library of their own (named libcrurl) that will offer (parts of) the libcurl API and be implemented using Cronet.

Cronet is the networking stack of Chromium put into a library for use on mobile. The same networking stack that is used in the Chrome browser.

There’s also a mentioned possibility that “if this works”, they might also create “crurl” tool which is then their own version of the curl tool but using their own library. In itself is a pretty strong indication that their API will not be fully compatible, as if it was they could just use the existing curl tool…

Why?

“Implementing libcurl using Cronet would allow developers to take advantage of the utility of the Chrome Network Stack, without having to learn a new interface and its corresponding workflow. This would ideally increase ease of accessibility of Cronet, and overall improve adoption of Cronet by first-party or third-party applications.”

Logically, I suppose they then also hope that 3rd party applications can switch to this library (without having to change to another API or adapt much) and gain something and that new applications can use this library without having to learn a new API. Stick to the old established libcurl API.

How?

By throwing a lot of man power on it. As the primary author and developer of the libcurl API and the libcurl code, I assume that Cronet works quite differently than libcurl so there’s going to be quite a lot of wrestling of data and code flow to make this API work on that code.

The libcurl API is also very versatile and is an API that has developed over a period of almost 20 years so there’s a lot of functionality, a lot of options and a lot of subtle behavior that may or may not be easy or straight forward to mimic.

The initial commit imported the headers and examples from the curl 7.65.1 release.

Will it work?

Getting basic functionality for a small set of use cases should be simple and straight forward. But even if they limit the subset to number of functions and libcurl options, making them work exactly as we have them documented will be hard and time consuming.

I don’t think applications will be able to arbitrarily use either library for a very long time, if ever. libcurl has 80 public functions and curl_easy_setopt alone takes 268 different options!

Given enough time and effort they can certainly make this work to some degree.

Releases?

There’s no word on API/ABI stability or how they intend to ship or version their library. It is all very early still. I suppose we will learn more details as and if this progresses.

Flattered?

I think this move underscores that libcurl has succeeded in becoming an almost defacto standard for network transfers.

A Google office building in New York.

There’s this saying about imitation and flattery but getting competition from a giant like Google is a little intimidating. If they just put two paid engineers on their project they already have more dedicated man power than the original libcurl project does…

How will it affect curl?

First off: this doesn’t seem to actually exist for real yet so it is still very early.

Ideally the team working on this from Google’s end finds and fixes issues in our code and API so curl improves. Ideally this move makes more users aware of libcurl and its API and we make it even easier for users and applications in the world to do safe and solid Internet transfers. If the engineers are magically good, they offer a library that can do things better than libcurl can, using the same API so application authors can just pick the library they find work the best. Let the best library win!

Unfortunately I think introducing half-baked implementations of the API will cause users grief since it will be hard for users to understand what API it is and how they differ.

Since I don’t think “libcrurl” will be able to offer a compatible API without a considerable effort, I think applications will need to be aware of which of the APIs they work with and then we have a “split world” to deal with for the foreseeable future and that will cause problems, documentation problems and users misunderstanding or just getting things wrong.

Their naming will possibly also be reason for confusion since “libcrurl” and “crurl” look so much like typos of the original names.

We are determined to keep libcurl the transfer library for the internet. We support the full API and we offer full backwards compatibility while working the same way on a vast amount of different platforms and architectures. Why use a copy when the original is free, proven and battle-tested since years?

Rights?

Just to put things in perspective: yes they’re perfectly allowed and permitted to do this. Both morally and legally. curl is free and open source and licensed under the MIT license.

Good luck!

I wish the team working on this the best of luck!

Updates after initial post

Discussions: the hacker news discussion, the reddit thread, the lobsters talk.

Rename? it seems the google library might change name to libcurl_on_cronet.

Update in April 2020:

According to an update to the bug entry dated February 28th 2020:

Remove libcurl_on_cronet and dependencies.

This project was never finished, and we have no current plans to
continue development.

cURL and libcurl

7.65.1 patched up and ready to go

4 Comments

(download it from curl.haxx.se of course!)

Whatever we do and whatever we try, no matter how hard we try to test, debug, review and do CI builds it does not change the eternal truth:

Nothing gets tested properly until released.

We worked hard on fixing bugs in the weeks before we shipped curl 7.65.0. We really did. Yet, several annoying glitches managed to creep in, remain unnoticed and cause problems to users when they first eagerly tried out the new release. Those were glitches that none in the development team had experienced or discovered but only took a few hours for users to detect and report.

The initial bad sign was that it didn’t even take a full hour from the release announcement until the first bug on 7.65.0 was reported. And it didn’t stop with that issue. We obviously had a whole handful of small bugs that caused friction to users who just wanted to get the latest curl to play with. The bugs were significant and notable enough that I quickly decided we should patch them up and release an update that has them fixed: 7.65.1. So here it is!

This patch release even got delayed. Just the day before the release we started seeing weird crashes in one of the CI builds on macOS and they still remained on the morning of the release. That made me take the unusual call to postpone the release until we better understood what was going on. That’s the reason why this comes 14 days after 7.65.0 instead of a mere 7 days.

Numbers

the 182nd release
0 changes
14 days (total: 7,747)
35 bug fixes (total: 5,183)
61 commits (total: 24,387)
0 new public libcurl function (total: 80)
0 new curl_easy_setopt() option (total: 267)
0 new curl command line option (total: 221)
27 contributors, 12 new (total: 1,965)
16 authors, 6 new (total: 687)
0 security fixes (total: 89)
0 USD paid in Bug Bounties

Bug-fixes

Let me highlight some of the fixes that went this during this very brief release cycle.

build correctly with OpenSSL without MD4

This was the initial bug report, reported within an hour from the release announcement of 7.65.0. If you built and installed OpenSSL with MD4 support disabled, building curl with that library failed. This was a regression since curl already supported this and due to us not having this build combination in our CI builds we missed it… Now it should work again!

CURLOPT_LOW_SPEED_* repaired

In my work that introduces more ways to disable specific features in curl so that tiny-curl would be as small as possible, I accidentally broke this feature (two libcurl options that allow a user to stop a transfer that goes below a certain transfer speed threshold during a given time). I had added a way to disable the internal progress meter functionality, but obviously not done a good enough job!

The breakage proved we don’t have proper tests for this functionality. I reverted the commit immediately to bring back the feature, and when now I go back to fix this and land a better fix soon, I now also know that I need to add tests to verify.

multi: track users of a socket better

Not too long ago I found and fixed a pretty serious flaw in curl’s HTTP/2 code which made it deal with multiplexed transfers over the same single connection in a manner that was far from ideal. When fixed, it made curl do HTTP/2 better in some circumstances.

This improvement ended up proving itself to have a few flaws. Especially when the connection is closed when multiple streams are done over it. This bug-fix now makes curl closing down such transfers in a better and cleaner way with fewer “loose ends”.

parse_proxy: use the IPv6 zone id if given

One more zone id fix that I didn’t get around to land in 7.65.0 has now landed: specifying a proxy with a URL that includes an IPv6 numerical address and a zone id – now works.

connection “bundles” on same host but different ports

Internally, libcurl collects connections to a host + port combination in a “bundle” (that’s just a term used for this concept internally). It does this to count number of connections to this combination and enforce limits etc. It is only used a bit for controlling when multiplexing can be done or not on this host.

Due to a regression, probably added already back in 7.62.0, this logic always used the default port for the protocol instead of the actual port number used in the given URL! An application that for example did parallel HTTP transfers to the hostname “example.org” on both port 80 and port 81, and used HTTP/1 on one of the ports and HTTP/2 on the other would be totally mixed up by curl and cause transfer failures.

But not anymore!

Coming up

This patch release was not planned. We will give this release a few days to stew and evaluate the situation. If we keep getting small or big bugs reported, we might not open the feature window at all in this release cycle and instead just fix bugs.

Ideally however, we’ve now fixed the most pressing ones and we can now move on and follow our regular development process. Even if we have, the feature window for next release will be open during a shorter period than normal.

cURL and libcurl

curl user survey 2019 analysis

3 Comments

The annual curl user survey 2019 ran for 14 days and ended a while ago. I’ve spent a good deal of time summing up the data, making graphs, tables and creating a document out of what I’ve learned.

Some quick insights:

  • HTTPS is now the most used protocol
  • Linux is the most used platform
  • Most of the users (who answered) are in Europe
  • Windows 10 grows as the dominant Windows version used for curl
  • 55% of users use HTTP/2 while 4.1% of users use HTTP/0.9

For all this and much much more. See the full report.

cURL and libcurl, Development, Open Source

curl: 3K forks

It’s just another meaningless number, but today there are 3,000 forks done of the curl GitHub repository.

This pops up just a little over three years since we reached our first 1,000 forks. Also, 10,000 stars no too long ago.

Why fork?

A typical reason why people fork a project on GitHub, is so that they can make a change in their own copy of the source code and then suggest that change to the project in the form of a pull-request.

The curl project has almost 700 individual commit authors, which makes at least 2,300 forks done who still haven’t had their pull-requests accepted! Of course those are 700 contributors who actually managed to work all the way through to inclusion. We can imagine that there is a huge number of people who only ever thought about doing a change, some who only ever just started to do it, many who ditched the idea before it was completed, some who didn’t actually manage to implement it properly, some who got their idea and suggestion shut down by the project and of course, lots of people still have their half-finished change sitting there waiting for inspiration.

Then there are people who just never had the intention of sending any change back. Maybe they just wanted to tinker with the code and have fun. Some want to do private changes they don’t want to offer or perhaps they already know the upstream project won’t accept.

We just can’t tell.

Many?

Is 3,000 forks a lot or a little? Both. It is certainly more forks than we’ve ever had before in this project. But compared to some of the most popular projects on GitHub, even comparing to some other C projects (on GitHub the most popular projects are never written in C) our numbers are dwarfed by the really popular ones. You can probably guess which ones they are.

In the end, this number is next to totally meaningless as it doesn’t say anything about the project nor about what contributions we get or will get in the future. It tells us we have (or had) the attention of a lot of users and that’s about it.

I will continue to try to make sure we’re worth the attention, both now and going forward!

(Picture from pixabay.)

cURL and libcurl, Security

Report from the curl bounty program

We announced our glorious return to the “bug bounty club” (projects that run bug bounties) a month ago, and with the curl 7.65.0 release today on May 22nd of 2019 we also ship fixes to security vulnerabilities that were reported within this bug bounty program.

Announcement

Even before we publicly announced the program, it was made public on the Hackerone site. That was obviously enough to get noticed by people and we got the first reports immediately!

We have received 19 reports so far.

Infrastructure scans

Quite clearly some people have some scripts laying around and they do some pretty standard things on projects that pop up on hackerone. We immediately got a number of reports that reported variations of the same two things repeatedly:

  1. Our wiki is world editable. In my world I’ve lived under the assumption that this is how a wiki is meant to be but we ended up having to specifically mention this on curl’s hackerone page: yes it is open for everyone on purpose.
  2. Sending emails forging them to look like the come from the curl web site might work since our DNS doesn’t have SPF, DKIM etc setup. This is a somewhat better report, but our bounty program is dedicated for and focused on the actual curl and libcurl products. Not our infrastructure.

Bounties!

Within two days of the program’s life time, the first legit report had been filed and then within a few more days a second arrived. They are CVE-2019-5435 and CVE-2019-5436, explained somewhat in my curl 7.65.0 release post but best described in their individual advisories, linked to below.

I’m thrilled to report that these two reporters were awarded money for their findings:

Wenchao Li was awarded 150 USD for finding and reporting CVE-2019-5435.

l00p3r was awarded 200 USD for finding and reporting CVE-2019-5436.

Both these issues were rated severity level “Low” and we consider them rather obscure and not likely to hurt very many users.

Donate to help us fund this!

Please notice that we are entirely depending on donated funds to be able to run this program. If you use curl and benefit from a more secure curl, please consider donating a little something for the cause!

cURL and libcurl

curl 7.65.0 dances in

After another eight week cycle was been completed, curl shipped a new release into the world. 7.65.0 brings some news and some security fixes but is primarily yet again a set of bug-fixes bundled up. Remember 7.64.1?

As always, download it straight from curl.haxx.se!

One fun detail on this release: we have 500 less lines of source code in the lib/ directory compared to the previous release!

Things that happened in curl since last release:

Numbers

the 181st release
3 changes
56 days (total: 7,733)
119 bug fixes (total: 5,148)
215 commits (total: 24,326)
0 new public libcurl function (total: 80)
1 new curl_easy_setopt() option (total: 267)
0 new curl command line option (total: 221)
50 contributors, 24 new (total: 1,953)
32 authors, 12 new (total: 681)
2 security fixes (total: 89)
350 USD paid in Bug Bounties

News

  1. libcurl has deprecated support for the global DNS cache.
  2. Pipelining support is now completely removed from curl.
  3. CURLOPT_MAXAGE_CONN is a new option that controls how long to keep a live connection in the connection cache for reuse.

Security

This release comes with fixes for two separate security problems. Both rated low risk. Both reported via the new bug bounty program.

CVE-2019-5435 is an issue in the recently introduced URL parsing API. It is only a problem in 32 bit architectures and only if an application can be told to pass in ridiculously long (> 2GB) strings to libcurl. This bug is similar in nature to a few other bugs libcurl has had in the past, and to once and for all combat this kind of flaw libcurl now (in 7.65.0 and forward) has a “maximum string length” limit for strings that you can pass to it using its APIs. The maximum size is 8MB. (The reporter was awarded 150 USD for this find.)

CVE-2019-5436 is a problem in the TFTP code. If an application decides to uses a smaller “blksize” than 504 (default is 512), curl would overflow a buffer allocated on the heap with data received from the server. Luckily, very few people actually download data from unknown or even remote TFTP servers. Secondly, asking for a blksize smaller than 512 is rather pointless and also very rare: the primary point in changing that size is to enlarge it. (The reporter was awarded 200 USD for this find.)

Bug-fixes

Over one hundred bug-fixes landed in this release, but some of my favorites from release cycle include…

mark connection for close on TLS close_notify

close_notify is a message in the TLS protocol that means that this connection is about to close. In most circumstances that message doesn’t actually provide information to curl that is needed, but in the case the connection is closed prematurely, understanding that this message preceded the closure helps curl act appropriately. This change was done for the OpenSSL backend only as that’s where we got the bug reported and worked on it this time, but I think we might have reasons to do the same for other backends going forward!

show port in the verbose “Trying …” message

The verbose message that says “Trying 12.34.56.78…” means that curl has sent started a TCP connect attempt to that IP address. This message has now been modified to also include the target port number so when using -v with curl 7.65.0, connecting to that same host for HTTPS will instead say “Trying 12.34.56.78:443…”.

To aid debugging really. I think it gives more information faster at a place you’re already looking.

new SOCKS 4+5 test server

The test suite got a brand new SOCKS server! Previously, all SOCKS tests for both version 4 and version 5 were done by firing up ssh (typically openssh). That method was decent but made it hard to do a range of tests for bad behavior, bad protocol replies and similar. With the new custom test server, we can basically add whatever test we want and we’ve already extended the SOCKS testing to cover more code and use cases than previously.

SOCKS5 user name and passwords must be shorter than 256

curl allows user names and passwords provided in URLs and as separate options to be more or less unrestricted in size and that include if the credentials are used for SOCKS5 authentication – totally ignoring the fact that the protocol SOCKS5 has a maximum size of 255 for the fields. Starting now, curl will return an error if the credentials for SOCKS5 are too long.

Warn if curl and libcurl versions do not match

The command line tool and the library are independent and separable, as in you can run one version of the curl tool with another version of the libcurl library. The libcurl API is solid enough to allow it and the tool is independent enough to not restrict it further.

We always release curl the command line tool and libcurl the library together, using the same version number – with the code for both shipped in the same single file.

There should rarely be a good reason to actually run curl and libcurl with different versions. Starting now, curl will show a little warning if this is detected as we have learned that this is almost always a sign of an installation or setup mistake. Hopefully this message will aid people to detect the mistake earlier and easier.

Better handling of “–no-” prefixed options

curl’s command line parser allows users to switch off boolean options by prefixing them with dash dash no dash. For example we can switch off compressed responses by using “–no-compression” since there regular option “–compression” switches it on.

It turned out we stripped the “–no-” thing no regarding if the option was boolean or not and presumed the logic to handle it – which it didn’t. So users could actually pass a proxy string to curl with the regular option “–proxy” as well as “–no-proxy”. The latter of course not making much sense and was just due to an oversight.

In 7.65.0, only actual boolean command line options can be used with “–no-“. Trying it on other options will cause curl to report error for it.

Add CURLUPART_ZONEID to the URL API

Remember when we added a new URL parsing API to libcurl back in 7.62.0? It wasn’t even a year ago! When we did this, we also changed the internals to use the same code. It turned out we caused a regression when we parsed numerical IPv6 addresses that provide the zone ID within the string. Like this: “https://[ffe80::1%25eth0]/index.html”

Starting in this release, you can both set and get the zone ID in a URL using the API, but of course setting it doesn’t do anything unless the host is a numeric IPv6 address.

parse proxy with the URL parser API

We removed the separate proxy string parsing logic and instead switched that over to more appropriately use the generic URL parser for this purpose as well. This move reduced the code size, made the code simpler and makes sure we have a unified handling of URLs! Everyone is happy!

longer URL schemes

I naively wrote the URL parser to handle scheme names as long as the longest scheme we support in curl: 8 bytes. But since the parser can also be asked to parse URLs with non-supported schemes, that limit was a bit too harsh. I did a quick research, learned that the longest currently registered URI scheme is 36 characters (“microsoft.windows.camera.multipicker”). Starting in this release , curl accepts URL schemes up to 40 bytes long.

Coming up next

There’s several things brewing in the background that might be ready to show in next release. Parallel transfers in the curl tool and deprecating PolarSSL support seem likely to happen for example. Less likely for this release, but still being worked on slowly, is HTTP/3 support.

We’re also likely to get a bunch of changes and fine features we haven’t even thought about from our awesome contributors. In eight weeks I hope to write another one of these blog posts explaining what went into that release…

cURL and libcurl

The curl user survey 2019

the survey

For the 6th consecutive year, the curl project is running a “user survey” to learn more about what people are using curl for, what think think of curl, what the need of curl and what they wish from curl going forward.

the survey

As in most projects, we love to learn more about our users and how to improve. For this, we need your input to guide us where to go next and what to work on going forward.

the survey

Please consider donating a few minutes of your precious time and tell me about your views on curl. How do you use it and what would you like to see us fix?

the survey

The survey will be up for 14 straight days and will be taken down at midnight (CEST) May 26th. We appreciate if you encourage your curl friends to participate in the survey.

Bonus: the analysis from the 2018 survey.

cURL and libcurl, wolfSSL

tiny-curl

10 Comments

curl, or libcurl specifically, is probably the world’s most popular and widely used HTTP client side library counting more than six billion installs.

curl is a rock solid and feature-packed library that supports a huge amount of protocols and capabilities that surpass most competitors. But this comes at a cost: it is not the smallest library you can find.

Within a 100K

Instead of being happy with getting told that curl is “too big” for certain use cases, I set a goal for myself: make it possible to build a version of curl that can do HTTPS and fit in 100K (including the wolfSSL TLS library) on a typical 32 bit architecture.

As a comparison, the tiny-curl shared library when built on an x86-64 Linux, is smaller than 25% of the size as the default Debian shipped library is.

FreeRTOS

But let’s not stop there. Users with this kind of strict size requirements are rarely running a full Linux installation or similar OS. If you are sensitive about storage to the exact kilobyte level, you usually run a more slimmed down OS as well – so I decided that my initial tiny-curl effort should be done on FreeRTOS. That’s a fairly popular and free RTOS for the more resource constrained devices.

This port is still rough and I expect us to release follow-up releases soon that improves the FreeRTOS port and ideally also adds support for other popular RTOSes. Which RTOS would you like to support for that isn’t already supported?

Offer the libcurl API for HTTPS on FreeRTOS, within 100 kilobytes.

Maintain API

I strongly believe that the power of having libcurl in your embedded devices is partly powered by the libcurl API. The API that you can use for libcurl on any platform, that’s been around for a very long time and for which you can find numerous examples for on the Internet and in libcurl’s extensive documentation. Maintaining support for the API was of the highest priority.

Patch it

My secondary goal was to patch as clean as possible so that we can upstream patches into the main curl source tree for the changes makes sense and that aren’t disturbing to the general code base, and for the work that we can’t we should be able to rebase on top of the curl code base with as little obstruction as possible going forward.

Keep the HTTPS basics

I just want to do HTTPS GET

That’s the mantra here. My patch disables a lot of protocols and features:

  • No protocols except HTTP(S) are supported
  • HTTP/1 only
  • No cookie support
  • No date parsing
  • No alt-svc
  • No HTTP authentication
  • No DNS-over-HTTPS
  • No .netrc parsing
  • No HTTP multi-part formposts
  • No shuffled DNS support
  • No built-in progress meter

Although they’re all disabled individually so it is still easy to enable one or more of these for specific builds.

Downloads and versions?

Tiny-curl 0.9 is the first shot at this and can be downloaded from wolfSSL. It is based on curl 7.64.1.

Most of the patches in tiny-curl are being upstreamed into curl in the #3844 pull request. I intend to upstream most, if not all, of the tiny-curl work over time.

License

The FreeRTOS port of tiny-curl is licensed GPLv3 and not MIT like the rest of curl. This is an experiment to see how we can do curl work like this in a sustainable way. If you want this under another license, we’re open for business over at wolfSSL!