Tag Archives: OpenSSL

cURL and libcurl, Open Source, Security

darwin native SSL for curl

1 Comment

I recently mentioned the new schannel support for libcurl that allows libcurl to do SSL natively without the use of any external libraries on Windows.

This “getting native support” obviously triggered Nick Zitzmann who stepped up and sent in Secure Transport support – the native API for doing SSL on Mac OS X and iOS. This ninth supported SSL library is now called ‘darwinssl’ in the curl code base. There have been some follow-up commits too to cleanup things and make use of that API for providing the necessary function calls when doing NTLM too etc.

This functionality is merged in to curl’s master git repository and will be part of the upcoming curl 7.27.0 release, planned to hit the public at the end of July 2012.

It could be noted that if your for example build curl/libcurl to also support SCP and SFTP, you’d be linking with libssh2 for that and libssh2 is still relying on a crypto library that is either OpenSSL or gcrypt so you may in fact still end up linking with a 3rd party crypto library… Nick mentioned in a separate mail how he has looked into making libssh2 use the Secure Transport API, but that he faced some issues regarding big numbers which made him hesitate and consider how to move forward.

cURL and libcurl, Security

Sloppily using SSL_OP_ALL

This story begins with a security flaw in OpenSSL. OpenSSL is truly a fundamental piece of software these days and I would go so far and say that lots of our critical infrastructure today is using it and needs it. Flaws in OpenSSL literally affect entire societies or at least risk doing so if the flaws can be exploited.

SSL/TLS is a rather old and well used protocol with many different implementations, both client and server side. In order to enhance how OpenSSL works with older SSL implementations or just those that have different views on how to implement things, OpenSSL provides an API call to tweak behaviors. The SSL_CTX_set_options function. In the curl project we’ve found good use of it for this purpose, and we use the generic define SSL_OP_ALL to switch on all “rather harmless” workarounds that OpenSSL offers. Rather harmless, that’s what the comment in the header file says.

Ok, enough background and dancing around the issue. The flaw that ignited my idea to write this blog post was a particular mistake made within SSL a long time ago within the code handling SSL 3.0 and TLS 1.0 protocols when speaking this protocol with a peer that could select the plain-text (see this explanation) – the problem is a generic one with the protocol so different SSL libraries would approach it differently. Ok, so OpenSSL fixed the flaw back in the days of 0.9.6d (we’re talking May 9th 2002). As a user of a library such as OpenSSL it always feels good to see them being on top of security problems and releasing fixes. It makes you feel that you’re being looked after to some extent.

Shortly thereafter, the OpenSSL developers discovered that some broken server implementations didn’t work with the work-around they had done…

Alas, on July 30th 2002 the OpenSSL team released version 0.9.6e which offered a way for programs to disable this particular work-around. By switching this off, it would of course make the protocol less secure again but it would inter-operate better with faulty servers. How do you switch off this security measure? By using the SSL_CTX_set_options function setting the bit SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS.

Ok, so far so good. But the next step is what changed everything from fine to not so fine anymore: they then added that new bit to the SSL_OP_ALL define.

Yes. In one blow every single application out there that use SSL_OP_ALL suddently started switching off this security measure as soon as they were recompiled against this version of OpenSSL. This change was made in 2002 and this is still like this today. It fixed the security problem from OpenSSL’s aspect, but the way the bit was later added to the SSL_OP_ALL define it was instead transferred to affect many programs.

In curl’s case, we were alerted about this flaw on January 19th 2012 and it resulted in a security advisory. I did a quick search for SSL_OP_ALL on koders.com and it is obvious that there are hundreds of programs out there still using this bitmask as-is. In the curl project we enabled the SSL_OP_ALL approach for the first time in the 7.10.6 release we did in July 2003. It was wrong already at the time we started using it. It turns out we’ve been enabling this flaw for almost nine years.

In the GnuTLS camp however, they simply stopped doing their work-around for this as soon as they started supporting TLS 1.1 due to the problems the work-around caused to some servers. This since TLS 1.1 isn’t vulnerable to the problem. OpenSSL 1.0.1 beta was released on Janurary 3 2012 and is the first OpenSSL version ever released to support TLS newer than 1.0… The browsers/NSS seem to have mitigated this problem in a different way and there’s a patch available for OpenSSL to implement the same work-around but there’s been no feedback on how or if it will be used.

cURL and libcurl

Top-3 curl changes in 2011

1 Comment

At the end of the year I thought it would be interesting to have a look back over the past twelve months and see what the biggest changes in the curl project were and what the most important bug-fixes were etc. I’ve turned into a little mini series of blog posts. The top-3s. First out, the top-3 changes.

Top-3 changes

In total I counted 29 notable changes brought during 2011. The most significant ones in my view are:

Change 1 – fancier protocol support in proxy strings

This may sound trivial, and the code certainly is, but with this change suddenly a lot of applications that use libcurl got better proxy support without having do anything at all. Previously the application would have to set what protocol type the proxy it would use is, even though libcurl has supported having the proxy specified as an environment variable for ages.

Having only the proxy name was useful but limiting. With this new change you can specify proxy type or rather proxy protocol by prefixing the proxy name like “socks4://magic-proxy.example.com” if you want a SOCKS4 proxy. libcurl now supports socks4, socks4a, socks5 or socks5h used as prefixes as well as http.

Change 2 – allow sending “empty” HTTP headers

Another minor change in code but possibly larger impact in usefulness for applications. We introduced a way for applications to change internal headers and to add new ones ages ago. We (or rather I) then made the choice that if you’d provide a header with only the name and a colon, that is with no contents on the right side, it would delete the internal header. That was not a clever move as later on people have wanted to add “blank” or “empty” headers that look exactly like that, but libcurl has then refused to.

There have been some more or less hackish work-arounds to trick libcurl into allowing an empty header, but now finally we introduce a nice and clean way for applications to pass in these kinds of empty headers:

Pass in an empty header that instead of a colon has a semicolon! This is an otherwise illegal header that wouldn’t make sense, but libcurl will use that as a trigger that an empty header should be used and it will then replace the semicolon with a colon and things will be fine.

Change 3 – Added support for cyassl and axTLS

Proving that libcurl moves forward and into more and more markets, the number of supported SSL libraries grew to 7 this year. The all new cyassl backend that replaced the previously done yassl backend that was using cyassl’s former OpenSSL emulation layer. Now we’re using the native and pure API and things are much cleaner. The possibly smallest available TLS library axTLS also got support.

Not all backends and not all SSL libraries are the same or support the same set of features, but then libcurl is used in many different scenarios and use cases and this way we offer more options to more users to craft libcurl for their particular needs. Our internal SSL backend API has managed quite well and proves to have been a worthy change. Adding support for yet another SSL library within libcurl is actually not a lot of work.

Change 4 – TSL-SRP

You may think number 4 of a top-3 list is weird, but I couldn’t cut it off here! =) TLS-SRP has been waiting in the shadows for so long and all of a sudden two of the major SSL libraries have support for it in released versions and libcurl got support for using these features in both libraries during 2011.

cURL

cURL and libcurl, libssh2, Open Source

libcurl, seven SSL libs and one SSH lib

1 Comment

I did a talk today at Fosdem with this title. The room only had 48 seats and it was completely packed with people standing everywhere it was possible around the seated guys.

The English slides from my talk are below. It was also recorded on video so I hope I’ll be able to post once it becomes available online

libcurl, seven SSL libraries and one SSH library
Linux, Open Source, Security

null-prefix domino

dominos

At the end of July 2009, Scott Cantor contacted us in the curl project and pointed out a security flaw in libcurl (in code that was using OpenSSL to verify server certificates). Having read his explanation I recalled that I had witnessed the discussion on the NSS list about this problem just a few days earlier (which resulted in their August 1st security advisory). The problem is basically that the cert can at times contain a name with an embedded zero in the middle, while most source code assumes plain C-style strings that ends with a zero. This turns out to be exploitable, and is explained in great detail in this document (PDF).

I started to work on a patch, and in the mean time I talked to Simon Josefsson of the GnuTLS team to see if GnuTLS was fine or not, only to get him confirm that GnuTLS did indeed have the same problem.

So I contacted vendor-sec, and then on the morning of August 5 I thought I’d just make a quick check how the other HTTPS client implementations do their cert checks.

Wget: vulnerable

neon: vulnerable

serf: vulnerable

So, Internet Explorer and Firefox were vulnerable. NSS and GnuTLS were. (OpenSSL wasn’t, but then it doesn’t provide this verifying feature by itself) (lib)curl, wget, neon, serf were all vulnerable. If that isn’t a large amount of the existing HTTPS clients then what is? I also think that this shows that it would be good for all of us if OpenSSL had this functionality, as even if it had been vulnerable we could’ve fixed a busload of different applications by repairing a single library. Now we instead need to hunt down all apps that use OpenSSL and that verify certificate names.

Quite clearly we (as implementers) have all had the same silly assumptions, and quite likely we’ve affected each other into doing these sloppy codes. SSL and certificates are over and over again getting hit by this kind of painful flaws and setbacks. Darn, getting things right really is very very hard…

(Disclaimer: I immediately notified the neon and serf projects but to my knowledge they have not yet released any fixed versions.)

cURL and libcurl

What is this yassl really?

yassl is said to be Yet Another SSL library and I’ve been told that for example it is the preferred library used by the mysql camp. I got interested in this several years ago when I learned about it since I thought it was fun to see an alternative implementation of OpenSSL that still offers the same API.

Since then, I’ve amused myself by trying to build and run curl with it like every six months or so. I’ve made (lib)curl build fine with yassl (and its configure script also detects that it is an OpenSSL API emulated by yassl), but I’ve never seen it run the entire curl test suite through without failing at least one test!

I asked the mysql guy about how yassl has worked for them, but he kind of shrugged and admitted that they hadn’t tried it much (and then I don’t know really who he spoke for, the entire team or just he and his closest friends) but he said it worked for them.

Today I noticed the yassl version 1.9.6 that I downloaded, built and tried against curl. This time curl completely fails to build with it…

Let me also point out that it’s not like I’ve not told the yassl team (person?) about these problems in the past. I have, and there have been adjustments that have been meant to address problems I’ve seen. I just can’t make curl use it successfully… libcurl can still be built and run with OpenSSL, GnuTLS or NSS so it’s not like we lack SSL library alternatives.

The same team/person seems to behind another SSL lib called Cyassl that’s aimed for smaller footprint systems and I’ve heard whispers about people trying to get libcurl to build against that and it surely is going to be interesting to see where that leads!

cURL and libcurl, Security, Web

Getting cacerts for your tools

As the primary curl author, I’m finding the comments here interesting. That blog entry “Teaching wget About Root Certificates” is about how you can get cacerts for wget by downloading them from curl’s web site, and people quickly point out how getting cacerts from an untrusted third party place of course is an ideal situation for an MITM “attack”.

Of course you can’t trust any files off a HTTP site or a HTTPS site without a “trusted” certificate, but thinking that the curl project would run one of those just to let random people load PEM files from our site seems a bit weird. Thus, we also provide the scripts we do all this with so that you can run them yourself with whatever input data you need, preferably something you trust. The more paranoid you are, the harder that gets of course.

On Fedora, curl does come with ca certs (at least I’m told recent Fedoras do) and even if it doesn’t, you can actually point curl to use whatever cacert you like and since most default installs of curl uses OpenSSL like wget does, you could tell curl to use the same cacert your wget install uses.

This last thing gets a little more complicated when one of the two gets compiled with a SSL library that doesn’t easily support PEM (read: NSS), but in the case of curl in recent Fedora they build it with NSS but with an additional patch that allows it to still be able to read PEM files.

cURL and libcurl, Open Source

CA cert bundle from Firefox

It could be interesting to note that extracting all the cacerts from your local Firefox installation isn’t that tricky, if you just use some of the magic that are at hand with the NSS certutil tool.

Users of OpenSSL or GnuTLS based tools or libraries (such as libcurl) might be pleased to learn this.

curl users in general of course should be aware that we no longer ship any ca-cert bundle with curl (as of curl 7.18.1), as it seems some ports haven’t yet updated or discovered this.

Update: this script is now present as lib/firefox-db2pem.sh in the curl CVS repository.

Development, Linux

Blaming Debian packaging

1 Comment

I happened to read the blog post called Open-Source Security Idiots which really is having a go at the poor Debian maintainer of OpenSSL for causing the recent much debated OpenSSL security problem in Debian and Debian-based distros.

While I think the author Steven J. Vaughan-Nichols is mostly correct about his criticism, I think he’s being far too specific and trying to pinpoint Debian and claiming that to be a single specific bad distro (and his additional confused complaint on Firefox vs Iceweasel just made the article lose focus).

As someone who’s involved in a bunch of projects that are being packed by a range of Linux distros, I can’t but to disagree. This habit of changing packages without passing the changes upstream is wide-spread and not limited to changes done by maintainers since it also includes mere bug reports. It is something that just about every distro is doing to at least some extent. It varies from package to package and over time, but given an overview I honestly can’t say that there’s a single specific distro that is worse than the others. It is a disease that follows the distros and we must all help out to exterminate it.

Of course, the upstream projects also need to be aware of this and help pushing packagers of their software to behave.

cURL and libcurl

libcurl built with yassl

yassl is a (in comparison) small SSL/TLS library that libcurl can be built to use instead of using one of the other SSL/TLS libraries libcurl supports (OpenSSL, GnuTLS, NSS and QSSL). However, yassl differs somewhat from the others in the way that it provides an OpenSSL-emulating API layer so that in libcurl we pretty much use exactly the same code for OpenSSL as we do for yassl.libcurl

yassl claimed this libcurl support for several years ago and indeed libcurl builds fine with it and you can even do some basic SSL operations with it, but the emulation is just not good enough to let the curl test suite go through so lots of stuff breaks when libcurl is built with yassl.

I did this same test 15 months ago and it had roughly the same problems then.

As far as I know, the other three main libraries are much more stable (the QSSL/QsoSSL library is only available on the AS/400 platform and thus I don’t personally know much about how it performs) and thus they remain the libraries I recommend users to select from if they want something that’s proven working.