Category Archives: Technology

Really everything related to technology

cURL and libcurl, Network

HTTP2 Expression of Interest: curl

For the readers of my blog, this is a copy of what I posted to the httpbis mailing list on July 12th 2012.

Hi,

This is a response to the httpis call for expressions of interest

BACKGROUND

I am the project leader and maintainer of the curl project. We are the open source project that makes libcurl, the transfer library and curl the command line tool. It is among many things a client-side implementation of HTTP and HTTPS (and some dozen other application layer protocols). libcurl is very portable and there exist around 40 different bindings to libcurl for virtually all languages/enviornments imaginable. We estimate we might have upwards 500
million users or so. We’re entirely voluntary driven without any paid developers or particular company backing.

HTTP/1.1 problems I’d like to see adressed

Pipelining – I can see how something that better deals with increasing bandwidths with stagnated RTT can improve the end users’ experience. It is not easy to implement in a nice manner and provide in a library like ours.

Many connections – to avoid problems with pipelining and queueing on the connections, many connections are used and and it seems like a general waste that can be improved.

HTTP/2.0

We’ve implemented HTTP/1.1 and we intend to continue to implement any and all widely deployed transport layer protocols for data transfers that appear on the Internet. This includes HTTP/2.0 and similar related protocols.

curl has not yet implemented SPDY support, but fully intends to do so. The current plan is to provide SPDY support with the help of spindly, a separate SPDY library project that I lead.

We’ve selected to support SPDY due to the momentum it has and the multiple existing implementaions that A) have multi-company backing and B) prove it to be a truly working concept. SPDY seems to address HTTP’s pipelining and many-connections problems in a decent way that appears to work in reality too. I believe SPDY keeps enough HTTP paradigms to be easily upgraded to for most parties, and yet the ones who can’t or won’t can remain with HTTP/1.1 without too much pain. Also, while Spindly is not production-ready, it has still given me the sense that implementing a SPDY protocol engine is not rocket science and that the existing protocol specs are good.

By relying on external libs for protocol and implementation details, my hopes is that we should be able to add support for other potentially coming HTTP/2.0-ish protocols that gets deployed and used in the wild. In the curl project we’re unfortunately rarely able to be very pro-active due to the nature of our contributors, which tends to make us follow the rest and implement and go with what others have already decided to go with.

I’m not aware of any competitors to SPDY that is deployed or used to any particular and notable extent on the public internet so therefore no other “HTTP/2.0 protocol” has been considered by us. The two biggest protocol details people will keep mention that speak against SPDY is SSL and the compression requirements, yet I like both of them. I intend to continue to participate in dicussions and technical arguments on the ietf-http-wg mailing list on HTTP details for as long as I have time and energy.

HTTP AUTH

curl currently supports Basic, Digest, NTLM and Negotiate for both host and proxy.

Similar to the HTTP protocol, we intend to support any widely adopted authentication protocols. The HOBA, SCRAM and Mutual auth suggestions all seem perfectly doable and fine in my perspective.

However, if there’s no proper logout mechanism provided for HTTP auth I don’t forsee any particular desire from browser vendor or web site creators to use any of these just like they don’t use the older ones either to any significant extent. And for automatic (non-browser) uses only, I’m not sure there’s motivation enough to add new auth protocols to HTTP as at least historically we seem to rarely be able to pull anything through that isn’t pushed for by at least one of the major browsers.

The “updated HTTP auth” work should be kept outside of the HTTP/2.0 work as far as possible and similar to how RFC2617 is separate from RFC2616 it should be this time around too. The auth mechnism should not be too tightly knit to the HTTP protocol.

The day we can clense connection-oriented authentications like NTLM from the HTTP world will be a happy day, as it’s state awareness is a pain to deal with in a generic HTTP library and code.

cURL and libcurl, Open Source, Security

Is there a case for a unified SSL front?

There are many, sorry very many, different SSL libraries today that various programs may want to use. In the open source world at least, it is more and more common that SSL (and other crypto) using programs offer build options to build with either at least OpenSSL or GnuTLS and very often they also offer optinal build with NSS and possibly a few other SSL libraries.

In the curl project we just added support for library number nine. In the libcurl source code we have an internal API that each SSL library backend must provide, and all the libcurl source code is internally using only that single and fixed API to do SSL and crypto operations without even knowing which backend library that is actually providing the functionality. I talked about libcurl’s internal SSL API before, and I asked about this on the libcurl list back in Feb 2011.

So, a common problem should be able to find a common solution. What if we fixed this in a way that would be possible for many projects to re-use? What if one project’s ability to select from 9 different provider libraries could be leveraged by others. A single SSL API with a simplified API but that still provides the functionality most “simple” SSL-using applications need?

Marc Hörsken and I have discussed this a bit, and pidgin/libpurble came up as a possible contender that could use such a single SSL. I’ve also talked about it with Claes Jakobsson and Magnus Hagander for postgresql and I know since before that wget certainly could use it. When I’ve performed my talks on the seven SSL libraries of libcurl I’ve been approached by several people who have expressed a desire in seeing such an externalized API, and I remember the guys from cyassl among those. I’m sure there will be a few other interested parities as well if this takes off.

What remains to be answered is if it is possible to make it reality in a decent way.

Upsides:

  1. will reduce code from the libcurl code base – in the amount of 10K or more lines of C code, and it should be a decreased amount of “own” code for all projects that would decide to use thins single SSL library
  2. will allow other projects to use one out of many SSL libraries, hopefully benefiting end users as a result
  3. should get more people involved in the code as more projects would use it, hopefully ending up in better tested and polished source code

There are several downsides with a unified library, from the angle of curl/libcurl and myself:

  1. it will undoubtedly lead to more code being added and implemented that curl/libcurl won’t need or use, thus grow the code base and the final binary
  2. the API of the unified library won’t be possible to be as tightly integrated with the libcurl internals as it is today, so there will be some added code and logic needed
  3. it will require that we produce a lot of documentation for all the functions and structs in the API which takes time and effort
  4. the above mention points will deduct time from my other projects (hopefully to benefit others, but still)

Some problems that would have to dealt with:

  1. how to deal with libraries that don’t provide functionality that the single SSL API does
  2. how to draw the line of what functionality to offer in the API, as the individual libraries will always provide richer and more complete APIs
  3. What is actually needed to get the work on this started for real? And if we do, what would we call the project…

PS, I know the term is more accurately “TLS” these days but somehow people (me included) seem to have gotten stuck with the word SSL to cover both SSL and TLS…

cURL and libcurl, Open Source, Security

darwin native SSL for curl

1 Comment

I recently mentioned the new schannel support for libcurl that allows libcurl to do SSL natively without the use of any external libraries on Windows.

This “getting native support” obviously triggered Nick Zitzmann who stepped up and sent in Secure Transport support – the native API for doing SSL on Mac OS X and iOS. This ninth supported SSL library is now called ‘darwinssl’ in the curl code base. There have been some follow-up commits too to cleanup things and make use of that API for providing the necessary function calls when doing NTLM too etc.

This functionality is merged in to curl’s master git repository and will be part of the upcoming curl 7.27.0 release, planned to hit the public at the end of July 2012.

It could be noted that if your for example build curl/libcurl to also support SCP and SFTP, you’d be linking with libssh2 for that and libssh2 is still relying on a crypto library that is either OpenSSL or gcrypt so you may in fact still end up linking with a 3rd party crypto library… Nick mentioned in a separate mail how he has looked into making libssh2 use the Secure Transport API, but that he faced some issues regarding big numbers which made him hesitate and consider how to move forward.

Network

lighting up that fiber

Exactly 10 hours and 34 minutes after Tyfon sent me the mail confirming they had received my order, the connection was up and I received an SMS saying so. Amazingly quick service I’d say. Unfortunately I wasn’t quite as fast to actually try it out…

Once I got home from work and got some time to fiddle, I inserted an RJ45 into port 1 of my media converter and the other end in my wifi router and wham, I was online.

My immediate reaction? First, check ping time to my server. Now it averages at 2.5 ms, down from some 32 ms over my ADSL line. Then check transfer speeds. Massive disappointment. Something is wrong since it goes very slow in both directions, with no more than 5-10KB/sec transfers. I emailed customer service at once, less than 24 hours after I ordered it… bredbandskollen.se says 0.20 mbit downlink and 75 mbit uplink! Weird.

They got back early this morning by email, and we communicated back and forth. For them to be able to file a report back to the fiber provider I need to report a MAC and IP address of a direct-connected (no router) computer, which of course had to wait until I get back home from work.

At home, when connecting two different windows-running laptops they don’t get an IP address. I’m suspecting this is due to packet-loss and thus it taking several DHCP retries to work and I didn’t have patience enough. I switched back to my ADSL connection again and emailed Tyfon the IP and MAC I believe my router used before…

Network

A network provider for my fiber

2 Comments

In late May I finally got my media converter installed inside my house so now my fiber gets terminated into a 4-port gigabit switch.

Now the quest to find the right provider started. I have a physical 1 gigabit connection to “the station”, and out of the 12 providers (listed on bredbandswebben.se) I can select to get the internet service delivered by, at least two offer 1000 mbit download speeds (with 100mbit upload). I would ideally like a fixed IPv4-address and an IPv6 subnet, and I want my company to subscribe to this service.

The companies are T3 and Alltele. Strangely enough both of them failed to respond in a timely manner, so I went on to probe a few of the other companies that deliver less than 1000mbit services.

The one company that responded fastest and with more details than any other was Tyfon. They informed me that currently nobody can sell a “company subscription” on this service and that on my address I can only get at most a 100/100 mbit service right now. (Amusingly most of these operators also offer 250/25 and 500/50 rates but I would really like to finally get a decent upstream speed so that I for example can backup to a remote site at a decent speed.)

So, I went with 100/100 mbit for 395 SEK/month (~ 44 Euro or 57 USD). I just now submitted my order and their confirmation arrived at 23:00:24. They say it may take a little while to deliver so we’ll see (“normally within 1-2 weeks“). I’ll report back when I have news.

(And I’ve not yet gotten the invoice for the physical installation…)

cURL and libcurl, Security, Windows

schannel support in libcurl

1 Comment

schannel is the API Microsoft provides to allow applications to for example implement SSL natively, without needing any third part library.

On Monday June 11th we merged the 30+ commits Marc Hörsken brought us. This is now the 8th SSL variation supported by libcurl, and I figure this is going to become fairly popular now in the Windows camp coming the next release: curl 7.27.0.

So now my old talk about the seven SSL libraries libcurl supported has become outdated…

It can be worth noting that as long as you build (lib)curl to also support SCP and SFTP, powered by libssh2, that library will still require a separate crypto library and libssh2 supports to get built with either OpenSSL or gcrypt. Marc mentioned that he might work on making that one use schannel as well.

cURL

cURL and libcurl, Network

shorter HTTP requests for curl

Starting in curl 7.26.0 (due to be released at the end of May 2012), we will shrink the User-agent: header that curl sends by default in HTTP(S) requests to something much shorter! I suspect that this will raise some eyebrows out there so even though I’ve emailed about it to the curl-users list before I thought I’d better write it up and elaborate.

A default ‘curl localhost’ on Debian Linux makes 170 bytes get sent in that single request:

GET / HTTP/1.1
User-Agent: curl/7.24.0 (i486-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0g zlib/1.2.6 libidn/1.23 libssh2/1.2.8 librtmp/2.3
Host: localhost
Accept: */*

As you can see, the user-agent description takes up a large portion of that request, and this for really no good reason at all. Without sacrificing any functionality I shrunk the same request down to 71 bytes:

GET / HTTP/1.1
User-Agent: curl/7.24.0
Host: localhost
Accept: */*

That means we shrunk it down to 41% of the original size. I’ll admit the example is a bit extreme and most other normal use cases will use longer host names and longer paths, but even for a URL like “https://daniel.haxx.se/docs/curl-vs-wget.html” we’re down to 50% of the original request size (100 vs 199).

Can we shrink it even more? Sure, we could leave out the version number too. I left it in there now only to allow some kind of statistics to get extracted. We can’t remove the entire header, we need to include a user-agent in requests since there are too many servers who won’t function properly otherwise.

And before anyone asks: this change is only for the curl command line tool and not for libcurl, the library. libcurl does in fact not send any user-agent at all by default…

cURL and libcurl, Mail

NFS has many meanings

Today I learned that Need for speed World (I first had to google what “NFS-world” actually means) uses curl when I received this email:

From: [removed]
Subject: NFS-world

I can not go into the game for 4 months my nickname “[removed]”. it writes the error “Login failed, please try again.” Please solve this problem. Support Group does not help.

But no, I don’t know why this guy emailed me…

I then went on to look for other Electronic Arts games using libcurl, and I fell over these forum posts that clearly indicate Game Face uses it, but I found no credits or other information page online.

Can you find any other?

Network, Technology

Digging the fiber

1 Comment

Finally the installation of my open fiber is moving along.

Roughly two weeks ago the team responsible for getting the thing from the boundary of my estate to my house arrived. They spent a great deal of time trying to piggyback the existing tube already running under my driveway for the telephone cable – until they gave up and had to use their shovels to dig a ditch through my garden. Apparently the existing tube was too tight and already too filled up with the existing cables. A little strike of bad luck I think since now they instead had to make a mess of my garden. Here’s a little picture of the dig work they did:

a ditch for the fiber through the garden

They aim at a depth of 25 cm for the cable while going through people’s estates, while outside of my garden they need 50 cm depth underneath the road and sidewalk down my little suburb street.

Once they were done we could see this orange cable sticking up next to my mailbox:

the outer end of the cable by my mailbox

… and the other end is sticking up here next to my front door. I expect the next team to get here and do the installation from here and pull it in through my wall and install the media converter etc possibly in the closet next to my front door. We’ll see…

the end of the cable next to the stairs by my front door

Today, when I arrived home after work the team that were digging up the sidewalk had already connected the cable side that was previously sticking out next to my mailbox (the middle picture).

Of course, they did their best at putting things (like soil) back as it was but I’ll admit that my better half used some rather colorful expressions to describe her sentiments about getting the garden remade like this.

I’ll get back with more reports later on when I get things installed internally and when the garden starts to repair.

Technology

Back to China

As the plan is currently, I’m going to Beijing China the last week in May for work. It’s now been something like 4.5 years since I was in China the last time, and I’m really looking forward to see how things have changed. This time I expect to get a slightly different insight as well since I’ll be visiting and talking to a bunch of Chinese employers of my customer.

ma dao cheng gong

This picture is hanging in my house, and apparently means “gain an immediate victory“, as I was told…

Also, this brings back the chance for me to show you all the picture of this awesome power socket we had in our hotel room the last time, allowing basically any plug to get inserted:

chinese-socket

In comparison to Jordan where I recently spent a week vacationing, where my hotel room had the British style of sockets, but in other places in the same (fancy) hotel they had euro plugs…