Category Archives: cURL and libcurl

curl and/or libcurl related

library for proxy detection

Only days after I wrote about the pacparser, another and in many ways more complete approach to detecting what proxy to use for accessing various internet resources emerge: libproxy.curl tiny

One of the main authors of it, Alex Panit, already submitted a feature-request for libcurl to support this. but I'm not at all convinced that is a good idea. It seems the authors submit "please include support for this"-requests all over in similar and related projects - similar in style to how metalink did.

As usual, I value your input and feedback so please raise your voice and speak up!

So far this young project lacks docs on API and install process, so I haven't yet even been able to build it for a test drive...

Jun-Ichiro

Today I just casually noticed the NetBSD 4.0 release, but what really caught my interest was the dedication of the release. It is dedicated to Jun-Ichiro "itojun" Hagino, who apparently passed away in October this year only 37 years old. See also this "memorial".

Anyway, the reason I noticed is because I remember him and his contributions in the curl project back in 2001 and 2002 when he provided a set of very good and useful patches (for example this) that made curl take a huge step forward in its IPv6-support. It is of course with sadness that I get to hear about him again this way. I'm sure he will be missed in many camps.

curl icons

I got a question on the #curl IRC channel and I have to admit that the curl and libcurl icons are a bit hard to find on the curl web site, so I thought I'd paste them here too to put the lights on them a bit more.

If you use (lib)curl in your projects or site, please consider making this noticeable by mentioning it and possibly using an existing curl icon to show:

powered by libcurl powered by libcurl powered by libcurl powered by libcurl powered by libcurl

powered by libcurl powered by libcurl powered by libcurl

small curl icon small curl icon with border libcurl icon

cURL libcurl

... as you can see there are plenty to choose from! 😉

And please, if you are capable of drawing logos, bitmaps, icons or the likes and you feel like you can do better than these, please step forward and bring on your work!

Parsing those Dreaded PACs

Since a long time companies all over have found it suitable to use "PAC" (proxy auto-config) scripts to control browsers and tell them what proxy to use for what particular URL and more, and this has been a major nuisance for people who use other (automated) means to go through the proxy than one of them huge gui-based browsers.

For a long time, Ralph Mitchell's take on this was the only half-decent way to get it working. Half-decent only because it was never packaged and provided in a convenient way to people, it was fully functional.

Now, Manu Garg seems to have stepped forward and introduces pacparser which is a C library for parsing PAC files. Period. It looks (very) similar to Ralph's approach, in that it incorporates the Mozilla's javascript engine Spidermonkey, with nothing really special added. So, it's basically the same approach but this time it is packaged, shipped and documented how it is used!

Announced on the curl-users mailing list today!

curl keeps connections alive

Just in the last few days we modified curl to enable the SO_KEEPALIVE option on connections it creates. It basically means that curl will now detect connections that are idle after a certain amount of time, even if that time is something around two hours by default and that's what most systems will have it set to.cURL

The main problem that caused us to finally enable this (you can still disable this by using --no-keep-alive) is when people do (long-lasting) FTP transfers and they use a NAT, firewall or router that detects and removes what it considers are idle connections. An FTP transfer is using two connections, but the control one where the commands are sent over is completely quiet while the actual data transfer is in progress so when the transfer is done, the control connection has been nuked by the router/NAT. Of course curl survives this as good as possible, but it can't do proper error-checking etc in this situation.

Funnily, there's no really good fix for the FTP situation since the two hours SO_KEEPALIVE timeout will many times be too long to help (although most modern systems allows you to change the timeout or a system or application level), but the other "obvious" fix is to send a "NOOP" command on the control channel every once in a while during the transfer. But no, that doesn't work fine either on most servers since it seems the servers often don't listen on the control connection during the transfer, so all we'd get is curl sending commands that won't be replied to until the end of the transfer, and thus it will end up causing problems.

Note: curl sets this option. libcurl still doesn't, so if you want your app to set the option you can use the same CURLOPT_SOCKOPTFUNCTION callback that curl uses. This requires libcurl 7.16.0 or later.

To fscons and back in 16 hours

I took the X2000 train to Gothenburg from Stockholm at 08:10 so I was at the conference place first at almost 11:30.

SELF

This meant I got to listen in on the end of Jonas Öberg's speech on SELF (an FSF Europe and others project on e-learning and a lot of related matters). This wasn't really my cup of tea, but the other track had a MySQL talk and that isn't really my thing so I had to just pick one... 🙂 Nothing bad about the subject or Jonas really, just a hint of where my interests are not so much.

Lunch

At lunch I got the opportunity to catch up with Squid-Henrik (Nordström) to talk about recent happenings in our projects. Squid being about to release v3 after several years, and I could report that curl has gotten support for SSH-related protocols during the last year or so... I also exchanged a few words with Peter Stuge who expressed interest in hacking on libssh2 recently, and well I have done some of that!

Qtopia

Trolltech’s GreenphoneKnut Yrvin from Trolltech Norway did an excellent talk on Qtopia in the Telecom business, which had changed name from something involving Greenphone since they have since ditched that project. Anyway, he spoke of the upcoming possible opportunities for free software and open source in consumer electronics, and then particularly in smart phones and then of course mostly related to Qt and Qtopia. He passed around a Greenphone to let us get a feel or it and fiddle a bit with it and yes, it seemed like a nice phone - not a lot bigger or heavier than my current Sony Ericsson thing. It featured a nice "sliding UI" even if I had serious troubles moving around in the system and I couldn't really figure out the maneuvering concept much! I suspect all it would take is a little more time and perhaps a manual or someone explaining it to me.

OpenMoko

Neo1973In the subsequent talk Ole Tange from the OpenMoko project also handed out "fiddle-versions" of their primary phone, Neo1973 for us in the audience to touch and hold. The major bad with these were however that both devices were dead so we couldn't see anything on them, just hold them and feel that... eh yeah. they're a bit on the biggish side and quite a bit bigger than my current phone in all three dimensions! He spoke about the upcoming 2nd version of the phone that is supposed to become available in Q1 2008 and given that it will feature wifi, bluetooth, accelerometers, GPS, 640x480 pixels touch screen, accelerated graphics card and a mini-B USB plug and run an entirely open and free Linux version with documented hardware is indeed thrilling. The Neo1973's size is not attractive, but its internals are. This is in fact a unit I will seriously consider buying/hacking when/if it becomes available for purchase.

Other details in his OpenMoko talk gave me the impression that the software is not yet very far advanced. Like he first made a comparison to the OLPC system with hw and sw items side-by-side both listing as GTK+ based UIs, but then he also mentioned a thanks to Trolltech for having ported their Greenphone Qtopia system to OpenMoko. On my direct question if that wasn't a bit contradictive since surely they must be focusing on ONE of these graphics/widgets systems for their main development, he went on to rant about how OpenMoko "is a computer" that can "run anything". I'm not sure, but it certainly gave me the impression that there just is no main development... Where is OpenMoko at right now really? Anyone knows? I guess I should spend some time on researching that, and also investigate a bit on the "running Rockbox on OpenMoko" front...

curl

cURLWhen the time came for my talk, at 15:00 we first had to mess about a bit since the computer I was supposed to borrow to run my presentation on was suddenly gone (and used for the other track's talk I later learned) but thanks to other people I soon had a replacement and I got on with it.

I know the topic by heart of course, curl being my primary open source project for ten years and I know every bit of it and its history and so on, but making a fine presentation based on that is an entirely different story. Also, giving it in English adds a layer of, well not complexity perhaps, but it makes it all bit more rough in the edges since even though I know English pretty well and all, my vocabulary isn't the largest and I don't always find the right synonyms and the phrasing etc when trying to explain or argue for my sake.

Also, since I don't quite know my own presentation by heart it isn't really the best possible performance I can do, but what the heck. I tried to present curl and libcurl, what they are and what they're good for, why people use it and how the development is done and why YOU should use it now and in the future. The guys at fscons got all talks on video so I hope to be able to see myself on video soon and I'll try to learn from that for my next talk. And of course those of you who weren't present at fscons will get your chance to see my pale face and listen to my Swedish-accented stumbling English! 😉 Oh, and I had to rush the presentation a bit towards the end when my 45 minutes ran out a little bit faster than I had anticipated, or was it the questions that popped up? Questions are good, since they make me aware the audience is with me and are interested.

I'm not sure if the topic of curl is somewhat boring, or if it felt too technical or what, but I think I had less than 50% of the audience listening. The other talk going on while I spoke was a lightning talk session with a bunch of people.

Here's the slides from my talk, in a 31 page 500K pdf: http://daniel.haxx.se/curl-20071208.pdf

LinuxBIOS

Slightly dry in my mouth after this, I recharged myself with a cup of coffee and some cinnamon-rolls and walked it to see the next talk. Or rather series of talks since this was a "lightning talks" session where five guys spoke quickly about various topics. They were about web development with perl, a weird ajax system called gaia that seemed to involve a lot of .NET, a web development system of some sorts named makumba, and a quick mentioning of a 10 gigabit full open source router. For me, the most interesting piece was Peter Stuge's brief talk about LinuxBIOS, what it is, what it does and so on. That's really a to-the-metal project and I like getting back to earth and on to real stuff. Much of what he said and explained about difficulties with documentation from hardware vendors etc are just so familiar to me based on Rockbox experiences. To the great enjoyment of the audience, Peter's live demo of LinuxBIOS booting up failed notoriously and after numerous resets it finally booted up and started playing loud music - when the following speaker already was half-through his router presentation!

Closing

I only got to hear the beginnings of the closing talk held by Georg Greve from FSF Europe as I had to leave after 20 minutes or so to catch my cab that took me back to the train station and I was on my way back to Stockholm again on the 18:42 train...

Did I mention that I got a tshirt? I planned to include a picture of the shirt here, but I took a shot with my mobile phone when I got home and the camera in it is just so extremely crappy in low-light situations (even if I had all the lights in the room turned on) so I can't torture you by including it. I'll have to make another attempt later or find a link to someone else who did...

In conclusion: even though I only did a quick visit and didn't get to see that many talks, I liked what I saw and I had fun. It sounded like the guys doing this are seriously planning on doing it again next year. I hope they'll do and that I'll manage to do there again, hopefully to do another talk!

Tunneling with libcurl

As I wrote a while ago, companies using http proxies make people feel a need to break out of their proxies.

Bryan is a friend who recently found out that his company is switching proxy to a different one and apparently both corkscrew and proxytunnel have problems with this new piece, and since libcurl offers quite a lot of functionality to accomplish almost this, a new project was born: curltunnel.libcurl

One immediate benefit of using libcurl is the support for multiple authentication methods, in fact more than any of the above mentioned tools.

However, it seems our first quick stab at making this tool (currently 278 lines of code), made it work for several common cases but... not for Bryan's new proxy.

The current theory is that the proxy actually checks for SSL traffic and only lets that through, and thus it prevents the ssh server banner to appear when we try to tunnel through the proxy to a remote ssh server on port 443. If further testing proves this correct, we will of course have to add a SSL layer to the mix.

URL Encode POST Data

Several months ago I did a job down south in Sweden - it was a three hour train ride (one way, and I went down and back the same day...) with the fastest train we have in this country. It gave me some time on the train to tinker with things and I didn't feel like bothering with the "Internet On Train" thing they so fancifully offer these days. I'm not saying that's a bad idea, I just felt that perhaps me and my Linux laptop would have to spend too much time fighting it to get it up to really enjoy it. Instead I wrote up a patch for curl for a feature we discussed ages ago: send POST data with the command line client that gets URL encoded automatically!cURL

The idea is of course that when you write a simple shell script of some sorts and want to automate POSTs to a web site, it is somewhat complicated to url encode the strings before you pass them on to curl. curl could instead get an option that does it for you.

Fast forward to current time and now I've dug up the old patch again, I had a discussion on the mailing list about it and what do you know! Today I've posted a patch that introduces --data-urlencode, and I'm very interested in feedback or suggestions on how to polish it further and then to commit it.

Human Connections

LinkedIn logoFor fun, I created two "groups" on LinkedIn for two of the open source projects I'm perhaps the most active in. I'm not quite sure what benefit and good use we'll get from them, but anyway they're created and if you feel in any way related to Rockbox or curl, here are the links you can use to do a join request:

Rockbox tiny Rockbox: http://www.linkedin.com/e/gis/42081/49AF807A7908

curl tiny cURL: http://www.linkedin.com/e/gis/42082/362F5916AFF1

And a link to my public LinkedIn profile...

Fresh CA Cert Bundle Anyone?

cURLThe popular ca extract service on the curl web site converts the Firefox ca certs into a PEM file suitable for use with curl, wget or anything else OpenSSL-based that likes PEM formatted CA cert bundles.

The main script was fixed yesterday as it was previously getting a nightly source code snapshot to get the "magic" file to convert from, but I noticed they stopped updating the nightly source snapshots a good while ago so the updates had stopped!

Now, the script only gets the actually needed certdata file and converts it, so now it downloads a lot less data in vain and it also thus runs much faster. Now the PEM files offered on that page are up-to-date with the most recent Firefox.