Category Archives: cURL and libcurl

curl and/or libcurl related

URL Encode POST Data

Several months ago I did a job down south in Sweden – it was a three hour train ride (one way, and I went down and back the same day…) with the fastest train we have in this country. It gave me some time on the train to tinker with things and I didn’t feel like bothering with the “Internet On Train” thing they so fancifully offer these days. I’m not saying that’s a bad idea, I just felt that perhaps me and my Linux laptop would have to spend too much time fighting it to get it up to really enjoy it. Instead I wrote up a patch for curl for a feature we discussed ages ago: send POST data with the command line client that gets URL encoded automatically!cURL

The idea is of course that when you write a simple shell script of some sorts and want to automate POSTs to a web site, it is somewhat complicated to url encode the strings before you pass them on to curl. curl could instead get an option that does it for you.

Fast forward to current time and now I’ve dug up the old patch again, I had a discussion on the mailing list about it and what do you know! Today I’ve posted a patch that introduces –data-urlencode, and I’m very interested in feedback or suggestions on how to polish it further and then to commit it.

Human Connections

LinkedIn logoFor fun, I created two “groups” on LinkedIn for two of the open source projects I’m perhaps the most active in. I’m not quite sure what benefit and good use we’ll get from them, but anyway they’re created and if you feel in any way related to Rockbox or curl, here are the links you can use to do a join request:

Rockbox tiny Rockbox: http://www.linkedin.com/e/gis/42081/49AF807A7908

curl tiny cURL: http://www.linkedin.com/e/gis/42082/362F5916AFF1

And a link to my public LinkedIn profile

Fresh CA Cert Bundle Anyone?

cURLThe popular ca extract service on the curl web site converts the Firefox ca certs into a PEM file suitable for use with curl, wget or anything else OpenSSL-based that likes PEM formatted CA cert bundles.

The main script was fixed yesterday as it was previously getting a nightly source code snapshot to get the “magic” file to convert from, but I noticed they stopped updating the nightly source snapshots a good while ago so the updates had stopped!

Now, the script only gets the actually needed certdata file and converts it, so now it downloads a lot less data in vain and it also thus runs much faster. Now the PEM files offered on that page are up-to-date with the most recent Firefox.

Crashing Firefox Goes libcurl

I guess I haven’t been paying attention lately, but I stumbled over the Breakpad project, which incidentally is gonna be used as crash reporting tool for Firefox 3 (the original link to that is no longer working), and it uses libcurl (the original link to that quote is no longer working): “On Linux, libcurl is used for this function, as it is the closest thing to a standard HTTP library available on that platform.

The wording implies that it uses something else on Mac OS X, but I’m not aware of any standard HTTP library on it. Am I missing something or are they going libcurl there too?

Also, I wonder if using different HTTP libraries on different platforms instead of a single one isn’t just begging for more problems than what it solves? As far as I know, libcurl has a few upsides compared to wininet for example. Of course, I’m not the man to tell how they should do their stuff.

libcurl DNS resolve problems on Leopard

libcurlI found this article by Jungle Dave titled Leopard DNS Issues (and work-around), which explains how libcurl built with IPv6 support may cause trouble on MacOS X 10.5 (Leopard).

According to him, that’s because getaddrinfo() causes a SRV lookup to be made and that may be either slow or get discarded completely and thus cause trouble.

This just adds another problem to getaddrinfo() resolves then, since we already have the problem with it when resolving round-robin DNSes since more or less every machine has a bad /etc/gai.conf setup that makes getaddrinfo() return a sorted list instead of the “random” one DNS admins in the wild would prefer the users to use…

aget compared to curl

As you should know, we maintain this curl comparison table on the curl web site, and it lists a set of free tools and how they compare against curl and each other in various aspects. If you want more features compared or other tools included, please tell. Also if you disagree with any of the facts stated there, just shout!cURL

The other day I got an email asking me to add aget to the table, and since it is a free tool (original BSD licensed) with a similar purpose it would indeed fit.

So I downloaded aget 0.4 and had a go at it.

  1. The “stable” 0.4 version doesn’t build out of the tarball. It does wrong assumptions on “errno” and thus I had to manually poke on 3 source files to make it generate a fine binary! While this error is claimed to be fixed in the “devel” version, the devel version fails to build on compiler errors instead!
  2. My first test was to download aget’s own home page with aget… and it failed. It says the page is 0 bytes and it doesn’t download anything and outputs something about a bad seek 0 bytes!
  3. This really turned me off, but then I thought I should report this back to the guys rather than just blog it… but there’s no email address in the package that seems suitable, and when checking on the site I find a reference to a mailing list but when trying to read the list’s archive it just redirects back to the main page! So blogging it is.
  4. aget 0.4 from 2002, the aget devel version is from June 2004. Development seems to have stopped.
  5. I decided aget isn’t going to be added to the table by me at this time. It’ll have to mature some more first (and given the age of the tarballs I doubt that’ll happen…). I also read through the source code a bit and it really gives the impression of being a young project that hasn’t yet have time to settle since there are numerous of suspicious conclusions and source code doing “funny” things.

curl and libcurl 7.17.1

7.17.1 – the 102nd release of curl is out, with less than 5 months left to our ten year anniversary!

The previous release (7.17.0) included a few larger internal changes and unfortunately that had the backside that it brought a whole array of new bugs in, that we now have spent almost two months polishing off.

cURL

Apart from the twenty or so bug fixes, a range of new things are introduced as well, including improved NSS support, –proxy-negotiate, –post301 (to make curl act more standards compliant on HTTP 301 responses), –hostpubmd.

libcurl hackers will appreciate CURLOPT_OPENSOCKETFUNCTION and CURLOPT_COPYPOSTFIELDS (the latter a complement to the existing CURLOPT_POSTFIELDS that got broken in 7.17.0 if you posted binary data that contains a zero byte).

7.17.1 contains contributions by at least 16 different people (me not included).

curl references

I amuse myself by occasionally reading up on articles and posts “out there” that talk about curl and libcurl, and I often find interesting snippets and data well worth reading. Here’s a few of the ones I’ve stumbled upon recently:

  1. Tony G wrote a post to a u2 database mailing list and did an indirect praise of curl.
  2. magnetk.com writes about how to build a recent libcurl with visual studio 2005

wget going libcurl?

Micah Cowan is the current maintainer of GNU Wget, and he recently posted a long mail to the wgetA gnu head! mailing list titled “Thoughts on Wget 1.x, 2.0“.

Two fun quotes for the curious who don’t feel like reading the whole post:

1. On the subject of making wget deal with multiple simultanous connections/requests: The obvious solution to that is to use c-ares, which does exactly that: handle DNS queries asynchronously. Actually, I didn’t know this until just now, but c-ares was split off from ares to meet the needs of the curl developers.

2. Following the first reasoning, they can indeed get away with even less work if they base that work on an existing solution: While I’ve talked about not reinventing the wheel, using existing packages to save us the trouble of having to maintain portable async code, higher-level buffered-IO and network comm code, etc, I’ve been neglecting one more package choice. There is, after all, already a Free Software package that goes beyond handling asynchronous network operations, to specifically handle asynchronous _web_ operations; I’m speaking, of course, of libcurl. There would seem to be some obvious motivation for simply using libcurl to handle all asynchronous web traffic, and wrapping it with the logic we need to handle retries, recursion, timestamping, traversing, selecting which files to download, etc. Besides async web code, of course, we’d also automatically get support for a number of various protocols (SFTP, for example) that have been requested in Wget.libcurl

I am of course happy to see that the consideration exists – even if this won’t go further than just expressed in a mail. I did ventilate this idea to the wget people back in 2001, and even though we’re now more than six years down the road since then the situation is now even more clear: libcurl is a much more capable and proven transport layer solution and it supports much more protocols than wget is/does.

Me biased? naaah… 🙂