Category Archives: cURL and libcurl

curl and/or libcurl related

curl and libcurl 7.17.1

7.17.1 - the 102nd release of curl is out, with less than 5 months left to our ten year anniversary!

The previous release (7.17.0) included a few larger internal changes and unfortunately that had the backside that it brought a whole array of new bugs in, that we now have spent almost two months polishing off.

cURL

Apart from the twenty or so bug fixes, a range of new things are introduced as well, including improved NSS support, --proxy-negotiate, --post301 (to make curl act more standards compliant on HTTP 301 responses), --hostpubmd.

libcurl hackers will appreciate CURLOPT_OPENSOCKETFUNCTION and CURLOPT_COPYPOSTFIELDS (the latter a complement to the existing CURLOPT_POSTFIELDS that got broken in 7.17.0 if you posted binary data that contains a zero byte).

7.17.1 contains contributions by at least 16 different people (me not included).

curl references

I amuse myself by occasionally reading up on articles and posts "out there" that talk about curl and libcurl, and I often find interesting snippets and data well worth reading. Here's a few of the ones I've stumbled upon recently:

  1. Tony G wrote a post to a u2 database mailing list and did an indirect praise of curl.
  2. magnetk.com writes about how to build a recent libcurl with visual studio 2005

wget going libcurl?

Micah Cowan is the current maintainer of GNU Wget, and he recently posted a long mail to the wgetA gnu head! mailing list titled "Thoughts on Wget 1.x, 2.0".

Two fun quotes for the curious who don't feel like reading the whole post:

1. On the subject of making wget deal with multiple simultanous connections/requests: The obvious solution to that is to use c-ares, which does exactly that: handle DNS queries asynchronously. Actually, I didn't know this until just now, but c-ares was split off from ares to meet the needs of the curl developers.

2. Following the first reasoning, they can indeed get away with even less work if they base that work on an existing solution: While I've talked about not reinventing the wheel, using existing packages to save us the trouble of having to maintain portable async code, higher-level buffered-IO and network comm code, etc, I've been neglecting one more package choice. There is, after all, already a Free Software package that goes beyond handling asynchronous network operations, to specifically handle asynchronous _web_ operations; I'm speaking, of course, of libcurl. There would seem to be some obvious motivation for simply using libcurl to handle all asynchronous web traffic, and wrapping it with the logic we need to handle retries, recursion, timestamping, traversing, selecting which files to download, etc. Besides async web code, of course, we'd also automatically get support for a number of various protocols (SFTP, for example) that have been requested in Wget.libcurl

I am of course happy to see that the consideration exists - even if this won't go further than just expressed in a mail. I did ventilate this idea to the wget people back in 2001, and even though we're now more than six years down the road since then the situation is now even more clear: libcurl is a much more capable and proven transport layer solution and it supports much more protocols than wget is/does.

Me biased? naaah... 🙂

curl work towards release #102

libcurlThis is a bug fix week in curl land, trying to get everything sorted and fine to be able to release a really fine 7.17.1 release within a week or so. We got some nasty memory-related problems with changed protocols re-using the same easy handle, but it was good that they crept up and I think we're doing good changes now that stabilize curl.

Release date now targeted perhaps around October 28-30.

this week in curl land

It has been another busy week, and this time people brought up a range of stupid and annoying bugs:

libcurl...but also included Patrick Monnerat's cool internal re-arrange to use handler structs for the various protocols.

Upcoming dates to check for:

  • October 14 - Feature freeze for the 7.17.1 release (and I take off for China)
  • October 21 - I come back home from China (to catch up with a million emails)
  • Octobert 27-28 (something) - Probable release date for the 7.17.1 release, assuming that all serious bugs have been fixed by then.

As usual, post curl-related stuff on the mailing lists and not to me personally!

curl vs wget, really

Ok, since people truly and actually often ask me about what the differences are between curl and Wget, it might be suitable to throw in my garbage here and state the main differences as I see them. Please consider my bias towards curl since after all, curl is my baby - but I have contributed code to Wget as well.

curl

  • Features and is powered by libcurl -a cross-platform library with a stable API that can be used by each and everyone. This difference is major since it creates a completely different attitude on how to do things internally. It is also slightly harder to make a library than a "mere" command line tool.
  • Pipes. curl is more in the traditional unix-style, it sends more stuff to stdout, and reads more from stdin in a "everything is a pipe" manner.cURL
  • Return codes. curl returns a range of defined and documented return codes for various (error) situations.
  • Single shot. curl is bascially made to do single-shot transfers of data.It transfers just the URLs that
    the user specifies, and does not contain any recursive downloading logic or any sort of HTML parser.
  • More protocols. curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS and FILE at the time of this writing. Wget supports HTTP, HTTPS and FTP.
  • More portable. Ironically curl builds and runs on lots of more platforms than wget, in spite of their attempts to keep things conservative. For example, VMS, OS/400, TPF and other more "exotic" platforms that aren't straight-forward unix clones.
  • More SSL libraries and SSL support. curl can be built with one out of four different SSL/TLS libraries, and it offers more control and wider support for protocol details.
  • curl (or rather libcurl) supports more HTTP authentication methods, and especially when you try over HTTP proxies.

wget

  • Wget is command line only. There's no lib or anything. Personally I've always disliked the project doesn't provide a man page as they stand in the GNU side of this and consider "info" pages to be the superior way to document things like this. I strongly disagree.
  • Recursive! Wget's major strong side compared to curl is its ability to download recursively, or even just download everything that is referred to from a remote resource, be it a HTML page or a FTP directory listing.A gnu head!
  • Older. Wget has traces back to 1995, while curl can be tracked back no longer than 1997.
  • Less developer activity. While this can be debated, I consider three metrics here: mailing list activity, source code commit frequency and release frequency. Anyone following these two projects can see that the curl project has a lot higher pace in all these areas, and it has indeed been so for several years.
  • HTTP 1.0. Wget still does its HTTP operations using HTTP 1.0, and while that is still working remarkably fine and hardly ever is troublesome to the end-users, it is still a fact. curl has done HTTP 1.1 since March 2001 (while still offering optional 1.0 requests).
  • GPL. Wget is 100% GPL v2, which I believe is going v3 really soon when they release their next release. curl is MIT licensed.
  • GNU. Wget is part of the GNU project and all copyrights are assigned to them etc. The curl project is entirely stand-alone and independent with no organization parenting at all.

This turned out to be a long post and it might in fact be usable to save for the future, so I'm also posting this as a more permanent doc on my site on this URL: http://daniel.haxx.se/docs/curl-vs-wget.html. So possible updates will be done there. Do let me know if you have further evident differences or if you disagree with me on details here!

libcurl in less than 50KB

Gary Maxwell enlightened us that his build (of a slightly older libcurl) is way below 50KB on an ARM7 architecture, while Dan Fandrich could squeeze the latest libcurl release to at least below 100KB on x86.libcurl

Of course these particular builds are fairly stripped down builds with only HTTP support left, but they are built from unmodified sources. Full-fledged builds with all protocols will of course be significantly larger.

libcurl bindings

Christopher Smith blogged about improving curlpp and not only did Jean-Philippe react immediately, it also showed me how far away I am from these C++ guys and their ideas and views of the world.

Not only I am not even aware of what functors and facets truly are (nor do I really care), but I find it interesting that the choice of them and whether or not one or the other is used or supported is such a religious thing...

You know what? The older I get, the less interested I get in the maze that is OO concepts. I just so have no interest whatsoever in C++ nor Java!

Of course I'm primarily happy they use libcurl and that they keep enhancing the ways people access it. I have work enough on the C API so I never really dive very deep in the various bindings (there are more than 30 these days).

On a slightly related note, there's a second Lua binding now called Lua-cURL - the other one is actually called luacurl and yeah the names are... not very imaginative and very very very similar to each other.