Category Archives: cURL and libcurl

curl and/or libcurl related

cURL and libcurl, Open Source

libcurl in package management

A few days ago I noticed that the “urlgrabber” project now has switched to using pycurl (the python libcurl binding) in their bleeding edge development. It means that projects using that, such well-known apps like yum and anaconda then use libcurl. Already since ages the Suse installer named YaST is using libcurl and a few months ago I learned that the opensolaris package management (pkg) is also switching to become pycurl based.

According to the lead man on the urlgrabber project, Seth Vidal, there are several reasons to switch from Python’s native urllib for (mostly) HTTP transport and he was friendly enough to mention a few to me. Clearly the two primary reasons are FIPS certification and urllib’s lacking HTTP proxy support. The FIPS certification is something the Fedora project has been pushing for a lot during recent time and thus they’ve worked hard on making libcurl support NSS for SSL/TLS, and the lack of HTTP proxy support is supposedly hard to push into urllib itself due to its stagnant development etc.

In Debian-esque worlds, libcurl and curl are already used by the package system in forms of apt-transport-https and apt-file.

It seems that when you run an open source operating system tomorrow, chances are that libcurl is in the back-end of the package system.

cURL and libcurl

curl 7.19.5

I’m happy to say that we’ve just shipped our 111th public release of curl and libcurl: 7.19.5

Notable changes this time include:

  • libcurl now closes all dead connections whenever you attempt to open a new connection
  • libssh2’s version number can now be figured out run-time instead of using the build-time fixed number
  • CURLOPT_SEEKFUNCTION may now return CURL_SEEKFUNC_CANTSEEK
  • curl can now upload with resume even when reading from a pipe
  • a build-time configured curl_socklen_t is now used instead of socklen_t

… and there are at least 29 bugs fixed. All this during 75 days since the last release.

Thanks everyone!

cURL and libcurl, Development

Dear Apple Inc

12 Comments

Dear Apple Inc,

As one of the primary authors of libcurl and curl, two parts that are included in every Mac OS X release since years back, I was only wondering if you would consider sponsoring me with a Mac, to make it easier for me to do (lib)curl development, tuning and bug-fixing on/for the Mac?green-apple

I really don’t have any particular income from Macs so I don’t see how I can personally motivate spending some 2000 USD on a Mac only for curl. And to be honest, I can’t think of any other reason to get a Mac either!

I did look around Apple’s web site to find an email adress of someone to send my plea to, but I failed. So I’ll just put it here. I have exactly no hope in actually accomplishing anything with this other than putting some attention on how things are.

This post was triggered by recent libcurl bugs that seem to show up only on Mac!

cURL and libcurl, Open Source

Getting support to curl

The other day I read this blog post by Stormy Peters, talking about getting people to sponsor or support Open Source projects and she continued to describe the Gnome approach and a bunch of projects that accept donations etc etc.

It made me (not too surprising) think about the situation for our little project cURL. We’re independent of any umbrella organization (GNU, ASF, etc) and we don’t have any vendor or company backing that pays for daily development or maintenance. We don’t have any legal entity or formal organization behind the project. We’re all just a bunch of people on some mailing lists.

We do have occasional companies and vendors who step up and pay individual developers to add features or provide various kinds of support, but they’re all basically single-shot occurrences and nothing that’s done on an ongoing basis.

Or products are used in all Linux distros, by hundreds of companies and so on. We’re a fairly active team, continuously working on bug fixes, tweaks and adding new features.

What can we do to make us more attractive for more support or active sponsoring by some vendor(s)?

Would joining an “umbrella” organization or forming a legal entity make it any more likely to happen?

Isn’t it so, that if the project is mature and good enough already, there’s actually very very little incentive for any company to take it under their wings and rather the market economy makes it a lot more profitable to simply use it as it is and if – at worst – in the end something really hits the fan, you can pay someone at that crisis point to fix up the immediate problem. And then continue like before.

And to be honest, I think we are proving to everyone that it works this way by continuing to deliver rock solid quality software. For no price. Completely open source. Year after year. Darnit, it’s just too fun to stop!

cURL

cURL and libcurl, libssh2

Adding known hosts support

… to libcurl and libssh2!

I’m about to start this little mini adventure, so if you’re one of the guys out there who’s been looking forward to be able to do even more (Open)SSH-like things with curl and libcurl when we use SCP and SFTP then consider this a little notification to start listening!

This will require improvements and changes in both projects, and funnily enough I’m already involved knee-deep in both so that shouldn’t cause any problems. I do however greatly appreciate feedback and reviews of my pending implementation proposals! I want this done in a way that benefits many and that isn’t too likely to break at least within the nearest future.

Ok, enough of that. Stand by for posts to the mailing lists. I’ll start off with the libcurl one which will thus be a slightly higher level API for all this. I’ll update this blog post later on to feature direct links to my proposals. Please consider posting responses to the suggestions to the appropriate mailing list!

The libcurl proposal

The first mail to libssh2-devel

cURL and libcurl, Linux, Mail, Network, Web

HTTP Status Report

1 Comment

Mark Nottingham Mark Nottingham held a very interesting one hour talk on the status of HTTP and the work on HTTPbis on a QCon conference recently, and luckily for us HTTP geeks there’s this great video/presentation from that.

curl is mentioned at least twice in the slides, unfortunately it has a wrong fact on the second mention where it says curl uses “Pragma: no-cache” as it isn’t true anymore. It used to do that, but we’ve stopped doing it in curl since a while ago.

I’m a subscriber to the httpbis mailing list and a casual contributor, but nonetheless his summary and overview of the state was refreshing as I’ve not been able to keep up with all the details and I haven’t been tracking that working group from its start either.

cURL and libcurl, Open Source

Code re-use is fun

Back in 2003 I wrote up support for the HTTP NTLM authentication method for libcurl. Happy with my achievement, I later that year donated a GPL licensed version of my code to the Wget project (which also was my first contact with the signed paper stuff with the GNU/FSF to waive my copyright claims and instead hand them over). What was perhaps not so amusing with this code was when both curl and Wget 2005 were discovered to have the same security flaw due to my mistakes in this code shared by both projects!

Just recently, the neon project seems to be interested in taking on the version I adjusted somewhat for them, so possibly the third HTTP code is soon using this. Yeah I posted it on their mailing list back then so it has been sitting there in the archives maturing for some 6 years by now…

I also happened to fall over the SSH Tunnel Creator tool, which I’ve never used myself, that apparently snatched my neon donation (quite according to what the license allowed of course) and used it in their tool to do NTLM!

It’s actually not until recent years I discovered libntlm, and while I don’t know how good it was back in the days when I wrote my first NTLM stuff I generally think using existing libs is the better idea…

cURL and libcurl, Network, Open Source

murl for extended curlness

1 Comment

I’m a firm believer in the old unix mantra of letting each tool do its job and do it well, and pass on the rest of the work to the next tool. I’ve always stated that curl should remain this way and that it should remain within its defined walls and not try to do everything.

But time passes and more and more ideas are thrown up in the air, or in some cases directly at me, and the list of things that we could do but don’t due to this philosophical limit of remaining focused has grown. It currently includes at least:

  • metalink support
  • recursive HTML downloads
  • recursive/wildcard FTP transfers
  • bittorrent support
  • automatic proxy configuration
  • simultaneous/parallel download support

Educated readers of course immediately detect that this list (if implemented) would make a tool that basically does what wget already does (and a lot more) and I’ve explicitly said for a decade that curl is not a wget clone. Maybe it is time for us (me?) to reevaluate that sentiment – at least in some sense.

I don’t want to sacrifice the concepts that have worked so fine for curl under so many years, so I’m still firmly against stuffing all this into curl (or libcurl). That simply will not happen with me at the wheel.

A much more interesting alternative would be to instead start working on a second tool within the curl project: murl. A tool that does basically everything that curl already does, but also opens the doors for adding just about everything else we can cram in and that is still related to data transfers. That would include, but not be restricted to, all the fancy stuff mentioned in the list above!

No the name murl is not set in stone, nor is this whole idea anything but plain and early thoughts thrown out at this point so it may or may not actually take off. It will probably depend on if I get support and help from fellow hackers to get started and moving along.

cURL

cURL and libcurl, Security

curl 7.19.4

curl and libcurl 7.19.4 has just been released! This time I think the perhaps most notable fix is the CVS-2009-0037 security fix which this release addresses. A little over 600 days passed since the previous vulnerability was announced.

Other than that major event, there are a bunch of interesting changes in this release:

  • Added CURLOPT_NOPROXY and the corresponding –noproxy
  • the OpenSSL-specific code disables TICKET (rfc5077) which is enabled by default in openssl 0.9.8j
  • Added CURLOPT_TFTP_BLKSIZE
  • Added CURLOPT_SOCKS5_GSSAPI_SERVICE and CURLOPT_SOCKS5_GSSAPI_NEC – with the corresponding curl options –socks5-gssapi-service and –socks5-gssapi-nec
  • Improved IPv6 support when built with with c-ares >= 1.6.1
  • Added CURLPROXY_HTTP_1_0 and –proxy1.0
  • Added docs/libcurl/symbols-in-versions
  • Added CURLINFO_CONDITION_UNMET
  • Added support for Digest and NTLM authentication using GnuTLS
  • CURLOPT_FTP_CREATE_MISSING_DIRS can now be set to 2 to retry the CWD even when MKD fails
  • GnuTLS initing moved to curl_global_init()
  • Added CURLOPT_REDIR_PROTOCOLS and CURLOPT_PROTOCOLS

We also did at least 15 documented bugfixes in this release and 25 people are credited for their help to make it happen.