Tag Archives: cURL and libcurl

cURL and libcurl

curl 7.19.6 is here!

Yet again we strike back with an update to the popular download tool curl and the transfer library libcurl.

Noticeable changes this time include:

  • A security related fix, for the flaw named CVE-2009-2417.
  • CURLOPT_FTPPORT (and curl’s -P/–ftpport) support port ranges
  • Added CURLOPT_SSH_KNOWNHOSTS, CURLOPT_SSH_KEYFUNCTION, CURLOPT_SSH_KEYDATA so that both the library and the curl tool now understand and work with OpenSSH style known_hosts file (if built with libssh2 1.2 or later)
  • CURLOPT_QUOTE, CURLOPT_POSTQUOTE and  CURLOPT_PREQUOTE can be told to ignore error responses when used with FTP. Handy if you want to run custom commands that may fail, but still enjoy persistent connections properly.

Let me just mention that the known_host support will make the SCP and SFTP transfers done with curl one step more secure. My work on this feature (both in libssh2 and in libcurl) was sponsored by a well-known company that shall remain unidentified at their request.

cURL

cURL and libcurl, Open Source

libcurl in package management

A few days ago I noticed that the “urlgrabber” project now has switched to using pycurl (the python libcurl binding) in their bleeding edge development. It means that projects using that, such well-known apps like yum and anaconda then use libcurl. Already since ages the Suse installer named YaST is using libcurl and a few months ago I learned that the opensolaris package management (pkg) is also switching to become pycurl based.

According to the lead man on the urlgrabber project, Seth Vidal, there are several reasons to switch from Python’s native urllib for (mostly) HTTP transport and he was friendly enough to mention a few to me. Clearly the two primary reasons are FIPS certification and urllib’s lacking HTTP proxy support. The FIPS certification is something the Fedora project has been pushing for a lot during recent time and thus they’ve worked hard on making libcurl support NSS for SSL/TLS, and the lack of HTTP proxy support is supposedly hard to push into urllib itself due to its stagnant development etc.

In Debian-esque worlds, libcurl and curl are already used by the package system in forms of apt-transport-https and apt-file.

It seems that when you run an open source operating system tomorrow, chances are that libcurl is in the back-end of the package system.

cURL and libcurl

curl 7.19.5

I’m happy to say that we’ve just shipped our 111th public release of curl and libcurl: 7.19.5

Notable changes this time include:

  • libcurl now closes all dead connections whenever you attempt to open a new connection
  • libssh2’s version number can now be figured out run-time instead of using the build-time fixed number
  • CURLOPT_SEEKFUNCTION may now return CURL_SEEKFUNC_CANTSEEK
  • curl can now upload with resume even when reading from a pipe
  • a build-time configured curl_socklen_t is now used instead of socklen_t

… and there are at least 29 bugs fixed. All this during 75 days since the last release.

Thanks everyone!

cURL and libcurl, Development

Dear Apple Inc

12 Comments

Dear Apple Inc,

As one of the primary authors of libcurl and curl, two parts that are included in every Mac OS X release since years back, I was only wondering if you would consider sponsoring me with a Mac, to make it easier for me to do (lib)curl development, tuning and bug-fixing on/for the Mac?green-apple

I really don’t have any particular income from Macs so I don’t see how I can personally motivate spending some 2000 USD on a Mac only for curl. And to be honest, I can’t think of any other reason to get a Mac either!

I did look around Apple’s web site to find an email adress of someone to send my plea to, but I failed. So I’ll just put it here. I have exactly no hope in actually accomplishing anything with this other than putting some attention on how things are.

This post was triggered by recent libcurl bugs that seem to show up only on Mac!

cURL and libcurl, Open Source

Getting support to curl

The other day I read this blog post by Stormy Peters, talking about getting people to sponsor or support Open Source projects and she continued to describe the Gnome approach and a bunch of projects that accept donations etc etc.

It made me (not too surprising) think about the situation for our little project cURL. We’re independent of any umbrella organization (GNU, ASF, etc) and we don’t have any vendor or company backing that pays for daily development or maintenance. We don’t have any legal entity or formal organization behind the project. We’re all just a bunch of people on some mailing lists.

We do have occasional companies and vendors who step up and pay individual developers to add features or provide various kinds of support, but they’re all basically single-shot occurrences and nothing that’s done on an ongoing basis.

Or products are used in all Linux distros, by hundreds of companies and so on. We’re a fairly active team, continuously working on bug fixes, tweaks and adding new features.

What can we do to make us more attractive for more support or active sponsoring by some vendor(s)?

Would joining an “umbrella” organization or forming a legal entity make it any more likely to happen?

Isn’t it so, that if the project is mature and good enough already, there’s actually very very little incentive for any company to take it under their wings and rather the market economy makes it a lot more profitable to simply use it as it is and if – at worst – in the end something really hits the fan, you can pay someone at that crisis point to fix up the immediate problem. And then continue like before.

And to be honest, I think we are proving to everyone that it works this way by continuing to deliver rock solid quality software. For no price. Completely open source. Year after year. Darnit, it’s just too fun to stop!

cURL

cURL and libcurl, libssh2

Adding known hosts support

… to libcurl and libssh2!

I’m about to start this little mini adventure, so if you’re one of the guys out there who’s been looking forward to be able to do even more (Open)SSH-like things with curl and libcurl when we use SCP and SFTP then consider this a little notification to start listening!

This will require improvements and changes in both projects, and funnily enough I’m already involved knee-deep in both so that shouldn’t cause any problems. I do however greatly appreciate feedback and reviews of my pending implementation proposals! I want this done in a way that benefits many and that isn’t too likely to break at least within the nearest future.

Ok, enough of that. Stand by for posts to the mailing lists. I’ll start off with the libcurl one which will thus be a slightly higher level API for all this. I’ll update this blog post later on to feature direct links to my proposals. Please consider posting responses to the suggestions to the appropriate mailing list!

The libcurl proposal

The first mail to libssh2-devel

cURL and libcurl, Linux, Mail, Network, Web

HTTP Status Report

1 Comment

Mark Nottingham Mark Nottingham held a very interesting one hour talk on the status of HTTP and the work on HTTPbis on a QCon conference recently, and luckily for us HTTP geeks there’s this great video/presentation from that.

curl is mentioned at least twice in the slides, unfortunately it has a wrong fact on the second mention where it says curl uses “Pragma: no-cache” as it isn’t true anymore. It used to do that, but we’ve stopped doing it in curl since a while ago.

I’m a subscriber to the httpbis mailing list and a casual contributor, but nonetheless his summary and overview of the state was refreshing as I’ve not been able to keep up with all the details and I haven’t been tracking that working group from its start either.

cURL and libcurl, Open Source

Code re-use is fun

Back in 2003 I wrote up support for the HTTP NTLM authentication method for libcurl. Happy with my achievement, I later that year donated a GPL licensed version of my code to the Wget project (which also was my first contact with the signed paper stuff with the GNU/FSF to waive my copyright claims and instead hand them over). What was perhaps not so amusing with this code was when both curl and Wget 2005 were discovered to have the same security flaw due to my mistakes in this code shared by both projects!

Just recently, the neon project seems to be interested in taking on the version I adjusted somewhat for them, so possibly the third HTTP code is soon using this. Yeah I posted it on their mailing list back then so it has been sitting there in the archives maturing for some 6 years by now…

I also happened to fall over the SSH Tunnel Creator tool, which I’ve never used myself, that apparently snatched my neon donation (quite according to what the license allowed of course) and used it in their tool to do NTLM!

It’s actually not until recent years I discovered libntlm, and while I don’t know how good it was back in the days when I wrote my first NTLM stuff I generally think using existing libs is the better idea…