Category Archives: Open Source

Open Source, Free Software, and similar

cURL and libcurl, Development

curl meetup at Fosdem 2012

The FOSDEM 2012 dates were recently revealed (4-5 February 2012).

A pint of guinness

I’d be happy to arrange a get-together for libcurl hackers at Fosdem this year. To me, Brussels, Belgium seems mid-europe enough to be able to attract a bunch of us:

  • libcurl application users/authors
  • libcurl binding hackers
  • libcurl contributors
  • … and everyone else who’s doing related activities or who just is interested

Potential subjects to discuss at such a meeting:

  • what’s the most important stuff libcurl still lacks?
  • what’s the least documented/understood parts of libcurl?
  • are there shared problems several/many libcurl bindings have to solve?
  • can we improve how we work/develop libcurl and bindings?
  • what kind of beer is best at a curl meetup?
  • [fill in your own curl related subject]

I would like at least 4-5 people voicing interest for this to be worthwhile for me to actually try to do anything. Please speak up on the libcurl mailing list, tweet me or mail me privately! The more people that are interested, the more planning and stuff we’ll do for it.

cURL and libcurl

curl 7.22.0

Another release of curl and libcurl just happened. 7.22.0 is released.

Apart from the 28 something documented bug fixes, we introduce a range of changes that could be noteworthy:

  • Added CURLOPT_GSSAPI_DELEGATION – remember that we explicitly disabled GSSAPI delegation in our previous release due to a security problem. Now we introduce an option for the application to control exactly how to behave.
  • Added support for NTLM delegation to Samba’s winbind daemon helper ntlm_auth. This lets libcurl use the external helper program to do things like NTLM single-sign on.
  • Display notes from setup file in testcurl.pl – provides a way for test clients to provide more information back to the centralized test summary on the primary server.
  • BSD-style lwIP TCP/IP stack experimental support on Windows – there are still flaws in lwIP on windows that prevents it from working properly
  • OpenSSL: Use SSL_MODE_RELEASE_BUFFERS if available – this is basically a way to ask OpenSSL to use less memory
  • –delegation was added to set CURLOPT_GSSAPI_DELEGATION – simply the new option exported to the command line tool
  • nss: start with no database if the selected database is broken – a slightly modified behavior
  • telnet: allow programatic use on Windows – basically making the windows implementation in sync with how the non-windows version already has worked for quite some time

This release is this great thanks to 25 friendly contributors.

cURL

cURL and libcurl

A libcurl postergirl?

1 Comment

google for libcurl

If you click the image you’ll see a full-resolution screendump for my recent search for “libcurl” on google. Where did that (image of a) girl come from? Judging from where it appears on the results page right next to the information about the cURL project you can’t but assume that she’s somehow related to the project.

That’s of course not true. When moving the mouse over the image I get a tooltip with a funny “hair curling” URL and that’s also where a click on the image takes me.

A mighty weird way of presenting a search result if you ask me!

Electronics, Open Source, Technology

I like a good firmware bump

1 Comment

So I have this TV that I got for Christmas 2009. As it happens the guys at Philips clearly kept fixing the software and removed bugs after that moment. No surprise there really. I’ve been an embedded software developer for some twenty years by now. I know that software never gets “done” and that what ships in products is only what seems to be “good enough” at some point in time. Sometimes of course not even that good.

So the other day I took a photo of my TV firmware version. It shows how the firmware was made in April 2009. I did it during a discussion with a friend who happens to have the exact same TV as I do, and it then of course turns out he has a different (newer) firmware.

Oh right, I wonder if I can upgrade to a newer one? Once I’ve mastered the maze of the Philips web site I eventually found a download link and PDFs that told me how to. The list of fixes since my version was extensive and I noticed a few flaws mentioned that I have actually experienced!

The TV firmware download was a whopping 43MB. I realize this is because it is a full-fledged Linux system with kernel and God knows what else they’ve crammed in there. I decided to give it a closer check! The result of that was a little disappointing. It is quite clearly encrypted after some basic initial header.

hexdump -C firmware image

The data that starts on offset 0x220 is not x86 instructions and in fact nothing in the beginning of the file looks like x86 code (I just ran a quick “objdump -D –target binary -m i386” on the file). Of course, I don’t know what architecture my TV runs so perhaps even checking for x86 is wrong. I know MIPS is popular in DVDs, settop-boxes and related graphics stuff but…. Nah, I decided it really wasn’t worth the effort so I stopped investigating. I have no real intention of hacking on it anyway.

So I instead proceeded to the actual procedure of upgrading the thing.

Unzip the zip file and put the file in the root dir of a FAT32-formatted usb-stick. The instructions of course didn’t say it needs to be FAT32 but I used that and it worked, and I just smug in the dark to how a manufacturer like this just assumes that we would have FAT32 on our usb-sticks…

But I digress. When I inserted the upgrade USB, the TV switched itself off, was dark for a short while and then turned itself on again and showed the firmware upgrade screen.

The process was very fast, just like 30-40 seconds or something like that and then it was done and asked me to remove the “media” and restart. Of course we know that a usb stick is “media” so I removed it from the TV set.

The instructions were very clear that to “restart” the TV I must only press the ON/OFF button on the remote once and only once. So I was careful to do just that… 😉

Nothing strange happened, but after a brief moment of black screen the regular and familiar interface.

I jumped into the firmware version menu to check it out and yes, it shows an updated version now:

I did a quick check to see if I could detect my previous quirks now, but they may really be gone. They’ve been related to sound through HDMI and some graphical “glitches” when feeding the TV with full HD from a laptop.

So, with this firmware that was shipped many months after I got my TV, I seem to have gotten a better product.

I haven’t yet tested this new version to a significant degree so I don’t know yet if I’ve gotten some new nasty side-effects from it, as sometimes these kinds of firmware upgrades really cause you pain when something that formerly used to work so good suddenly turns out to not work that good any longer.

Rockbox

Rockbox Devcon 2011

Rockbox

Hoards of hackers in similar-looking t-shirts with funny logos having the b in front of the K (see below for some sort of explanation) were seen on the streets of London on Friday June 3rd 2011.

Thanks a lot to  Google UK who hosted our Rockbox developers conference this time in central London.

We had some short-time visitors but we were 16-18 reverse engineering happy persons in a single room most of the weekend, where we hacked away on code, whined on the amount of outstanding patches and bugs and generally made a large amount of bad jokes and Monthy Python references.

The happy core team was caught on a picture:

Rockbox team Devcon 2011

On the Saturday we plowed through a lengthy list of discussion points to really make the most of all of us gathering physically. Among the outcomes from that is that we decided we want to change to git, we think a lot of future of Rockbox lies in the app for Android, we keep the Archos support and more. The Android builds are going to get into the build system ASAP and we’re gonna setup a system where (only) trusted build clients will participate in the building of Android builds that will be distributed to users – this since applications on phones will have a much greater risk of causing harm if some “bad guy” would try to infect our system with stupid things.

Dominik “bluebrother” Riebling brought up the very interesting point that none of us had noticed: we have two different logos being used in the project: one with the K being in front of the b (like the one on the web page) and one with the K being behind the b – which is used in SVG logos and on just about all Rockbox t-shirts made so far! If you zoom in on the tshirts on the group picture you’ll see!

We will also start allowing GPLv3 code into Rockbox in order to be able to use espeak, but all our code will remain GPLv2 or later. I could only find a single USB header file left that comes from the Linux source tree and has a GPLv2 only license.

Even more than this was discussed but I figure the rest of the details will be posted properly on rockbox.org for those seriously interested.

All in all, it was a very enjoyable weekend with a lot of fun and great friends. We stayed at a hotel just a few blocks from the devcon office which was really convenient. even though its wakeup routine was a bit non-standard. Peter “petur” D’Hoye took a lot of pictures as usual.

We also managed to break the Tower of Rockbox record.

Daniel "Bagder" Stenberg Rockbox Devcon 2011

The group picture was taken by a Google person I don’t know the name of who helped us out, and the one of me was taken by Peter D’Hoye.

Rockbox

Rockbox bridge and tower

4 Comments

Keeping to the tradition and subtle arts of Rockbox Towers, but doing it with a twist to celebrate the place we have Rockbox devcon 2011, we decided to make a Rockbox bridge.

We started out by gathering all devices we had in the room that can run Rockbox and distributed them on the construction floor area. As the Android app runs fine on tablets now there’s actually a rather good way to get some solid base into the construction…

Many Rockbox devices

Once all material was known, the construction started with a large amount of eager engineers contributing with good and bad ideas and at times very shaky hands:

constructing a Rockbox bridge

(wods, scorche, gevaerts and paumary)

The result, involving an iRiver beneath the bridge catching the digital flow, became what might be the longest Rockbox construction done so far:

Rockbox bridge

Rockbox bridge closeup

After the bridge, the work started on the real stuff. Building the tallest Rockbox tower ever made. After a couple of accidents and crashes, the tireless team managed to break the previous 104 cm record and the new Rockbox tower record is now officially 117cm:

Rockbox devcon 2011 tower 117cm

(Pictures in this post were all taken by Peter D’Hoye.)

Open Source

foss-sthlm, the sixth, the controversial one

I’m happy to say that I was the main planner and organizer of yet another foss-sthlm meetup, the sixth. Last week we attracted about one hundred eager FOSS hackers to attend to this meeting to which I had found and cooperated a somewhat controversial sponsor.

We have had a discussion within foss-sthlm now due to this event’s sponsor: what kinds of companies are acceptable as sponsor for FOSS events and what are not? It is obvious that we have a lot of different opinions here and several people have expressed that they deliberately didn’t go to meetup #6 simply because of the sponsor’s involvement and relationship to the Swedish defence industry. Do you think a defence manufacturer or weapon systems creator can also have its good sides and sponsor good activities or should we distance ourselves from them?

I’m honestly still interested in more opinions on this. We have not formed a policy around this subject because I simply don’t think that’s a good idea. We’re not a formal organization and we all have our different views. I think we have our members hand around and participate as long as we stay within a reasonable “boundary”. If we would get involved with the wrong kind of companies, a larger portion of the group would boycott the meeting or just plainly leave foss-sthlm. But why would we ever? I’m the one who’s mostly been in touch with sponsors and I would certainly not get involved and ask for money from companies that I believe have crossed the magic line in the sand.

The meeting was perhaps the most techy and most advanced of them all so far, and I found the talks very inspiring and educating. I’ve not had time or energy to put up a page with pictures or descriptions of them, and I think I’ll just skip it. You should probably try harder to attend next time instead!

cURL and libcurl, Technology

US patent 6,098,180

5 Comments

(I am not a lawyer, this is not legal advice and these are not legal analyses, just my personal observations and ramblings. Please correct me where I’m wrong or add info if you have any!)

At 3:45 pm on March 18th 2011, the company Content Delivery Solutions LLC filed a complaint in a court in Texas, USA. The defendants are several bigwigs and the list includes several big and known names of the Internet:

  • Akamai
  • AOL
  • AT&T
  • CD Networks
  • Globalscape
  • Google
  • Limelight Networks
  • Peer 1 Network
  • Research In Motion
  • Savvis
  • Verizon
  • Yahoo!

The complaint was later amended with an additional patent (filed on April 18th), making it list three patents that these companies are claimed to violate (I can’t find the amended version online though). Two of the patents ( 6,393,471 and 6,058,418) are for marketing data and how to use client info to present ads basically. The third is about file transfer resumes.

I was contacted by a person involved in the case at one of the defendants’. This unspecified company makes one or more products that use “curl“. I don’t actually know if they use the command line tool or the library – but I figure that’s not too important here. curl gets all its superpowers from libcurl anyway.

This Patent Troll thus basically claims that curl violates a patent on resumed file transfers!

The patent in question that would be one that curl would violate is the US patent 6,098,180 which basically claims to protect this idea:

A system is provided for the safe transfer of large data files over an unreliable network link in which the connection can be interrupted for a long period of time.

The patent describes several ways in how it may detect how it should continue the transfer from such a break. As curl only does transfer resumes based on file name and an offset, as told by the user/application, that could be the only method that they can say curl would violate of their patent.

The patent goes into detail in how a client first sends a “signature” and after an interruption when the file transfer is about to continue, the client would ask the server about details of what to send in the continuation. With a very vivid imagination, that could possibly equal the response to a FTP SIZE command or the Content-Length: response in a HTTP GET or HEAD request.

A more normal reader would rather say that no modern file transfer protocol works as described in that patent and we should go with “defendant is not infringing, move on nothing to see here”.

But for the sake of the argument, let’s pretend that the patent actually describes a method of file transfer resuming that curl uses.

The ‘180 (it is referred to with that name within the court documents) patent was filed at February 18th 1997 (and issued on August 1, 2000). Apparently we need to find prior art that was around no later than February 17th 1996, that is to say one year before the filing of the stupid thing. (This I’ve been told, I had no idea it could work like this and it seems shockingly weird to me.)

What existing tools and protocols did resumed transfers in February 1996 based on a file name and a file offset?

Lots!

Thank you all friends for the pointers and references you’ve brought to me.

  • The FTP spec RFC 959 was published in October 1985. FTP has a REST command that tells at what offset to “restart” the transfer at. This was being used by FTP clients long before 1996, and an example is the known Kermit FTP client that did offset-based file resumed transfer in 1995.
  • The HTTP header Range: introduces this kind of offset-based resumed transfer, although with a slightly fancier twist. The Range: header was discussed before the magic date, as also can be seen on the internet already in this old mailing list post from December 1995.
  • One of the protocols from the old days that those of us who used modems and BBSes in the old days remember is zmodem. Zmodem was developed in 1986 and there’s this zmodem spec from 1988 describing how to do file transfer resumes.
  • A slightly more modern protocol that I’ve unfortunately found no history for before our cut-off date is rsync, as I could only find the release mail for rsync 1.0 from June 1996. Still long before the patent was filed obviously, and also clearly showing that the one year margin is silly as for all we know they could’ve come up with the patent idea after reading the rsync releases notes and still rsync can’t be counted as prior art.
  • Someone suggested GetRight as a client doing this, but GetRight wasn’t released in 1.0 until Febrary 1997 so unfortunately that didn’t help our case even if it seems to have done it at the time.
  • curl itself does not pre-date the patent filing. curl was first released in March 1998, and the predecessor was started around summer-time 1997. I don’t have any remaining proofs of that, and it still wasn’t before “the date” so I don’t think it matters much now.

At the time of this writing I don’t know where this will end up or what’s going to happen. Time will tell.

This Software patent obviously is a concern mostly to US-based companies and those selling products in the US. I am neither a US citizen nor do I have or run any companies based in the US. However, since curl and libcurl are widely used products that are being used by several hundred companies already, I want to help bring out as much light as possible onto this problem.

The patent itself is of course utterly stupid and silly and it should never have been accepted as it describes trivially thought out ideas and concepts that have been thought of and implemented already decades before this patent was filed or granted although I claim that the exact way explained in the patent is not frequently used. Possibly the protocol using a method that is closed to the description of the patent is zmodem.

I guess I don’t have to mention what I think about software patents.

I’m convinced that most or all download tools and browsers these days know how to resume a previously interrupted transfer this way. Why wouldn’t these guys also approach one of the big guys (with thick wallets) who also use this procedure? Surely we can think of a few additional major players with file tools that can resume file transfers and who weren’t targeted in this suit!

I don’t know why. Clearly they’ve not backed down from attacking some of the biggest tech and software companies.

patent drawing

(Illustration from the ‘180 patent.)

c-ares, cURL and libcurl

libcurl’s name resolving

Recently we’ve put in some efforts into remodeling libcurl’s code that handles name resolves, and then in particular the two asynchronous name resolver backends that we support: c-ares and threaded.

Name resolving in general in libcurl

libcurl can be built to do name resolves using different means. The primary difference between them is that they are either synchronous or asynchronous. The synchronous way makes the operation block during name resolves and there’s no “decent” way to abort the resolves if they take longer time than the program wants to allow it (other than using signals and that’s not what we consider a decent way).

Asynch resolving in libcurl

This is done using one of two ways: by building libcurl with c-ares support or by building libcurl and tell it to use threads to solve the problem. libcurl can be built using either mechanism on just about all platforms, but on Windows the build defaults to using the threaded resolver.

The c-ares solution

c-ares’ primary benefit is that it is an asynchronous name resolver library so it can do name resolves without blocking without requiring a new thread. It makes it use less resources and remain a perfect choice even if you’d scale up your application up to and beyond an insane number of simultaneous connections. Its primary drawback is that since it isn’t based on the system default name resolver functions, they don’t work exactly like the system name resolver functions and that causes trouble at times.

The threaded solution

By making sure the system functions are still used, this makes name resolving work exactly as with the synchronous solution, but thanks to the threading it doesn’t block. The downside here is of course that it uses a new thread for every name resolve, which in some cases can become quite a large number and of course creating and killing threads at a high rate is much more costly than sticking with the single thread.

Pluggable

Now we’ve made sure that we have an internal API that both our asynchronous name resolvers implement, and all code internally use this API. It makes the code a lot cleaner than the previous #ifdef maze for the different approaches, and it has the side-effect that it should allow much easier pluggable backends in case someone would like to make libcurl support another asynchronous name resolver or system.

This is all brand new in the master branch so please try it out and help us polish the initial quirks that may still exist in the code.

There is no current plan to allow this plugging to happen run-time or using any kind of external plugins. I don’t see any particular benefit for us to do that, but it would give us a lot more work and responsibilities.

cURL