cURL and libcurl

curl keeps connections alive

Just in the last few days we modified curl to enable the SO_KEEPALIVE option on connections it creates. It basically means that curl will now detect connections that are idle after a certain amount of time, even if that time is something around two hours by default and that’s what most systems will have it set to.cURL

The main problem that caused us to finally enable this (you can still disable this by using –no-keep-alive) is when people do (long-lasting) FTP transfers and they use a NAT, firewall or router that detects and removes what it considers are idle connections. An FTP transfer is using two connections, but the control one where the commands are sent over is completely quiet while the actual data transfer is in progress so when the transfer is done, the control connection has been nuked by the router/NAT. Of course curl survives this as good as possible, but it can’t do proper error-checking etc in this situation.

Funnily, there’s no really good fix for the FTP situation since the two hours SO_KEEPALIVE timeout will many times be too long to help (although most modern systems allows you to change the timeout or a system or application level), but the other “obvious” fix is to send a “NOOP” command on the control channel every once in a while during the transfer. But no, that doesn’t work fine either on most servers since it seems the servers often don’t listen on the control connection during the transfer, so all we’d get is curl sending commands that won’t be replied to until the end of the transfer, and thus it will end up causing problems.

Note: curl sets this option. libcurl still doesn’t, so if you want your app to set the option you can use the same CURLOPT_SOCKOPTFUNCTION callback that curl uses. This requires libcurl 7.16.0 or later.

Books, Development

The Mythical Man-Month

The Mythical Man-MonthFrederick Brooks wrote this classical book already back in 1975 and added a few extra chapters for the twenty years anniversary 1995…

Large portions of it feels of the age and there’s a lot of talk about Fortran, System/360 and PL-1 as if we should know about them (which made me fast forward over some chapters). But there are gems as well, and the most significant things people seem to remember Brooks’ book for are still pretty valid and fine.

Adding more people to a project leads to the need for more communication and thus it may slow down development rather than speed it up. Also known as Brooks’s law.

Given the complexity of software and software development, there’s no single method or concept that will lead to an improvement by an order of magnitude – within a decade. There’s No Silver Bullet. (This section was not in the original edition of the book.)

The risks involved when rewriting something and wants to fix everything that was wrong in the previous version so you over-work and over-design the successor. The so called Second system effect.

A lot of the book is spent on thoughts and theories around how to manage really really large software projects, like when you involve thousands of persons. Is it even possible to make such huge projects successful and if so, what does it take? The extra chapters do indeed add value since they offered Brooks a chance to re-evaluate his earlier claims and ideas and to check what seemed to be truths and what mistakes he did in the original edition.

A very interesting read that I’m glad I finally got time to get through!

Rockbox

A thankful Rockbox user

I just wanted to share with you some very friendly and encouraging words I received in a mail from an individual that shall remain unidentified. It is actually rather unusual to get this kind of cheerful words as people who get things and enjoy them rarely get back, but instead we hear more from the ones who get problems or are otherwise unhappy. sansa e200R - looks almost like a e200 v2

“From someone not nearly so gifted and talented, thanks for your efforts to develop Rockbox on the Sansa e200 v2.
Your efforts are deeply appreciated by many, and I’m glad that there is someone as clever and kind as you working on this project in your spare time (which I’m sure is precious to you.)
Work on the project when you can, but don’t deprive your family of your time and presence – they are your most important treasure. I’m just a private individual, and I know this little expression of thanks doesn’t mean much, but here it is, just the same. All the very best wishes to you and your family for Christmas and the coming new year!”

Thanks a lot. (linkified by me)

Linux, Open Source

The Scandinavian Free Software Award went to…

1 Comment

skolelinux logoYou might remember that I mentioned that I was nominated for this award. The final winner of it was the Skolelinux project, as was announced on the fsf-europe press-release list just an hour ago (well the prize was handed out on Friday night but I forgot to ask yesterday so I didn’t know until now…). I guess the fscons site will be updated in due time as well…

As I said before, my own contributions to free software is rather tiny and insignificant compared to giants such as this. I think this project is a well-deserved winner of the award. Congratulations to all involved!

cURL and libcurl, Open Source

To fscons and back in 16 hours

I took the X2000 train to Gothenburg from Stockholm at 08:10 so I was at the conference place first at almost 11:30.

SELF

This meant I got to listen in on the end of Jonas Öberg‘s speech on SELF (an FSF Europe and others project on e-learning and a lot of related matters). This wasn’t really my cup of tea, but the other track had a MySQL talk and that isn’t really my thing so I had to just pick one… 🙂 Nothing bad about the subject or Jonas really, just a hint of where my interests are not so much.

Lunch

At lunch I got the opportunity to catch up with Squid-Henrik (Nordström) to talk about recent happenings in our projects. Squid being about to release v3 after several years, and I could report that curl has gotten support for SSH-related protocols during the last year or so… I also exchanged a few words with Peter Stuge who expressed interest in hacking on libssh2 recently, and well I have done some of that!

Qtopia

Trolltech’s GreenphoneKnut Yrvin from Trolltech Norway did an excellent talk on Qtopia in the Telecom business, which had changed name from something involving Greenphone since they have since ditched that project. Anyway, he spoke of the upcoming possible opportunities for free software and open source in consumer electronics, and then particularly in smart phones and then of course mostly related to Qt and Qtopia. He passed around a Greenphone to let us get a feel or it and fiddle a bit with it and yes, it seemed like a nice phone – not a lot bigger or heavier than my current Sony Ericsson thing. It featured a nice “sliding UI” even if I had serious troubles moving around in the system and I couldn’t really figure out the maneuvering concept much! I suspect all it would take is a little more time and perhaps a manual or someone explaining it to me.

OpenMoko

Neo1973In the subsequent talk Ole Tange from the OpenMoko project also handed out “fiddle-versions” of their primary phone, Neo1973 for us in the audience to touch and hold. The major bad with these were however that both devices were dead so we couldn’t see anything on them, just hold them and feel that… eh yeah. they’re a bit on the biggish side and quite a bit bigger than my current phone in all three dimensions! He spoke about the upcoming 2nd version of the phone that is supposed to become available in Q1 2008 and given that it will feature wifi, bluetooth, accelerometers, GPS, 640×480 pixels touch screen, accelerated graphics card and a mini-B USB plug and run an entirely open and free Linux version with documented hardware is indeed thrilling. The Neo1973’s size is not attractive, but its internals are. This is in fact a unit I will seriously consider buying/hacking when/if it becomes available for purchase.

Other details in his OpenMoko talk gave me the impression that the software is not yet very far advanced. Like he first made a comparison to the OLPC system with hw and sw items side-by-side both listing as GTK+ based UIs, but then he also mentioned a thanks to Trolltech for having ported their Greenphone Qtopia system to OpenMoko. On my direct question if that wasn’t a bit contradictive since surely they must be focusing on ONE of these graphics/widgets systems for their main development, he went on to rant about how OpenMoko “is a computer” that can “run anything”. I’m not sure, but it certainly gave me the impression that there just is no main development… Where is OpenMoko at right now really? Anyone knows? I guess I should spend some time on researching that, and also investigate a bit on the “running Rockbox on OpenMoko” front…

curl

cURLWhen the time came for my talk, at 15:00 we first had to mess about a bit since the computer I was supposed to borrow to run my presentation on was suddenly gone (and used for the other track’s talk I later learned) but thanks to other people I soon had a replacement and I got on with it.

I know the topic by heart of course, curl being my primary open source project for ten years and I know every bit of it and its history and so on, but making a fine presentation based on that is an entirely different story. Also, giving it in English adds a layer of, well not complexity perhaps, but it makes it all bit more rough in the edges since even though I know English pretty well and all, my vocabulary isn’t the largest and I don’t always find the right synonyms and the phrasing etc when trying to explain or argue for my sake.

Also, since I don’t quite know my own presentation by heart it isn’t really the best possible performance I can do, but what the heck. I tried to present curl and libcurl, what they are and what they’re good for, why people use it and how the development is done and why YOU should use it now and in the future. The guys at fscons got all talks on video so I hope to be able to see myself on video soon and I’ll try to learn from that for my next talk. And of course those of you who weren’t present at fscons will get your chance to see my pale face and listen to my Swedish-accented stumbling English! 😉 Oh, and I had to rush the presentation a bit towards the end when my 45 minutes ran out a little bit faster than I had anticipated, or was it the questions that popped up? Questions are good, since they make me aware the audience is with me and are interested.

I’m not sure if the topic of curl is somewhat boring, or if it felt too technical or what, but I think I had less than 50% of the audience listening. The other talk going on while I spoke was a lightning talk session with a bunch of people.

Here’s the slides from my talk, in a 31 page 500K pdf: http://daniel.haxx.se/curl-20071208.pdf

LinuxBIOS

Slightly dry in my mouth after this, I recharged myself with a cup of coffee and some cinnamon-rolls and walked it to see the next talk. Or rather series of talks since this was a “lightning talks” session where five guys spoke quickly about various topics. They were about web development with perl, a weird ajax system called gaia that seemed to involve a lot of .NET, a web development system of some sorts named makumba, and a quick mentioning of a 10 gigabit full open source router. For me, the most interesting piece was Peter Stuge’s brief talk about LinuxBIOS, what it is, what it does and so on. That’s really a to-the-metal project and I like getting back to earth and on to real stuff. Much of what he said and explained about difficulties with documentation from hardware vendors etc are just so familiar to me based on Rockbox experiences. To the great enjoyment of the audience, Peter’s live demo of LinuxBIOS booting up failed notoriously and after numerous resets it finally booted up and started playing loud music – when the following speaker already was half-through his router presentation!

Closing

I only got to hear the beginnings of the closing talk held by Georg Greve from FSF Europe as I had to leave after 20 minutes or so to catch my cab that took me back to the train station and I was on my way back to Stockholm again on the 18:42 train…

Did I mention that I got a tshirt? I planned to include a picture of the shirt here, but I took a shot with my mobile phone when I got home and the camera in it is just so extremely crappy in low-light situations (even if I had all the lights in the room turned on) so I can’t torture you by including it. I’ll have to make another attempt later or find a link to someone else who did…

In conclusion: even though I only did a quick visit and didn’t get to see that many talks, I liked what I saw and I had fun. It sounded like the guys doing this are seriously planning on doing it again next year. I hope they’ll do and that I’ll manage to do there again, hopefully to do another talk!

Books

80 Novels in One Go

80 romaner för dig som har brÃ¥ttomHenrik Lange is the author and illustrator of this book in Swedish that contains 80 international and some Swedish novels, each concentrated down to a single-page comic strip consisting of four squares only, and then one of them is “wasted” on the title!

They’re often really witty and yet they capture the core and culprit of the novels, and with fine illustrations to go with that!

Sagan om ringen - excerp from the book It’s a great and fun read and one of those books you can have lying around and just read a random page every now and then and enjoy them every time.

The example strip here on the right is typical for the book. A couple of thousand pages crammed into 3 squares…

Oh for the english readers, the title of the book translates to something like “80 novels for people in a hurry”. The Kindle only takes 120 more! 🙂

Rockbox

I Solved the AMS-Sansas’ Firmware Checksum Puzzle

On my Sansa v2 web page, I’m collecting firmware binaries for these new targets in order to figure them out and kickstart the Rockbox effort for them. All firmware files have a .bin extension.

It is quite clear (by simple human inspection) that the first 0x400 bytes in each .bin file is a header (padded with 0xff bytes), as on the 0x400 index there is the ARM exception vector and then there’s ARM code following.SanDisk marked chip, an AMS AS3525

In the header there are numerous values, but the 32 bit value at index 4 immediately looked like it could be a checksum of some sorts.

We found two very similar firmwares for the M200 model, one for the European and one for the American in which the “checksum” values only differed by 2 even though there were clearly multiple (although not extensive) differences in the files.

A checksum that differs with so little indicates a simple algorithm. With something more fancy, like CRC32 or similar, a very small change in the files would cause a major change in the checksum value. Two checksum values near each other rather hinted on a simple addition, subtraction, xor or similar.

So I did a hexdump of the two files, cut off the headers and ran a ‘diff -u’ on them. That showed me that the first lines that differed (on index 0x15990) looked like this in the euro version:

00 00 a0 e3 a4 40 9f e5 00 00 c1 e5 04 00 a0 e1
00 fb ff eb 04 00 80 e0 45 10 a0 e3 01 10 40 e5

And like this in the US version.

02 00 a0 e3 a4 40 9f e5 00 00 c1 e5 04 00 a0 e1
00 fb ff eb 04 00 80 e0 41 10 a0 e3 01 10 40 e5

The differences are shown in bold above to make them more obvious. Plus 4, minus 2… Or the other way, minus 4 plus 2. That was almost too good to be true! The fact that these particular differences seemed to be 2 when the values were added just have to mean that the checksum is done with addition (if I was lucky). And if so, this was the only change that mattered to the header so therefore the checksum didn’t take the whole file into account…

SanDisk Sansa ClipI wrote up a small tool that would try out some variations of an “addition algorithm” with 32 bit adds and with 8 bit adds and then I tried with XORs instead to the same effect. Then it struck me that the value in the header at index 0x0c was not changing by a lot between firmwares and it had a number which was an index after the change I mentioned above, but before the subsequent changes…

The program still didn’t spit out the right value when I restricted the algorithm to the size mentioned in the header… until I realized my tool didn’t skip the header when it did the checksum, and when I added a 0x400 bytes skip the values matched! It was as simple as that. Here’s checksum.c.

There are still a few other unidentified fields in the header.

Rockbox

Rockbox on iPod Touch

iPod TouchWith the recently published jailbreak for iPod Touch, combined with the SDL port for iPhone there should be little in the way for running Rockbox on it as an application, pretty much in exctly the same way I mentioned how Rockbox could be made to run on mobile phones.

It seems a suitable place to start this venture is the iphone-dev project page.

While the iPhone and iPod Touch aren’t 100% identical internally, it seems they’re similar enough to make the differences possible to ignore. Also, the fact that what everyone does is build applications that run under the normal Apple-provided OS, there’s no need to know or learn how to poke on the actual hardware so subtle differences in audio chips etc is abstracted away by the operating system even for applications put on the unit this way.

Update nov 2008: With the recent developments on the linuxoniphone blog, it looks like an iPod Touch version of Rockbox is now a lot more likely to be possible. Still, nobody has yet volunteered to start this work and I won’t even say that it is likely that anyone will make an attempt.

Rockbox

Rockbox on a Mobile Phone?

Sony Ericsson w580i has no RockboxMany of the hackers involved in the Rockbox project of course have had the dream to hack our phones as well one day to make them start behaving and working more in the way we want them to. Many times people have brought up the possibilities of (and the inherent problems with) porting Rockbox to a mobile phone. Mobile phones being even more locked down protected devices than your ordinary average mp3 player. These days mobile phones get more room for additional data (music and videos etc) and more music and media player abilities so many times Rockbox on the phone would make sense.

With the introduction of Linux-based phones, the chances have greatly improved. A Rockbox “simulator” can already be built for Linux, and that’s basically all Rockbox app-layer code that does everything Rockbox does but on a native Linux/cygwin instead of on a target device. It’s just about only SDL and a POSIX system needed.

While the Rockbox simulator so far in its 5.5 year’s life has had its primary use as just a simulator for Rockbox developers, it should be fairly easy and straight-forward to adapt it to a life as a stand-alone application. I figure the current work with get UI code adapted for use with touch screens also might help.

Here’s some platforms that might be suitable:

Motorola EZX

Motorola E680iMotorola has a series of phones usually referred to as the EZX phone platform (the Motorola E680, E680i, E680g, 780, A1200, ROKR E6, and ROKR E2 phones) that run Linux on an XScale PXA27x (ARM core) and have a working SDL port. A guy from Indonesia known as ‘blackhawk’ has ported Rockbox to it, but the details are scarce and the patches are infrequent and out-of-date. (Yes that’s a license violation.) I also just hate it when people store info about anything in nothing but a (weird and ad-packed) forum… Update: blackhawk just showed up in the Rockbox forum!

Android

Google’s huge effort is an operating system that is Linux-based (on ARM as well), so there’s basically just a fine SDL for it that’s missing before Rockbox can run on it. My suspicion is that Android is so java-focused and targeted so that SDL won’t be a primary goal for any phone that’ll run Android, and most probably it won’t be as easy to install and mess around with native code anyway.

OpenMokoNeo 1973

OpenMoko is another Linux-based operating system and environment for mobile phones, primarily for the Neo1973. SDL support is still only mentioned on their wish list.

Nokia N8x0

Nokia N810While the N800 series isn’t exactly a phone (although you can do IP-telephony over wifi with it), it is an “internet tablet” from Nokia that runs Linux and I would suspect that if it just becomes successful enough there will eventually be a phone based on that platform. And anyway, Rockbox for the N800 series might make sense even while it isn’t a phone… SDL is ported for this target (maemo).

Volunteers

Of course, all these are just possibilities that remain theoretical until there’s actually interested and willing volunteers that step forward and make it happen…

curl, open source and networking