Decrypting ipods

Recently we’ve seen progress by the linux4nano guys in their quest to get custom code to run on an Ipod Nano 2nd generation. They’ve apparently managed to extract the bootrom off a 2nd gen ipod nano (my copy of their extracted data is here – a reminder on objdump usage: “arm-elf-objdump -D --target binary -marm [file]“). I believe their intent is to port Linux to the newer ipods. Possibly ipodlinux. They do mention providing the necessary info to Rockbox and yes we will welcome it.

A large crowd of Rockbox hackers have joined their IRC channel and have been hanging out with them and helped out discussing ideas and pushed them towards publishing their news and infos on how this all is accomplished etc. Their SVN repo hosts some (most?) of the tools made so far.

The Rockbox wiki page for nano2g has been updated and hopefully it will keep track of what happens.

There have been speculations, but I don’t yet know based on what facts, that this recent news and hacks will be usable on other recent (encrypted) ipod models.

Summary: very interesting progress has been made. Lots of it is still left to figure out. There seems to be a bunch of skilled people around and now we’re seeing information and documentation for this getting published so I can’t but to hope for a bright future!

Concepts of a new distributed build

It was time to make an overhaul of our distributed builds system for Rockbox. The one currently in place is quite fancy and it does build 106 builds in around 7-8 minutes, but during the years it has served us we have found a few areas where we want to improve.

The goals for the new system were primarily:

  • do all the builds faster
  • reverse the connection so that people can contribute clients easier
  • make a system that is more allowing for slower machines to contribute

The biggest weaknesses of the existing system:

  • The master uses ssh to the distributed clients, which forces them to have an accessible ssh server and port etc. It also makes it awkward for people behind NATs who wants to run more clients.
  • It only hands out a particular build to one client, so thus if a large build happens to get handed to a slow client towards the end of a build round, all the other clients will sit idle waiting for the last client to finish.
  • The build and the subsequent upload of results to the master are synchronous, so thus a client with a very slow uplink may spend a significant time on the upload before it can start the next build.

The  new system is currently in development. It consists of a server that runs on one of our main servers, and there’s a client script that each volunteer contributor runs on their systems.

The clients connect to the master on a dedicated TCP port, specifying user name, password, name of the particular client instance, what particular architectures the client can build and how many bogomips the client boasts. While bogomips is a bogus way to measure anything, we’ve started out using it for a rough way to sort the the build clients based on speed.

The clients keep connected to the server all the time. There’s a ping message from the master every N second of idleness to make sure the connection is kept alive. As soon as the master wants the client to do a build, it sends a message to it detailing exactly how it should build it and using what SVN revision. The client will then do the build at once, upload the results using HTTP to a dedicated place and then tell the server the build is complete.

The server knows about all builds to do at a  commit, what we call a build round. It has a rough “score” or “weight” for each build that grades them in a slow to fast order. When a build round starts, the server will first sort all builds based on number of times they’ve been handed out and as secondary sort key the “weight” of it. Then it loops over the currently connected build clients and hand out builds from the sorted build table. The server then continues to do that until all clients have three builds each to build. As soon as a build is reported to have been completed by a client, that client will get the next build from the sorted build list.

If a client connects to the server and the server deems the client to be too old (since it does specify its version in the handshake message), it will be told to update to a specific version instead and come back then. This way the server can update all build clients when important things are fixed.

The clients will soon start to get assigned builds that already have been assigned to another client. This is not a problem but in fact our intention. The client that completes the build first will simply tell the server, and the server will then tell all the other clients that build that same build that they should cancel that particular build.

A client that joins the server in the middle of a build round will simply get a bunch of builds immediately and join in. A client that disconnects during a build round simply won’t complete its builds and other clients will instead do them. The system is also tolerant against the fact that bogomips is lame to compare computers with, and that the build “score” may not be very accurate or even that some server will have very slow or very fast upload speeds at unpredictable times.

The build master itself does not know when to start a new build round. It simply knows about the concept and it knows how to tell clients to complete a round. To make the master to start a new round, you need to connect to the server’s listening port and issue a special command and provide a password and then you can tell the server to start a build of a specific SVN revision. Or to queue up a build to be performed after the current one if there happens to be one in progress already.

When a full build round is complete, a hundred or so builds have been done, and full packages and log files are now in a directory on the build server, the server will simply trigger an external script that then takes care of updating our build table etc. In fact, every single completed build will optionally trigger an external script to allow web pages or stats pages to get updated as we go.

This build system is currently pretty Rockbox-specific as this is the project and development system we’re writing this for, but there’s really nothing in this that must be this way. I’m sure that if someone (you?) wants to adapt this for another project, I’d be more than happy to assist and to help ensuring that this becomes a more generic distributed build system. Just raise your hand and step forward!

At the time of this writing, (primarily) me and Björn are still ironing out quirks in this new system to hopefully get it going live real soon…

Rockbox

encrypted file transfer protocols compared

I like putting up some explanatory “this versus that” documents on stuff I know a little about. I’ve done things like curl vs wget, ftp vs http and http vs bittorrent in the past.

This time, I decided it was about time to do a technical comparison of the four major encrypted file transfer protocols SCP, SFTP, FTPS and HTTPS and explain how they differ in as many aspects and viewpoints as possible. I quite often get questions about how some of these compare against some of the others and why you’d use one instead of another etc. I hope this document will help people to find such answers themselves.

Of course I do mistakes and sometimes express myself in muddy ways, so your feedback and help is important. You can help me make this comparison become better!

http://daniel.haxx.se/docs/encrypted-transfer-protocols-compared.html

It’s still rough and all, but what question and comparisons between them do you miss? What mistakes have I done? What parts aren’t spelled out clear enough?

Rockbox Devcon 2009 Summary

Rockbox Team Devcon in Ghent Belgium 2009

The Rockbox team that gathered in Ghent for this weekend of talk, hacking and socializing (drinking beer) is caught on this group picture. Click the image for a slightly larger version. Photo by Petur.

The people on the photo

The top line from the left: amiconn, markun, bertrik, gevaerts, GodEater, AlexP, Zagor, domonoky, Bagder (me!)

The lower line from the left: kugel, pixelma, scorche, petur

We did have a 2-hour discussion session on the saturday, and I expect to post an mp3 of it later on. The short and compressed outcome in plain text is found here. Petur was a great host. The facilities were nice, the hotel was great, the food arrangements worked out perfectly. A swell weekend!

As our tradition demands, we did bring out all our targets (portable music devices that can run Rockbox or at least have some code in the Rockbox repo) to be used as building bricks to create a Tower of Rockbox.

This first picture shows that we have a pretty wide selection of players in this room:

Rockbox Tower 2009 Device Overview

With all those “bricks” put in an imaginative order on top of each other, the result could look something like this:

Rockbox Rower 2009

you may enjoy comparing this building with last year’s creation.

More pictures from this year can be found in Petur’s collection and gevaerts’ collection.

libssh2 vs libssh

There are only two open source libraries for SSH that I am aware of. At least that are at the fundamental layer, written in C.

I researched the SSH library market years ago when I stuck with libssh2 as the one I thought was most promising, and since then I and others have taken it much further. The lib that I didn’t go with at that time, confusingly enough named libssh, recently came out with a new release.

Since there is now clearly two active open source SSH libraries it feels like we should help our users and potential newcomers by explaining how our projects and libraries differ. As a little teaser: one of the libraries turned out more than twice as fast as the other in my test…

While I admit to not having actually used libssh for real, I’ve read the docs and I’ve tried it a little bit. My take at a comparison is now online at:

http://www.libssh2.org/libssh2-vs-libssh.html

I will highly appreciate your feedback and additional things that differ between the two! The list isn’t really much to boast about as it currently looks!

HTTPbis at IETF75

Mark, one of the editors of the ongoing HTTPbis efforts, first mentioned that there wasn’t going to be any HTTPbis meeting on the upcoming IETF75 meeting in Stockholm July 26-31, 2009. I felt a bit sorry for that since I live in Stockholm, I’m a bit involved in the HTTPbis work and I’ve never been to a IETF meeting.

It simply must have been due to my almighty powers, but apparently two of the editors are going here anyway and there has now been a request for a HTTPbis session during the meeting.

I’m looking forward to this! Hopefully it’ll bring some fun talks on tech we care about, but also meeting cool people in real life that I never met before.

Stockholm

Oh, and am I the only one who can’t find the dates anywhere on ietf75.se?

Kernels on those phones

So Google says there could be 18 phones running Android by the end of this year. In Sweden we just days ago got HTC Magic, the first ever Android phone showing up here (tied to a ridiculous operator deal that makes me and lots of my friends not go that route). Then Palm shipped their Palm Pre just days ago, also based on Linux.

This has brought the interesting questions: how is the state of these kernel HTC Magicports in regards to the mainline Linux tree? They’re both using ARM cores (of course).

The ARM kernel maintainer Russell King himself is not impressed. Apparently Google hasn’t even tried to push their work upstream to the kernel in a long while. The tone in that discussion did make it sound as if they might be starting to work on this again now.

The Palm guys apparently haven’t even yet shown any code at all, but is said to be releasing their code within two weeks to opensource.palm.com.  They have not even tried to push their work upstream, so I figure they’re either not even going to bother or they are facing a rather steep uphill battle in the future.

Lyre

I’ve previously blogged about the initiative to build an own open hardware platform that can run Rockbox fine, and just today I noticed their new site is up and alive at:

http://lyre.sourceforge.net/

The hardware has changed quite significantly since the last blog entry of mine, and they’re now using a LPC3130 from NXP instead of the Atmel they had before, and I believe they’ve also changed codec/DAC etc. Me knowingly, Rockbox does not yet run on this newly produced board.

Lyre PCB

I should probably also add that this board is of course still quite far from being portable and there’s no news or info anywhere on how or if you can actually get one of these yourself yet.

curl, open source and networking