daniel.haxx.se

I previously blogged when my network hardware died. Here’s the recap and continuation of that story and how things evolved…

One day my ADSL modem could no longer get sync, I couldn’t send data and my (landline) phone was dead. My phone is connected into the ADSL modem through which it does IP telephony. Other times this has happened I could just switch off the modem for 10 seconds and then back on again it would work again for another 6 months or a year or so.

I’ve had ADSL at roughly 12mbit working flawlessly for several years so this was an unexpected breakage.

On 14 sep 16:16 I called my operator’s (Bredbandsbolaget) support about the issue when the modem hadn’t been able to get contact for a whole day – I was suspecting some kind of glitch in the service from the other end. The support person said that I had a “very old modem” and they immediately decided to send me a new modem by mail that would fix my problems.

At 16 sep 18:51 I called support again. I received modem #2 and installed it this day. The modem, Xavi Technologies X5258-P2, is a much more fancy model than what I had been using for the last couple of years – the new one had 4 Ethernet ports and wifi. Not that I really care about that cruft as I want to use my own wifi router anyway to get control of things better.

When I plugged in modem #2 I noticed that it lit up the ‘phone’ LED at once (which normally would only be on if I use the phone) and while internet data seemed to work, the phone did not. When I called support again to ask about this, they decided it was a broken modem they had sent me and would send me a replacement at once.

A few days later I got modem #3 and installed it. I also got the joy of sending back two ADSL modems.

3 oct 20:25 – I called the support again. Modem #3 hung occasionally and I wanted to get their help to fix the problem. The support guy I talked to claimed his sometimes happens if a wifi router is too close to the modem and adviced me to put my ADSL modem and wifi router further apart. It sounded like a suspicious analysis and theory to me, as why would the modem completely hang from this and if it did, why would it keep on running for days at times after a reboot? The support person also revealed that he had detailed logs going back a few weeks at least where he could see my ADSL modem power recycles and he could also see “bad CRC” counters going up before my restarts. I moved my devices two meters apart.

A little side-story: the modem has wifi support, but as I run my own wifi router behind it I don’t want the modem’s wifi. I noticed it ran on a different channel than my regular one so it wasn’t an immediate concern. It did however turn out that in order to switch it off I had to configure that with a Windows program and in order to install that program I had to enter a username and password that I didn’t have. Asking support for the credentials, they instead offered to simply disable the wifi from their end instead. That was fine by me, but again showed what fancy controls they have over these things.

For a week or so my connection actually was better and I actually thought my suspicions about the fishy advice were wrong. But no. It turned out I was only lucky for a few days as then it started hanging again every few days. It would stop transfering data in/out, and the “phone” led would blink slowly. How on earth could a device like this hang in any circumstance? I’ve been an embedded developer all my professional life, I know hanging is the worst possible thing. I much better but still ugly way to resolve a problem without any obvious way out, would be to reboot. A reboot would’ve been annoying as well, but far from as annoying as this.

Now, after all, I have a fiber installation coming “soon” so I figured I could possibly just shut up and endure this ADSL mess and it will go away or at least change drastically once I get my new connection…

But eventually it got too tedious, also partly because my kids and my wife also found it annoying and troubling – I had to give up the eduring. The fiber installtion also seemed to be delayed. Who knows how long I was supposed to remain on ADSL.

So, on 5 dec 18:38 I was back on the phone with the support people and complained about the hangs I frequently get with modem #3. The guy listened to me explaining the issue, he checked the reboot logs from his side and swiftly decided he would send me a new modem. He decided to send a modem of a different brand this time to see if this made things work better in my end.

On dec 8th I got modem #4. A different model this time compared to #2 and #3. It was now a Zyxel P-2601. I got home from work at 18:15, had a quick dinner and then I connected the new equipment. Would this really be the end of my troubles? Anticipation!

- Oh harsh reality, how thee can be rough and cold.

This modem can’t be powered on. If I flip the power switch and turns it on, all the leds switch on but as soon as my finger leaves the power-on toggle again the modem turns itself off… At 18:52 I tried to call support, but a voice claimed they had “internal systems problems” so I gave up.

12:45 on Friday Dec 9th I called again and reported my broken modem and the friendly support woman was a bit surprised I had gotten a broken device as she said “straight from the factory”. She even expressed some sympathy about the replacement unit, modem #5, not being able to reach me until Monday.

On Monday the 12th I got an invoice wanting to charge me 500 SEK for one of the broken modems they claimed I never sent back so I had to call customer service again and have them not do that. (I find 500 SEK for a broken ADSL modem quite a hefty charge when that’s basically the price for a completely new and working unit…)

December 13, modem #5 arrived and I connected it. It didn’t work at once but the phone worked which gave me a clue, so I connected a laptop directly to the ADSL modem and when I then tried to use a browser on that network I reached an admin interface web server and by using that I could switch the modem over to “bridge mode”. It turned out the default setting for this device is to function as a DHCP server and all sorts of other funny things that I didn’t want it to do.

At the time of this writing, number five has been running without problems for 72 hours.

So I have this TV that I got for Xmas 2009. As it happens the guys at Philips clearly kept fixing the software and removed bugs after that moment. No surprise there really. I’ve been an embedded software developer for some twenty years by now. I know that software never gets “done” and that what ships in products is only what seems to be “good enough” at some point in time. Sometimes of course not even that good.

So the other day I took a photo of my TV firmware version. It shows how the firmware was made in April 2009. I did it during a discussion with a friend who happens to have the exact same TV as I do, and it then of course turns out he has a different (newer) firmware.

Oh right, I wonder if I can upgrade to a newer one? Once I’ve mastered the maze of the Philips web site I eventually found a download link and PDFs that told me how to. The list of fixes since my version was extensive and I noticed a few flaws mentioned that I have actually experienced!

The TV firmware download was a whopping 43MB. I realize this is because it is a full-fledged Linux system with kernel and God knows what else they’ve crammed in there. I decided to give it a closer check! The result of that was a little disappointing. It is quite clearly encrypted after some basic initial header.

The data that starts on offset 0×220 is not x86 instructions and in fact nothing in the beginning of the file looks like x86 code (I just ran a quick “objdump -D –target binary -m i386″ on the file). Of course, I don’t know what architecture my TV runs so perhaps even checking for x86 is wrong. I know MIPS is popular in DVDs, settop-boxes and related graphics stuff but…. Nah, I decided it really wasn’t worth the effort so I stopped investigating. I have no real intention of hacking on it anyway.

So I instead proceeded to the actual procedure of upgrading the thing.

Unzip the zip file and put the file in the root dir of a FAT32-formatted usb-stick. The instructions of course didn’t say it needs to be FAT32 but I used that and it worked, and I just smug in the dark to how a manufacturer like this just assumes that we would have FAT32 on our usb-sticks…

But I digress. When I inserted the upgrade USB, the TV switched itself off, was dark for a short while and then turned itself on again and showed the firmware upgrade screen.

The process was very fast, just like 30-40 seconds or something like that and then it was done and asked me to remove the “media” and restart. Of course we know that a usb stick is “media” so I removed it from the TV set.

The instructions were very clear that to “restart” the TV I must only press the ON/OFF button on the remote once and only once. So I was careful to do just that…

Nothing strange happened, but after a brief moment of black screen the regular and familiar interface.

I jumped into the firmware version menu to check it out and yes, it shows an updated version now:

I did a quick check to see if I could detect my previous quirks now, but they may really be gone. They’ve been related to sound through HDMI and some graphical “glitches” when feeding the TV with full HD from a laptop.

So, with this firmware that was shipped many months after I got my TV, I seem to have gotten a better product.

I haven’t yet tested this new version to a significant degree so I don’t know yet if I’ve gotten some new nasty side-effects from it, as sometimes these kinds of firmware upgrades really cause you pain when something that formerly used to work so good suddenly turns out to not work that good any longer.

http://www.rfc-editor.org/rfc/rfc6265.txt is out!

Back when I was a HTTP rookie in the late 90s, I once expected that there was this fine RFC document somewhere describing how to do HTTP cookies. I was wrong. A lot of others have missed that document too, both before and after my initial search.

I was wrong in the sense that sure there were RFCs for cookies. There were even two of them (RFC2109 and RFC2965)! The only sad thing was however that both of them were totally pointless as in effect nobody (servers nor clients) implemented cookies like that so they documented idealistic protocols that didn’t exist in the real world. This sad state has made people fall into cookie problems all the way into modern days when they’ve implemented services according to those RFCs and then blame their browser for failing.

It turned out that the only document that existed that were being used, was the original Netscape cookie document. It can’t even be called a specification because it is so short and is so lacking in details that it leaves large holes open and forces implementors to guess about the missing pieces. A sweet irony in itself is the fact that even Netscape removed the document from their site so the only place to find this document is at archive.org or copies like the one I link to above at the curl.haxx.se site. (For some further and more detailed reading about the history of cookies and a bunch of the flaws in the protocol/design, I recommend Michal Zalewski’s excellent blog post HTTP cookies, or how not to design protocols.)

While HTTP was increasing in popularity as a protocol during the 00s and still is, and more and more stuff get done in browsers and everything and everyone are using cookies, the protocol was still not documented anywhere as it was actually used.

Somewhat modeled after the httpbis working group (which is working on updating and bugfixing the HTTP 1.1 spec), IETF setup a mailing list named httpstate in the early 2009 to start discussing what problems there are with cookies and all related matters. After lively discussions throughout the year, the working group with the same name as the mailinglist was founded at December 11th 2009.

One of the initial sparks to get the httpstate group going came from Bill Corry who said this about the start:

In late 2008, Jim Manico and I connected to create a specification for HTTPOnly — we saw the security issues arising from how the browser vendors were implementing HTTPOnly in varying ways[1] due to a lack of a specification and formed an ad-hoc working group to tackle the issue[2].

When I approached the IETF about forming a charter for an official working group, I was told that I was <quote> “wasting my time” because cookies itself did not have a proper specification, so it didn’t make sense to work on a spec for HTTPOnly.  Soon after, we pursued reopening the IETF httpstate Working Group to tackle the entire cookie spec, not just HTTPOnly.  Eventually Adam Barth would become editor and Jeff Hodges our chair.

I must stress that the work has only involved to document how things work today and not to invent or create anything new. We don’t fix any of the many known problems with cookies, but we describe how you write your protocol implementation if you want to interact fine with existing infrastructure.

The new spec explicitly obsoletes the older RFC2965, but doesn’t obsolete RFC2109. That was done already by RFC2965. (I updated this paragraph after my initial post.)

Oh, and yours truly is mentioned in the ending “acknowledgements” section. It’s actually the second RFC I get to be mentioned in, the first being RFC5854.

Future

I am convinced that I will get reason to get back to the cookie topic soon and describe what is being worked on for the future. Once the existing cookies have been documented, there’s a desire among people to design something that overcomes the problems with the existing protocol. Adam’s CAKE proposal being one of the attempts and ideas in the pipe.

Another parallel IETF effort is the http-auth mailing list in which lots of discussions around HTTP authentication is being held, and as they often today involve cookies there’s a lot of talk about them there as well. See for example Timothy D. Morgan’s document Weaning the Web off of Session Cookies.

I’ll certainly track the development. And possibly even participate in shaping how this will go. We’ll see.

(cookie image source)

After my printer died, I immediately ordered a new one online and not long afterwards I could pick it up from my local post office. As I use both the scanner and the printer features pretty much I went with another “all-in-one” model and I chose an HP model (again) basically because I’ve been happy with how my previous worked (before its death). “HP Officejet Pro 8500 A910″ seems to be the whole name. And yeah, it really is as black as the picture here shows it.

This model is less “photo-focused” than my previous but I never print my own photos so that’s no loss. What did annoy me was however that this model uses 4 ink cartridges instead of the 6 in my previous, but of a completely different design so I can’t even re-use my half-full ink containers from the corpse!

My new printer has some fancy features. It is one of them that I can give an email address and then print on by sending email to it. The email address then gets a really long one with lots of seemingly random letters, it is in the hp.com domain and I can set up a white-list of people (From: addresses) that is allowed to print on it via email.

It also has full internet access itself so it could fetch a firmware upgrade file and install that entirely on its own without the use of a computer. (Which made me wonder if they use libcurl, but I realize there’s no way for me to tell and of course there are many alternatives they might use.)

Driver-wise, it seems like a completely different set for Windows (hopefully this won’t uninstall itself) and on Linux I could install it fine to print, but xsane just won’t find it to scan. I intend to instead try to use the printer’s web service for scanning, hopefully that will be roughly equivalent for my limited use – I mostly scan documents, bills and invoices for my work.

I finally got my act together and bought myself a Synology DS211j NAS  with two 2TB drives. I’ll use it as a shared network disk at home and I intend to backup to it – as I also have a home office it’ll feel better to be able to also backup company related data somewhat more safely. My previous backup was only copying data from one HDD to another within the same physical machine.

To make it slightly more resistant to disk and hardware errors I’ve configured the disks to do RAID1 (all data is stored on both disks simultaneously).

The GPLv2 license was provided printed on paper in the package.

After a very long time of work, a very very long time since these devices were introduced on the mp3 player market, the hard working guys from freemyipod.org have produced something on yet another device. This is the same group that previously was called linux4nano and worked so long and fiercely to get code running on the 2nd generation iPod Nano and the 4th generation iPod Nano.

At the end of December 2010, Michael “TheSeven” Sparmann announced that he was running custom code including music playback on the iPod Classic. The (sometimes) so called 6th generation.

Robert Menes spiced up the story today by showing us a live picture of a Classic device that now actually is running Rockbox:

Awesome work Michael, truly impressive. I hope a lot of Classic owners soon will be able to try out Rockbox for real. Rockbox is said to not yet be very stable or functional, so there’s a lot of room for more hackers and developers to join in and help us improve!

A long while ago I posted my first version of the comparison of libssh vs libssh2. I have since then kept it updated and modified it over time. (Reminder: I am the libssh2 maintainer)

In that page, I included the performance differences I had measured which at the time showed libssh2 to be significantly faster when doing SCP operations.

The libssh guys always claimed I was wrong:

Please don’t be ridiculous. No competent network developer will take you seriously when you tell that libssh2 is 2.3 times faster that libssh.

and have even used rather harsh words when saying so.

you read this FUD page on the libssh2 website. I don’t want to start arguing here, the page is complete crap

(These two quotes are from the two leading libssh developers.)

Due to their complaints I withdrew the mentioning of the speed differences from the comparison page. Maybe I had done something wrong after all and since I didn’t care properly to go back and verify my methods and redo everything, I decided to just take it off until I have more backing or more accurate tests.

Fast forward to current time and Mark Riordan does his extensive performance tests of various SSH/SFTP implementations. He mailed the libssh mailing list about it, and his test results are interesting. I’m including it below for easier reading and just in case Mark’s original won’t be around as long as this.

It repeats very similar numbers to mine and shows the same speed difference that I was told cannot happen. Isn’t that funny? Am I still ridiculous?

SSH file transfer performance

The following table summarizes performance of SSH clients.

					LAN: 1 Gbit/sec		WAN: 6 Mb down, 0.9 Mb up
Solaris x86 server
Client	Client OS	Server	Comp Enable	File Cmp	UL MB/s	DL MB/s	UL MB/s	DL MB/s
libssh2	Win	Solaris	No	No	0.147	12.2
libssh2	Win	Solaris	Yes	No
libssh2	Linux	Solaris	No	No	0.82	11.8
libssh2	Linux	Solaris	Yes	No
libssh 0.4.6	Win	Solaris	No	No
Bitvise Tunnelier	Win	Solaris	No	No	13.50	3.95
Bitvise Tunnelier	Win	Solaris	Yes	No	8.541	10.2
psftp	Win	Solaris	No	No	9.4	5.06 or 0.46
WS_FTP 12.3	Win	Solaris	No	No	8.07	7.65
Ubuntu sftp	Linux	Solaris	?	No	29.6	11.5
Linux server
libssh2	Win	Linux	No	No	9.5	8.1	0.059	0.26
libssh2	Win	Linux	Yes	No
libssh2	Linux	Linux	No	No	7.4	7.4	0.083	0.267
libssh 0.4.6	Win	Linux	No	No	15.4	2.8	0.10	0.13
libssh 0.4.6	Linux	Linux	No	No	8.97	2.8	0.099	0.189
libssh 0.4.6	Linux	Linux	Yes	Yes	19.7	3.3
libssh latest	Win	Linux	No	No	14.1	1.38
psftp	Win	Linux	No	No	4.59	6.58	0.070	0.10
WS_FTP 12.3	Win	Linux	No	No	23.0	8.5	0.113	0.361
Bitvise Tunnelier	Win	Linux	No	No
Ubuntu sftp	Linux	Linux	No	No	16.2	6.6	0.11	0.51

What about SFTP?

It should be noted that in my original claim and in this test above we speak SSH speeds (like SCP), not SFTP. SFTP has its own slew of problems and libssh2 is in fact not very good at doing SFTP speedily yet. We have work in progress to improve this situation, but we’re not there yet. I’ll post a follow-up on SFTP speeds soonish as things have been developing nicely in there recently.

What about speeds compared to other clients?

libssh2 is not fully on par with for example openssh when it comes to raw SCP speed, but it is in the same “neighborhood”.

On ebay there’s a fancy S3C244-based board named mini 2440 with a 3.5″ touch LCD attached on sale for 85 USD. 64MB ram, 400MHz CPU, a nand flash and more. Lots of stuff for the money.

The guys in the lyre project seem to have adopted this as yet another hardware platform to attempt to run Rockbox on. After their Atmel AT91SAM target was ditched, they went the ARMopendous route and now this seems to have entered. This third hardware platform is called the Lyre prototype 2…

You should note that this Mini 2440 board has no batteries or anything and thus is not really meant to be a portable device in this shape.

“Bob” seems to have initial Rockbox code running on this device, and well-established Rockbox hackers JdGordon and domonoky have both ordered their own kits so the future looks bright.

Some owners of the Sansa Clip player from SanDisk noticed that it does playback of all songs with a minor pitch. Due to a flawed HW setup they don’t do a proper 44,100 Hz but instead 43,791 Hz (0.993 times the target value) or something like that. According to some sources this problem might be fixed in the Sansa Fuze players now.

Bugs aren’t really so surprising, perhaps what is surprising is that this bug has been around now for almost two years. To make matters worse, SanDisk now decided that due to them making cheap players people shouldn’t expect them to be very good sound quality wise and therefore they can simply not fix the problems:

due to trade-off decisions that were made in engineering these products to deliver superior consumer value at what we believe are extremely attractive price points, our sound fidelity isn’t perfect.  We have re-evaluated the possibility of reducing the pitch variation and due to the engineering trade-offs the decision was made to stay with the current design. Very few listeners, however, have noticed or complained about it as an issue in actual practice.  For those who can detect sound differences with their naked ears during actual use and not via frequency analysis, our products may not be the best choice for them.

Clip owners out there now put their hope even more on Rockbox for Clip.

It was time to make an overhaul of our distributed builds system for Rockbox. The one currently in place is quite fancy and it does build 106 builds in around 7-8 minutes, but during the years it has served us we have found a few areas where we want to improve.

The goals for the new system were primarily:

• do all the builds faster
• reverse the connection so that people can contribute clients easier
• make a system that is more allowing for slower machines to contribute

The biggest weaknesses of the existing system:

• The master uses ssh to the distributed clients, which forces them to have an accessible ssh server and port etc. It also makes it awkward for people behind NATs who wants to run more clients.
• It only hands out a particular build to one client, so thus if a large build happens to get handed to a slow client towards the end of a build round, all the other clients will sit idle waiting for the last client to finish.
• The build and the subsequent upload of results to the master are synchronous, so thus a client with a very slow uplink may spend a significant time on the upload before it can start the next build.

The  new system is currently in development. It consists of a server that runs on one of our main servers, and there’s a client script that each volunteer contributor runs on their systems.

The clients connect to the master on a dedicated TCP port, specifying user name, password, name of the particular client instance, what particular architectures the client can build and how many bogomips the client boasts. While bogomips is a bogus way to measure anything, we’ve started out using it for a rough way to sort the the build clients based on speed.

The clients keep connected to the server all the time. There’s a ping message from the master every N second of idleness to make sure the connection is kept alive. As soon as the master wants the client to do a build, it sends a message to it detailing exactly how it should build it and using what SVN revision. The client will then do the build at once, upload the results using HTTP to a dedicated place and then tell the server the build is complete.

The server knows about all builds to do at a  commit, what we call a build round. It has a rough “score” or “weight” for each build that grades them in a slow to fast order. When a build round starts, the server will first sort all builds based on number of times they’ve been handed out and as secondary sort key the “weight” of it. Then it loops over the currently connected build clients and hand out builds from the sorted build table. The server then continues to do that until all clients have three builds each to build. As soon as a build is reported to have been completed by a client, that client will get the next build from the sorted build list.

If a client connects to the server and the server deems the client to be too old (since it does specify its version in the handshake message), it will be told to update to a specific version instead and come back then. This way the server can update all build clients when important things are fixed.

The clients will soon start to get assigned builds that already have been assigned to another client. This is not a problem but in fact our intention. The client that completes the build first will simply tell the server, and the server will then tell all the other clients that build that same build that they should cancel that particular build.

A client that joins the server in the middle of a build round will simply get a bunch of builds immediately and join in. A client that disconnects during a build round simply won’t complete its builds and other clients will instead do them. The system is also tolerant against the fact that bogomips is lame to compare computers with, and that the build “score” may not be very accurate or even that some server will have very slow or very fast upload speeds at unpredictable times.

The build master itself does not know when to start a new build round. It simply knows about the concept and it knows how to tell clients to complete a round. To make the master to start a new round, you need to connect to the server’s listening port and issue a special command and provide a password and then you can tell the server to start a build of a specific SVN revision. Or to queue up a build to be performed after the current one if there happens to be one in progress already.

When a full build round is complete, a hundred or so builds have been done, and full packages and log files are now in a directory on the build server, the server will simply trigger an external script that then takes care of updating our build table etc. In fact, every single completed build will optionally trigger an external script to allow web pages or stats pages to get updated as we go.

This build system is currently pretty Rockbox-specific as this is the project and development system we’re writing this for, but there’s really nothing in this that must be this way. I’m sure that if someone (you?) wants to adapt this for another project, I’d be more than happy to assist and to help ensuring that this becomes a more generic distributed build system. Just raise your hand and step forward!

At the time of this writing, (primarily) me and Björn are still ironing out quirks in this new system to hopefully get it going live real soon…