Category Archives: Web

web stuff

50 hours offline

Several sites in the haxx.se domain and other stuff related to me and my fellows were completely offline for almost 50 hours between August 24th 19:00 UTC and August 26th 20:30 UTC.

The sites affected included the main web sites for the following projects: curl, c-ares, trio, libssh2 and Rockbox. It also affected mailing lists and CVS repositories etc for some of those.

The reason for the outage has been explained by the ISP (Black Internet) to be because of some kind of sabotage. Their explanation given so far (first in Swedish):

Strax efter kl 20 i måndags drabbades Black Internet och Black Internets kunder av ett mycket allvarligt sabotage. Sabotaget gjordes mot flera av våra core-switchar, våra knutpunkter. Detta resulterade i ett mer eller mindre totalt avbrott för oss och våra kunder. Vi har polisanmält händelsen och har ett bra samarbete med dem.

Translated to English (by me) it becomes:

Soon after 8pm on Monday, Black Internet and its customers were struck by a very serious act of sabotage. The sabotage was made against several of our core switches. This resulted in a more or less total disruption of service for us and our customers. We have reported the incident to the police and we have a good cooperation with them.

Do note that you could keep track of this situation by following me on twitter.

It’s good to be back. Let’s hope it’ll take ages until we go away like that again!

Update: according to my sources, someone erased/cleared Black Internet’s core routers and then they learned that they had no working backups so they had to restore everything by hand.

I host www.libssh2.org

Sara Golemon, the founder and former maintainer of libssh2, pointed over the main site www.libssh2.org to my server the other day and now my previously unofficial libssh2 web site suddenly turned out to be the only and official one.

The plan is now to get the web contents push into a separate git repo to allow all libssh2’ers to modify it.

I’m also open and interested in feedback and ideas on how to improve the web site in whatever kind of way you think. Consider the current site mostly a placeholder for the info we have. How can we make it better?

libssh2

HTTP cookies IETF working group

So finally (remember I mentioned this list when it was created back in January 2009) an IETF http-state working group was created, with the following description:

The HTTP State Management Mechanism (Cookies) was original created by Netscape Communications in their Netscape cookie specification, from which a formal specification followed (RFC 2109, RFC 2965). Due to years of implementation and extension, several ambiguities have become evident, impairing interoperability and the ability to easily implement and use HTTP State Management Mechanism.

I’m on the list from the start and I hope to be able to contribute some of my cookie experiences and knowledge to aid the document to actually end up with something useful. The ambition, while it was “toned down” somewhat since the initial posts of the mailing lists, is still fairly high I would claim:

The working group will refine RFC2965 to:

  • Incorporate errata and updates
  • Clarify conformance requirements
  • Remove known ambiguities where they affect interoperability
  • Clarify existing methods of extensibility
  • Remove or deprecate those features that are not widely implemented and also unduly affect interoperability
  • Add features that are already widely implemented or have a critical mass of support
  • Where necessary, add implementation advice
  • Document the security properties of HTTP State Management Mechanism and its associated mechanisms for common applications

In doing so, it should consider:

  • Implementer experience
  • Demonstrated use of HTTP State Management Mechanism
  • Impact on existing implementations and deployments
  • Ability to achieve broad implementation.
  • Ability to address broader use cases than may be contemplated by the original authors.

The Working Group’s specification deliverables are:

  • A document that is suitable to supersede RFC 2965
  • A document cataloging the security properties of HTTP State Management Mechanism

I think this is a scope that is manageable enough to actually have a chance to succeed and its planning is quite similar to that of the IETF httpbis group. Still, RFC2965 lists a huge pile of stuff that has never been implemented by anyone and even though it was a while since I did read that spec I also expect it to lack several things existing cookie parsers and senders already use. The notorious IE httpOnly is an example I can think of right now.

My HTC Magic Review

This is my first “smartphone” I’ve owned myself so of course I have nothing else this fancy to actually compare against. I’ve played around with others’ a few times but that doesn’t really count. I’ve owned perhaps 8 mobile phones since I got my first one 1996, and they have all been Nokias and Sony Ericssons.

I was never really interested in iPhone due to many reasons. It is not open. It has a (very) restricted app distribution mechanism. It forbids apps from running simultaneously etc. And it has a pretty strong connection with itunes with no proper mass-storage syncing supported. But I admit that it has a slick UI and many cool apps.

My plan is to get some Android hacking going eventually and this is basically the first Android phone that has reached Swedish soil. I mean without requiring me to bend over backwards to get it, as I’m sure I could’ve bought previous Android phones from obroad if I really wanted to.

Random good things:

  • it’s fast, most things run faster than on my previous Sony Ericsson thing and yet this is way more advanced with much bigger screen estate and fancier UI
  • it has a nice gui that you mostly can guess how to work with
  • I love being able to use a qwerty-style keyboard when messaging instead of relying on T9 etc
  • wifi is fun, but with a decent data plan it basically only brings me slightly improved speed and I often can’t even tell the difference!
  • the integration with the Google services are nice, gmail and maps most noticeably
  • there really are a bunch of existing cool apps (I know iphone has lots more, but there are still thousands)
  • it has a much better approach to messaging, similar to what I’ve seen in the iphone, than I’ve ever experienced in a Nokia or Sony Ericsson. It focuses on conversations and keeps the “thread”.HTC Magic
  • I really really like the feeling of it being a networked thing that also can make phone calls. I can browse, use maps, use gmail just as easily as I can message or call people. With my previous phones all the internet-related services always felt tacked on like a very late afterthought.
  • The notification system is nice, and the three-screen wide “home” with its widget-system is really neat.

Bad stuff:

  • I’ve had some apps crash on me on occasion. But it’s rarely a problem as they’re restarted automatically for me.
  • Toggling wifi on/off a lot can sometimes lead to me not getting any data network at all, and I’ve had to reboot the phone to get back to phone-based (Edge/3G) data.

On-screen keyboard

Of course any and all geek friend I have ask me about how I deal with the on-screen keyboard. I must admit I’m still quite fond of it. Mostly because a physical keyboard makes the phone clonky and it adds physical contraints and wear-points that I don’t like. So the keyboard is a bit small, especially when the phone is in portrait mode, but the suggested completions are fine and I believe I’m already typing pretty quickly on the thing. When I ssh’ed from the phone to one of my servers I did find the obvious lack of cursor keys (to for example navigate an ordinary ncurses-based app or the command line history of a bash prompt) but other than that I really can’t complain.

Background Applications

One obvious advantage compared to iphones is of course the ability to run applications exactly the way I’d like. I can actually run the irc client and then have it in the background while I go browse the web or answer a call or whatever and then at my choice go back to the still connected irc client. In fact when playing with this it feels like a really ridiculous restriction of the iphone.

Comparing to my SE w550i

My previous phone is 94 grams compared to the Magic’s 116. The magic has a much bigger screen. The magic is roughly 11mm wider and 14mm taller. That makes it use 30% more volume (85 cm2) but still fits fine in the front pocket of any set of pants I use. The magic claims a lot longer battery life, but given that it has so much functionality I can’t help to play with all the time I doubt it’ll notice. It’ll more likely run down fast simply because I’ll use it more.

I’m also pleased that there’s no problem to just plug in the Magic to my Linux desktop and copy/sync the photos and the videos etc.

Google Integration

I realize some people will feel that the very tight integration with Google and Google’s services is a downside as it adds just another item that Google “owns” in your life. Still, it makes the experience very slick and as a user I get a lot of stuff “for free” as it just connects to lots of things that I already used and had accounts on. So gmail, sharing photos on picasaweb etc “just works”.

Concepts of a new distributed build

It was time to make an overhaul of our distributed builds system for Rockbox. The one currently in place is quite fancy and it does build 106 builds in around 7-8 minutes, but during the years it has served us we have found a few areas where we want to improve.

The goals for the new system were primarily:

  • do all the builds faster
  • reverse the connection so that people can contribute clients easier
  • make a system that is more allowing for slower machines to contribute

The biggest weaknesses of the existing system:

  • The master uses ssh to the distributed clients, which forces them to have an accessible ssh server and port etc. It also makes it awkward for people behind NATs who wants to run more clients.
  • It only hands out a particular build to one client, so thus if a large build happens to get handed to a slow client towards the end of a build round, all the other clients will sit idle waiting for the last client to finish.
  • The build and the subsequent upload of results to the master are synchronous, so thus a client with a very slow uplink may spend a significant time on the upload before it can start the next build.

The  new system is currently in development. It consists of a server that runs on one of our main servers, and there’s a client script that each volunteer contributor runs on their systems.

The clients connect to the master on a dedicated TCP port, specifying user name, password, name of the particular client instance, what particular architectures the client can build and how many bogomips the client boasts. While bogomips is a bogus way to measure anything, we’ve started out using it for a rough way to sort the the build clients based on speed.

The clients keep connected to the server all the time. There’s a ping message from the master every N second of idleness to make sure the connection is kept alive. As soon as the master wants the client to do a build, it sends a message to it detailing exactly how it should build it and using what SVN revision. The client will then do the build at once, upload the results using HTTP to a dedicated place and then tell the server the build is complete.

The server knows about all builds to do at a  commit, what we call a build round. It has a rough “score” or “weight” for each build that grades them in a slow to fast order. When a build round starts, the server will first sort all builds based on number of times they’ve been handed out and as secondary sort key the “weight” of it. Then it loops over the currently connected build clients and hand out builds from the sorted build table. The server then continues to do that until all clients have three builds each to build. As soon as a build is reported to have been completed by a client, that client will get the next build from the sorted build list.

If a client connects to the server and the server deems the client to be too old (since it does specify its version in the handshake message), it will be told to update to a specific version instead and come back then. This way the server can update all build clients when important things are fixed.

The clients will soon start to get assigned builds that already have been assigned to another client. This is not a problem but in fact our intention. The client that completes the build first will simply tell the server, and the server will then tell all the other clients that build that same build that they should cancel that particular build.

A client that joins the server in the middle of a build round will simply get a bunch of builds immediately and join in. A client that disconnects during a build round simply won’t complete its builds and other clients will instead do them. The system is also tolerant against the fact that bogomips is lame to compare computers with, and that the build “score” may not be very accurate or even that some server will have very slow or very fast upload speeds at unpredictable times.

The build master itself does not know when to start a new build round. It simply knows about the concept and it knows how to tell clients to complete a round. To make the master to start a new round, you need to connect to the server’s listening port and issue a special command and provide a password and then you can tell the server to start a build of a specific SVN revision. Or to queue up a build to be performed after the current one if there happens to be one in progress already.

When a full build round is complete, a hundred or so builds have been done, and full packages and log files are now in a directory on the build server, the server will simply trigger an external script that then takes care of updating our build table etc. In fact, every single completed build will optionally trigger an external script to allow web pages or stats pages to get updated as we go.

This build system is currently pretty Rockbox-specific as this is the project and development system we’re writing this for, but there’s really nothing in this that must be this way. I’m sure that if someone (you?) wants to adapt this for another project, I’d be more than happy to assist and to help ensuring that this becomes a more generic distributed build system. Just raise your hand and step forward!

At the time of this writing, (primarily) me and Björn are still ironing out quirks in this new system to hopefully get it going live real soon…

Rockbox

encrypted file transfer protocols compared

I like putting up some explanatory “this versus that” documents on stuff I know a little about. I’ve done things like curl vs wget, ftp vs http and http vs bittorrent in the past.

This time, I decided it was about time to do a technical comparison of the four major encrypted file transfer protocols SCP, SFTP, FTPS and HTTPS and explain how they differ in as many aspects and viewpoints as possible. I quite often get questions about how some of these compare against some of the others and why you’d use one instead of another etc. I hope this document will help people to find such answers themselves.

Of course I do mistakes and sometimes express myself in muddy ways, so your feedback and help is important. You can help me make this comparison become better!

http://daniel.haxx.se/docs/encrypted-transfer-protocols-compared.html

It’s still rough and all, but what question and comparisons between them do you miss? What mistakes have I done? What parts aren’t spelled out clear enough?

The web shop timeout mystery

Another one of the things in the modern world I’ve not yet understood:

why on earth do some web-based shops timeout your shopping and automatically clear you “shopping cart” if you just leave it around for a few hours/days? Why why why? What harm does it do them if I don’t hurry on to purchase?

I love being able to press ‘buy’ on lots of stuff (that then are added to the “cart”) and then ponder a few days if I want more stuff, if I selected the right models, alter a few things and similar. So when they time-out on me like this, it’s like a blow in the face and I need to start over again. It’s simply crazy that I have to backup my list of things to buy just in case they’ll flush me before I’m done!

Yes, I’m aware that some sites offer “save lists” etc if you’re registered and logged in and all, but I don’t want to have to do that.

I can imagine that at times things run out of stock or they even change the prices of merchandise that’s in my cart, but they could still solve that in other ways than just clearing everything.

HTTP Status Report

Mark Nottingham Mark Nottingham held a very interesting one hour talk on the status of HTTP and the work on HTTPbis on a QCon conference recently, and luckily for us HTTP geeks there’s this great video/presentation from that.

curl is mentioned at least twice in the slides, unfortunately it has a wrong fact on the second mention where it says curl uses “Pragma: no-cache” as it isn’t true anymore. It used to do that, but we’ve stopped doing it in curl since a while ago.

I’m a subscriber to the httpbis mailing list and a casual contributor, but nonetheless his summary and overview of the state was refreshing as I’ve not been able to keep up with all the details and I haven’t been tracking that working group from its start either.

Not social enough

There’s this concept that’s very popular these days. Social networking web sites. I’ve always been intrigued by the six degrees of separation idea so I joined Facebook and I’ve given it a try. Result: yawn.

Of course I realize everything depends on who you are, how your social network works and so on, but for me the Facebook experiment has only proven to me what I already suspected: I’m not “social enough” to care about all my friends’ teeny weeny little issues and expressions. I don’t have many friend added (35 at this particular moment) but already at this low number I get terribly uncomfortable after reading too much personal goings-on. And I’m not interested in everyones’ top-lists, what IKEA furniture they would be or which of the characters in the Muppet Show they resemble the most. I’m not going to use Facebook much until something changes.

Twitter is another one of the more trendy sites and services. This is very chaotic and most of the stuff posted there is utter crap. But there are some interesting people to follow and I do my best at following the tradition and contribute with my junk: My Twitter feed. More seriously I kind of use and view twitter as chatter around the coffee machine at a virtual office. You can select who to listen to. You can say whatever you feel like and the ones who might care could be reading it… The good part – for me of course – being that I can stay all geeky and techy and avoid that facebookish stuff I don’t like. Oh, and if you’re a friend in this manner, do tell me so that I can follow you!

LinkedIn is different. Here’s a site with a different goal and perspective, and keeping in touch with people I’ve been involved with professionally is a totally different matter. This makes a lot of sense to me, and it’s actually proven to pay off – several times. I believe me being a contract developer of course also make me value having a large network to reach out to so that I keep getting myself interesting assignments on a regular basis! My LinkedIn page.

User data probably for sale

It’s time for a little “doomsday prophesy”.

Already seen happen

As was reported last year in Sweden, mobile operators here sell customer data (Swedish article) to companies who are willing to pay. Even though this might be illegal (Swedish article), all the major Swedish mobile phone operators do this. This second article mentions that the operators think this practice is allowed according to the contract every customer has signed, but that’s far from obvious in everybody else’s eyes and may in fact not be legal.

For the non-Swedes: one mobile phone user found himself surfing to a web site that would display his phone number embedded on the site! This was only possible due to the site buying this info from the operator.

While the focus on what data they sell has been on the phone number itself – and I do find that a pretty good privacy breach in itself – there’s just so much more the imaginative operators just very likely soon will offer companies who just pay enough.

Legislations going the wrong way

There’s this EU “directive” from a few years back:

Directive 2006/24/EC of the European Parliament and of the Council of 15 March 2006 on the retention of data generated or processed in connection with the provision of publicly available electronic communications services or of public communications networks and amending Directive 2002/58/EC

It basically says that Internet operators must store information of users’ connections made on the net and keep them around for a certain period. Sweden hasn’t yet ratified this but I hear other EU member states already have it implemented…

(The US also has some similar legislation being suggested.)

It certainly doesn’t help us who believe in maintaining a level of privacy!

What soon could happen

There’s hardly a secret that operators run network supervision equipments on their customer networks and thus they analyze and snoop on network data sent and received by each and every customer. They do this for network management reasons and for such legislations I mentioned above. (Disclaimer: I’ve worked and developed code for a client that makes and sells products for exactly this purpose.)

Anyway, it is thus easy for the operators to for example spot common URLs their users visit. They can spot what services (bittorrent, video sites, Internet radio, banks, porn etc) a user frequents. Given a particular company’s interest, it could certainly be easy to check for specific competitors in users’ visitor logs or whatever and sell that info.

If operators can sell the phone numbers of their individual users, what stops them from selling all this other info – given a proper stash of money from the ones who want to know? I’m convinced this will happen sooner or later, unless we get proper legislation that forbids the operators from doing this… In Sweden this sell of info is mostly likely to get done by the mobile network operators and not the regular Internet providers simply because the mobile ones have this end user contract to lean on that they claim gives them this right. That same style of contract and terminology, is not used for regular Internet subscriptions (I believe).

So here’s my suggestion for Think Geek to expand somewhat on their great shirt:

i-read-your-everything

(yeah, I have one of those boring ones with only the first line on it…)