Tag Archives: Haxx

Rockbox services transition

Remember Rockbox? It is a free software firmware replacement for mp3 players. I co-founded the project back in 2001 together with Björn and Linus. I officially left the project back in 2014.

The project is still alive today, even of course many of us can’t hardly remember the concept of a separate portable music player and can’t figure out why that’s a good idea when we carry around a powerful phone all days anyway that can do the job – better.

Already when the project took off, we at Haxx hosted the web site and related services. Heck, if you don’t run your own server to add fun toy projects to, then what kind of lame hacker are you?

None of us in Haxx no longer participates in the project and we haven’t done so for several years. We host the web site, we run the mailing lists, we take care of the DNS, etc.

Most of the time it’s no biggie. The server hosts a bunch of other things anyway for other project so what is a few extra services after all?

Then there are times when things stop working or when we get a refreshed bot attack or web crawler abuse against the site and we get reminded that here we are more than eighteen years later hosting things and doing work for a project we don’t care much for anymore.

It doesn’t seem right anymore. We’re pulling the plug on all services for Rockbox that occasionally gives us work and annoyances. We’re offering to keep hosting DNS and the mailing lists – but if active project members rather do those too, feel free. It never was a life-time offer and the time has come for us.

If people still care for the project, it is much better if those people will also care for these things for the project’s sake. And today there are more options than ever for an open source project to get hosting, bug tracking, CI systems etc setup for free with quality. There’s no need for us ex-Rockboxers to keep doing this job that we don’t want to do.

I created a wiki page to detail The Transition. We will close down the specified services on January 1st 2021 but I strongly urge existing Rockboxers to get the transition going as soon as possible.

I’ve also announced this on the rockbox-dev mailing list, and I’ve mentioned it in the Rockbox IRC.

A curl delivery network

I’ve run my own public web sites on hardware I’ve administered myself for over twenty years now. I’ve hosted the curl web site myself since it’s inception.

The curl web site at curl.haxx.se has recently been delivering roughly 1.5 terabyte of data to the world per month. The CA bundle we convert to PEM from the Mozilla source code, is alone downloaded more than 100,000 times per day. Occasional blog entries I’ve posted here on my blog have climbed very fast on popular sites such as Hacker news and Reddit, and have resulted in intense visitor storms hitting this same server – sometimes reaching visitor counts above 200,000 “uniques” – most of them within the first few hours of the publication. At times, those visitor spikes have effectively brought the server to its knees.

Yes, my personal web site and the curl web site are both sharing the same physical server. It also hosts more than a dozen other sites and numerous services for our own pleasures and fun, providing services for a handful of different open source projects. So when the server has to cease doing work because it runs out of memory or hits other resource restraints, that causes interruptions all over. Oh yes, and my email doesn’t reach me.

Inconvenient and annoying.

The server

Haxx owns and runs this co-located server that we have a busload of web servers on – for the good of the projects and people that run things on it. This machine’s worst bottle neck is available RAM memory and perhaps I/O performance. Every time the server goes down to a crawl due to network traffic overload we discuss how we should upgrade it. Installing a new machine and transferring over all the sites and services is work. Work that none of us at Haxx are very happy to volunteer to do. So it hasn’t been done yet, and frankly the server handles the daily load just fine and without even a blink. Which is ninety nine point something percent of the time…

Haxx pays for a certain amount of network traffic so as long as we’re below some threshold we remain paying the same monthly fee. We don’t want to increase the traffic by magnitudes as that would cost more.

The specific machine, that sits deep inside a server room in Stockholm Sweden, is a five(?) years old Dell Poweredge E310, Intel Xeon X3440 2.53GHz with 8GB ram, This model is shown on the image at the top.

Alternatives that hasn’t helped

Why not a mirror system? We had a fair amount of curl site mirrors a few years ago, but it never worked well because they were always less reliable than the main site and they often turned stale and out of sync with the master site which eventually just hurt users.

They also trick visitors into bookmark or otherwise go back to the mirror site instead of the real one and there were always the annoying people who couldn’t resist but to fill the mirror with ads and stuff. Plus, they didn’t help much with with the storms to the main site.

Why not a cloud server? Because with the amount of services, servers and various things we do on our server, it would be inconvenient and expensive. But perhaps even more because we started out like this so we have invested time and energy into the infrastructure as it works right now. And I enjoy rowing my own boat!

The CDN

Fastly reached out and graciously offered to help us handle the load. Both on the account of traffic amounts but also to save our machine from struggling this hard the next time I’ll write something that tickles people’s curiosity (or rage) to that level when several thousands of visitors want to read the same article at the same time.

Starting now, the curl.haxx.se and the daniel.haxx.se web sites are fronted by Fastly. It should give web site visitors from all over the world faster response times and it will make the site more reliable and less likely to have problems due to traffic load going forward.

In case you’re not familiar with what a CDN is, a simplified explanation would say it is a globally distributed network of reverse proxy servers deployed in multiple data centers. These CDN servers front the Internet and will to the largest extent possible serve the visitors with the right content directly from their own caches instead of them reaching the actual lowly backend server I run that hosts the original content. Fastly has lots of servers across the globe for this purpose. Users who are a long way away from Sweden will probably be the ones who will notice this change the most, as you may suddenly find haxx.se content much closer (network round-trip wise) than before.

Standards

These new servers will host the sites over HTTPS just like before, and they will require TLS 1.2 and SNI. They will work over IPv6 and support HTTP/2.  Network standard wise, there shouldn’t be any step down – and honestly, I haven’t exactly been on the cutting edge of these technologies myself for these sites in the past.

Editing the site

We will keep editing and maintaining the site like before. It is made up of an old system with templates and include files that generate mostly static web pages. The site is mostly available on github and using that, you can build a local version for development and trying out changes before they land.

Hopefully, this move to Fastly will only make the site faster and more reliable. If you notice any glitches or experience any problems with the site, please let us know!

Snaxx delivers

A pint of guinnessLate in the year 1999 I quit my job. I handed over a signed paper where I wrote that I quit and then I started my new job first thing in the year 2000. I had a bunch of friends at the work I left and together with my closest friends (who coincidentally also switched jobs at roughly the same time) we decided we needed a way to keep in touch with friends that isn’t associated with our current employer.

The fix, the “employer independent” social thing to help us keep in touch with friends and colleagues in the industry, started on the last of February 2000. The 29th of February, since it was a leap year and that fact alone is a subject that itself must’ve been discussed at that meetup.

Snaxx was born.

Snaxx is getting a bunch of friends to a pub somewhere in Stockholm. Preferably a pub with lots of great beers and a sensible sound situation. That means as little music as possible and certainly no TVs or anything. We keep doing them at a pace of two or three per year or so.

Bishops Arms logo

Yesterday we had the 31st Snaxx and just under 30 guests showed up (that might actually have been the new all time high). We had many great beers, food and we argued over bug reporting, discussed source code formats, electric car charging, C64 nostalgia, mentioned Linux kernel debugging methods, how to transition from Erlang to javascript development and a whole load of other similarly very important topics. The Bishops Arms just happens to be a brand of pubs here that have a really sensible view on how to run pubs to be suitable for our events so yesterday we once again visited one of their places.

Thanks for a great time yesterday, friends! I’ll be setting up a date for number 32 soon. I figure it’ll be in the January 2015 time frame…If you want to get notified with an email, sign up yourself on the snaxx mailing list.

A few pictures from yesterday can be found on the Snaxx-31 G+ event page.

A payment on a flight story

“Three beers and three chicken sandwiches, thanks” – I said, graciously handing over my VISA card to the flight stewardess to pay for everything for me and my two Haxx friends Björn and Linus. 20 something Euros. Neither of us were carrying any cash.

We were happy that we had seats in the 7th row on the way back to Stockholm since when we flew down to FOSDEM Brussels just two days earlier we were in the 18th row and by then they had ran out of sandwiches. Brussels Airlines on a direct flight.

The lady inserted the card into her handheld card-reader, messed around a while since it wouldn’t “take” at the first attempt and then she handed it to me to enter my PIN. So I did.

credit cardsMs stewardess pressed some buttons or something and then she said, “no it doesn’t work” and continued to try inserting my card in basically every variation you can (especially having the chip side turned out has to be a very clever way) while repeating to me that it doesn’t work. This is the same card I’ve used numerous times during the weekend and I used it several times at the airport less than an hour ago. I know it works.

“OK let’s say you’re right”, I sighed as I really didn’t think I nor my card were to blame but I also didn’t feel like just causing trouble. I handed her my second VISA card. “Here, try this instead then”.

“No it doesn’t work either”. This certainly wasn’t right.

Linus reached over and offered his MasterCard. Perhaps there was a VISA problem with the device but “Sorry sir, it doesn’t work” came back. We’re now at farce level. Björn joins the fun and offers a 4th card, another MasterCard.

By this time we’ve eaten most of the sandwiches and were enjoying the beers.

The lady continued to struggle and it still didn’t work. She was starting to act a bit troubled about this and I asked her if she really insisted that all our cards are broken and she admitted that she suspected the machine wasn’t working quite as it should. Then she leaned towards the male steward who was serving drinks a couple of steps away. They fiddled a bit more and then she came back to us.

“That worked, I swiped it” she said without much further explanation and returned with Björn’s card and the receipt for the purchase. I felt quite done with this by now so I didn’t ask nor pushed her why she didn’t do that earlier. I was glad it finally was fixed. So much for me paying, now Björn did it instead. Oh well, truly fascinating that they would do like this on an airline where people have to do purchases with credit cards all the time.

Then Björn looked at the receipt he got back:

Pringles and a bottle of water, 5 Euro.

It made all three of us burst out in uncontrolled laughter. It was then followed by some contemplation what it actually meant. What exactly did Björn pay for? Did he only get the wrong receipt or did he pay for it? If so, who paid for our food and drinks?

It is 2014 and we haven’t come further than this.

(Update: Björn reminded me that we did verify the last 4 digits of the card number on the receipt and it didn’t match any of our cards…)

My FOSDEM 2014

I’m back home after FOSDEM 2014.Lots of coffee A big THANK YOU from me to the organizers of this fine and totally free happening.

Europe’s (the World’s?) biggest open source conference felt even bigger and more crowded this year. There seemed to be more talks that got full, longer lines for food and a worse parking situation.

Nothing of that caused any major concern for me though. I had a great weekend and I met up with a whole busload of friends from all over. Many of them I only meet at FOSDEM. This year I had some additional bonuses by for example meeting up with long-term committers Steve and Dan from the curl project whom I had never met before IRL. Old buddies from Haxx and Rockbox are kind of default! 🙂

Talk-wise this year was also extra good. I’ve always had a soft spot for the Embedded room but this year there was fierce competition for my attention so I spread my time among many rooms and got to see stuff about: clang the compiler, lots of really cool stuff on GDB, valgrind and helgrind, power efficient software, using the GPU to accelerate libreoffice, car automation and open source, how to run Android on low-memory devices, Firefox on Android and more.

I missed out the kdbus talks since it took place in one of them smaller devrooms even though it was “celebrity warning” all over it with Lennart Poettering. In general there’s sometimes this problem at FOSDEM that devrooms have very varying degrees of popularity on the different talks so the size of the room may be too large or too small depending on the separate topics and speakers. But yeah, I understand it is a very hard problem to improve for the organizers.

As a newbie Firefox developer at Mozilla I find it fun to first hear the Firefox on Android talk for an overview on how things  run on that platform now and then I also got references to Firefox both in the helgrind talk and the low-memory Android talk. In both negative and positive senses.

As always on FOSDEM some talks are not super good and we get unprepared speakers who talk quietly, monotone and uninspired but then there’s the awesome people that in spite of accents and the problem of speaking in English as your non-native language, can deliver inspiring and enticing talks that make me just want to immediately run home and try out new things.

The picture on the right is a small tribute to the drinks we could consume to get our spirits up during a talk we perhaps didn’t find the most interesting…

This year I found the helgrind and the gdb-valgrind talks to be especially good together with Meeks’ talk on using the GPU for libreoffice. We generally found that the wifi setup was better than ever before and worked basically all the time.

Accordingly, there were 8333 unique MAC addresses used on the network through the two days, which we then can use to guesstimate the number of attendees. Quite possibly upwards 6000…

See you at FOSDEM 2015. I think I’ll set myself up to talk about something then. I didn’t do any this year.

This is my first day at Mozilla

I’m now a Senior Network Engineer at Mozilla. Today is my first day as an employee. Here’s my summary of how I got here and how the process has been so far, from my perspective.

Mozilla is an American company

OK, that’s not news to anyone and neither is it to me. I’m starting off with this because I’m not American. I’m a Swede, and I live in Sweden. When working and communicating with my fellow people at the mothership I of course get the slight cultural differences that are American versus Swedish. It’s not a problem because I’ve gotten quite used to working with Americans at work over the years but going through an entire recruiting and starting-a-new-job process entirely in English with Americans is new to me.

Interviewing

So when I was opening up to my social network to start looking for new assignments since I decided to end my previous one, I was approached by a friend who works for Mozilla. Of course I was interested to work for Mozilla!

So I had a Skype interview with a recruiter first, and then with two engineers. During evenings of course since Mozilla like many other tech companies are mainly in the west coast of the US, meaning -9 hours from me.

After those initial interviews I had to struggle with myself since I was in this luxury position of having no less than two other very interesting projects being suggested. How do you pick the best one out of three really good alternatives? I actually had to wrestle with all the details and factors involved and I decided that this lead was the best out of the three. If Mozilla wanted me, I’d go for that option.

So I took 4 days off from my current work and flew out to Mountain View California one day in November 2013. It takes some 17 hours to get there, I had a spare day to get somewhat adjusted to the time zone and then I fired off no less than five 45 minute interviews in a row that Wednesday. Then I got a ride back to SFO and I took the plane back home.

I’ve been a consultant for 16 years and I’ve done way over 30 projects during this time. I’ve been interviewed for all of them, and a bunch more that I didn’t end up getting. You can say I’m quite experienced in getting interviewed for work. What I’ve learned is to stay honest, just be me and have an as good time as possible but in the end it really isn’t possible to tell how the other end interprets me and my answers and if I match what they desire.

Getting hired

They didn’t reject me. I got the offer. I said yes. I signed the papers.

So one of the most frequent questions I’ve got when I told friends about my new job:

So you’re not gonna be a consultant anymore?

And no, I’m not. This option only came in the form of an employment or not at all, and I decided I wanted to do this rather than the alternatives. I’ve done consultant based development since 1996 and now I’m not anymore! Of course I still have tight bounds to Haxx and I certainly won’t exclude that I’ll return there one day.

My agreement also allows me to spend some time on curl development. Don’t you worry about that. I feel confident that I will only increase my commitment and efforts there.

Getting anxious

Once I had accepted the job I immediately wanted to start and by golly I got a harder time trying to motivate myself do good the period until that assignment ended.

I’ve had more phone meetings, I’ve checked out code and repeatedly rebuilt Firefox since then, I’ve read Bugzilla entries and perhaps most of all I’ve participated in Mozilla’s “onboarding program” which is a web based system that gives me tasks and asks questions and provides information, all in an effort to give them the information about me that they need and give me the proper info on Mozilla, what it is about, its background and what we’re doing ahead. (See how smoothly I used “we” there?)

Anxiously waiting for The Day. This may sound a bit melodramatic to some, but I do kind of feel like my life is changing a bit now into something it wasn’t before.

Starting a new job working remotely at day one

Today is The Day. I assume not everyone of you reading this have done it, but I start my first day at my new work for my new employer on the upper floor of my house, completely alone. I work from home, remotely and I’m also mostly isolated from a large portion of my colleagues by time zones.

So I grab my coffee and walk up the stairs to my home office. I sit down and I dive in. Source code. Lots of source code.

There’s a steep learning curve here, but I’m not afraid of some hard work and I like a challenge, I like code, I like open source and I like internet protocols. And I have good coffee.

Mozilla. I’m here. I’ve started. Today.

Mozilla dinosaur head logo

I go Mozilla

Mozilla dinosaur head logo

In January 2014, I start working for Mozilla

I’ve worked in open source projects for some 20 years and I’ve maintained curl and libcurl for over 15 years. I’m an internet protocol geek at heart and Mozilla seems like a perfect place for me to continue to explore this interest of mine and combine it with real open source in its purest form.

I plan to use my experiences from all my years of protocol fiddling and making stuff work on different platforms against random server implementations into the networking team at Mozilla and work on improving Firefox and more.

I’m putting my current embedded Linux focus to the side and I plunge into a worldwide known company with worldwide known brands to do open source within the internet protocols I enjoy so much. I’ll be working out of my home, just outside Stockholm Sweden. Mozilla has no office in my country and I have no immediate plans of moving anywhere (with a family, kids and all established here).

I intend to bring my mindset on protocols and how to do things well into the Mozilla networking stack and world and I hope and expect that I will get inspiration and input from Mozilla and take that back and further improve curl over time. My agreement with Mozilla also gives me a perfect opportunity to increase my commitment to curl and curl development. I want to maintain and possibly increase my involvement in IETF and the httpbis work with http2 and related stuff. With one foot in Firefox and one in curl going forward, I think I may have a somewhat unique position and attitude toward especially HTTP.

I’ve not yet met another Swedish Mozillian but I know I’m not the only one located in Sweden. I guess I now have a reason to look them up and say hello when suitable.

Björn and Linus will continue to drive and run Haxx with me taking a step back into the shadows (Haxx-wise). I’ll still be part of the collective Haxx just as I was for many years before I started working full-time for Haxx in 2009. My email address, my sites etc will remain on haxx.se.

I’m looking forward to 2014!

Rpi night in GBG

pelagicore logo

Daniel talking So I flew down to and participated at yet another embedded Linux hacking event that was also co-organized by me, that took place yesterday (November 20th 2013) in Gothenburg Sweden.

The event was hosted by Pelagicore in their nice downtown facilities and it was fully signed up with some 28 attendees.

I held a talk about the current situation of real-time and low latency in the Linux kernel, a variation of a talk I’ve done before and even if I have modified it since before you can still get the gist of it on this old slideshare upload. As you can see on the photo I can do hand-wavy gestures while talking! When I finally shut up, we were fed tasty sandwiches and there was some time to socialize and actually hack on some stuff.

Embedded Linux hackers in GBG

I then continued my tradition and held a contest. This time I did raise the complexity level a bit as I decided I wanted a game with more challenges and something that feels less like a quiz and more like a game or a maze. See my separate post for full details and for your chance to test your skills.

This event was also nicely synced in time with the recent introduction of the foss-gbg mailing list, which is an effort to gather people in the area that have an interest in Free and Open Source Software. Much in the same way foss-sthlm was made a couple of years ago.

Pelagicore also handed out 9 Raspberry Pis at the event to lucky attendees.

Haxx turns 0x10

On October 17th 1997 we registered Haxx as a company. Today we’re 0x10 years old. We like hexadecimal numbers.

BirthdaycakeHaxx was founded before the first curl version was released.

It was before Google.

It was long before Firefox and Chrome and even before the Mozilla browser appeared.

Heck, a lot of things of today didn’t exist those 16 long years ago.

It was a different and in many ways simpler world back then, but I would say that we’ve manage quite good to keep with the times and we’ve progressed fine as a company and as individuals all since then.

Me, Björn and Linus are still going strong with more contacts, more customers and possibly more fun than ever.

Happy birthday to us!

Better pipelining in libcurl 7.30.0

Back in October 2006, we added support for HTTP pipelining to libcurl. The implementation was naive and simple: it basically preferred to pipeline everything on the single connection to a given host if it could. It works only with the multi interface and if you do a second request to the same host it will try to pipeline that.

pipelines

Over the years the feature was bugfixed and improved slightly, which proved that at least a couple of applications actually used it – but it was never any particularly big hit among libcurl’s vast amount of features.

Related background information that gives details on some of the problems with pipelining in the wild can be found in Mark Nottingham’s Making HTTP Pipelining Usable on the Open Web internet-draft, Mozilla’s bug report “HTTP pipelining by default” and Chrome’s pipelining docs.

Now, more than six years later, Linus Nielsen Feltzing (a colleague and friend at Haxx) strikes back with a much improved and almost completely revamped HTTP pipelining support (merged into master just hours before the new-feature window closed for the pending 7.30.0 release). This time, the implementation features and provides:

  • a configurable number of connections and pipelines to each unique host name
  • a round-robin approach that favors starting new connections first, and then pipeline on existing connections once the maximum number of connections to the host is reached
  • a max-depth value that when filled makes the code not add any more requests on that connection/pipeline
  • a pipe penalization system that avoids adding new requests to pipes that are known to be receiving very large contents and thus possibly would stall subsequent requests for an extended period of time
  • a server blacklist that allows the application to specify a list of servers for which HTTP pipelining should not be attempted – real world tests has proven that some servers are too broken to be allowed to play the game

The code also adds a feature that helps out applications to do massive amount of requests in a controlled manner:

A hard maximum amount of connections, that when reached makes libcurl queue up easy handles internally until they can create a new connection or re-use a previously used one. That allows an application to for example set the limit to 50 and then add 400 handles to the multi handle but it will still only use 50 connections as a maximum so over time when requests get completed it will start new transfers on the requests that are waiting in line and thus shrinking the queue and keeping the maximum amount of connections until there’s less than 50 left to do…

Previously that kind of queuing had to be done by the application itself, but now with the much more extensive pipelining support it really isn’t as easy for an application to know when the new request can get pipelined or create a new connection so this logic is now provided by libcurl itself. It is likely going to be appreciated and used also by non-pipelining applications…

This implementation is accompanied with a bunch of new test cases for libcurl (and even a new HTTP test server for the purpose), and it has been tested in the wild for a while with libcurl as the engine in a web browser implementation (the company doing that has requested to remain anonymous). We believe it is in fairly decent state, but as this is a large step and the first release it is shipped with I expect there to be some hiccups along the way.

Two things to take note of:

  1. pipelining is only available for users of libcurl’s multi interface, and only if explicitly enabled with CURLMOPT_PIPELINING
  2. the curl command line tool does not use the multi interface and thus it will not use pipelining