daniel.haxx.se

We started off this second embedded hacking day (the first one being the one we had in October) when I sent out the invitation email on April 22nd asking people to sign up. We limited the number of participants to 40, and within two hours all seats had been taken! Later on I handed out more tickets so we ended up with 49 people on the list and interestingly enough only 13 of these were signed up for the previous event as well so there were quite a lot of newcomers.

Arrival

At 10 in the morning on Saturday June 1st, the first people had already arrived and more visitors were dropping in one by one. They would get a goodie-bag from our gracious host with t-shirt (it is the black one you can see me wearing on the penguin picture on the left), some information and a giveaway thing. This time we unfortunately did not have a single female among the attendees, but the all-male crowd would spread out in the room and find seating, power and switches to use. People brought their laptops and we soon could see a very wide range of different devices, development boards and early design ideas showing up on the tables. Blinking leds and cables everywhere. Exactly the way we like it!

Giveaway

We decided pretty early on the planning for this event that we wouldn’t give away a Raspberry Pi again like we did last time. Not that it was a bad thing to give away, it was actually just a perfect gift, but simply because we had already done that and wanted to do something else and we reasoned that by now a lot of this audience already have a Raspberry pi or similar device.

So, we then came up with a little device that could improve your Raspberry Pi or similar board: a USB wifi thing with Linux drivers so that you easily can add wifi capabilities to your toy projects!

And in order to provide something that you can actually hack on during the event, we decided to give away an Arduino Nano version. Unfortunately, the delivery gods were not with us or perhaps we had forgot to sacrifice the correct animal or something, so this second piece didn’t arrive in time. Instead we gathered people’s postal addresseAns and once the package arrives in a couple of days we will send it out to all attendees. Sort of a little bonus present afterwards. Not the ideal situation, but hey, we did our best and I think this is at least a decent work-around.

So the fun begun

In the big conference room next to the large common room, I said welcome to everyone at 11:00 before I handed over to Magnus from Xilinx to talk about Xilinx Zynq and combining ARM and FPGAs. The crowd proved itself from the first minute and Magnus got a flood of questions immediately. Possibly it was also due to the lovely combo that Magnus is primarily a HW-guy while the audience perhaps was mostly SW-persons but with an interest in lowlevel stuff and HW and how to optimize embedded systems etc.

After this initial talk, lunch was served.

Contest

I got lots of positive feedback the last time on the contest I made then, so I made one this time around as well and it was fun again. See my separate post on the contest details.

Flying

After the dust had settled and everyones’ pulses had started to go back to normal again after the contest, Björn Stenberg “took the stage” at 14:00 and educated us all in how you can use 7 Arduinos when flying an R/C plane.

It seemed as if Björn’s talk really hit home among many people in the audience and there was much talking and extra interest in Björn’s large pile of electronics and “stuff” that he had brought with him to show off. The final video Björn showed during his talk can be found here.

Stuff to eat

People actually want to get something done too during a day like this so we can’t make it all filled up with talks. Enea provided candy, drinks and buns. And of course coffee and water during the entire day.

Even with buns and several coffee refills, I think people were slowly getting soft in their brains when the afternoon struck and to really make people wake up, we hit them with Erik Alapää’s excellent talk…

Aliasing in C and C++

Or as Erik specified the full title: “Aliasing in C99/C++11 and data transfer between hard real-time systems on modern RISC processors”…

Erik helped put the light on some sides of the C programming language that perhaps aren’t the most used or understood. How aliasing can be used and what pitfalls it can send us down into!

Personally I don’t really had a lot of time or comfort to get much done this day other than making sure everything ran smooth and that everyone was happy and the schedule was kept. My original hopes was to get some time to do some debugging on a few of my projects during the day but I failed that ambition…

We made sure to videofilm all the talks so we should hopefully be able to provide online versions of them later on.

Real-time Linux

I took the last speaker slot for the day. I think lots of brains were soft by then, and a few people had already started to drop off. I talked for a while generically about how the real-time problem (or perhaps low-latency) is being handled with Linux these days and explained a bit about PREEMPT_RT and full dynamic ticks and what the differences of the methods are.

The end

At 20:00 we forced everyone out of the facilities. A small team of us grabbed a bite and a couple of beers to digest the day and to yap just a little bit more before we split up for the evening and took off home…

Thank you everyone who was there for making it another great event. Thank you all speakers for giving the event the extra brightness! Thank you Enea for sponsoring, hosting and providing all the goodies in such an elegant manner! It is indeed possible that we make a 3rd embedded hacking day in the future…

Okay, so here are the correct answers to the embedded hacking #2 contest (click for larger pictures):

The fact that you get the clues as hexadecimal uppercase ASCII was pretty quickly clear to everybody. I found it interesting to hear how people attacked the problem of decoding the hex into letters. Most people seem to have made a lookup-table fairly soon, and at least one contestant I talked to made a mistake in his table that turned W into X instead! This year’s winner did the conversion completely without a written down table…

So all the pieces are decoded like this:

Of course, now a pedant would argue that FORK() isn’t correct, but I decided to use all uppercase just to make the conversion slightly easier. At least I think converting only uppercase ASCII as hex is easier. So the question is “What does fork() return in the child process?”

The answer to the question is 0 (zero). Short and simple. See fork’s man page.

After 13 minutes and 20 seconds since I clicked start on the timer, Linus Nielsen Feltzing approached me with a little note with the correct answer and we had a winner!

The very happy Linus was very disappointed in the previous competition when he was very close to winning but was beaten just within seconds by last time’s winner.

Now, the Chromebook that Enea donated to the winner of the contest was handed over to Linus. (The Samsung Cortex-A15 version.)

I created another contest for the Embedded hacking event we just pulled off again, organized with foss-sthlm and Enea. Remember that I made one previously at our former hacking day?

The lesson from that time was that the puzzle ingredient then was slightly too difficult so people had to work a bit too long. It made many people give up and the ones who didn’t had to spend a significant time on solving it.

This time, I decided to use the same basic principle: ask N questions that all provide hints for the (N+1)th question, so that the first one to give me the answer to that final question is the winner. It makes it very easy for me to judge and it is a rather neat competition style game. I decided 10 questions should be enough.

To reduce some of the complexity from last time, I decided to provide the individual clues in the correct chronological order but instead add another twist: they aren’t in plain text! But since they’re chronological, the participants can go back and quite “easily” try other alternatives if there are some strange words appearing in the output. I made sure that all alternatives always have fine English alternatives so that if you pick the wrong answer it might still sound or look like English for a while…

I was very happy to see over 30 persons in the room that decided to accept the challenge. I suspect the prize did its part in attracting people to give it a go.

The rules in slightly longer terms as I put them (click it to see a higher resolution version):

And I clarified how the questions work:

I then started my timer, and I showed all the questions on the projector to everyone. I gave them around 40 seconds per question. It thus took almost seven minutes to go through them and then I left a final slide up showing all questions:

To allow readers to give this contest a go first before checking the answers. See the full answer and explanation.

Here are the videos from the embedded Linux hacking day foss-sthlm arranged on October 20th 2012. They are all speaking Swedish:

Linux och open source inom inbyggda system

med Daniel Stenberg

Yocto-projektet

med Björn Stenberg

Utveckling och trender av multicorekretsar inom halvledarindustrin

med Jonas Svennebring

Reverse engineering – egen kod på andras hårdvara

med Linus Nielsen Feltzing

During our embedded Linux hacking event in Stockholm on October 20th I ran a little contest for the ones who wanted to participate. I created it entirely by myself to allow as many people as possibly to participate with them knowing me or me knowing them etc limiting the fun.

For your amusement I include the full contest here. If you want to try it out, then make sure you don’t attempt to google for any answers or otherwise use a machine/computer as a help.

Here I just want to mention that, as is shown in the above example question, ‘ace‘ is the correct character sequence and the letters should then be kept in that order in the final question. Also note that a character sequence can legally contain a dash as well. You will get 16 similar sequences of 1 to 3 letters, and those 16 sequences should be moved around to form the 17th question.

… at this point I fired off all the questions one by one at about 15-20 seconds per question. In this blog post I’ll take a shortcut and instead show you the final page I made that showed all questions at once, which I then left displayed for the remainder of the competition  time. Click the image to get a full resolution version that is perfectly readable:

My take away from this contest is that it was harder than I anticipated and took a longer time to crack than I thought. I gave away a few additional clues and hints as the time went by, but in the end I believe there were several persons who were very close to breaking it at almost the same time. In the end, Klas and Jonas presented the correct answer first and won the bottle of Champagne. I’m sure you appreciate their efforts after having tried this yourself!

The answers? Are you really sure? The correct answers and the final question with its answer is available…

I had a great time creating the competition and I believe the competitors appreciated it.

Additional trivia: I created the letter sequences for the other alternatives by writing other English phrases and chopped them up, so that they were from actual English and hence possibly more believable.

In one of the fast-moving early scenes in episode 16 of Person of Interest at roughly 2:05 into the thing I caught this snapshot:

(click the image to see a slightly bigger version)

It is only in sight for a fraction of a second. What is seen in the very narrow terminal screen on the right is source code scrolling by. Which source code you say? Take a look again. That my friends is kernel/groups.c from around line 30 in a recent Linux kernel. I bet that source file never had so many viewers before, although perhaps not that many actually appreciated this insight!

And before anyone asks: no, there’s absolutely no point or relevance in showing this source code in this section. It is just a way for the guys to look techy. And to be fair, in my mind kernel code is fairly techy!

As a protocol geek I love working in my open source projects curl, libssh2, c-ares and spindly. I also participate in a few related IETF working groups around these protocols, and perhaps primarily I enjoy the HTTPbis crowd.

Meanwhile, I’m a consultant during the day and most of my projects and assignments involve embedded systems and primarily embedded Linux. The protocol part of my life tends to be left to get practiced during my “copious” amount of spare time – you know that time after your work, after you’ve spent time with your family and played with your kids and done the things you need to do at home to keep the household in a decent shape. That time when the rest of the family has gone to bed and you should too but if you did when would you ever get time to do that fun things you really want to do?

IETF has these great gatherings every now and then and they’re awesome places to just drown in protocol mumbo jumbo for several days. They’re being hosted by various cities all over the world so often I deem them too far away or too awkward to go to, also a lot because I rarely have any direct monetary gain or compensation for going but rather I’d have to do it as a vacation and pay for it myself.

IETF 83 is going to be held in Paris during March 25-30 and it is close enough for me to want to go and HTTPbis and a few other interesting work groups are having scheduled meetings. I really considered going, at least to meet up with HTTP friends.

Something very rare instead happened that prevents me from going there! My customer (for whom I work full-time since about six months and shall remain nameless for now) asked me to join their team and go visit the large embedded conference ESC in San Jose, California in the exact same week! It really wasn’ t a hard choice for me, since this is my job and being asked to do something because I’m wanted is a nice feeling and position – and they’re paying me to go there. It will also be my first time in California even though I guess I won’t get time to actually see much of it.

I hope to write a follow-up post later on about what I’m currently working with, once it has gone public.

This is not news. This is only facts that seem to still be unknown to many people so I just want to help out documenting this to help educate the world. I’ll dance around the subject first a bit by providing the full background info…

round robin basics

Round robin DNS has been the way since a long time back to get some rough and cheap load-balancing and spreading out visitors over multiple hosts when they try to use a single host/service with static content. By setting up an A entry in a DNS zone to resolve to multiple IP addresses, clients would get different results in a semi-random manner and thus hitting different servers at different times:

server     A   192.168.0.1
server     A   10.0.0.1
server     A   127.0.0.1

For example, if you’re a small open source project it makes a perfect way to feature a distributed service that appears with a single name but is hosted by multiple distributed independent servers across the Internet. It is also used by high profile web servers, like for example www.google.com and www.yahoo.com.

host name resolving

If you’re an old-school hacker, if you learned to do socket and TCP/IP programming from the original Stevens’ books and if you were brought up on BSD unix you learned that you resolve host names with gethostbyname() and friends. This is a POSIX and single unix specification that’s been around since basically forever. When calling gethostbyname() on a given round robin host name, the function returns an array of addresses. That list of addresses will be in a seemingly random order. If an application just iterates over the list and connects to them in the order as received, the round robin concept works perfectly well.

but gethostbyname wasn’t good enough

gethostbyname() is really IPv4-focused. The mere whisper of IPv6 makes it break down and cry. It had to be replaced by something better. Enter getaddrinfo() also POSIX (and defined in RFC 3943 and again updated in RFC 5014). This is the modern function that supports IPv6 and more. It is the shiny thing the world needed!

not a drop-in replacement

So the (good parts of the) world replaced all calls to gethostbyname() with calls to getaddrinfo() and everything now supported IPv6 and things were all dandy and fine? Not exactly. Because there were subtleties involved. Like in which order these functions return addresses. In 2003 the IETF guys had shipped RFC 3484 detailing Default Address Selection for Internet Protocol version 6, and using that as guideline most (all?) implementations were now changed to return the list of addresses in that order. It would then become a list of hosts in “preferred” order. Suddenly applications would iterate over both IPv4 and IPv6 addresses and do it in an order that would be clever from an IPv6 upgrade-path perspective.

no round robin with getaddrinfo

So, back to the good old way to do round robin DNS: multiple addresses (be it IPv4 or IPv6 or both). With the new ideas of how to return addresses this load balancing way no longer works. Now getaddrinfo() returns basically the same order in every invoke. I noticed this back in 2005 and posted a question on the glibc hackers mailinglist: http://www.cygwin.com/ml/libc-alpha/2005-11/msg00028.html As you can see, my question was delightfully ignored and nobody ever responded. The order seems to be dictated mostly by the above mentioned RFCs and the local /etc/gai.conf file, but neither is helpful if getting decent round robin is your aim. Others have noticed this flaw as well and some have fought compassionately arguing that this is a bad thing, while of course there’s an opposite side with people claiming it is the right behavior and that doing round robin DNS like this was a bad idea to start with anyway. The impact on a large amount of common utilities is simply that when they go IPv6-enabled, they also at the same time go round-robin-DNS disabled.

no decent fix

Since getaddrinfo() now has worked like this for almost a decade, we can forget about “fixing” it. Since gai.conf needs local edits to provide a different function response it is not an answer. But perhaps worse is, since getaddrinfo() is now made to return the addresses in a sort of order of preference it is hard to “glue on” a layer on top that simple shuffles the returned results. Such a shuffle would need to take IP versions and more into account. And it would become application-specific and thus would have to be applied to one program at a time. The popular browsers seem less affected by this getaddrinfo drawback. My guess is that because they’ve already worked on making asynchronous name resolves so that name resolving doesn’t lock up their processes, they have taken different approaches and thus have their own code for this. In curl’s case, it can be built with c-ares as a resolver backend even when supporting IPv6, and c-ares does not offer the sort feature of getaddrinfo and thus in these cases curl will work with round robin DNSes much more like it did when it used gethostbyname.

alternatives

The downside with all alternatives I’m aware of is that they aren’t just taking advantage of plain DNS. In order to duck for the problems I’ve mentioned, you can instead tweak your DNS server to respond differently to different users. That way you can either just randomly respond different addresses in a round robin fashion, or you can try to make it more clever by things such as PowerDNS’s geobackend feature. Of course we all know that A) geoip is crude and often wrong and B) your real-world geography does not match your network topology.

happy eyeballs

During this period, another connection related issue has surfaced. The fact that IPv6 connections are often handled as a second option in dual-stacked machines, and the fact is that IPv6 is mostly present in dual stacks these days. This sadly punishes early adopters of IPv6 (yes, they unfortunately IPv6 must still be considered early) since those services will then be slower than the older IPv4-only ones.

There seems to be a general consensus on what the way to overcome this problem is: the Happy Eyeballs approach. In short (and simplified) it recommends that we try both (or all) options at once, and the fastest to respond wins and gets to be used. This requires that we resolve A and AAAA names at once, and if we get responses to both, we connec() to both the IPv4 and IPv6 addresses and see which one is the fastest to connect.

This of course is not just a matter of replacing a function or two anymore. To implement this approach you need to do something completely new. Like for example just doing getaddrinfo() + looping over addresses and try connect() won’t at all work. You would basically either start two threads and do the IPv4-only route in one and do the IPv6 route in the other, or you would have to issue non-blocking resolver calls to do A and AAAA resolves in parallel in the same thread and when the first response arrives you fire off a non-blocking connect() …

My point being that introducing Happy Eyeballs in your good old socket app will require some rather major remodeling no matter what. Doing this will most likely also affect how your application handles with round robin DNS so now you have a chance to reconsider your choices and code!

Pádraig Brady taught me a great trick in a comment to a previous blog post and it was so neat I feel a need to highlight it further as it also makes it easier for me to find it again later!

To simulate a far away server, add RTT time to the localhost device. For example if we add 100 milliseconds (which then makes 200ms ping time to localhost):

tc qdisc add dev lo root handle 1:0 netem delay 100msec

Restore it back to normal again with:

tc qdisc del dev lo root

Android, the software environment, has gotten a lot of press, popularity and interest from all over lately. People on the streets know there’s something called Android, companies know people know and so on. Everyone (well apart from a few competitors perhaps) likes Android it seems.

Being an embedded guy I like keeping an eye on the embedded world and Android being pretty embedded this at least tangents my universe. What is Android anyway? android.com says Android is “an open-source software stack for mobile devices, and a corresponding open-source project led by Google“. Not very specific, is it?

Android on different devices

You can already find Android on mobile phones, media players, tablets, TVs and more. Very soon we’ll see it in car infotainment equipment, GPSes and all sorts of things that have displays. Clearly Android is not only for mobile phones and not even necessarily for mobile things. TVs often aren’t that mobile… And not touchscreen either.

Android with tweaks

The fact that there hardly are two Android installs completely alike is frequently debated. Lots of manufacturers patch and change the look and feel of Android to differentiate. Android is not associated with any particular look or feel quite clearly.

Android with binary drivers

Almost all Android installations you get on the Android phones and devices today have a fair amount of closed, proprietary drivers. It means that even if the companies provide the source code for all the free parts in time (which they sometimes have a hard time to do it seems), there are still parts that you don’t get to see the code for. So getting a complete Android installation from source to run on your newly purchased Android device can be a challenge. Also, it shows that Android can consist of an unspecified amount of extra proprietary pieces that don’t disqualify it from being Android.

Android without apps

I have friends who work on devices where the customer has request them to run Android, although they don’t have any ability to run 3rd party apps. Android is then only there for the original developers writing specific code for that device. Potential buyers of that device won’t get any particular Android benefits that they might be used to from their mobile phones running Android, as the device is completely closed in all practical aspects.

Android without market

Devices that don’t meet Google’s demands and don’t get to be “Google certified” don’t get to install the google market app etc, but at least a company who wants to can then in fact install their own market app or offer another way for customers to get new apps. The concept of getting and installing apps aren’t bound to the market app being there. In fact, I’ve always been expecting that some other companies or parties would come along and provide an alternative app that would offer apps even to non-google-branded devices but obviously nobody has yet stepped up to provide that in any significant way.

Android without Java

I listened  to a talk at an embedded conference recently where the person did a 40 minute talk on why we should use Android on our embedded systems. He argued that Android was (in this context) primarily good for companies because it avoids GPL and LGPL to a large extent. He talked about using “Android” in embedded devices and cutting out everything that is java, basically only leaving the Linux kernel and the BSD licensed bionic libc implementation. Of course, bionic may now also provide features to the rest of the system that glibc and uClibc do not, they being designed as more generic libcs.

Personally, I would never call anything shipped without the Java goo layers to be Android. But since it was suggested, I decided to play with the idea that a platform can be “Android” even without Java…

That particular license-avoiding argument of course was based on what I consider is a misunderstanding. While yes, lots of companies have problems with or are downright scared of the GPL and LGPL licences, but I’ve yet to meet a company who have any particular concerns about the licensing of the libc. I regularly meet and discuss with companies who have thoughts and worries about GPL in the kernel and they certainly often don’t like *GPL in regular libraries that they linked with in their applications. I have yet to find a customer who is worried about the glibc or uClibc licenses.

In fact, most embedded Linux customers also happily run busybox that is GPL although we know from history that many companies do that in violation with the license rules, only to get the lawyers running after them.

Android is what?

As far as I know Android is a trademark of some sorts, and so is Linux. If you can run an “Android” that is just a kernel and libc (and I’m not saying this is true beyond doubt because I’ve not heard anyone authoritative say this), isn’t that then basically a very very small difference to any normal vanilla embedded Linux?

The latter examples above are even without any kind of graphical UI or user-visible interface, meaning that particular form of “Android” can just as well run your microwave or your wifi router.

Without the cruft can we change the kernel?

The Android team decided that a bunch of changes to the Linux kernel are necessary.to make Android. The changes have been debated back and forth, some of them were merged into mainline Linux only to later get backed out again while the greater part of them never even got that far. You cannot run a full-fledged Android system on a vanilla kernel: you need the features the patches introduce.

If we’re not running all the java stuff do we still need those kernel patches? Is bionic made to assume one or more of them? That brings me to my next stepping stone along this path:

Without the patches can we change libc?

If we don’t run the java layers, do we really have to run the bionic libc? Surely the Android kernel allows another libc and if we use another libc we don’t need the Android kernel patches – unless we think they provide functionality and features that really improve our device.

Android is the new Linux?

Android as a name to describe something really already is just as drained as Linux. All these Android devices are just as much Linux devices and just as a “linux device” doesn’t really tell anything about what it is actually more than what kernel it runs, neither does it seem “Android device” will mean in the long run or perhaps already.

Android however has reached some brand recognition already among mortals. I think Android is perceived as something more positive in the minds of the consumer electronic consumers than Linux is. Linux is that OS that nobody uses on their desktops, Android is that cool phone thing.

I will not be the slightest surprised if we start to see more traditional Linux systems call themselves Android in the future. Some of them possibly without changing a single line of code. Linux one day, Android the next. Who can tell the difference anyway? Is there a difference?