Tag Archives: Development

10G and Direct Cache Access

As some of you might know, I currently work with a client doing 10G network stuff. 10G as in 10 gigabit/second Ethernet. That’s a lot of data. It’s actually so much data it’s hard to even generate network loads of this magnitude to be able to do good tests, as a typical server using SATA harddrives hardly fills a one gigabit pipe due to “slow” I/O: ordinary SATA drives don’t even reach 100MB/sec. You need RAID solutions or putting the entire thing in RAM first. So generating 10 gigabit network loads thus requires some extraordinary solutions.

Having a server that tries to “eat” a line speed 10G is a big challenge, and in fact we can’t do it as 1.25 GB/sec is just too much and yet we run a quad-core 3.00GHz Xeon thing here which is at least near the best “off-the-shelf” CPU/server you can get at the moment. Of course our software does a little bit more with the data than just receiving it as well.

Anyway, recently I’ve been experimenting with 10G cards from Myricom and when trying to maximize our performance with these beauties, I fell over the three-letter acronym DCA. Direct Cache Access. A terribly overused acronym consisting of often-used words make it hard to research and learn about! But here’s a great document describing some of the gory details:

Direct Cache Access for High Bandwidth Network I/O

Summary: it is an Intel technology for delivering data directly into the CPU’s cache, to reduce the bandwidth requirement to memory (note: it only decreases the bandwidth requirement at that moment, not the total requirement as it still needs to be read from memory into the cache, as noted in a comment below). Using this technique it should be possible to drastically reduce the time for getting the traffic. Support for this tech has been added to the Linux kernel as well since a while back.

It seems DCA is (only?) implemented in Intel’s 7300 chipset family which seems to only exist for Xeon 7300 and 7400. Too bad we don’t have one of these monsters so I haven’t been able to try this out for real yet…

Currently we can generate 10G network loads using two different approaches: one is uploading a specially crafted binary blob embedded with the FPGA image to a Xilinx-equipped board with a 10G MAC that then can do some fiddling with the packages (like increasing a counter) so that they aren’t all 100% identical. It makes a pretty good load test, even if the traffic isn’t at all shaped like the “real” traffic our product will receive. Our other approach has been even less good: upload a custom firmware to the network card and have that send the same Ethernet frame… This latter approach didn’t get better because it was a bit too complicated and badly documented on how to make a really good generator out of it. Even if I liked being able to upload custom code to my network card! 😉

Allow me to also mention that the problems with generating 10G is with small packet sizes, like 100 bytes or so as the main problem in the hardwares seem to the number of packets, not the payload part. Thus it is easier to do full line speed with 9000 bytes packets (jumbo frames) than the tiny ones we are likely to get when this product is in use by customers in the wild.

Update: this article was written in 2008. Please note that many things may have changed since then.

Open source in my day job

From people in the open source community and then especially friends and fellow hackers in projects I am involved in, I sometimes get questions on how my open source participations affect my “real job”.

I work as a consultant during the days and I do contract development for hire. I’ve been a consultant since 1996 and during this time I’ve participated in more than 25 “full-time” projects for almost as many customers.

I contribute to numerous open source projects and I’ve done it for many years. I lead and maintain several open source projects. I’ve committed many thousands of times to public source code repositories.

Does the contributions make me more attractive to potential customers? Not particularly is my rather sad experience. While some of my customers notice my track record (my CV does of course mention my most notable contributions) most of my day-job clients focus solely on other paid projects I’ve done and that exact technologies and products I worked with and created in the past. It may of course not be too strange as things I do and get paid for is then potentially “good enough for someone to pay me for” while the stuff I do for free in open source projects are… well, not paid for and thus it can’t be qualified by that ruler.

Do I get new assigments thanks to my open source projects? Yes, I do, but usually they tend to be on the smallish side and not of the bigger kinds my regular assignments at work are.

And the reality is of course also that the vast vast vaaast majority of all software consultants that people hire to do development have no public record of open source involvement so it could just be a result of that this is so rare the customers never had a reason to learn or adapt to using open source contributions as a “factor”.

Or maybe I’m just ignorant and haven’t figured out how my customers truly work.

Do I work with open source in my day job? Yes almost exclusively. I’ve been working with Linux in various embedded systems basically the last 8 years and working with Linux systems pretty much implies a wide range of open source development tools as well.

4 ohloh improvements I’d like

I am a stats junkie so I like my stats in large amounts. But I like the stats to be right and as accurate as possible, and when I look at what ohloh produces I like the concepts and ideas in general, I just think their implementation is lacking in a few vital areas that need improvement:

1. There are no dependencies or hierarchies between packages, so “I use this” counters get worthless since people mark end-user packages they use. Low-level support packages and libraries that are used indirectly don’t get many “use counts”

2. Doing very few commits in a very well used project with few authors gives you way way more points than doing a bus-load of commits in something less used with many fellow contributors. This makes the top-list of people very skewed as some of the top-64 people only did a few hundred commits ever. I doubt many mortals would consider someone who only ever did 300 commits to be a top community person. At the very moment I write this, the #1 ranked person has done 20 commits during 5 months…!

3. Too few versioning systems are supported, leaving out huge chunks of the open source world. Bazaar, mercurial and a few more are a bit too popular to be ignored without the results getting skewed.

4. I’d like to see the “number of users” of products as a percentage, as the total number of users they show include all contributors to all projects. Out of the 140,000 users (which undoubtedly include a lot of duplicates), it would surprise me if more than 10,000 have actually registered what products they use. I’ve tried to find the exact number but I failed. So 3,000 users don’t mean 3,000 out of 140,000 but 3,000 out of 10,000…

Cure coming for Wrap Rage?

This phenomena you thought you were alone to experience, the rage and anger you feel when you’ve bought some new toy and you get it packaged in tight and nearly un-enforceable plastic that demands a decent amount of violence and persistence to crack. It’s called Wrap Rage.

I’ve been told the packages (called blister packs or clam shells) are designed to be this way to be able to show off the merchandise while at the same time prevent thefts: it is hard for a customer to just extract something out of those things in your typical physical store.

Amazon’s initiative Frustration-Free packaging is indeed a refreshing take on this and apparently an attempt to reverse this development. Online stores really cannot have any good reasons to use this kind of armor around products since there’s no risk of stealing. I wish others will follow to make the manufacturers realize that there is a market for this. This needs to be done by manufacturers of stuff, the stores cannot be made to repackage stuff due to warranties and what not.

It wouldn’t surprise me if you could even find cheaper ways to package products once you let go of some of the requirements that no longer apply for online stores. Visibility of the products once packaged is another thing that is pointless for online stores but I would expect is very important to sales in physical stores. I’ve always thought it is pretty pointless and expensive that every single package is made to be able to be a display model. To be able to attract customers to buy it. When you buy the thing online it’s no longer just pointless, it’s plain stupid.

Imagine a future when you can just open your new toy without getting bruises or scratch marks!

My million users

I’ve been working professionally with computers since 1991 and explicitly as a developer since 1993. I’ve written one or two lines of code since then. How many users could there be out there that are using something that includes my code?

Open source

I’ve participated in a wide range of open source projects, so of course all direct users of those projects would count: curl, Rockbox and let’s include subversion and others. I would guess that there are at least one million users of curl, quite likely more than so of subversion and Rockbox may also reach a million users or so. It’s of course impossible to know for sure…

Lots of open source projects use libraries that I work on now and have worked with in the past. Primarily libcurl and c-ares. Such as Boinc, git, bazaar, darcs. Millions of users, no doubt (Boinc alone has some 1.5 million users). The OLPC’s XO laptop comes with (lib)curl. I think most Linux distros these days come with curl installed. How many linux installations are there? libcurl is rather popular when used within PHP as well and there are many many million installations of PHP out there. I have code in wget, also used by millions.

Closed source users of open source I’ve participated in

Adobe acrobat reader (for non-windows platforms), Adobe’s flash player and various other Adobe products, Second life, Google Earth and others. They’re bound to have several million users. curl is included in Mac OS X.

There are also a lot of devices that use libcurl that are even harder to track: Sandisk makes mp3 players that use libcurl, Sony makes a video device that uses libcurl, Tilgin, Neuros and others make IPTV-devices that use libcurl. libcurl is used for multiple “installers” such as the one AOL provide for a specific router. There are many company users.

Closed source stuff I’ve worked with on my day-job

… is of course also used widely and all over, but me being an embedded guys I mostly work on software in products and most of the products I’ve worked within have been for various niche markets in which I have little or no knowledge about how much the products (and thus my code) are actually used. I’ve left my fingerprints on several networking products, IPTV/Digital TV settop boxes, railroad equipments, a car ignition tester, 3g/telecom switches, rfid receivers, laser-using positioning systems and more.

How many millions?

Ok, let’s for the sake of the argument say that there’s somewhere around 100 million devices with my code from me included – I really have no idea how to make a sensible estimate. Let’s for simplicity also say that there are 100 million users of these devices. I would also guess that about half of the world’s population isn’t near using devices I may have programmed. Thus, if you’re using “devices” in general there’s a probability of 3 billion/100 million = 1/30 that you’re using something that includes code that I’ve worked on…

In fact, that number is then valid for any random “device” user – if you’re reading this on my blog I don’t expect you to be very random but rather a specialized person and then I would say the likeliness of you having at least something with my code in it is almost 100% guaranteed…

Where would you say my biggest weaknesses in this reasoning are?

strcasecmp in Turkish

A friendly user submitted the (lib)curl bug report #2154627 which identified a problem with our URL parser. It doesn’t treat “file://” as a known protocol if the locale in use is Turkish.

This was the beginning of a minor world-moving revelation for me. Of course this is already known to mankind and I’m just behind, but really: lots of my fellow hacker friends had no idea either.

So “file” and “FILE” are not the same word case insensitively in Turkish because ‘i’ is not the lowercase version of ‘I’.

Back to strcasecmp: POSIX pretty much makes the function useless by saying that “The results are unspecified in other locales [than POSIX]”.

I’m a bit annoyed by this fact, as now I have to introduce my own function (which thus cannot use tolower() or toupper() since they also are affected by the locale) and use since the strings in our code is clearly used for “English” strings so file and FILE truly are the same string when compared case insensitively…

popen() in pthreaded program confuses gdb

I just thought I’d share a lesson I learned today:

I’ve been struggling for a long time at work with a gdb problem. When I set a break-point and then single-step from that point, it sometimes (often) decides to act as if I had done ‘continue‘ and not ‘next‘. It is highly annoying and makes debugging nasty problems really awkward.

Today I searched around for the topic and after some experiments I can now confirm: if I remove all uses of popen() I no longer get the problems! I found posts that indicated that forking could confuse threaded programs, and since this program at work uses threads I could immediately identify that it uses both popen() and system() and both of them use fork() internally. (And yes, I believe my removal of popen() also removed the system() calls.)

And now I can finally debug my crappy code again to become less crappy!

My work PC runs glibc 2.6.1, gcc 4.1.3 and gdb 6.6. But I doubt the specific versions matter much.

Swedish Top Developers?

I find it hilarious that IDG.se out of all publications put together the “best developers in Sweden” and lists the top-75 ones (article in Swedish). It is funny because IDG is not exactly a place flooding over with technical (or any kind of in-depth) knowledge, so obviously they got this list by getting input from others and how on earth can they then compare person A against person B when they’ve been mentioned by different sources? Also, just lumping every kind of “developer” into the same pile and then trying to order them is also an interesting challenge. Clearly some of these devs are more project managers, theorists and similar, while others are hardcore kernel-hackers, C coders or Java dudes.

I don’t mean to bash the people present on this list, as I’m sure I would also liked being present if I had been that. I just think the list fits so well into IDG’s style of populistic journalism. The audience wants top-lists, let’s give them another one!

Or perhaps I’m just jealous that I’m not included! 😉

Conference in my living room

a kids drawing easel thing by IKEAToday (uh make that yesterday since we’re now past midnight here…) around lunch I drove my two kids over to my parents in law and got back to my house to host four friends (associated with a company that shall remain nameless in this blog – at least for now) coming over to discuss some work stuff.

It was great fun sitting in my living room chatting for a few hours, having a cup of coffee and instead of using a fancy company white board I brought my kids’ drawing easel (oh we love IKEA). The picture is the actual model we used, called “MÅLA”.

And we did indeed manage to get some good decisions done and some proper architectural stuff set. Admittedly, my kids’ drawing pens were a bit thin and not as thick and “powerful” as the ordinary office white board pends tend to be.

The Mythical Man-Month

The Mythical Man-MonthFrederick Brooks wrote this classical book already back in 1975 and added a few extra chapters for the twenty years anniversary 1995…

Large portions of it feels of the age and there’s a lot of talk about Fortran, System/360 and PL-1 as if we should know about them (which made me fast forward over some chapters). But there are gems as well, and the most significant things people seem to remember Brooks’ book for are still pretty valid and fine.

Adding more people to a project leads to the need for more communication and thus it may slow down development rather than speed it up. Also known as Brooks’s law.

Given the complexity of software and software development, there’s no single method or concept that will lead to an improvement by an order of magnitude – within a decade. There’s No Silver Bullet. (This section was not in the original edition of the book.)

The risks involved when rewriting something and wants to fix everything that was wrong in the previous version so you over-work and over-design the successor. The so called Second system effect.

A lot of the book is spent on thoughts and theories around how to manage really really large software projects, like when you involve thousands of persons. Is it even possible to make such huge projects successful and if so, what does it take? The extra chapters do indeed add value since they offered Brooks a chance to re-evaluate his earlier claims and ideas and to check what seemed to be truths and what mistakes he did in the original edition.

A very interesting read that I’m glad I finally got time to get through!