Parallel Spaghetti – decoded

Thursday, November 28th, 2013

Here’s the decoding procedure for the Parallel Spaghetti Decode challenge.

Step 1, the answers to all the questions. You will notice that I did have some fun in D6 and E2, but since they were boxes that weren’t on the right track anyway I thought you’d still enjoy them.

Step 2, let me illustrate how the above answers will take you through the maze. The correct path is made up out of yellow boxes and the correct answers are shown with red arrows leading forward. Click it for full resolution version.

The parallel spaghetti challenge correct track shown

Step 3, those different colors in the “Word” column give you the words used for the two questions. If you rearrange them, the two questions become:

which tr command line option specifies delete characters

and

what curl command line option specifies POST requests

So, it took about 14 minutes at our event for Oscar Andersson to bring the correct answer to me:

-d

source code survival rate

Tuesday, November 5th, 2013

The curl project has its roots in the late 1996, but we haven’t kept track of all of the early code history. We imported our code to Sourceforge late 1999 and that’s how far back we can see in our current git repository. The exact date is “Wed Dec 29 14:20:26 UTC 1999″. So, almost 14 years of development.

Warning: this blog post contains more useless info and graphs than many mortals can handle. Be aware!

How much old code remain in the current source tree? Or perhaps put differently: how is the refresh rate of the code? We fix bugs, we change things, we add features. Surely we’ll slowly over time rewrite the old code and replace it with new more shiny and better working code? I decided to check this. Here’s what I found!

The tools

We have all code in git. ‘git blame’ is the primary tool I used as it lists all lines of all source code and tells us when it was added. I did some additional perl scripting around it.

The code

I decided to check all code in the src/ and lib/ directories in the curl and libcurl source tree. The source code is used to create both the curl tool and the libcurl library and back in 1999 there was no libcurl like today so we do get a slightly better coverage of history this way.

In total this sums up to some 112000 lines in the current .c and .h files.

To count the total amount of commits done to those specific files through history I ran:

git log --oneline src/*.[ch] lib/*.[ch] | wc -l

6047 commits in total. (if I don’t specify the files and count all commits in the repo it ends up at 16954)

git stats

We run gitstats on the curl repo every day so you can go there for some more and current stats. Right now it tells us that average number of commits is 4.7 per active day (that means days when actually something was committed), or 3.4 per all days over the entire time. There was git activity 3576 days in total. By 224 authors.

Surviving commits

How much of the code would you think still remains that were present already that December day 1999?

How much of the code in the current code base would you think was written the last few years?

Commit vs Author vs Date

I wanted to see how much old code that exists, or perhaps how the age of the code is represented in the current code base. I decided to therefore base my logic on the author time that git tracks. It is basically the time when the author of a change commits it to his/her local tree as then the change can be applied later on by a committer that can be someone else, but the author time remains the same. Sometimes a committer commits multiple patches at once, possibly at a much later time etc so I figured the author time would be a better time stamp. I also decided to track the date instead of just the commit hash so that I can sort the changes properly and also make interesting graphs that are based on that time. I use the time with a second precision so changes done a second apart will be recorded as two separate changes while two commits done with the same author time stamp will be counted as the same time.

I had my script run ‘git blame –line-porcelain’ for all files and had my script sum up all changes done on the same time.

Some totals

The code base contains changes written at 4147 different times. Converted to UTC times, they happened on 2076 unique days. On 167 unique months. That’s every month since the beginning.

We’re talking about 312 files.

Number of lines changed over time

A graph with changes over time. The Y axis is number of lines that were changed on that particular time. (click for higher res)

Lines changed over time

Ok you object, that doesn’t look very appealing. So here’s the same data but with all the changes accumulated over time.

accumulated

Do you think the same as I do? Isn’t it strangely linear? It seems that the number of added lines that remain in the code today is virtually the same over time! But fair enough, the changes in the X axis are not distributed according to the time/date they represent so we shouldn’t be fooled by the time, but certainly we can see that changes in general only bring in a certain amount of surviving modified lines.

Another way to count the changes is then to check all the ~4000 change times of the present code, and see how many days between them there are:

delta

Ah, now finally we’re seeing something. Older code that is still present clearly was made with longer periods in between the changes that have lasted. It makes perfect sense to me, since the many years of development probably have later overwritten a lot of code that was written in between.

Also, it is clearly that among the more recent changes that have survived they were often done on the same day or just a few days away from another lasting change.

Grouped on date ranges

The number of modified lines split up on the individual year the change came in.

year

Interesting! The general trend is clear and not surprising. Two years stand out from the trend, 2004 and 2011. I have not yet investigated what particular larger changes that were made those years that have survived. The bump for 1999 is simply the original import and most of those lines are preprocessor lines like #ifdef and #include or just opening and closing braces { and }.

Splitting up the number of surviving lines on the specific year+month they were added:

month

This helps us analyze the previous chart. As we can see, the rather tall bars from 2004 and 2011 are actually several months wide and explains the bumps in the year-chart. Clearly we made some larger effort on those periods that were good enough to still remain in the code.

Correlate to added or removed lines?

So, can we perhaps see if some years’ more activity in number of added or removed source lines can be tracked back to explain the number of surviving source code lines? I ran “git diff [hash1]..[hash2] –stat — lib/*.[ch] src/*.[ch]” for all years to get a summary of number of added and removed source code lines that year. I added those number to the table with surviving lines and then I made another graph:

year-again

Funnily enough, we see almost an exact correlation there for the first eight years and then the pattern breaks. From the year 2009 the number of removed lines went down but still the amount of surving lines went up quite a bit and then the graphs jump around a bit.

My interpretation of this graph is this boring: the amount of surviving code in absolute numbers is clearly correlating to the amount of added code. And that we removed more code yearly in the 2000-2003 period than what has survived.

But notice how the blue line is closing the gap to the orange/red one over time, which means that percentage wise there’s more surviving code in more recent code! How much?

Here’s the amount of surviving lines/added lines and a second graph looking at surviving lines/(added + removed) to see if the mere source code activity would be a more suitable factor to compare against…

relation survival vs added and removed lines

Code committed within the last 5 years are basically 75% left but then it goes downhill down to the 18% survival rate of the 1999 code import.

If you can think of other good info to dig out, let me know!

1999,1699
2000,1115
2001,3061
2002,2432
2003,2578
2004,7644
2005,4016
2006,5101
2007,7665
2008,7292
2009,9460
2010,11762
2011,19642
2012,11842
2013,16844

Say hello to Moo

Tuesday, November 20th, 2012

I decided it was about time to upgrade my main development machine to something modern and snappy. It is 5.5 years ago since I bought my current work horse, a dual-core AMD Athlon 64 X2 5600+ (2.8GHz) equipped thing.Fractal Design I’m using my machine primarily for development. I never game. I decided to go for the higher end of what’s available to get me something to live with for several years to come.

Motherboard: Asus P8Z77-M. Micro-ATX. Intel Z77 chipset.

CPU: Intel Core i7 3770K 3,5Ghz Socket 1155. This is a 22nm monster featuring 8 MB L3-cache

Memory: TridentX DDR3 PC19200/2400MHz CL10 2×8GB. 16GB of ram.

HDDSeagate Barracuda ST3000DM001 64MB 3TB.

Chassi: Fractal Design Define R3 USB3. See picture. Rather big and fits a lot more drives and stuff than what I have now…

SSD: OCZ Vertex 4 256GB

CPU cooler: Cooler Master Hyper 412S

Graphics: ASUS Radeon HD5450 512MB (very simple and cheap thing but supports 2560×1600 which the MB doesn’t do)

PSU: Plexgear PS-500 500W

(a prisjakt list with the full setup)

All in all, this has two 120mm chassi fans, one 135mm fan on the big CPU cooler and there’s one fan in the PSU. I hope they won’t be causing too much noise or problems for me. The rather low-end graphics should keep the total power consumption (and thus heat production) at a decent level. ASUS p8z77-m

I purchased all the individual parts separately as I dislike how I can’t get an as optimized machine prebuilt from anywhere  - I basically have to pay around 50% more, and then I still wouldn’t get the exact set of pieces I’d like. This way I also avoid the highly disturbing Microsoft tax prebuilt systems come with.

Unfortunately I got some bad luck included too, as when I first put everything together and pressed the power button nothing happened. Well, a single led was turned on but nothing else happened. It took me a while and some sweat to figure out where the problem lied and once I replaced the broken motherboard it would start properly and then I could proceed and install it.gskill TridentX ddr3

Once my new machine (which now goes under the name Moo) gets settled, my old box will become my daughter’s new machine as hers existing tired old PIII machine isn’t really fun to do a lot with.

Three static code analyzers compared

Thursday, July 12th, 2012

I’m a fan of static code analyzing. With the use of fancy scanner tools we can get detailed reports about source code mishaps and quite decently pinpoint what source code that is suspicious and may contain bugs. In the old days we used different lint versions but they were all annoying and very often just puked out far too many warnings and errors to be really useful.

Out of coincidence I ended up getting analyses done (by helpful volunteers) on the curl 7.26.0 source base with three different tools. An excellent opportunity for me to compare them all and to share the outcome and my insights of this with you, my friends. Perhaps I should add that the analyzed code base is 100% pure C89 compatible C code.

Some general observations

First out, each of the three tools detected several issues the other two didn’t spot. I would say this indicates that these tools still have a lot to improve and also that it actually is worth it to run multiple tools against the same source code for extra precaution.

Secondly, the libcurl source code has some known peculiarities that admittedly is hard for static analyzers to figure out and not alert with false positives. For example we have several macros that look like functions and on several platforms and build combinations they evaluate as nothing, which causes dead code to be generated. Another example is that we have several cases of vararg-style functions and these functions are documented to work in ways that the analyzers don’t always figure out (both clang-analyzer and Coverity show problems with these).

Thirdly, the same lesson we knew from the lint days is still true. Tools that generate too many false positives are really hard to work with since going through hundreds of issues that after analyses turn out to be nothing makes your eyes sore and your head hurt.

Fortify

The first report I got was done with Fortify. I had heard about this commercial tool before but I had never seen any results from a run but now I did. The report I got was a PDF containing 629 pages listing 1924 possible issues among the 130,000 lines of code in the project.

fortify-curl

Fortify claimed 843 possible buffer overflows. I quickly got bored trying to find even one that could lead to a problem. It turns out Fortify has a very short attention span and warns very easily on lots of places where a very quick glance by a human tells us there’s nothing to be worried about. Having hundreds and hundreds of these is really tedious and hard to work with.

If we’re kind we call them all false positives. But sometimes it is more than so, some of the alerts are plain bugs like when it warns on a buffer overflow on this line, warning that it may write beyond the buffer. All variables are ‘int’ and as we know sscanf() writes an integer to the passed in variable for each %d instance.

sscanf(ptr, "%d.%d.%d.%d", &int1, &int2, &int3, &int4);

I ended up finding and correcting two flaws detected with Fortify, both were cases where memory allocation failures weren’t handled properly.

LLVM, clang-analyzer

Given the exact same code base, clang-analyzer reported 62 potential issues. clang is an awesome and free tool. It really stands out in the way it clearly and very descriptive explains exactly how the code is executed and which code paths that are selected when it reaches the passage is thinks might be problematic.

clang-analyzer report - click for larger version

The reports from clang-analyzer are in HTML and there’s a single file for each issue and it generates a nice looking source code with embedded comments about which flow that was followed all the way down to the problem. A little snippet from a genuine issue in the curl code is shown in the screenshot I include above.

Coverity

Given the exact same code base, Coverity detected and reported 118 issues. In this case I got the report from a friend as a text file, which I’m sure is just one output version. Similar to Fortify, this is a proprietary tool.

coverity curl report - click for larger version

As you can see in the example screenshot, it does provide a rather fancy and descriptive analysis of the exact the code flow that leads to the problem it suggests exist in the code. The function referenced in this shot is a very large function with a state-machine featuring many states.

Out of the 118 issues, many of them were actually the same error but with different code paths leading to them. The report made me fix at least 4 accurate problems but they will probably silence over 20 warnings.

Coverity runs scans on open source code regularly, as I’ve mentioned before, including curl so I’ve appreciated their tool before as well.

Conclusion

From this test of a single source base, I rank them in this order:

  1. Coverity – very accurate reports and few false positives
  2. clang-analyzer – awesome reports, missed slightly too many issues and reported slightly too many false positives
  3. Fortify – the good parts drown in all those hundreds of false positives

Join the SPDY library development

Friday, January 6th, 2012

Back in October I posted about my intentions to work on getting curl support for SPDY to be based on libspdy. I also got in touch with Thomas, the primary author of libspdy and owner of libspdy.org.

Unfortunately, he was ill already then and he was ill when I communicated with him what I wanted to see happen and I also posted a patch etc to him. He mentioned to me (in a private email) a lot of work they’ve done on the code in a private branch and he invited me to get access to that code to speed up development and allow me to use their code.

I never got any response on my eager “yes, please let me in!” mail and I’ve since mailed him twice over the period of the latest months and as there have been no responses I’ve decided to slowly ramp up my activities on my side while hoping he will soon get back.

I’ve started today by setting up the spdy-library mailing list. I hope to attract fellow interested hackers to join me on this. The goal is quite simply to make a libspdy that works for us. It is to be C89 code that is portable with an API that “makes sense”. I don’t know yet if we will work on libspdy as it currently looks, if Thomas’ team will push their updated work soon or if going with my current spindly fork off github is the way. I hope to get help to decide this!

Join the effort by simply adding yourself the mailing list and participate in the discussions: http://cool.haxx.se/cgi-bin/mailman/listinfo/spdy-library.

And a wiki on github.

Update: I’ve created a hub collecting all related info and pointers over at spindly.haxx.se.

Welcome!

curl meetup at Fosdem 2012

Thursday, September 15th, 2011

The FOSDEM 2012 dates were recently revealed (4-5 February 2012).

A pint of guinness

I’d be happy to arrange a get-together for libcurl hackers at Fosdem this year. To me, Brussels, Belgium seems mid-europe enough to be able to attract a bunch of us:

  • libcurl application users/authors
  • libcurl binding hackers
  • libcurl contributors
  • … and everyone else who’s doing related activities or who just is interested

Potential subjects to discuss at such a meeting:

  • what’s the most important stuff libcurl still lacks?
  • what’s the least documented/understood parts of libcurl?
  • are there shared problems several/many libcurl bindings have to solve?
  • can we improve how we work/develop libcurl and bindings?
  • what kind of beer is best at a curl meetup?
  • [fill in your own curl related subject]

I would like at least 4-5 people voicing interest for this to be worthwhile for me to actually try to do anything. Please speak up on the libcurl mailing list, tweet me or mail me privately! The more people that are interested, the more planning and stuff we’ll do for it.

Haxx – the first year

Friday, October 1st, 2010

Last year I left my former employment, and focused on Haxx full-time. My brother joined me a few months afterward (January 2010). Today, at October 1 2010 we celebrate the official one year anniversary of Haxx AB as employer.Haxx

The history of Haxx goes far longer back than so. Linus Nielsen Feltzing and I first registered the company Haxx back in October 1997 and we used it then primarily as a way to market and do business on the side of our “real” jobs. To have a way to charge and do things we wanted to, that wasn’t conflicting with our day jobs. And of course we also bought the domain and could setup our “permanent” email addresses etc, which turned out great since I’ve thus used the same email address since back then and I hope I never need to change it again!

The first year of Haxx has been nothing but great fun and a major success.

As we’re contract developers and consultants, we of course need to make sure that our employees are sold to customers to a high degree with as little gaps as possible. Our projects are typically going on from a few months up to a year or two. During this year, both me and Björn have worked with several end customers and we’ve thus both managed to change assignments several times and none of the times caused any gaps – at all. Our services seem to be in high demand.

Being only two employees brings challenges on how to deal with sales, financial accounting etc as we’re just a few guys and we’re experts on development! We have found a few great partners that “sell” us (and of course we pay them a certain amount of percentage, but that’s a price we need to accept and is nothing but fair anyway since we can then remain doing what we’re good at and what we love) and we’re buying the bookkeeping etc from another company that is specialized at doing it for companies like us.

We’re looking forward to many more years of great fun. We also hope to be able to grow the company slightly over time, so if you’re a kick-ass embedded open source guy with networking experience and some 10+ years in the business and you live in the Stockholm Sweden area, do get in touch! As I’ve mentioned before, we’re gonna start out our second year with Linus onboard.

I’ll get back with an update next year! :-)

poll vs select

Saturday, July 17th, 2010

I’m a person working a lot with networking and development around it. I mostly do this on Linux, often involving drivers or otherwise very close to the operating system and C and the core libraries.

The other day I once again fell over some random inaccuracy about poll compared to select and instead of trying to whine on some IRC channel or complain on their mailing list, I decided I would instead strike back by writing up and presenting a web page of my own. It details as much as possible about poll vs select and related event-based functions. I want it to become a placeholder for everything that is relevant to say about poll and select in a comparison aspect and when comparing them to event-based alternatives like libevent and libev.

So the next time I face someone not quite understanding this whole situation or perhaps when someone reiterates something that isn’t quite true, I have a resource to point to.

Not to mention that I think this new poll vs select page fits in nicely with my other “X vs Y” articles and docs pages I’ve written the last few years.

If you find flaws, or miss details or have questions about this page. Please do not hesitate to comment here, or to mail me about it or tweet me on twitter or whatever method you prefer. I appreciate your feedback!

poll vs select

professional libcurl hackers look this way!

Tuesday, April 20th, 2010

In my company, Haxx, we work as consultants and we do contract development for customers who pay for our skill, time and dedication. We help them develop stuff.Haxx

We’re a small company, with basically two full-time employees. Most of our working days, we are involved with a single customer each who pays for our full-time involvement during a number of months. This is all good and fine. We love our jobs and we love our customers. We’re in it for the fun.

Now, these days we can see that the economy is slowly but surely gaining ground again and is getting up to speed. We hear more and more requests for help and potential assignments are starting to pour in. That’s great and all. Except that we’re only two guys and can’t accept very many projects…

Recently we’ve experienced a noticeable increase in amount of requests for support and other development help that involves curl and libcurl. I am the originator and maintainer of curl, there’s really no surprise or wonder that these companies contact me and us about it. I’m always very happy to see that there are companies and persons who are willing to pay for support of open source and in many cases pay for extending and bug fixing libcurl and have those fixes going back to the mainline sources without complaints.

Since we fail to accept a lot of requests, I’m interested in finding you who are interested in helping out with such work. Are you interested in helping out customers with curl related problems? Customers often come to us when they’ve got stuck within something they can’t easily solve themselves and they turn to us as experts in general, and experts on curl and libcurl in particular. And we are.

Before you think this is a great idea and you send me an email introducing yourself and your greatness in this area, please be aware that I will require proof of your qualifications. Most preferably, that proof is at least one good patch posted to the libcurl mailing list and accepted into the mainline libcurl code, but I’m open to accepting slightly less ideal proofs as well if you can just motivate why you failed to provide the ideal ones. Of course you will also need to be able to communicate in English without problems. Your geographical location, gender, race, religion, skin-color and shoe size are completely uninteresting.

I’m looking for someone interested in contract development, not full-time employment. We still do these kinds of jobs on a case by case basis and there may be one every two days, one per week or sometimes even less frequently. I want to increase my network of people I know and trust can deliver quality code and services for this kind of projects.

Can you help us?

Autotool alternatives

Wednesday, September 23rd, 2009

Lots of people whine and complain on the set of build tools we often refer to as a collective by the term ‘autotools’. That term tends to include autoconf, libtool and automake.

I think a certain amount of criticism is warranted against this family of aged tools that are unix-centric, have cryptic ways to control them (I think there’s a reason m4 macros  is not widely used…) and they are several independent tools with a tricky mix of cross-breeding.A build tool

The upsides include them being well tested, fairly well known, there’s a wide range of existing tests done for them, they work fine when cross-compiling and they support building out-of-source tree just fine.

But what about the alternatives?

I spend time in projects where the discussion of ditching autoconf come up every once in a while, as sure as that the sun will rise tomorrow. The discussion is always that tool Z is much better and easier to deal with and that everything gets shiny if we just switch. That Z is a lot of different tools that are available today, including CMake, scons, waf or cDetect.

The problem as I always see and why I almost always argue against Z is that autoconf is old, trusty, proven and I know it. The Z tool is often much newer, less proven, less peoeple involved in the project know Z, use Z or know how to customize it (since new tests will be needed and some tests will need to be changed etc). So even though Z is sometimes accepted as a testing ground in my projects, a year or two after the Z was accepted – unless I myself have accepted it and joined its efforts – Z has lagged behind to a point where it isn’t good anymore since I don’t know it and most people are rather fixing the traditional autoconf stuff. So we extract the Z support again.

But if we would never accept new tools we would never evolve, and yes indeed autoconf and friends have their share of flaws.

The question is of course when to switch – what kind of project in what development state etc – and which alternative that is useful for a particular project. Me being a developer primarily working with plain C and working with lowlevel code and libraries mostly will no doubt have a different view than those who use other languages, who do more “apps” or perhaps even GUI programming…

Can you help me point out good build system comparisions and overiews? I’ve tried to find good comparisions but I failed. Just about all of them are written by the authors of one of these tools.

My ambition is to create some sort of comparison document myself. I think the comparison could include autotools, cmake, waf, scons, cdetect, qmake and ant. Any more?

(I got triggered to write this blog post after my post to the trio mailing list on this topic.)