cURL and libcurl

Top-3 curl changes in 2011

1 Comment

At the end of the year I thought it would be interesting to have a look back over the past twelve months and see what the biggest changes in the curl project were and what the most important bug-fixes were etc. I’ve turned into a little mini series of blog posts. The top-3s. First out, the top-3 changes.

Top-3 changes

In total I counted 29 notable changes brought during 2011. The most significant ones in my view are:

Change 1 – fancier protocol support in proxy strings

This may sound trivial, and the code certainly is, but with this change suddenly a lot of applications that use libcurl got better proxy support without having do anything at all. Previously the application would have to set what protocol type the proxy it would use is, even though libcurl has supported having the proxy specified as an environment variable for ages.

Having only the proxy name was useful but limiting. With this new change you can specify proxy type or rather proxy protocol by prefixing the proxy name like “socks4://magic-proxy.example.com” if you want a SOCKS4 proxy. libcurl now supports socks4, socks4a, socks5 or socks5h used as prefixes as well as http.

Change 2 – allow sending “empty” HTTP headers

Another minor change in code but possibly larger impact in usefulness for applications. We introduced a way for applications to change internal headers and to add new ones ages ago. We (or rather I) then made the choice that if you’d provide a header with only the name and a colon, that is with no contents on the right side, it would delete the internal header. That was not a clever move as later on people have wanted to add “blank” or “empty” headers that look exactly like that, but libcurl has then refused to.

There have been some more or less hackish work-arounds to trick libcurl into allowing an empty header, but now finally we introduce a nice and clean way for applications to pass in these kinds of empty headers:

Pass in an empty header that instead of a colon has a semicolon! This is an otherwise illegal header that wouldn’t make sense, but libcurl will use that as a trigger that an empty header should be used and it will then replace the semicolon with a colon and things will be fine.

Change 3 – Added support for cyassl and axTLS

Proving that libcurl moves forward and into more and more markets, the number of supported SSL libraries grew to 7 this year. The all new cyassl backend that replaced the previously done yassl backend that was using cyassl’s former OpenSSL emulation layer. Now we’re using the native and pure API and things are much cleaner. The possibly smallest available TLS library axTLS also got support.

Not all backends and not all SSL libraries are the same or support the same set of features, but then libcurl is used in many different scenarios and use cases and this way we offer more options to more users to craft libcurl for their particular needs. Our internal SSL backend API has managed quite well and proves to have been a worthy change. Adding support for yet another SSL library within libcurl is actually not a lot of work.

Change 4 – TSL-SRP

You may think number 4 of a top-3 list is weird, but I couldn’t cut it off here! =) TLS-SRP has been waiting in the shadows for so long and all of a sudden two of the major SSL libraries have support for it in released versions and libcurl got support for using these features in both libraries during 2011.

cURL

cURL and libcurl

curlers rest on Sundays and during July

We now run gitstats on the curl git repository daily and provides fun graphs.

We have almost 11 years of source code history covered and I personally have done some ~68% of all commits. Given this long history it is fun to see some very clear trends. Like this first one: look at the distribution of commits per weekday over the entire period. The amount of commits done during weekends are significantly lower than during the work week, and the Sunday amount is clearly even lower than Saturday:

day_of_week

Similarly, we can see how the activity is spread out over calendar months. This shows an obvious correlation to the slower periods in my life, which means that July is vacation times and the numbers show it:

month_of_year

Electronics, Network, Open Source

My five ADSL modems

1 Comment

bredbandsbolagetI previously blogged when my network hardware died. Here’s the recap and continuation of that story and how things evolved…

One day my ADSL modem could no longer get sync, I couldn’t send data and my (landline) phone was dead. My phone is connected into the ADSL modem through which it does IP telephony. Other times this has happened I could just switch off the modem for 10 seconds and then back on again it would work again for another 6 months or a year or so.

I’ve had ADSL at roughly 12mbit working flawlessly for several years so this was an unexpected breakage.

On 14 sep 16:16 I called my operator’s (Bredbandsbolaget) support about the issue when the modem hadn’t been able to get contact for a whole day – I was suspecting some kind of glitch in the service from the other end. The support person said that I had a “very old modem” and they immediately decided to send me a new modem by mail that would fix my problems.

xavi technologies x5258-p2At 16 sep 18:51 I called support again. I received modem #2 and installed it this day. The modem, Xavi Technologies X5258-P2, is a much more fancy model than what I had been using for the last couple of years – the new one had 4 Ethernet ports and wifi. Not that I really care about that cruft as I want to use my own wifi router anyway to get control of things better.

When I plugged in modem #2 I noticed that it lit up the ‘phone’ LED at once (which normally would only be on if I use the phone) and while internet data seemed to work, the phone did not. When I called support again to ask about this, they decided it was a broken modem they had sent me and would send me a replacement at once.

A few days later I got modem #3 and installed it. I also got the joy of sending back two ADSL modems.

3 oct 20:25 – I called the support again. Modem #3 hung occasionally and I wanted to get their help to fix the problem. The support guy I talked to claimed his sometimes happens if a wifi router is too close to the modem and adviced me to put my ADSL modem and wifi router further apart. It sounded like a suspicious analysis and theory to me, as why would the modem completely hang from this and if it did, why would it keep on running for days at times after a reboot? The support person also revealed that he had detailed logs going back a few weeks at least where he could see my ADSL modem power recycles and he could also see “bad CRC” counters going up before my restarts. I moved my devices two meters apart.

A little side-story: the modem has wifi support, but as I run my own wifi router behind it I don’t want the modem’s wifi. I noticed it ran on a different channel than my regular one so it wasn’t an immediate concern. It did however turn out that in order to switch it off I had to configure that with a Windows program and in order to install that program I had to enter a username and password that I didn’t have. Asking support for the credentials, they instead offered to simply disable the wifi from their end instead. That was fine by me, but again showed what fancy controls they have over these things.

For a week or so my connection actually was better and I actually thought my suspicions about the fishy advice were wrong. But no. It turned out I was only lucky for a few days as then it started hanging again every few days. It would stop transfering data in/out, and the “phone” led would blink slowly. How on earth could a device like this hang in any circumstance? I’ve been an embedded developer all my professional life, I know hanging is the worst possible thing. I much better but still ugly way to resolve a problem without any obvious way out, would be to reboot. A reboot would’ve been annoying as well, but far from as annoying as this.

Now, after all, I have a fiber installation coming “soon” so I figured I could possibly just shut up and endure this ADSL mess and it will go away or at least change drastically once I get my new connection…

But eventually it got too tedious, also partly because my kids and my wife also found it annoying and troubling – I had to give up the eduring. The fiber installtion also seemed to be delayed. Who knows how long I was supposed to remain on ADSL.

So, on 5 dec 18:38 I was back on the phone with the support people and complained about the hangs I frequently get with modem #3. The guy listened to me explaining the issue, he checked the reboot logs from his side and swiftly decided he would send me a new modem. He decided to send a modem of a different brand this time to see if this made things work better in my end.

zyxel-p-2601hn

On dec 8th I got modem #4. A different model this time compared to #2 and #3. It was now a Zyxel P-2601. I got home from work at 18:15, had a quick dinner and then I connected the new equipment. Would this really be the end of my troubles? Anticipation!

– Oh harsh reality, how thee can be rough and cold.

This modem can’t be powered on. If I flip the power switch and turns it on, all the leds switch on but as soon as my finger leaves the power-on toggle again the modem turns itself off… At 18:52 I tried to call support, but a voice claimed they had “internal systems problems” so I gave up.

12:45 on Friday Dec 9th I called again and reported my broken modem and the friendly support woman was a bit surprised I had gotten a broken device as she said “straight from the factory”. She even expressed some sympathy about the replacement unit, modem #5, not being able to reach me until Monday.

On Monday the 12th I got an invoice wanting to charge me 500 SEK for one of the broken modems they claimed I never sent back so I had to call customer service again and have them not do that. (I find 500 SEK for a broken ADSL modem quite a hefty charge when that’s basically the price for a completely new and working unit…)

December 13, modem #5 arrived and I connected it. It didn’t work at once but the phone worked which gave me a clue, so I connected a laptop directly to the ADSL modem and when I then tried to use a browser on that network I reached an admin interface web server and by using that I could switch the modem over to “bridge mode”. It turned out the default setting for this device is to function as a DHCP server and all sorts of other funny things that I didn’t want it to do.

At the time of this writing, number five has been running without problems for 72 hours.

cURL and libcurl, Haxx, Rockbox

I’m interviewed by foss-magasin

foss-magasin

Claes at foss-magasin.se asked a bunch of questions about me, my commitments within the FOSS community and related matters recently over email. This Swedish interview just now went public: Daniel Stenberg cURL, Rockbox och FOSS-Sthlm (dead link).

For my international friends who don’t understand the Swedish: I am quite happy with the questions and being allowed to answer them at this lengths etc, so I am considering doing a full translation of it and posting it at a later date.

Open Source, Rockbox

Ten years of Rockbox

2 Comments

In December 2001 the mailing list was setup and the first mail was sent out on December 7th. This was months before the project had any name. We just gathered eager reverse-engineers wanting to improve the Archos Player firmware.

We were just a few friends who like hacking low level code, both as professionals but also in our spare time – and we really thought that these kinds of devices had much larger potential than what the firmwares they were given allowed them. “Rewriting Archos firmware from scratch, how hard can it be?” as we used to joke. Oh well, we did.

archosplayer-front

From that moment we worked on mp3 players. A couple of months later we started on the next target (Archos Recorder) and so we continued. We got ourselves the name Rockbox for the project and people joined up from everywhere, wanting to contribute their knowledge and enthusiasm.

Rewriting Archos firmware from scratch - the tshirts

(Björn “Zagor” Stenberg, Linus “LinusN” Nielsen and Daniel “Bagder” Stenberg in September 2002.)

We got our logo in 2002. In 2003 we supported the FM recorder model. We ported code to and run our first stuff on a “software codec” target in 2004. During 2005 we added support for our first color screen targets and in 2006 we added ipod to our “family”. The flood gates opened and new targets have poured in ever since. iAudio X5 and the Sansa e200 were also added that year.

Today, we have code running natively on 75 something targets (on SH1, m68k, ARM and MIPS architectures) and we run Rockbox as an app on top of other operating systems such as Android and Maemo. The project keeps up a fast pace and even in the last few months we’ve seen several new ports having been added to the source code tree.

Being a large project with lots of strong personalities and committed developers we’ve had our share of politics and flame fests. The real name policy was originally a reason for lots of heated debates, as we only accept contributions from people who provide real names – no nick names, but as time has passed the arguments have more and more been over technical details or over how the development is or isn’t run.

Rockbox has participated in the Google summer of Code program four years as a mentor organization and in this time we’ve had perhaps 15 students that have worked on Rockbox, and a bunch of them were successful and a fair amount of those students stayed in the project after having finished their summer projects.

The Android version hasn’t been released on the Android market so far because lots of developers think that first impressions is very important and as Rockbox has been designed with fixed-size screens there has been no support for platforms with varying screen resolutions. This has forced Rockbox to provide different versions for different Android targets (screens really). In addition to that, the GUI of Rockbox has been all native Rockbox and not very Android-like which has also been mentioned as a con. These issues are being worked on, although I cannot provide any estimate for when we’ll see Rockbox “for real” on Android.

I’ll stick to my story about what I think of Rockbox’s future: I think the dedicated music player market is going away slowly and that phones and other portable devices is what people will use to play music on. Rockbox is a very capable music player, but the question is if there’s really a demand for it on the new generation of devices…

cURL and libcurl

A special thank you from Google

5 Comments

Believe me, this kind of seemingly random act of kindness warms up even the most seasoned and cold hacker’s heart. Look at this excerpt from a mail I received:

From: Stephanie T <…@google.com>
Subject: 2 Googlers would like to recognize your hard work on cURL

Hello Daniel,

As you know, we here in Google’s Open Source Programs Office are always interested in learning about new projects and people in the open source community. To that end, we asked our co-workers to help us by nominating people outside of Google that they thought were doing great things in the world of open source. This information helps us determine which organizations to fund and which projects to select for the Google Summer of Code and related programs.

Chris C and John M (Googlers) believe your work on cURL deserves to be recognized. To show our appreciation for this work, we would like to send you a special thank you gift and 2- $175 USD gift codes (totaling $350 USD) to the Google online store so you can pick out tee shirts or other gifts for yourself and loved ones.

[snip]

Thanks a lot!

(The names of the persons in the mail have been shortened by me to reduce their exposure here.)

Open Source, Technology, Web

Swedish FOSS-magasin

1 Comment

foss-magasinClaes, our friend from foss-sthlm and several Open Source adventures, has just fired off a new initiative: FOSS-Magasin. The site launched for real on the evening November 19th.

Where there’s no real content on the site yet, Claes has set out a mission for himself and future contributors to create a site with technical content in Swedish that we geeks miss. This would be within areas such as FOSS, *nix, networking and more.

Tired of the poor state of technical and IT related media in Sweden that always seem to try to capture the really large audience and therefore always dumb down everything to a silly level, this is meant to be directed on more competent and interested readers.

The site is free and Claes is looking around for contributors to help hem get content to publish. I can only urge my Swedish friends to join up and help it get going, as I think it would be nice to get a proper Swedish tech site. For me, it will be especially interesting for things that actually happen in or otherwise is related to Sweden, as for all the rest I personally have no problems accessing English sites to get the info.

IT Politics, Licensing, Open Source

IDG prints lies about RMS

8 Comments

Joel Åsblom works as a “technical writer” at the Swedish “IT magazine” consortium IDG. He got assigned the job of interviewing Richard M Stallman when he was still in Stockholm after his talk at the foss-sthlm event. I had been mailing with another IDG guy (Sverker Brundin) on and off for weeks before this day to try to coordinate a time and place for this interview.

During this time, I forwarded the “usual” requests from RMS himself about how the writer should read up on the facts, the background and history behind Free Software, the GNU project and more. The recommended reading includes a lot of good info. My contact assured me that they knew this stuff and that they had interviewed mr Stallman before.

This November day after the talk done in Stockholm, Roger Sinel had volunteered to drive Richard around with his car to show him around the city and therefore he was also present in the IDG offices when Joel interviewed RMS. Roger recorded the entire interview on his phone. I’ve listened to the complete interview. You can do it as well: Part one as mp3 and ogg, and part 2 as mp3 and ogg. Roughly an hour playback time all together.

The day after the interview, Joel posted a blog entry on the computersweden.se blog (in Swedish) which not only showed disrespect towards his interviewee, but also proved that Joel has not understood very many words of Stallman’s view or perhaps he misread them on purpose. Joel’s blog post translated to English:

Yesterday I got an exclusive interview with legend Richard Stallman, who in the mid 80’s, published his GNU Manifesto on thoughts of a free operating system that would be compatible with Unix. Since then he has traveled the world with his insistent message that it is a crime against humanity to charge for the program.

As the choleric personality he is, I got the interview once I’ve made a sacred promise to never (at least in this interview) write only Linux but also add Gnu before each reference to this operating system. He thinks that his beloved GNU (a recursive acronym for GNU is Not Unix) is the basis of Linux in 1991 and thus should be mentioned in the same breath.

Another strange thing is that this man who KTH and a whole lot of other colleges have appointed an honorary doctorate has such a difficulty to understand the realities of the labor market. During the interview, I take notes on a computer running Windows, which makes him get really upset. He would certainly never condescend to work in an office where he could not run a computer that contains nothing but free software. I try to explain to him that the vast majority of office slaves depend on quite a few programs that are linked to mission-critical systems that are only available for Windows. No, Stallman insists that we must dare to stand up for our rights and not let ourselves be guided by others.

Again and again he returns to the subject that software licensing is a crime against humanity and completely ignores the argument that someone who has done a great job on designing programs also should be able to live from this.

The question then is whether the man is drugged. Yes, I actually asked if he (as suggested in some places) uses marijuana. This is because he has propagated for the drug to be allowed to get used in war veteran wellness programs. The answer is that he certainly think that cannabis should be legalized, but that he has stopped using the drug.

He confuses freedom with price – RMS never refuses anyone the right to charge for programs. Joel belittles the importance of GNU in a modern Linux system. He calls him “choleric”. He claims you cannot earn money on Free Software (maybe he needs to talk to some of the Linux kernel hackers) and he seems to think that Windows is crucial to office workers. Software licenses a crime against humanity? From the person who has authored several very widely used software licenses?

The final part about the drugs is just plain rude.

During the interview, Joel mentions several times that he is using Ubuntu at home (and Stallman explains that it is one of the non-free GNU/Linux systems). It is an excellent proof that just because someone is using a Linux-based OS, they don’t have to know one iota or care the slightest about some of the values and ethics that lie behind its creation.

In the end it leaves you wondering if Joel wrote this crap deliberately or just out of ignorance. It is hard to see that you actually can miss the point to this extent. It is just another proof what kind of business IDG is.

The reaction

Ok, so I felt betrayed and badly treated by IDG as I had helped them get this interview. I emailed Sverker and Joel with my complaints and I pointed out the range of errors and faults in this “blogpost”. I know others did too, and RMS himself of course wasn’t too thrilled with seeing yet another article with someone completely missing the point and putting words into his mouth that he never said and that he doesn’t stand for.

During the weekend I discussed this at FSCONS with friends and there were a lot of head-shakes, sighs and rolling eyes.

The two writers both responded to my harsh criticisms and brushed it off, claiming you can have different views on free vs gratis and so on, and both said something in the style “but wait for the real article”. Ok, so I held off this blog post until the “real article”.

The real article

Stallman – geni och kolerisk agitator, which then is supposedly the real article, was posted on November 15th. It basically changed nothing at all. The same flaws are there – none of the complaint mails and friendly efforts to help them straighten out the facts had any effect. I would say the most fundamental flaws ones are:

With opinions that it is a crime against humanity to charge for software Richard Stallman has made many enemies at home. In South America, he has more friends, some of which are presidents whom he persuaded to join the road to free source code.

Joel claims RMS says you can’t charge for software. The truth is that he repeatedly and with emphasis says that free software means free as in freedom, it does not necessarily means gratis. Listen to the interview, he said this clearly this time as well. And he says so every time he does a public talk.

Richard Stallman is also the founder of the Free Software Foundation, and his big show-piece is the fight against everything regarding software licenses.

Joel claims he has a “fight against everything regarding software licenses”. That’s so stupid I don’t know where to begin. The article itself even has a little box next to it describing how RMS wrote the GPL license etc. RMS is behind some of the most used software licenses in the world.

The fact that Joel tries to infer that Free Software is mostly a deal in South America is just a proof that this magazine (and writer) has no idea about for example the impact of Linux and GNU/Linux in just about all software areas except desktops.

My advice

All this serves just as a proof and a warning: please friends, approach this behemoth known as IDG with utmost care and be sure that they will not understand what you’re talking about if you’re not into their mainstream territory. They deliberately will write crap about you, even after having been told about errors and mistakes. Out of spite or just plain stupidity, I’m not sure.

[I deliberately chose not to include the full article translated to English here since it is mostly repetition.]

curl, open source and networking