Seen on my wordpress dashboard before I posted this:
That’s 500 posts since my first post on this blog (August 28 2007).
All posts by Daniel Stenberg
curl goes git
Just a few days ago the curl project turned twelve years old, and I decided that it was time for us to ditch our trusty old CVS setup and switch over to use git instead for source code control.
Why Switch at All
I’ve been very content with CVS over the years and in our small project we don’t really have any particularly weird or high demands on the version control software.
Lately (like in recent years) I’ve dipped my toes into various projects that have been using git, and more and more over time I’ve learned to appreciate the little goodies that git does that CVS simply cannot. I’m then not even speaking about branches or merges etc that git does a whole lot better and easier than CVS, I’m in fact even more in love with git’s way to ease handling with diffs sent by email and its great way of keeping track of authors separately from the committer etc. git am and git commit –author are simply two very handy tools missing in CVS.
Why Git
So if we want to switch from CVS to another tool what would we chose? That wasn’t really the question in my case so I didn’t answer it. In my case, it was rather that I’ve been using git in several projects and it is used in some of the biggest projects I work with so it was some git’s features I wanted. I didn’t consider any of the other distributed version tools as quite frankly: they wouldn’t be much better for me than what CVS already is. I want to reduce the number of different tools I need, and I’m quite sure anyway that git is one of the top contenders even if I would do an actual comparison.
So the choice to go git was quite selfish and done by me, but I felt that quite a few guys in the curl community supported this decision and very few actually believe remaining with CVS was a better idea.
The fact that git itself uses libcurl for its HTTP access of course also proves its good taste! 🙂
How did the conversion go
Very easy and swiftly. First, as I mentioned above we never used branches much so we basically had a linear development with a set of tags. I did an rsync of the full repo to get a local copy to work with, then I ran ‘git cvsimport’ on that to created a new repo. I did run it a couple of times to make sure I had done a correct mapping of all CVS user names to their git equivalents. Converting >10 years of CVS commits took roughly 10 minutes on my desktop machine so it wasn’t that tedious even.
Once I had a local repo created with all authors looking good, I simply followed the instructions on github.com on how to add a remote origin to a local branch and when I pushed to that, git sent off all commits ever made to curl to the remote repo now exposed to the world from github.com.
When that part was done, I did a quick read on the ‘git help daemon’ docs and 30 seconds later I had a local repo setup that is a mirror of the github one, so that users can still opt to get the code from haxx.se.
Unchanged work flow
Git allows different ways of working with the code, but I’ve decided that at least as a start we won’t change the way we work. I’ll offer all committers push rights to the master branch on the repository and we will simply all push to that, as our head development branch.
We will prefer patches made with git format-patch sent to the mailing list, but as before you can still produce patches by diffing source code using extracted tarballs or whatever approach you prefer.
All details on how to get the code for curl using git is available online.
two competitors or one united
In a discussion on the libssh2 mailing list, the founder and maintainer of the libssh project got involved. The actual discussion is not what I want to talk about here, but something he touched on in one of his mails:
“I think a bit of competition in open source is fair and leads to innovation”
Recall that there are only two existing SSH libraries written in C that are free and open. These two libs have roughly the same features, the same goals and a lot of other similarities.
So, does it help open source projects to have an identified and known competitor to compete against and to try to steal users from and to try to outperform in other ways? Is there any research done that proves this theory?
Or is it just so that we have a number of developers with a certain amount of time, and if we divide those in two or more groups we therefore make sure that neither is advancing as fast as they could if all those persons would participate in the same project? Open Source projects are so often driven and developed by volunteers, and within a given area (say people interested in a SSH library) there is only a given amount of people and to make the best use of that limited set, wouldn’t it be better if they all would work together in a single project?
Would OpenBSD, NetBSD and FreeBSDÂ have been better off by not having split up? Would KDE and GNOME have reached further today if they had been a united project?
I don’t know the answers here, and most probably the answer isn’t very clear or binary applying to all cases anyway as I bet all situations are slightly different and thus should be considered separately.
In the history of FOSS, many forked projects end up getting merged again but we don’t often see two independently created projects merge. I guess the Compiz and Beryl fusion is an exception. I think the sense of “my pet project” is often too strong, not to mention that different licenses and different development cultures make mergers hard to take place.
Look, I’m not really advocating that libssh2 and libssh should merge at this point. I’m just playing with the idea and trying to see the issue from different angles.
There are of course several things that would speak against projects to merge: different views on what licenses that are suitable, religious things such as how to indent source code or what build system to use. Quite possibly also other social aspects: development and team “culture” and behavior and why not just the “Not Invented Here” syndrome – it isn’t always that easy to give up what you made yourself or to appreciate someone else’s work.
140 foss hackers
At last, the first meeting with our recently started foss-sthlm effort took place. The amount of attention and attendance we achieved by far surpassed our wildest assumptions, and around 130-140 persons interested in open source showed up (we don’t know the exact number). The facilities in Kista where we held the event, were graciously let to our use by Stockholm’s University (DSV) and they were very good. MSC and Nohup were our two sponsors who donated great sub sandwiches and drinks to all of us. Thank you!
I’m glad we manage to offer this event completely free of charge to the attendees, and hey we had quality talkers speaking up on really interesting subject that I think the audience appreciated on the subjects of PostgreSQL, Upstart, Open Source Sweden, Rockbox and Debian packaging.
I did a 20 minute talk about curl – in Swedish. The slides are available (they are thus in Swedish too), see below, and hopefully there will soon be video available online with my presentation.
I also hope that we will gather all the slides at one single point to offer on the foss-sthlm web site, so check that out later on if you want the lot!
Foss Sthlm #1 – curl
View more presentations from bagder.
And here are Björn‘s slides:
Presentation av Rockbox
View more presentations from bjornhaxx.
The Rockbox future is an app
We rarely discuss what the future of Rockbox looks like, as it rarely actually matters. We work on whatever we think is fun and all things are fine.
Now, I’d like to do something different and actually lift my nose and stare towards the horizon. What’s in the future for Rockbox?
Dedicated mp3 players vanish
I claim that the current world of portable mp3 and music players is dying. Within just a few years, there will only be a few low-end players left being manufactured and most portable music will be done on much more capable (CPU and OS wise) devices such as the current smartphones, be it Android, Maemo or iPhone based ones.
While maintaining Rockbox to work and prosper on the existing targets is not a bad thing, the end of the line as a stand-alone firmware is in sight. I say the only viable future for Rockbox when the players go very advanced, is not to make Rockbox handle networking etc, it is to make sure that it can run and function as an app on these new devices and to take advantage of their existing network stacks etc.
An app for that?
Rockbox as an app has been a story we’ve told the kids around the campfires for a good while by now and yet we haven’t actually seen it take off in any significant way. I’m now building up my own interest in working on making this happen. In a chat after my Rockbox talk at Fosdem 2010, two other core Rockbox developers (Zagor and gevaerts) seemed to agree to the general view that a Rockbox future involves it running as an app.
Out of the existing systems mentioned above, I’d prefer to start this work focused on Android. It has the widest company backing combined with open source and it’s also the most used open phone OS. I don’t think there’s anything that will prevent us from working on all those platforms as the back-bone should be able to remain the same and portable code we already have and use. Heck, it could then also become more of a regular app for common desktops too.
Challenges
Changing Rockbox to become an application instead of a full operating system and application suite involves a lot of changes. Some of the things we need to solve include:
- We need to make sure we can use the OS’ native threading without any particular performance loss. The simulator shows that there are might be problems with this right in the current code as it runs dog slow on a Nokia n900.
- The full suite of plugins and games and so on doesn’t really fit that well when Rockbox is just an app in itself.
- Different screen sizes. Lots of stuff in Rockbox assume a compile-time fixed screen size, while a good phone app would have to be able to work with whatever size the particular happens to feature. I don’t think we need to care about live resizing (yet).
Update, July 2010: The app part II.
My Rockbox talk at Fosdem 2010
As I’ve mentioned several times before in this blog, I did a talk about Rockbox and reverse engineering at Fosdem 2010 Feburary 6-7 in Brussels, and since there was no “pre-arranged” video recording of the talks in the embedded devroom, Peter D’Hoye stepped up and recorded the whole thing using his Nokia n900 phone.
I decided to not make the slides for this talk available separately, as they were more or less the same as the ones I used for my FSCONS 2009 talk, so you can go watch them instead if this video isn’t enough!
To view it, I suggest you use VLC or similar and tell it to stream directly from one of these URLs, the file is a 1.1GB one with 848×480 resolution running for 51 minutes. Annoyingly, none of the usual free online video services allow this long ones.
a big curl forward
We’re proudly presenting a major new release of curl and libcurl and we call it 7.20.0.
The primary reason we decided to bump the minor number this time was that we introduce a range of new protocols, but we also did some other rather big works. This is the biggest update to curl and libcurl that have been made in recent years. Let me mention some of the other noteworthy changes and bugfixes:
We fixed a potential security issue, that would occur if an application requested to download compressed HTTP content and told libcurl to automatically uncompress it (CURLOPT_ENCODING) as then libcurl could wrongly call the write callback (CURLOPT_WRITEFUNCTION) with a larger buffer than what is documented to be the maximum size.
TFTP was finally converted to a “proper” protocol internally. By that I mean that it can now be used with the multi interface in an asynchronous way and it has far less special treatments. It is now “just another protocol” basically and that is a good thing. Also, the BLKSIZE problem with TFTP that has haunted us for a while was fixed so I really think this is the best version ever for TFTP in libcurl.
In several different places in the code older versions of libcurl didn’t properly call the progress callback while waiting for some special event to happen. This made the curl tool’s progress meter less responding but perhaps more importantly it prevented apps that use libcurl to abort the transfer during those phases. The affected periods included the ftp connection phase (including the initial FTP commands and responses), waiting for the TCP connect to complete and resolving host names using c-ares.
The DNS cache was found to have at least two bugs that could make entries linger in the database eternally and in another case too long. For apps that use a lot of connections to a lot of hosts, these problems could result in some serious performance punishments when the DNS cache lookups got slower and slower over time.
Users of the funny ftp server drftpd will appreciate that (lib)curl now support the PRET command, which is needed when getting data off such servers in passive mode. It’s a bit of a hack, but what can we do? We didn’t invent it nor can we help that it’s a popular thing to use! 😉
My Fosdem 2010
Friday
Björn and I left work on the Friday afternoon and took a flight down to Brussels, Belgium. After having checked in to our hotel, we met up with Frank from the Rockbox project and we headed to the Fosdem beer event that took place on a pub quite nearby to the hotel.
The Beer event was crowded. I mean really really crowded. But we still managed to get seated and we got fine Belgium beers and we had a good time. We met a few other Swedes that turned out to be the first in a long series of Swedes that were there. Petur from Rockbox joined up there as well and together we went over a fair share of their beer selection…
Saturday
For us tech guys, the Saturday morning had no really exciting subjects and weirdly enough the morning had only one track and the massive amount of parallel tracks didn’t start until after lunch. This gave us an opportunity to go sight-seeing, and we visited the city square and the Atomium before we headed into the FOSDEM premises and squeezed our way in to a presentation.
Peter Stuge from the Coreboot project explained to us that we were by far too many people crammed into that little room so if one of the responsible guys would come around a fair lot of us would get thrown out of there. With that heads up given, he started his talk and gave us insights in what coreboot is, what it does and so on. I’ve heard Peter talk about this topic before, but he’s still a good talker and the topic still is techy and interesting enough to listen to.
Embedded software development best practices by Adrien Ampelas turned out to be a bit boring. Basically we got the feeling that Adrien re-used a company slide show or something and told the audience a lot of things I bet the majority of people already knew. Yes we know we must use version control. Yes we know we should send patches upstream. No we don’t 
agree with you that there never exist any reason not to use git.
Sascha Hauer from the Barebox project (the project that was previously known as U-Boot v2) told us about this U-Boot project and what they’re trying to accomplish. It seems like an interesting approach to fix some of the worst mistakes of U-Boot but still leverage on all the things U-Boot did right. It’ll be fun to see if it gets adoption from board makers and companies in general. I guess there’s a lot of investment in U-Boot so lots of things will probably stick with that for a long time ahead…
Flash enable BIOS reverse engineering by Luc Verhaegen gave us an insight in the x86 based reverse engineering they do in the Coreboot project to figure out how to enable flashes and to make them possible to write to when you want to upgrade them to use Coreboot. It was only a quick run-through, but my general feeling was still that compared to Rockbox-style reverse engineering, their tasks actually seem a lot easier! Still interesting, as Luc is a good speaker.
Sunday
Sunday morning started earlier than yesterday. Interesting talks started right away, and we actually were too slow at breakfast so we missed the first part of the interesting Introduction to RTEMS talk by Thomas Doerfler. RTEMS is a fully open source RTOS that’s been around for ages and that has some very good realtime skills and can get shrunk to a rather small size. A slight downside with it is its slightly odd license, as it is a GPLv2+ license with a rather big exception that is made to allow proprietary applications link with it. It makes it incompatible with regular GPLv2 code.
The RepRap project was presented by Adrian Bowyer and I must admit that these 3D-printers are mighty cool and even more fun to see and witness in the real world than they are to see on tiny pictures on web sites.
Back in the embedded room, Roberto Jacinto told us about apt-get for android – with GUI which pretty much described the Aptoide project. It has nothing in common with apt: it doesn’t do dependencies and it doesn’t use its file formats. It has some pretty significant bugs still, and it generally seemed like a rather immature project that I’m not even sure I agree is on the right track. I’d rather actually see the real apt-get for android, with or without GUI.
The Cross build systems: Present & Future workshop could’ve become interesting. A lot of projects (PTXdist, Buildroot, Crosstool-NG, Openembedded, Emdebian etc) spoke about what they are, what they hope to do and how they’d like to collaborate. Unfortunately it took a bit too long time so by the time all had presented their projects the time was pretty much up. The most controversial and slightly off-topic of them all was Andy Green (formerly involved in Openmoko) who talked about how we all should stop cross-compiling and build directly on the target instead(!) and how booting Linux shouldn’t need a boot-loader and that designing PCBs with NAND is stupid(! again). I didn’t hear anyone agreeing with his ideas.
Next up was my talk on Rockbox. I did it in about 40 minutes and I think I covered a bit of what Rockbox is and how we work when we work with new potential targets. It later struck that I should perhaps have had a slide about what the future holds etc, but hey I think it went pretty smooth anyway! Peter recorded my talk on his n900 so hopefully it’ll soon be available online somewhere. After my talk we met a lot of guys wanting to talk Rockbox, ask about particular players and so on and it was mighty fun and interesting.
Greg Kroah-Hartman did the final talk and he is a very good and engaging speaker that really can catch the big audience in Fosdem’s biggest room. Write and Submit your first Linux kernel patch is his “standard talk” but he’s doing it so good and with such elegance that it is a pleasure to watch and learn from. And I’ll admit I wasn’t aware of the get_maintainers.pl script in the kernel tree. A very useful little thing!
Reflections
Some conclusions and general thoughts about the event:
Lack of gaps – there’s a problem when all talks in all rooms are made gapless. It makes people get up and leave 5-10 minutes before the end of each talk so that they will get in time to the next talk that will start on the full hour in another room. It causes pretty much all question-sessions towards the end to fail since the questions (and answers) can’t be heard.
Hard to find people – it is such a huge event and lots of people I have no idea what they look like, so trying to meet friends and people I’ve only emailed with or chatted with on IRC is very hard. Name tags would be really cool. I did have some benefits
from using my shirt with a big Haxx logo on the back since a fair amount of people recognized it and approached me!
Audio systems – the quality of the different rooms varied a lot (not only sound-wise but the sound was what bothered me). Unfortunately for me, the embedded room was one of the worst ones when it came to audio. It was a big room sure, but the biggest room had an excellent audio system and thus proved size is not what matters. In this case, I think a lot was to blame on the actual microphone we had there.
Phone apps – having phone apps with the entire schedule and a little map for each room etc was a great service. The app also reminded us when a talk you had marked as “favorite” was about to start. It was a bit strange though how the android and n900 versions of the app differed. The n900 version was buggy and slow, but it did offer the schedule in a time-based view while the android version only allowed us to view the schedule based on rooms.
Next year – yes. I think it was great fun and I will really try to attend next year again. Hopefully other friends will too, since meeting friends at the place really doubles the fun! Thank you all for a nice event!
libssh2 version 1.2.3
The team behind libssh2 isn’t very big, but we’ve managed to yet again ship a new release (version 1.2.3) that adds new features such as support for SSH-agent and the new libssh2_trace_sethandler() function, while also fixing a few bugs.
Enjoy!
The big protocols
OWASP Sweden once again arranged another interesting meeting, this time with three talks.
The title of the meeting on January 21st here in Stockholm called the protocols “the big ones” (but in Swedish) but I have no idea what kind of measurement they’ve used or what the small ones are or what other “big protocols” there might be! 😉
First we got to hear HÃ¥vard Eidnes tell us about BGP and that protocol seems to suffer from its share of security problems with the protocol itself but perhaps even more with the actual implementations as one of the bigger recent BGP-related incidents that was spoken about was about how internal routes were leaked to the outside from Pakistan in Feb 2008 which made them block the entire world’s access to Youtube. This talk also gave us some insights on the “wild west” of international routing and the lack of control and proper knowledge about who’s allowed to route what to where.
There then was a session by Rickard Bellgrim about DNSSEC and even though I’ve heard talks about this protocol in the past I couldn’t but to again feel that man they have a lot of terminology in that world that makes even a basic description fairly hard to keep up with in some parts of it all. And man do they have a lot of signing and keys and fingerprints and trusts going on… Of course DNSSEC is the answer to lots of existing problems with DNS and DNSSEC certainly opens up a range of new fun. The idea to somehow replace the need for ca-certs by storing keys in DNS is interesting, but even though technically working and sound I fear the browser vendors and the CAs of the SSL world won’t be very fast to turn the wheels to roll in that direction. DNSSEC certainly makes name resolving a lot more complicated, and I wonder if c-ares should ever get into that game… And BTW, DNSSEC of course doesn’t take away the fact that specific implementations may still be vulnerable to security flaws.
The last talk of the evening was about SSL, or rather TLS, held by Fredrik Hesse. He gave us a pretty detailed insight into how the protocol works, and then a fairly detailed overview of the flaws discovered during the last year or so, primarily MD5 and rogue ca certs, the null-prefix cert names and the TLS renegotiation bug. I felt good about already knowing just about everything of what he told us. I can also boast with having corrected the speaker afterward at the pub where we were having our post-talk-beers as he was evidently very OpenSSL focused when he spoke about what SSL libraries can and cannot do.
A great evening. And with good beers too. Thanks to the organizers!