Category Archives: Open Source

Open Source, Free Software, and similar

cURL and libcurl, Open Source

Code re-use is fun

Back in 2003 I wrote up support for the HTTP NTLM authentication method for libcurl. Happy with my achievement, I later that year donated a GPL licensed version of my code to the Wget project (which also was my first contact with the signed paper stuff with the GNU/FSF to waive my copyright claims and instead hand them over). What was perhaps not so amusing with this code was when both curl and Wget 2005 were discovered to have the same security flaw due to my mistakes in this code shared by both projects!

Just recently, the neon project seems to be interested in taking on the version I adjusted somewhat for them, so possibly the third HTTP code is soon using this. Yeah I posted it on their mailing list back then so it has been sitting there in the archives maturing for some 6 years by now…

I also happened to fall over the SSH Tunnel Creator tool, which I’ve never used myself, that apparently snatched my neon donation (quite according to what the license allowed of course) and used it in their tool to do NTLM!

It’s actually not until recent years I discovered libntlm, and while I don’t know how good it was back in the days when I wrote my first NTLM stuff I generally think using existing libs is the better idea…

Open Source, Rockbox

Rockbox in gsoc 2009

Gsoc 2009

For the third year in a row, the Rockbox project participates in Google’s Summer of Code and has just now officially been accepted as a mentor organization.

This is a great chance for every student or others who have been looking for a good chance to join in an open source project, and in particular if you’re interested in embedded systems developing for (relatively) low-resource devices.

The Rockbox project also has a very large developer base, a very large community and a healthy and friendly surrounding that is welcoming to newbies.

If you’re even just the slightest intrigued, please don’t hesitate to read further on the Rockbox gsoc 2009 page.

The two previous years we’ve had four students each year with 15-20 applications.

Let’s make this a great gsoc year!

libssh2, Network

libssh2 upped a notch

1 Comment

There have been some well-founded criticism against libssh2 for a long time for its bad transfer performance when doing SCP and SFTP based transfers. Tests have proved it to be significantly slower than the openssh based alternatives in comparisons done in similar conditions. We’re talking down to a tenth(!) of the speed for SFTP.

Luckily I have a unnamed (by agreement) sponsor who pays me for improving this.

Giving it some love

I basically started out reading the SFTP code and cleaned it up as I went over it, and I added some clarifying comments etc. I found some irregularities that I fixed. Soon I could spot an obvious performance boost, like perhaps 3-4 times the previous speed. But since SFTP was painfully slow originally, this was still very crappy compared to openssh.

I then switched over to plain SCP tests. SCP is basically just an “scp” command sent over SSH and then streaming the data over a plain SSH “channel”, while SFTP is a whole additional protocol layer on top. Thus SCP is more low-level, on the actual SSH level, and the foundation on which SFTP runs anyway so getting SCP faster was fundamental.

Make it speedier

My initial tests with libssh2 1.0 showed libssh2 to download data at roughly 25% of the speed of openssh when SCPing a 1GB file from an openssh server running on localhost. The openssh client shows roughly 40MB/sec on my test box.

Also, just checking my CPU load meter while doing the libssh2 transfers showed that it certainly wasn’t hitting the roof or anything. It was barely even noticeable! Of course something was really wrong but what was it?

SSH has a lower protocol layer that does the entire encryption thing, the transport layer, but on top of that is the “channel layer” that is packet based for sending data back and forth over the transport layer. This channel thing has a receive window concept, much like TCP itself has, which tells the remote side how much data it is allowed to send until it gets further notice.

libssh2 1.0 had a very conservative windowing logic. It started with a default window size of 64KB and it upped it at every read with the same amount that was read (which then could be 1K to 16KB something depending on the app).

My remake of this was to simplify the logic, read data from the network more evenly distributed over time, update the window size much less frequent and increase the window size by magnitudes! I found that when using a window size of 38MB (600 times the previous default size!!) things started flying.

Improved

With these modifications, libssh2 transfers SCP at close to 40MB/sec! SFTP is still left behind at a “mere” 14MB/sec on the same test setup but it has its own set of problems and solutions. Now this discussion on the libssh2 list is more about how to sensibly size the window to work the best way for different situations.

SFTP is a protocol that works more on file operations. The client sends OPEN, READ and CLOSE requests and the server replies with status and data. The READ request asks for N bytes starting at offset Z so a simple implementation like libssh2 asks for chunk after chunk in a serial manner, increasing the offset as it loops over the range. This causes a back-and-forth effect that certainly does not make optimized use of the network bandwidth.

SFTP ping pong

openssh has a nifty approach to enhance throughput for SFTP: it sends off and handles multiple outstanding READ requests in parallel so that it better can keep things busy (and the reverse when doing uploads). That concept is slightly harder to do with an API like the one libssh2 offers but it is of course still quite doable. I suspect that we might achieve results somewhat faster by simply use multiple connections as then we can remain using this simplistic approach but still use the full bandwidth. (Yes, I realize multiple connections may not be feasible for all applications.)

Previous tests we’ve done with SFTP uploads using multiple connections have proven libssh2 to be on par or even better than competitors on both Windows and Mac.

Please test

I’ll leave it like this for now. I’ll be very happy if people could test this version and report findings so that we make sure this is working and stable enough to release soonish. We’ll need to do something that offers window size controlling to apps, but we’ll discuss that further on the mailing list. Join in!

logo1-250

Rockbox

Rockboxitunes

6 Comments

Frank Gevaerts and Jonas Häggqvist in the Rockbox project spent some time the other night when they should’ve been sleeping and made the Rockbox USB stack possible to fake being an iPod and thereby tricking iTunes to work with a non-iPod player quite transparently. As the picture shows, the target used here is a SanDisk Sansa e2x0 player…

frankosansa

This is still work in progress and not yet in SVN. Keep up with the bleeding edge activity in the #rockbox IRC channel on freenode!

cURL and libcurl, Network, Open Source

murl for extended curlness

1 Comment

I’m a firm believer in the old unix mantra of letting each tool do its job and do it well, and pass on the rest of the work to the next tool. I’ve always stated that curl should remain this way and that it should remain within its defined walls and not try to do everything.

But time passes and more and more ideas are thrown up in the air, or in some cases directly at me, and the list of things that we could do but don’t due to this philosophical limit of remaining focused has grown. It currently includes at least:

  • metalink support
  • recursive HTML downloads
  • recursive/wildcard FTP transfers
  • bittorrent support
  • automatic proxy configuration
  • simultaneous/parallel download support

Educated readers of course immediately detect that this list (if implemented) would make a tool that basically does what wget already does (and a lot more) and I’ve explicitly said for a decade that curl is not a wget clone. Maybe it is time for us (me?) to reevaluate that sentiment – at least in some sense.

I don’t want to sacrifice the concepts that have worked so fine for curl under so many years, so I’m still firmly against stuffing all this into curl (or libcurl). That simply will not happen with me at the wheel.

A much more interesting alternative would be to instead start working on a second tool within the curl project: murl. A tool that does basically everything that curl already does, but also opens the doors for adding just about everything else we can cram in and that is still related to data transfers. That would include, but not be restricted to, all the fancy stuff mentioned in the list above!

No the name murl is not set in stone, nor is this whole idea anything but plain and early thoughts thrown out at this point so it may or may not actually take off. It will probably depend on if I get support and help from fellow hackers to get started and moving along.

cURL

cURL and libcurl, Security

curl 7.19.4

curl and libcurl 7.19.4 has just been released! This time I think the perhaps most notable fix is the CVS-2009-0037 security fix which this release addresses. A little over 600 days passed since the previous vulnerability was announced.

Other than that major event, there are a bunch of interesting changes in this release:

  • Added CURLOPT_NOPROXY and the corresponding –noproxy
  • the OpenSSL-specific code disables TICKET (rfc5077) which is enabled by default in openssl 0.9.8j
  • Added CURLOPT_TFTP_BLKSIZE
  • Added CURLOPT_SOCKS5_GSSAPI_SERVICE and CURLOPT_SOCKS5_GSSAPI_NEC – with the corresponding curl options –socks5-gssapi-service and –socks5-gssapi-nec
  • Improved IPv6 support when built with with c-ares >= 1.6.1
  • Added CURLPROXY_HTTP_1_0 and –proxy1.0
  • Added docs/libcurl/symbols-in-versions
  • Added CURLINFO_CONDITION_UNMET
  • Added support for Digest and NTLM authentication using GnuTLS
  • CURLOPT_FTP_CREATE_MISSING_DIRS can now be set to 2 to retry the CWD even when MKD fails
  • GnuTLS initing moved to curl_global_init()
  • Added CURLOPT_REDIR_PROTOCOLS and CURLOPT_PROTOCOLS

We also did at least 15 documented bugfixes in this release and 25 people are credited for their help to make it happen.

Open Source, Technology

A stream of streamings

I’m a last.fm fan. I love its ability to not only stream music without needing a dedicated client installed (yes a flash application I think suits a purpose) and I think it’s ability to provide music I might also like is amazingly nice. I’m a “random it all” kind of guy when I listen to my local music collection in most situations as well. It is not specifly well suited for listening on exact the songs you want, as if you select a specific song it won’t even play the full-length version of it.

Lately there’s been a lot of buzz in Swedish tech media about spotify, which is a similar idea (at the moment still an on invitation-only thing in Sweden). They stream music, but only to a proprietary Windows or Mac client and currently they offer free listening with ads (embedded in the audio and visible in the client) or 99 SEK (== 9 Euros == 11 USD) per month. The client is highly focused on specific songs or artists and it has nothing in the way of “random artitists I generally like and similar ones”. I’m not too thrilled.

Spotify offers its service in several places, and I hear in the UK it’s not even invitation-only (which of course is useful for the more forward-thinking hacking kind of guys who thus use a UK based proxy to reach them). There’s however no sign of a Linux client. We’re forced to run their windows client with Wine.

I’ve gotten the impression that Pandora is a similar concept to play with if you happen to be based in the US. I’m in Sweden and Pandora just shows me a “We are deeply, deeply sorry to say that due to licensing constraints, we can no longer allow access to Pandora for listeners located outside of the U.S.”

The other day despotify.se showed up. A bunch of clever hackers reverse engineered the Spotify protocol and stream and offer a full unofficial open sourced ncurses/libvorbis/pulse-audio/gstreamer/expat/zlib/openssl-based player! Reading the code shows that these guys certainly had to crack some hard nuts, but the activity in their IRC channel seems fierce and the code is rather clean so I expect it to turn out to eventually become a fine player if Spotify just doesn’t decide to play hard ball with them. Unfortunately, despotify hasn’t yet been able to produce a single sound for me since it has just died on assert()s on basically any attempts I’ve tried. The interface is also a bit… strange and not the easiest to figure out. (It should be noted that the despotify client still requires you who have an actual spotify account.)

It’ll be interesting to see how Spotify, or perhaps the big media companies owning all the music rights, will act on this initiative. This client does open up abilities for new fancy features. How about ripping the stream? How about re-distributing the stream like as a proxy? And of course it being open, it does open up for adding features I want to add.

Update: just hours after I posted this, Spotify closed access to their service using the despotify client as long as you’re not a “premium” (paying) user…

cURL and libcurl

Saying lib out loud

9 Comments

Not being a native English speaker, I’ve always pronounced libcurl with a ‘lib’ part as if it was part of ‘liberty’, and ‘curl’ with a K sound and ending with ‘earl’. I don’t know of any Swedes who would not pronounce ‘lib’ like that, but when speaking Swedish we’re of course highly influenced by other things so it’s not really relevant.

It wasn’t until I got on FLOSS Weekly that I fully realized that some English speakers would actually pronounce the ‘lib’ part as the first syllable of ‘library’ and that does make sense considering lib is short for library.

But libcurl is just a smaller player here. How do you English speakers pronounce libc? libxml? libgcc?

And yes, this is another one of those really important issues in life. Almost as important as how to close parenthetical expressions with emoticons!


Rockbox

Rockbox USB – A Long Journey

4 Comments

The road to get a fully working and enabled mass-storage USB stack in Rockbox has been long and bumpy.

Bridge-Chip

Back in the early days of mp3 players, they all had bridge-chips which basically meant that the entire USB handling was handled by a dedicated chip that also “took control” of the hard drive (right, most of the interesting players used hard drives initially) so Rockbox didn’t have to do much except hand over control, show something nice on the screen in the mean time and then when the plug was pulled, regain control of the drive again and continue working.

But similar to how early players used dedicated HW-solutions to do music playback and then ditched that for pure CPU-powered solutions, the HW-solutions for USB also died out and all recent models do USB in software.

Research

The PortalPlayer chips being notorious for their complete lack of public docs were more or less considered impossibly hard to get USB working on by us. Until MrH came along 2006 and figured out that the USB parts of the PP5024 chip (used in the sansa e200v1 models) was identical to the USB core used by Freescale’s imx31 chip and there are public docs available for that.

Software Stack

The first step towards a software USB stack for Rockbox was taken in the 2007 gsoc, when Christian “austriancoder” Gmeiner took on the task. He went for a Linux-kernel inspired and quite ambitious approach that at the end of the gsoc project still didn’t quite work. The plan was at that point to primarily write a driver for the Sandisk Sansa e200 series, and secondary for other PortalPlayer based targets (such as ipods etc).

Made Working

Enter Björn “Zagor” Stenberg who had a much more minimalistic approach. Not being pleased with the “big” approach Christian had provided, the existing code was simply cut out and replaced with the new leaner version. Focus was then to get something small and simple to work, and then add fancier layers or features later on instead of providing the architecture for that already from the start. The stack started to partially work for mass-storage.

Not Yet Stable

USB wizard Frank Gevaerts joined the effort and worked fiercely on the thing and not only did he bring USB serial support (that does enable things like logf() support to debug rockbox) he also made the mass storage support almost reliable. The non-existing docs for the hardware made the remaining few flaws very hard to catch and the development of the stack stood still at this point for a great while.

Meanwhile, the USB stack could successfully be used on various new (in the Rockbox sense, these models are no longer new on the mp3 player market) targets like the Toshiba Gigabeat S and the Onda series etc. Parts of the Cowon D2 reverse engineering was greatly enhanced thanks to the USB stack as figuring out the FTL is even more tricky without a fine way to get data over to the host PC. All this proved that the stack itself worked nicely but that there were target-specific inits or setups that were missing for the PortalPlayer targets.

Of course the other new targets were not PortalPlayer based so it took efforts from clueful hackers like Vitja Makarov to implement the hardware driver for the TCC chips and Maurus Cuelenaere who did the driver for the Onda.

Stable?

Now, in 2009 our reverse engineering hero Boris “dreamlayers” Gjenero banged his head against the problem and by doing really nice disassembly he figured out previously not done initiation sequences. These have now been committed to the Rockbox SVN respository, and the early tests of this are very promising showing the mass storage support to be very close to get supported “out of the box” on the PortalPlayer 5022 based targets now. The PP5020 ones still seem unstable, and we have no support on the PP5002 ones.

If things just proceed this fine, we’ll be able to ship Rockbox’s own USB stack by default in the upcoming 3.2 release!

Commit

At Wed, 25 Feb 2009 23:40:15 +0100, the USB code was enabled in the SVN code: “Enable USB mass storage on PP5020, PP5022 and PP5024 targets”

Rockbox

Updates: this post was updated several times to gain accuracy and truths pointed out to me by Dave Chapman, Björn Stenberg, Frank Gevaerts, Jens Arnold and others. Thanks!