Monthly Archives: March 2009

libssh2

libssh2.haxx.se

libssh2I’ve played around with a possible new design for the libssh2 web site and I’ve put it up on libssh2.haxx.se for everyone to play around with and comment (on the libssh2-devel list please).

The original and actual home page for the project is still over at www.libssh2.org but I’m not happy with that because of a few things:

The wiki duplicates info that we these days write in man pages and hardly anyone updates the wiki so it lags behind or just contains false or outdated info. I also think having the entire site a wiki a bit problematic for things like menus and generic site layout etc.

I want daily snapshots, web versions of the man pages and mailing list archives to be somewhat integrated in the site to be easier to find.

I already hosted libssh2-related stuff anyway so I’ve just packaged it in a slightly more friendly way. In my view.

What do you think?

Security, Technology

Haxx for you

1 Comment

So our company is named Haxx and it has been named like this for more than a decade, but the name is considered by some people be a mark of evil or something.

In my closest circle of friends we’ve kind of “always” liked using silly names and we’ve since long had a fascination with double Xes. Once upon the time in the early 90s we teamed up under the name Frexx and we did some funky programs on the Amiga. Most notably a programming language called FPL and the text editor FrexxEd.

When we then during the second half of the 90s needed to start an actual company to easier cater for our “spare time businesses” we wanted a new name but still one in a similar spirit. Being big friends and practitioners of writing “quick hacks” (“hack” in the sense that it is a quickly done program/script that perhaps isn’t always written very solidly or nice but works for the moment) to solve our own problems both at work and at home, we found Haxx to be a perfect name for us – Hack in pluralis, spelled with double-x.

Already at the time we took the name we knew about this bad habit at places that seemed to lump Hackers with Crackers or similar so we knew there would be a risk that some could assume us to be something else based on our name, but what the heck, we liked the name and we are and were hackers and we do and did a lot of hacks. Haxx it was. Haxx it is.

These days we get some minor problems due to this. At some companies (let’s not name any specific but you know the kind) they have black-listed haxx.se web sites (presumably because of the name ‘haxx’ in the domain name), some people get mails from us our the mailing lists we host easier filtered as spam and we get our share of strange suggestions etc.

I guess the upside of it is that we get our chances to whine on people and systems who decide to filter contents purely based on the presence of a single 4-letter word, either in a domain name or in web page or mail contents, and that is actually hilariously stupid.

Haxx

Rockbox, Technology

HD is the thing

Thomson apparently brought the new mp3hd format for music the other day. “HD” is apparently the thing we need to have included when a new term is announced. Why does the world need another lossless music format?

It seems they’ve introduced a crafty dual-format thing where they stuff MPEG-4 SLS lossless encoded data in a new id3 XHD3 tag within the mp3 and then stuff a “regular” mp3 as the normal data in there. This way it is supposed to still work fine with existing and older mp3 players. Of course the total size of all id3v2 tags is limited to 256MB, which could be a limiting factor for it.

As usual, you can find a thriving discussion on this topic on hydrogenaudio.

Rockbox should of course be possible to at first use the mp3 parts and if this truly is an existing established lossless codec there’s a chance it might be able to play that part in the future.

Rockbox

Rockbox 3.2

1 Comment

The never-ending flow of creativity in the Rockbox project was today turned into a release that we label 3.2. The goodies this time include the things below. The three-months release cycle does prevent the list from growing terribly big…

  • Faster text/graphics rendering on colour targets and in the greyscale library, speeding up list scrolling noticeably on ipod Video.
  • PictureFlow supports all targets except Archos Player, and can function during playback on all non-Archos targets.
  • Add LCD sleep/wakeup for iPod Video (5G, 5.5G) which allows significant increase of battery runtime.
  • New game, Goban plugin.
  • Battery charging on Sansa e200v1/c200v1.
  • PictureFlow resizes cover art on load, and supports greyscale targets.
  • Preliminary support for Ipod accessories.

What didn’t get included:

  • The ‘natsort’ which sorts files with numbers as the number and not by ascii. This caused quite a lot of discussions and will be sorted out for 3.3
  • The Rockbox USB stack. It has been enabled in SVN build for several weeks already, but due to it causing some pretty drastic problems to some users we decided to play it safe and disable it in the release. We really hope it’ll be fine for 3.3.
  • Support for any new targets. The Gigabeat S, the Ondas and the AMS sansas aren’t terribly far away, but still not “there”.

A more detailed list can be found in the Release notes for 3.2.

Rockbox

cURL and libcurl, Open Source

Code re-use is fun

Back in 2003 I wrote up support for the HTTP NTLM authentication method for libcurl. Happy with my achievement, I later that year donated a GPL licensed version of my code to the Wget project (which also was my first contact with the signed paper stuff with the GNU/FSF to waive my copyright claims and instead hand them over). What was perhaps not so amusing with this code was when both curl and Wget 2005 were discovered to have the same security flaw due to my mistakes in this code shared by both projects!

Just recently, the neon project seems to be interested in taking on the version I adjusted somewhat for them, so possibly the third HTTP code is soon using this. Yeah I posted it on their mailing list back then so it has been sitting there in the archives maturing for some 6 years by now…

I also happened to fall over the SSH Tunnel Creator tool, which I’ve never used myself, that apparently snatched my neon donation (quite according to what the license allowed of course) and used it in their tool to do NTLM!

It’s actually not until recent years I discovered libntlm, and while I don’t know how good it was back in the days when I wrote my first NTLM stuff I generally think using existing libs is the better idea…

Open Source, Rockbox

Rockbox in gsoc 2009

Gsoc 2009

For the third year in a row, the Rockbox project participates in Google’s Summer of Code and has just now officially been accepted as a mentor organization.

This is a great chance for every student or others who have been looking for a good chance to join in an open source project, and in particular if you’re interested in embedded systems developing for (relatively) low-resource devices.

The Rockbox project also has a very large developer base, a very large community and a healthy and friendly surrounding that is welcoming to newbies.

If you’re even just the slightest intrigued, please don’t hesitate to read further on the Rockbox gsoc 2009 page.

The two previous years we’ve had four students each year with 15-20 applications.

Let’s make this a great gsoc year!

libssh2, Network

libssh2 upped a notch

1 Comment

There have been some well-founded criticism against libssh2 for a long time for its bad transfer performance when doing SCP and SFTP based transfers. Tests have proved it to be significantly slower than the openssh based alternatives in comparisons done in similar conditions. We’re talking down to a tenth(!) of the speed for SFTP.

Luckily I have a unnamed (by agreement) sponsor who pays me for improving this.

Giving it some love

I basically started out reading the SFTP code and cleaned it up as I went over it, and I added some clarifying comments etc. I found some irregularities that I fixed. Soon I could spot an obvious performance boost, like perhaps 3-4 times the previous speed. But since SFTP was painfully slow originally, this was still very crappy compared to openssh.

I then switched over to plain SCP tests. SCP is basically just an “scp” command sent over SSH and then streaming the data over a plain SSH “channel”, while SFTP is a whole additional protocol layer on top. Thus SCP is more low-level, on the actual SSH level, and the foundation on which SFTP runs anyway so getting SCP faster was fundamental.

Make it speedier

My initial tests with libssh2 1.0 showed libssh2 to download data at roughly 25% of the speed of openssh when SCPing a 1GB file from an openssh server running on localhost. The openssh client shows roughly 40MB/sec on my test box.

Also, just checking my CPU load meter while doing the libssh2 transfers showed that it certainly wasn’t hitting the roof or anything. It was barely even noticeable! Of course something was really wrong but what was it?

SSH has a lower protocol layer that does the entire encryption thing, the transport layer, but on top of that is the “channel layer” that is packet based for sending data back and forth over the transport layer. This channel thing has a receive window concept, much like TCP itself has, which tells the remote side how much data it is allowed to send until it gets further notice.

libssh2 1.0 had a very conservative windowing logic. It started with a default window size of 64KB and it upped it at every read with the same amount that was read (which then could be 1K to 16KB something depending on the app).

My remake of this was to simplify the logic, read data from the network more evenly distributed over time, update the window size much less frequent and increase the window size by magnitudes! I found that when using a window size of 38MB (600 times the previous default size!!) things started flying.

Improved

With these modifications, libssh2 transfers SCP at close to 40MB/sec! SFTP is still left behind at a “mere” 14MB/sec on the same test setup but it has its own set of problems and solutions. Now this discussion on the libssh2 list is more about how to sensibly size the window to work the best way for different situations.

SFTP is a protocol that works more on file operations. The client sends OPEN, READ and CLOSE requests and the server replies with status and data. The READ request asks for N bytes starting at offset Z so a simple implementation like libssh2 asks for chunk after chunk in a serial manner, increasing the offset as it loops over the range. This causes a back-and-forth effect that certainly does not make optimized use of the network bandwidth.

SFTP ping pong

openssh has a nifty approach to enhance throughput for SFTP: it sends off and handles multiple outstanding READ requests in parallel so that it better can keep things busy (and the reverse when doing uploads). That concept is slightly harder to do with an API like the one libssh2 offers but it is of course still quite doable. I suspect that we might achieve results somewhat faster by simply use multiple connections as then we can remain using this simplistic approach but still use the full bandwidth. (Yes, I realize multiple connections may not be feasible for all applications.)

Previous tests we’ve done with SFTP uploads using multiple connections have proven libssh2 to be on par or even better than competitors on both Windows and Mac.

Please test

I’ll leave it like this for now. I’ll be very happy if people could test this version and report findings so that we make sure this is working and stable enough to release soonish. We’ll need to do something that offers window size controlling to apps, but we’ll discuss that further on the mailing list. Join in!

logo1-250

Rockbox

Rockboxitunes

6 Comments

Frank Gevaerts and Jonas Häggqvist in the Rockbox project spent some time the other night when they should’ve been sleeping and made the Rockbox USB stack possible to fake being an iPod and thereby tricking iTunes to work with a non-iPod player quite transparently. As the picture shows, the target used here is a SanDisk Sansa e2x0 player…

frankosansa

This is still work in progress and not yet in SVN. Keep up with the bleeding edge activity in the #rockbox IRC channel on freenode!