libssh2, Network, Security

Making SFTP transfers fast

12 Comments

SFTP, the SSH File Transfer Protocol, is a misleading name. It gives you the impression that it might be something like a secure version of FTP, perhaps something like FTPS but modeled over SSH instead of SSL. But it isn’t!The OpenSSH fish

I think a more suitable name would’ve been SNFS or FSSSH. That is: networked file system operations over SSH, as that is in fact what SFTP is. The SFTP protocol is closer to NFS in nature than FTP. It is a protocol for sending and receiving binary packets over a (secure) SSH channel to read files, write files, and so on. But not on the basis of entire files, like FTP, but by sending OPEN file as FILEHANDLE, “WRITE this piece of data at OFFSET using FILEHANDLE” etc.

SFTP was being defined by a working group with IETF but the effort died before any specification was finalized. I wasn’t around then so I don’t know how this happened. During the course of their work, they released several drafts of the protocol using different protocol versions. Version 3, 4, 5 and 6 are the ones most used these days. Lots of SFTP implementations today still only implement the version 3 draft. (like libssh2 does for example)

Each packet in the SFTP protocol gets a response from the server to acknowledge it was received. It also includes an error code etc. So, the basic concept to write a file over SFTP is:

[client] OPEN <filehandle>
[server] OPEN OK
[client] WRITE <data> <filehandle> <offset 0> <size N>
[server] WRITE OK
[client] WRITE <data> <filehandle> <offset N> <size N>
[server] WRITE OK
[client] WRITE <data> <filehandle> <offset N*2> <size N>
[server] WRITE OK
[client] CLOSE <filehandle>
[server] CLOSE OK

This example obviously assumes the whole file was written in three WRITE packets. A single SFTP packet cannot be larger than 32768 bytes so if your client could read the entire file into memory, it can only send it away using very many small chunks. I don’t know the rationale for selecting such a very small maximum packet size, especially since the SSH channel layer over which SFTP packets are transferred over doesn’t have the same limitation but allows much larger ones! Interestingly, if you send a READ of N bytes from the server, you apparently imply that you can deal with packets of that size as then the server can send packets back that are N bytes (plus header)…

Enter network latency.

More traditional transfer protocols like FTP, HTTP and even SCP work on entire files. Roughly like “send me that file and keep sending until the entire thing is sent”. The use of windowing in the transfer layer (TCP for FTP and HTTP and within the SSH channels for SCP) allows flow control to work without having to ACK every single little packet. This is a great concept to keep the flow going at high speed and still allow the receiver to not get drowned. Even if there’s a high network latency involved.

The nature of SFTP and its ACK for every small data chunk it sends, makes an initial naive SFTP implementation suffer badly when sending data over high latency networks. If you have to wait a few hundred milliseconds for each 32KB of data then there will never be fast SFTP transfers. This sort of naive implementation is what libssh2 has offered up until and including libssh2 1.2.7.

To achieve speedy transfers with SFTP, we need to “pipeline” the packets. We need to send out several packets before we expect the answers to previous ones, to make the sending of an SFTP packet and the checking of the corresponding ACKs asynchronous. Like in the above example, we would send all WRITE commands before we wait for/expect the ACKs to come back from the server. Then the round-trip time essentially becomes a non-factor (or at least a very small one).

libssh2

We’ve worked on implementing this kind of pipelining for SFTP uploads in libssh2 and it seems to have paid off. In some measurements libssh2 is now one of the faster SFTP clients.

In tests I did over a high-latency connection, I could boost libssh2’s SFTP upload performance 8 (eight) times compared to the former behavior. In fact, that’s compared to earlier git behavior, comparing to the latest libssh2 release version (1.2.7) would most likely show an even greater difference.

My plan is now to implement this same concept for SFTP downloads in libssh2, and then look over if we shouldn’t offer a slightly modified API to allow applications to use pipelined transfers better and easier.

Linux

What is Android anyway

2 Comments

Android, the software environment, has gotten a lot of press, popularity and interest from all over lately. People on the streets know there’s something called Android, companies know people know and so on. Everyone (well apart from a few competitors perhaps) likes Android it seems.

Being an embedded guy I like keeping an eye on the embedded world and Android being pretty embedded this at least tangents my universe. What is Android anyway? android.com says Android is “an open-source software stack for mobile devices, and a corresponding open-source project led by Google“. Not very specific, is it?

Android on different devicesAndroid

You can already find Android on mobile phones, media players, tablets, TVs and more. Very soon we’ll see it in car infotainment equipment, GPSes and all sorts of things that have displays. Clearly Android is not only for mobile phones and not even necessarily for mobile things. TVs often aren’t that mobile… And not touchscreen either.

Android with tweaks

The fact that there hardly are two Android installs completely alike is frequently debated. Lots of manufacturers patch and change the look and feel of Android to differentiate. Android is not associated with any particular look or feel quite clearly.

Samsung Galaxy Tab

Android with binary drivers

Almost all Android installations you get on the Android phones and devices today have a fair amount of closed, proprietary drivers. It means that even if the companies provide the source code for all the free parts in time (which they sometimes have a hard time to do it seems), there are still parts that you don’t get to see the code for. So getting a complete Android installation from source to run on your newly purchased Android device can be a challenge. Also, it shows that Android can consist of an unspecified amount of extra proprietary pieces that don’t disqualify it from being Android.

Android without apps

I have friends who work on devices where the customer has request them to run Android, although they don’t have any ability to run 3rd party apps. Android is then only there for the original developers writing specific code for that device. Potential buyers of that device won’t get any particular Android benefits that they might be used to from their mobile phones running Android, as the device is completely closed in all practical aspects.

Android without market

Devices that don’t meet Google’s demands and don’t get to be “Google certified” don’t get to install the google market app etc, but at least a company who wants to can then in fact install their own market app or offer another way for customers to get new apps. The concept of getting and installing apps aren’t bound to the market app being there. In fact, I’ve always been expecting that some other companies or parties would come along and provide an alternative app that would offer apps even to non-google-branded devices but obviously nobody has yet stepped up to provide that in any significant way.

Android without Java

I listened  to a talk at an embedded conference recently where the person did a 40 minute talk on why we should use Android on our embedded systems. He argued that Android was (in this context) primarily good for companies because it avoids GPL and LGPL to a large extent. He talked about using “Android” in embedded devices and cutting out everything that is java, basically only leaving the Linux kernel and the BSD licensed bionic libc implementation. Of course, bionic may now also provide features to the rest of the system that glibc and uClibc do not, they being designed as more generic libcs.

Personally, I would never call anything shipped without the Java goo layers to be Android. But since it was suggested, I decided to play with the idea that a platform can be “Android” even without Java…

mini2440

That particular license-avoiding argument of course was based on what I consider is a misunderstanding. While yes, lots of companies have problems with or are downright scared of the GPL and LGPL licences, but I’ve yet to meet a company who have any particular concerns about the licensing of the libc. I regularly meet and discuss with companies who have thoughts and worries about GPL in the kernel and they certainly often don’t like *GPL in regular libraries that they linked with in their applications. I have yet to find a customer who is worried about the glibc or uClibc licenses.

In fact, most embedded Linux customers also happily run busybox that is GPL although we know from history that many companies do that in violation with the license rules, only to get the lawyers running after them.

HTC Magic

Android is what?

As far as I know Android is a trademark of some sorts, and so is Linux. If you can run an “Android” that is just a kernel and libc (and I’m not saying this is true beyond doubt because I’ve not heard anyone authoritative say this), isn’t that then basically a very very small difference to any normal vanilla embedded Linux?

The latter examples above are even without any kind of graphical UI or user-visible interface, meaning that particular form of “Android” can just as well run your microwave or your wifi router.

Without the cruft can we change the kernel?

The Android team decided that a bunch of changes to the Linux kernel are necessary.to make Android. The changes have been debated back and forth, some of them were merged into mainline Linux only to later get backed out again while the greater part of them never even got that far. You cannot run a full-fledged Android system on a vanilla kernel: you need the features the patches introduce.

If we’re not running all the java stuff do we still need those kernel patches? Is bionic made to assume one or more of them? That brings me to my next stepping stone along this path:

Without the patches can we change libc?

If we don’t run the java layers, do we really have to run the bionic libc? Surely the Android kernel allows another libc and if we use another libc we don’t need the Android kernel patches – unless we think they provide functionality and features that really improve our device.

Android is the new Linux?ASUS M2NPV MX Motherboard

Android as a name to describe something really already is just as drained as Linux. All these Android devices are just as much Linux devices and just as a “linux device” doesn’t really tell anything about what it is actually more than what kernel it runs, neither does it seem “Android device” will mean in the long run or perhaps already.

Android however has reached some brand recognition already among mortals. I think Android is perceived as something more positive in the minds of the consumer electronic consumers than Linux is. Linux is that OS that nobody uses on their desktops, Android is that cool phone thing.

I will not be the slightest surprised if we start to see more traditional Linux systems call themselves Android in the future. Some of them possibly without changing a single line of code. Linux one day, Android the next. Who can tell the difference anyway? Is there a difference?

Electronics, libssh2, Network, Security

Re-evaluating the criticism

2 Comments

libssh2

A long while ago I posted my first version of the comparison of libssh vs libssh2. I have since then kept it updated and modified it over time. (Reminder: I am the libssh2 maintainer)

In that page, I included the performance differences I had measured which at the time showed libssh2 to be significantly faster when doing SCP operations.

The libssh guys always claimed I was wrong:

Please don’t be ridiculous. No competent network developer will take you seriously when you tell that libssh2 is 2.3 times faster that libssh.

and have even used rather harsh words when saying so.

you read this FUD page on the libssh2 website. I don’t want to start arguing here, the page is complete crap

(These two quotes are from the two leading libssh developers.)

Due to their complaints I withdrew the mentioning of the speed differences from the comparison page. Maybe I had done something wrong after all and since I didn’t care properly to go back and verify my methods and redo everything, I decided to just take it off until I have more backing or more accurate tests.

Fast forward to current time and Mark Riordan does his extensive performance tests of various SSH/SFTP implementations. He mailed the libssh mailing list about it, and his test results are interesting. I’m including it below for easier reading and just in case Mark’s original won’t be around as long as this.

It repeats very similar numbers to mine and shows the same speed difference that I was told cannot happen. Isn’t that funny? Am I still ridiculous?

SSH file transfer performance

The following table summarizes performance of SSH clients.

LAN: 1 Gbit/sec WAN: 6 Mb down, 0.9 Mb up
Solaris x86 server
Client Client OS Server Comp
Enable		 File
Cmp		 UL MB/s DL MB/s UL MB/s DL MB/s
libssh2 Win Solaris No No 0.147 12.2
libssh2 Win Solaris Yes No
libssh2 Linux Solaris No No 0.82 11.8
libssh2 Linux Solaris Yes No
libssh 0.4.6 Win Solaris No No
Bitvise Tunnelier Win Solaris No No 13.50 3.95
Bitvise Tunnelier Win Solaris Yes No 8.541 10.2
psftp Win Solaris No No 9.4 5.06 or 0.46
WS_FTP 12.3 Win Solaris No No 8.07 7.65
Ubuntu sftp Linux Solaris ? No 29.6 11.5
Linux server
libssh2 Win Linux No No 9.5 8.1 0.059 0.26
libssh2 Win Linux Yes No
libssh2 Linux Linux No No 7.4 7.4 0.083 0.267
libssh 0.4.6 Win Linux No No 15.4 2.8 0.10 0.13
libssh 0.4.6 Linux Linux No No 8.97 2.8 0.099 0.189
libssh 0.4.6 Linux Linux Yes Yes 19.7 3.3
libssh latest Win Linux No No 14.1 1.38
psftp Win Linux No No 4.59 6.58 0.070 0.10
WS_FTP 12.3 Win Linux No No 23.0 8.5 0.113 0.361
Bitvise Tunnelier Win Linux No No
Ubuntu sftp Linux Linux No No 16.2 6.6 0.11 0.51

What about SFTP?

It should be noted that in my original claim and in this test above we speak SSH speeds (like SCP), not SFTP. SFTP has its own slew of problems and libssh2 is in fact not very good at doing SFTP speedily yet. We have work in progress to improve this situation, but we’re not there yet. I’ll post a follow-up on SFTP speeds soonish as things have been developing nicely in there recently.

What about speeds compared to other clients?

libssh2 is not fully on par with for example openssh when it comes to raw SCP speed, but it is in the same “neighborhood”.

Haxx

Getting a new look

2 Comments

Haxx logo

Recently we have refreshed our logo design, and subsequently we’ve now also refreshed our web site to use this new look and design that already have influenced how our presentations, business cards and more look and will look in the future.

When I say “we” did it, there should be little surprise that we did engage with someone else to do this for us, since all of us at Haxx are quite incapable of doing designs that look tasteful.

We’re quite happy with the new look. We like the cool blue colors. More machine, less human, less colorful. This logo should also work slightly better in grayscale than before and getting rid of the border will also make it easier to use on various merchandise.

Linux

my first embedded Linux course

I’m happy to announce that I did my first ever full-day training course for eleven embedded developers Monday November 15th 2010. I had the pleasure to write all the materials myself, come up with three exercises for them and then actually stand in front of the team and deliver a complete session 9 – 17.

Let's say this illustrates Embedded SystemsI did my day as part of a three-day course, and I got to do the easy part: user-space development. My day covered the topics of: Embedded Linux development introduction, how to build, autobuilding, how to run, git basics, debugging, profiling and finally some brief words on testing.

Doing stuff outside of your ordinary schedule and “comfort zone” is certainly a bit scary and encouraging and that’s the sort of thing that makes you grow as a person and as a professional. I mean, I know the topics by heart and certainly pretty much without even thinking (I’ve been working with embedded systems for over 17 years!), but from that into making a decent training course is not just a coffee break worth of work.

I was quite happy and satisfied that I pretty much kept to the program, I managed to go through all the topics I had set myself out to, I think we had a really nice conversation going during the day and the audience gave me really good feedback and high “grades” in the evaluation forms they filled in before they left. Of course there were flaws in the presentation and I got some valuable ideas from my audience on how to improve it.

Now I feel like doing it again!

Open Source, Web

s/Firefox/Chrome/g

3 Comments

Google Chrome BallA few weeks ago I decided to give Chrome a good ride on my main machine, a Debian Linux unstable. I use it a lot, every day, and I of course use my browser during a large portion of my time in front of it. I’m a long time Firefox fan and when I’ve heard and read other people converting I’ve always thought it’d be hard for me due to my heavy use of certain plugins, old habits and so on.

(Of course, in Debian lingo the browsers are actually called Chromium and Iceweasel, but I’ve decided to ignore that fact in this post.)

Here’s the story on how it went, what’s good with Chrome and what’s lacking in comparison to Firefox. As compared on my Linux box here.

Obvious benefints:

  • Less wasted window/screen estate. The tabs up in the window title is brilliant.
  • Faster. It’s generally faster in almost every aspect, but what’s most noticeable is when starting it.
  • Less memory hungy. At times I’ve found my Firefox installation to spend an annoying amount of my precious RAM (I have 4GB installed) and even though I would expect Chrome’s a process-per-tab concept to be more expensive memory wise, I’ve had less such problems with it.
  • The unified address/search bar, back to how Firefox once had it, is only sensible.
  • In my Firefox I’ve had two minor quirks for a while that have annoyed me: 1) when I start to search for something, I get a few seconds “freeze” immediately after I’ve started searching. Like I enter a few letters, waaaaaaait, then I can continue. This is certainly nothing life-threatening or something I can’t live through but it is annoying. 2) I occasionally get problems with flash video playbacks that the video pause or studder, most often a few seconds into it. Chrome has not given me these quirks.
  • Mailman! I administrate more than 20 mailing lists on the same host (cool.haxx.se) using mailman. Each list has iFirefox Ballts own URL and its own password. But Firefox just cannot remember them separately!!! These are pages I visit several times each day to ack or reject posts etc. Chrome remembers the passwords excellently for all the individual lists. This makes me a much happier person.

Problems I didn’t get:

  • The adblock version for Chrome is as good. I’m not sure exactly how well they compare but I haven’t noticed anything that’s given me reason to get annoyed.
  • The resizeable text edit areas in Chrome is excellent and removes the need for some of the fancier edit plugins in Firefox.

Things still nicer in Firefox:

  • I love the plugin to force unknown content-types to still be displayed by the browser. Far too many resources are still done using the wrong one and Chrome’s only option is to save it locally and then force me to run a local tool to display the file. Sure, it works fine but when I want to do that on many files it gets tedious.
  • In general Chrome, is a bit worse at handing content it doesn’t know about. I’ve managed to fiddle with my /etc/mozpluggerrc so that at least PDFs are now saved instead of saying “missing plug-in” but so far I’ve failed to get evince to display them directly. Even if it still is possible to make it happen, it is certainly a bit quirky to have to manually edit a text file to make it happen…

Conclusion

I’ll be running Chrome here now for a while!

Network, Web

Future transports

On Sunday morning during FSCONS 2010, in the room “Torg 4 South” I did a 30 minute talk about a few future, potentially coming network protocols for transport. A quick look at the current state, some problems of today and 4 different technologies that have been and are being developed to solve the problem.

I got a fair amount of questions and several persons approached me afterwards to make sure they got a copy of my slides.

The video recording is hopefully going to be made available later on, but until then you can read the slides below and imagine my Swedish  accent talking about these matters!

Future transports

You can also download the slides directly as a PDF.
IT Politics, Open Source

FSCONS 2010 day 2

2 Comments

[continued from FSCONS 2010 day 1]

With the previous night’s social event ending fairly late and involving a fair amount of good beers, it was nice to be able to sleep in a bit and have one of those great hotel breakfasts in a slow and relaxed manner.

I then checked out from the hotel and walked over to the venue, this time not as mislead by google maps’ directions as I was yesterday.

The economics of open innovation and FOSS. was the first session I attended and the talker Karthik Jayaraman did a good job of explaining and showing how things can happen fast and why and he did some interesting predictions of the future.

I followed Kathik’s talk in the same room with my 30 minute session Future transports in which I discussed and explained a bit about transport protocols today and what might come tomorrow.

Glyn Moody is a bit of a celebrity (he has for example written several books) and you can tell he’s done this before. He’s an excellent speaker and as he’s a native English person he has a bit of an edge compared to most speakers at this conference. Mr Moody got the biggest room at FSCONS filled up to the last available chair and there were still a bunch who had to sit on the floor.

Glyn talked about Ethical Monopolies, the history of patents and copyright and how they have changed and how they today no longer are even close to having the purposes they were created for.

He advocates that we stop talking about “Intellectual Properties” but instead refer to them as “Intellectual government-granted monopolies” as that is a much better use of words when these subjects are brought up. The ordinary person thinks people should be able to keep properties, but would in many cases object to (more) monopolies.

The last session I got to enjoy this year was Mikael von Knorring’s “Who are the free users” which did present a good view of things but I wasn’t very focused during the talk so I’ll refrain from judging it in any direction.

I took a last stroll over to the cafeteria area where I found some friends, said hello and good bye and then I took off towards the train station and my 3+ hours train ride back to Stockholm on the other coast of Sweden. (It could be noted that I left early, there were at least two session slots that I missed.)

My head is packed with impressions. I met lots of great people and friends, both old and many new ones. We had awesome discussions and I hope at least some of the ideas that were brought up will be turned into reality. I will post more about those here if/when they happen.

The future for Free Software (in the Nordic region) is bright!

cURL and libcurl, Network

Scalable application layer transfers

At FSCONS 2010 I had the pleasure to do a talk about how to make your client-side networking applications really scale when upping the number of simultaneous connections. Including some details that libcurl will support you all the way!

My talk was named “Scalable application layer transfers” and the slides from it is available online. See below. Hopefully the video recording of it will appear later and I’ll post a  follow-up then. A little extra bonus material as background would be my poll vs select vs event-based article.

As I mentioned in a previous post, the room was shock full when I started preparing my equipment for the talk since the session before me was a keynote, but by the time I actually starter presenting there were only the limited set of hardcore geeks left.

In the FSCONS program there were several talks over the weekend about women in FOSS and so on, while I on the other hand certainly only contributed to enforcing the stereotypes by being white, male, middle-aged, very techy and I delivered my two speeches for audiences in which I believe not a single woman attended. Whether I am part of the problem or the solution we can discuss in a separate post later on… 🙂

Scalable application layer transfers

Open Source

FSCONS 2010 day 1

07:02: The alarm of my mobile never rang, because I was already up. I got to play with my two kids a while before the taxi arrived to pick me up at 07:30.

The X2000 train from Stockholm to Göteborg took off exactly on schedule and we were off. I learned that the on-board Internet service wasn’t possible to sign on to with Chrome but I had to fire up my good old Firefox for it. Going with first class X2000 offer free Internet all the way, and free coffee. Two of my favorite frees.

The train arrived only 10 minutes late in Gothenburg. I took a taxi over to my hotel, checked in, put my smaller laptop in my backpack and walked over to the FSCONS venue.

After having had lunch and caught up with some friends, I sat down in the big audience listening to the presentation about the Inhana project, by Kyrah. Interactive storytelling and about helping female artists in Syria to play/work with technology in the form av Arduino boards.

Kyrah had the room full. Not that many remained when I entered the stage after her and did my talk on scalable application layer transfers.

I followed up with a cup of coffee after some private discussions on SCTP and how to do fast transfers in the Tor project and then I headed towards the talk about data structures in the Linux kernel by Allesandro Rubini.

Mr Rubini is a long-time involved Linux kernel hacker (and well known co-author of the Linux Device Drivers bible) and in a very casual and effective style he taught us how we can use regular Linux code for lists and trees in a GPL licensed project and how the clever container_of macro works. To me, its biggest drawback is that it relies on a gcc-specific feature: typeof(), but otherwise it is a beautiful craftsmanship.

Allesandro brought down the biggest spontaneous applause when he responded to someone’s question “but couldn’t you also do this using templates in C++?” by suitably and appropriately bashing C++…

.Allesandro Rubini

Anders Arnholm followed along in the embedded track and he talked about using Linux in the automotive world and I think many of us thought the best part of his talk was the numbers and comparisons he had when trying different flash file systems to increase boot performance and really cut down startup time to a minimum. The initial kernel startup time was 6 something seconds when using JFFS2 and they managed to get down to below 200 milliseconds with the use of the AXFS (Advanced XIP Filesystem, where XIP is short for execute in place).

Anders Arnholm

boot-times-axfs

… after Anders’ talk I followed the crowds, got a seat in a bus and we were transported over to the social event. I mentioned a little bit about that in my previous post, the award for me.

My FSCONS 2010 day 2 entry will be posted within shortly.

tech, open source and networking