I did a talk today at Fosdem with this title. The room only had 48 seats and it was completely packed with people standing everywhere it was possible around the seated guys.
The English slides from my talk are below. It was also recorded on video so I hope I’ll be able to post once it becomes available online
SFTP, the SSH File Transfer Protocol, is a misleading name. It gives you the impression that it might be something like a secure version of FTP, perhaps something like FTPS but modeled over SSH instead of SSL. But it isn’t!
I think a more suitable name would’ve been SNFS or FSSSH. That is: networked file system operations over SSH, as that is in fact what SFTP is. The SFTP protocol is closer to NFS in nature than FTP. It is a protocol for sending and receiving binary packets over a (secure) SSH channel to read files, write files, and so on. But not on the basis of entire files, like FTP, but by sending OPEN file as FILEHANDLE, “WRITE this piece of data at OFFSET using FILEHANDLE” etc.
SFTP was being defined by a working group with IETF but the effort died before any specification was finalized. I wasn’t around then so I don’t know how this happened. During the course of their work, they released several drafts of the protocol using different protocol versions. Version 3, 4, 5 and 6 are the ones most used these days. Lots of SFTP implementations today still only implement the version 3 draft. (like libssh2 does for example)
Each packet in the SFTP protocol gets a response from the server to acknowledge it was received. It also includes an error code etc. So, the basic concept to write a file over SFTP is:
[client] OPEN <filehandle>
[server] OPEN OK
[client] WRITE <data> <filehandle> <offset 0> <size N>
[server] WRITE OK
[client] WRITE <data> <filehandle> <offset N> <size N>
[server] WRITE OK
[client] WRITE <data> <filehandle> <offset N*2> <size N>
[server] WRITE OK
[client] CLOSE <filehandle>
[server] CLOSE OK
This example obviously assumes the whole file was written in three WRITE packets. A single SFTP packet cannot be larger than 32768 bytes so if your client could read the entire file into memory, it can only send it away using very many small chunks. I don’t know the rationale for selecting such a very small maximum packet size, especially since the SSH channel layer over which SFTP packets are transferred over doesn’t have the same limitation but allows much larger ones! Interestingly, if you send a READ of N bytes from the server, you apparently imply that you can deal with packets of that size as then the server can send packets back that are N bytes (plus header)…
Enter network latency.
More traditional transfer protocols like FTP, HTTP and even SCP work on entire files. Roughly like “send me that file and keep sending until the entire thing is sent”. The use of windowing in the transfer layer (TCP for FTP and HTTP and within the SSH channels for SCP) allows flow control to work without having to ACK every single little packet. This is a great concept to keep the flow going at high speed and still allow the receiver to not get drowned. Even if there’s a high network latency involved.
The nature of SFTP and its ACK for every small data chunk it sends, makes an initial naive SFTP implementation suffer badly when sending data over high latency networks. If you have to wait a few hundred milliseconds for each 32KB of data then there will never be fast SFTP transfers. This sort of naive implementation is what libssh2 has offered up until and including libssh2 1.2.7.
To achieve speedy transfers with SFTP, we need to “pipeline” the packets. We need to send out several packets before we expect the answers to previous ones, to make the sending of an SFTP packet and the checking of the corresponding ACKs asynchronous. Like in the above example, we would send all WRITE commands before we wait for/expect the ACKs to come back from the server. Then the round-trip time essentially becomes a non-factor (or at least a very small one).
We’ve worked on implementing this kind of pipelining for SFTP uploads in libssh2 and it seems to have paid off. In some measurements libssh2 is now one of the faster SFTP clients.
In tests I did over a high-latency connection, I could boost libssh2’s SFTP upload performance 8 (eight) times compared to the former behavior. In fact, that’s compared to earlier git behavior, comparing to the latest libssh2 release version (1.2.7) would most likely show an even greater difference.
My plan is now to implement this same concept for SFTP downloads in libssh2, and then look over if we shouldn’t offer a slightly modified API to allow applications to use pipelined transfers better and easier.
A long while ago I posted my first version of the comparison of libssh vs libssh2. I have since then kept it updated and modified it over time. (Reminder: I am the libssh2 maintainer)
In that page, I included the performance differences I had measured which at the time showed libssh2 to be significantly faster when doing SCP operations.
The libssh guys always claimed I was wrong:
Please don’t be ridiculous. No competent network developer will take you seriously when you tell that libssh2 is 2.3 times faster that libssh.
and have even used rather harsh words when saying so.
you read this FUD page on the libssh2 website. I don’t want to start arguing here, the page is complete crap
(These two quotes are from the two leading libssh developers.)
Due to their complaints I withdrew the mentioning of the speed differences from the comparison page. Maybe I had done something wrong after all and since I didn’t care properly to go back and verify my methods and redo everything, I decided to just take it off until I have more backing or more accurate tests.
Fast forward to current time and Mark Riordan does his extensive performance tests of various SSH/SFTP implementations. He mailed the libssh mailing list about it, and his test results are interesting. I’m including it below for easier reading and just in case Mark’s original won’t be around as long as this.
It repeats very similar numbers to mine and shows the same speed difference that I was told cannot happen. Isn’t that funny? Am I still ridiculous?
SSH file transfer performance
The following table summarizes performance of SSH clients.
LAN: 1 Gbit/sec WAN: 6 Mb down, 0.9 Mb up Solaris x86 server Client Client OS Server Comp
EnableFile
CmpUL MB/s DL MB/s UL MB/s DL MB/s libssh2 Win Solaris No No 0.147 12.2 libssh2 Win Solaris Yes No libssh2 Linux Solaris No No 0.82 11.8 libssh2 Linux Solaris Yes No libssh 0.4.6 Win Solaris No No Bitvise Tunnelier Win Solaris No No 13.50 3.95 Bitvise Tunnelier Win Solaris Yes No 8.541 10.2 psftp Win Solaris No No 9.4 5.06 or 0.46 WS_FTP 12.3 Win Solaris No No 8.07 7.65 Ubuntu sftp Linux Solaris ? No 29.6 11.5 Linux server libssh2 Win Linux No No 9.5 8.1 0.059 0.26 libssh2 Win Linux Yes No libssh2 Linux Linux No No 7.4 7.4 0.083 0.267 libssh 0.4.6 Win Linux No No 15.4 2.8 0.10 0.13 libssh 0.4.6 Linux Linux No No 8.97 2.8 0.099 0.189 libssh 0.4.6 Linux Linux Yes Yes 19.7 3.3 libssh latest Win Linux No No 14.1 1.38 psftp Win Linux No No 4.59 6.58 0.070 0.10 WS_FTP 12.3 Win Linux No No 23.0 8.5 0.113 0.361 Bitvise Tunnelier Win Linux No No Ubuntu sftp Linux Linux No No 16.2 6.6 0.11 0.51
It should be noted that in my original claim and in this test above we speak SSH speeds (like SCP), not SFTP. SFTP has its own slew of problems and libssh2 is in fact not very good at doing SFTP speedily yet. We have work in progress to improve this situation, but we’re not there yet. I’ll post a follow-up on SFTP speeds soonish as things have been developing nicely in there recently.
libssh2 is not fully on par with for example openssh when it comes to raw SCP speed, but it is in the same “neighborhood”.
We’ve mostly been fixing bugs and making things internally look better in the libssh2 source code during the recent months so the new release I just uploaded, called version 1.2.7, isn’t really exciting to any particular level for outsiders. Existing users should however be fairly happy as we’ve addressed a fair bunch of bugs and some of them have been annoying us in the project for a long time.
I’m convinced this is the best libssh2 release we’ve ever made.
The list of bug-fixes include these:
If you find other bugs or have patches, just bring them all to us!
Many moons ago I created a little tool I named roffit. It is just a tiny perl script that converts a man page written in the nroff format to good-looking HTML. I should perhaps also add that I didn’t find any decent alternatives then so I wrote up my own version. I’ve been using it since in projects such as curl, c-ares and libssh2 to produce web versions of the docs.
It has just done its job and I haven’t had any needs to fiddle with it. The project page lists it as last modified in 2004, even though I actually moved it to a sourceforge CVS repo back in 2007.
Just a few days ago, I got emailed and was notified that Debian has it included as a package in the distribution and someone was annoyed on some particular flaws.
This resulted in a bunch of bugs getting submitted to the Debian bug tracker, I started up the brand new roffit-devel mailing list to easier host roffit discussions and I switched over the CVS repo to a git one on github.
If you like seeing man pages turned into web pages, consider joining up and help us improve this thing!
In a discussion on the libssh2 mailing list, the founder and maintainer of the libssh project got involved. The actual discussion is not what I want to talk about here, but something he touched on in one of his mails:
“I think a bit of competition in open source is fair and leads to innovation”
Recall that there are only two existing SSH libraries written in C that are free and open. These two libs have roughly the same features, the same goals and a lot of other similarities.
So, does it help open source projects to have an identified and known competitor to compete against and to try to steal users from and to try to outperform in other ways? Is there any research done that proves this theory?
Or is it just so that we have a number of developers with a certain amount of time, and if we divide those in two or more groups we therefore make sure that neither is advancing as fast as they could if all those persons would participate in the same project? Open Source projects are so often driven and developed by volunteers, and within a given area (say people interested in a SSH library) there is only a given amount of people and to make the best use of that limited set, wouldn’t it be better if they all would work together in a single project?
Would OpenBSD, NetBSD and FreeBSD have been better off by not having split up? Would KDE and GNOME have reached further today if they had been a united project?
I don’t know the answers here, and most probably the answer isn’t very clear or binary applying to all cases anyway as I bet all situations are slightly different and thus should be considered separately.
In the history of FOSS, many forked projects end up getting merged again but we don’t often see two independently created projects merge. I guess the Compiz and Beryl fusion is an exception. I think the sense of “my pet project” is often too strong, not to mention that different licenses and different development cultures make mergers hard to take place.
Look, I’m not really advocating that libssh2 and libssh should merge at this point. I’m just playing with the idea and trying to see the issue from different angles.
There are of course several things that would speak against projects to merge: different views on what licenses that are suitable, religious things such as how to indent source code or what build system to use. Quite possibly also other social aspects: development and team “culture” and behavior and why not just the “Not Invented Here” syndrome – it isn’t always that easy to give up what you made yourself or to appreciate someone else’s work.
The team behind libssh2 isn’t very big, but we’ve managed to yet again ship a new release (version 1.2.3) that adds new features such as support for SSH-agent and the new libssh2_trace_sethandler() function, while also fixing a few bugs.
Enjoy!
For many years I’ve always kept up to speed with my commitments in my primary open source projects. I’ve managed to set aside enough time to close the bug reports as fast as they have poured in. This, while still having time to work on new features every now and then.
During this last year (or so) however, I’ve come to realize that I no longer can claim to be in that fortunate position and I now find myself seeing the pile of open bugs get bigger and bigger over time. I get more bug reports than I manage to close.
There are of course explanations for this. In both ends of the mix actually. I’ve got slightly less time due my recent decision to go working for Haxx full-time, and how I’ve decided to focus slightly more on paid work which thus leads to me having less time for the unpaid work I’m doing.
Also, I’ve seen activity raise in the curl project, in the libssh2 project and in the c-ares project. All of these projects have the same problem of various degrees: a lack of participating developers working on fixing bugs. Especially bugs reported by someone else.
Since this situation is still fairly new to me, I need to learn on how to adapt to it. How to deal with a stream of issues that is overwhelming and I must select what particular things I care about and what to “let through”. This of course isn’t ideal for the projects but I can’t do much more than proceed to the best of my ability, to try to make people aware of that this is happening and try to get more people involved to help out!
Don’t get fooled by my focus on “time” above. Sometimes I even plainly lack the energy necessary to pull through. It depends a lot on the tone or impression I get from the report or reporter how I feel, but when a reporter is rude or just too “demanding” (like constantly violating the mailing list etiquette or just leaving out details even when asked) I can’t but help to feel that at times working as a developer during my full-day paid hours can make it a bit hard to then work a couple of hours more in the late evening debugging further.
The upside, let’s try to see it as a positive thing, is that now I can actually “punish” those that clearly don’t deserve to get helped since I now focus on the nice people, the good reports, the ones which seem to be written by clever people with an actual interest to see their problems addressed. Those who don’t do their part I’ll just happily ignore until they shape up.
I will deliberately just let issues “slip through” and not get my attention and require that if they are important enough people will either report it again, someone else will step up and help fix them or perhaps someone will even consider paying for the fix.
I also did a talk about libssh2 at FSCONS 2009 (in addition to the Rockbox talk) and the slides from it are now available to browse or download.
The talk was recorded on video, and I’ll make sure to let you know as soon as it is available online somewhere. That might take a while.
Remember how I mentioned back in August thatI held a libssh2 talk at Slackathon? The other day I recalled that I never saw that video recording so I tracked it down and found it. So while you’re waiting to see the video of my libssh2 or Rockbox talks I did today at FSCONS, you can enjoy a slightly shorter version:
