Open Source, Free Software, and similar
Late Wednesday evening (middle European time zone) on January 7th 2009 I was up doing a live recording of the podcast show FLOSS Weekly with Leo Laporte and Randal Schwartz. This recording is now available for download as episode #51.
We chatted a bit about curl and libcurl and I think I did a decent job of keeping to the subject and not making a total fool of myself. Enjoy!
(The talk was done using skype and yes my laptop was running Windows at the time…!)
Jorge “casainho” Pinto is known in the Rockbox circles as the main guy behind the “Rockbox Player” project which strives to build their own portable music player to run Rockbox.
They’ve made some progress latetly, and they’ve now run Rockbox far enough to display stuff on their screen:
Click the image for the full photo. Cortesy of Casainho himself. “I hope to take no more than 1 month to finish the port.“
The target is using an Atmel AT91SAM9260 at 200MHz and the screen is a 12bit color 128×128 one.
Over at the IETF another group was just created named http-state (with an associated mailing list) with the specific goal:
Ultimately, the purpose of this group is to create an updated HTTP State Management Mechanism RFC (aka cookies) that will supersede the Netscape spec, RFCs 2109, 2964, 2965 then add in real-world usage (e.g. HTTPOnly), and possibly add in additional features and possibly merge in draft-broyer-http-cookie-auth-00.txt and draft-pettersen-cookie-v2-03.txt.
I’ve joined the list and I hope to follow and participate in this, as I believe the current state of HTTP cookies is a rather sorry mess and the Netscape spec is still what closest describes how cookies work in the wild. Of course I’ll do it with my libcurl experience in my luggage.
While it perhaps would be cool to join the group in more formal way, there’s no way for me to participate in that IETF meeting in San Francisco in March.
I haven’t said it here before, but I feel I really should. I’ve been an avid emacs user since I started to learn it back in 1991 on emacs 18. I worked at IBM with their RS/6000 machines at the time and I learned C on AIX with emacs as my editor.
To me there was no alternative at that time and I soon learned all the quirks, got used to things and appreciated all the beautiful parts – my fellow colleagues being emacs fans of course helped pushing me into that team. For the fun of it, I’ve checked when vim was started and I’ve learned that it got available for “unix” in 1992. And it probably was quite far away from a real emacs competitor at that time. The vi editor was of course there, but it had no C syntax indent support and it used and still uses that quirky “mode” approach that I’ve never liked.
I came from hacking the C64 to programming Amiga over to AIX and I was used to and liked full-screen editors without any particular mode switching necessary. Still today I find it a bit curious how vi (in the shape of vim) can be this popular given that (in my view) funny concept.
While I do see some of the benefits of XEmacs over GNU Emacs I always disable all menus, icons and toolbars so to me in real-life editing the differences don’t mean a lot, so I tend to go with the plain GNU Emacs.
So, I’ve pretty much used Emacs just about daily at work and in my spare time hacking since then. Even during the few times I’ve been locked in at a windows desktop I’ve managed to get a windows version installed to survive my days!
Thank you emacs team!
I had a great and relaxing winter/Christmas holiday and hence my silence here.
I’m now back up to speed, with a podcast interview done yesterday (I’ll post another entry when it gets available), I do some funded development on libcurl and libssh2 in the background while I’m spending my days at my client’s place working on a 10G traffic analyzer product.
It was rather calm during “the break” but I’ve now noticed that at least the curl project has gotten significantly increased activity again. We’re on a feature freeze now for the January release, but there seems to be at least 4 patches pending adding new stuff for the release planned to come after this (around March if things go well).
I’m happy to say that I’ve just uploaded the 1.0 release of libssh2 to sourceforge! (I must confess that I strongly dislike the “file release” thing of sourceforge but libssh2 has always been released there so I’m just continuing the tradition really…)
The changes can be read closer in the package but the main things are these:
Added libssh2_sftp_tell64()
Added libssh2_session_block_directions()
Added libssh2_channel_request_pty_size_ex()
Added libssh2_sftp_seek64()
Added the beginning of a test suite
Deprecated libssh2_base64_decode()
Fixed many bugs – possibly the biggest item really since several of the fixed bugs were of the kind that prevented the lib to do successful transfers in many cases.
This is the primary library for SSH-based communication that I know of. Note that this is not the same project as the libssh one. I once did a thorough comparison with all SSH libraries I could find, and libssh2 was then the one nearest to my feature requirements. We have since then taken it a lot further and it is now a fairly stable and good library for SSH-based transfers and generic communication.
There’s of course still (a lot of) things left to do, but here’s the 1.0 as a sign that this is now a lib ready to get used!
After three months of work since the last release, we manage to keep the schedule and ship Rockbox 3.1. The list of news since 3.0 include the following:
Of course you can find a more detailed list in the MajorChanges wiki page, and the full release notes for 3.1.
My personal contribution has been very tiny this time around and I’ve basically just built the release builds and admined the distributed build system somewhat.
Eddy Nigg found out and blogged about how he could buy SSL certificates for a domain he clearly doesn’t own nor control. The cert is certified by Comodo who apparently has outsourced (parts of) there cert business to a separate company who obviously does very little or perhaps no verification at all of the buyers.
As a result, buyers could buy certificates from there for just about any domain/site name, and Comodo being a trusted CA in at least Firefox would thus make it a lot easier for phishers and other cyber-style criminals to setup fraudulent sites that even get the padlock in Firefox and looks almost perfectly legitimate!
The question is now what Mozilla should do. What Firefox users should expect their browser to do when HTTPS sites use Comodo-verified certs and how Comodo and their resellers are going to deal with everything…
Read the scary thread on the mozilla dev-tech-crypto list.
Update: if you’re on the paranoid/safe side you can disable trusting their certificates by doing this:
Select Preferences -> Advanced -> View Certificates -> Authorities. Search for
AddTrust AB -> AddTrust External CA Root and click “Edit”. Remove all Flags.
As some of you might know, I currently work with a client doing 10G network stuff. 10G as in 10 gigabit/second Ethernet. That’s a lot of data. It’s actually so much data it’s hard to even generate network loads of this magnitude to be able to do good tests, as a typical server using SATA harddrives hardly fills a one gigabit pipe due to “slow” I/O: ordinary SATA drives don’t even reach 100MB/sec. You need RAID solutions or putting the entire thing in RAM first. So generating 10 gigabit network loads thus requires some extraordinary solutions.
Having a server that tries to “eat” a line speed 10G is a big challenge, and in fact we can’t do it as 1.25 GB/sec is just too much and yet we run a quad-core 3.00GHz Xeon thing here which is at least near the best “off-the-shelf” CPU/server you can get at the moment. Of course our software does a little bit more with the data than just receiving it as well.
Anyway, recently I’ve been experimenting with 10G cards from Myricom and when trying to maximize our performance with these beauties, I fell over the three-letter acronym DCA. Direct Cache Access. A terribly overused acronym consisting of often-used words make it hard to research and learn about! But here’s a great document describing some of the gory details:
Direct Cache Access for High Bandwidth Network I/O
Summary: it is an Intel technology for delivering data directly into the CPU’s cache, to reduce the bandwidth requirement to memory (note: it only decreases the bandwidth requirement at that moment, not the total requirement as it still needs to be read from memory into the cache, as noted in a comment below). Using this technique it should be possible to drastically reduce the time for getting the traffic. Support for this tech has been added to the Linux kernel as well since a while back.
It seems DCA is (only?) implemented in Intel’s 7300 chipset family which seems to only exist for Xeon 7300 and 7400. Too bad we don’t have one of these monsters so I haven’t been able to try this out for real yet…
Currently we can generate 10G network loads using two different approaches: one is uploading a specially crafted binary blob embedded with the FPGA image to a Xilinx-equipped board with a 10G MAC that then can do some fiddling with the packages (like increasing a counter) so that they aren’t all 100% identical. It makes a pretty good load test, even if the traffic isn’t at all shaped like the “real” traffic our product will receive. Our other approach has been even less good: upload a custom firmware to the network card and have that send the same Ethernet frame… This latter approach didn’t get better because it was a bit too complicated and badly documented on how to make a really good generator out of it. Even if I liked being able to upload custom code to my network card! 😉
Allow me to also mention that the problems with generating 10G is with small packet sizes, like 100 bytes or so as the main problem in the hardwares seem to the number of packets, not the payload part. Thus it is easier to do full line speed with 9000 bytes packets (jumbo frames) than the tiny ones we are likely to get when this product is in use by customers in the wild.
Update: this article was written in 2008. Please note that many things may have changed since then.
With the newly established three-month release cycle in the Rockbox project it means we’re about to soon do a version 3.1 release. Planned to happen in a week or so.
In preparation for that, we just now branched trunk to a 3.1 branch. Help us polish it up, update translations and correct the release notes!