Fun with executable extensions in viewvc

A few years ago I wrote up silly little perl script (let’s call it script.pl) that would fetch a page from a site that returns a “random URL off the internet”. I needed a range of URLs for a test program of mine and just making up a thousand or so URLs is tricky. Thus I wrote this script that I would run and allow to get a range of URLs on each invoke and then run it again later and append to the log file. It wasn’t a fancy script, but it solved my task.

The script was part of a project I got funded to work on, that was improving libcurl back in 2005/2006 so I thought adding and committing the script to CVS felt only natural and served a good purpose. To allow others to repeat what I did.

Fast forward to late 2008. The script is now browsable via viewvc on a site that… eh, doesn’t have “.pl” disabled as a cgi extension in its config! The result of course is that each time someone tries to view the script using the web interface, the web server invokes the script locally!

All of a sudden I get a mail from someone, who apparently is admin or something of the site this old script was using, and he mentions that a machine on our network is hammering his site with many requests per second (38 requests/second apparently) and asked me to stop this. It turns out a search engine crawler has indexed the viewvc output several times, and now some 8 processes or so were running this script.pl and they were all looping around getting a page, outputting the URL, getting another page…

While I think 38 requests second is a bit low to even be considered a DOS, it certainly wasn’t intended nor friendly and I was greatly surprised when I slowly realized how it all came to end up like this! Man I suck! It reminds me of my other extension mess from just a few months ago…

Maybe I’ll learn how to do things right in the future when I grow up!

libssh2 1.0!

I’m happy to say that I’ve just uploaded the 1.0 release of libssh2 to sourceforge! (I must confess that I strongly dislike the “file release” thing of sourceforge but libssh2 has always been released there so I’m just continuing the tradition really…)

The changes can be read closer in the package but the main things are these:

Added libssh2_sftp_tell64()

Added libssh2_session_block_directions()

Added libssh2_channel_request_pty_size_ex()

Added libssh2_sftp_seek64()

Added the beginning of a test suite

Deprecated libssh2_base64_decode()

Fixed many bugs – possibly the biggest item really since several of the fixed bugs were of the kind that prevented the lib to do successful transfers in many cases.

This is the primary library for SSH-based communication that I know of. Note that this is not the same project as the libssh one. I once did a thorough comparison with all SSH libraries I could find, and libssh2 was then the one nearest to my feature requirements. We have since then taken it a lot further and it is now a fairly stable and good library for SSH-based transfers and generic communication.

There’s of course still (a lot of) things left to do, but here’s the 1.0 as a sign that this is now a lib ready to get used!

Rockbox 3.1

After three months of work since the last release, we manage to keep the schedule and ship Rockbox 3.1. The list of news since 3.0 include the following:

  • A bitmap scaler was added to Rockbox, which means that album art no longer has to be pre-scaled to the correct dimensions on your computer. See AlbumArt for more information.
  • The calendar plugin which has existed for the Archos units for a long time is now available on all devices equipped with a clock.
  • The spacerocks plugin which was removed from version 3.0 due to a major bug has been brought back.
  • Optimised MP3 decoder on dual-core targets, giving several more hours of battery life in most situations.
  • Optimizations for AAC and APE decoding
  • Backlight fading is now available on most targets.
  • When recording in mono, you can now chose between recording the left or right channel, or a mix of both.
  • It is now possible to configure which items are shown in the Quick Screen.
  • Several new features were added to the WPS syntax
  • The build system received a major overhaul. This only matters for people who compile their own builds.

Of course you can find a more detailed list in the MajorChanges wiki page, and the full release notes for 3.1.

My personal contribution has been very tiny this time around and I’ve basically just built the release builds and admined the distributed build system somewhat.

Rockbox

SSL certs crash without trust

Eddy Nigg found out and blogged about how he could buy SSL certificates for a domain he clearly doesn’t own nor control. The cert is certified by Comodo who apparently has outsourced (parts of) there cert business to a separate company who obviously does very little or perhaps no verification at all of the buyers.

As a result, buyers could buy certificates from there for just about any domain/site name, and Comodo being a trusted CA in at least Firefox would thus make it a lot easier for phishers and other cyber-style criminals to setup fraudulent sites that even get the padlock in Firefox and looks almost perfectly legitimate!

The question is now what Mozilla should do. What Firefox users should expect their browser to do when HTTPS sites use Comodo-verified certs and how Comodo and their resellers are going to deal with everything…

Read the scary thread on the mozilla dev-tech-crypto list.

Update: if you’re on the paranoid/safe side you can disable trusting their certificates by doing this:

Select Preferences -> Advanced -> View Certificates -> Authorities. Search for
AddTrust AB -> AddTrust External CA Root and click “Edit”. Remove all Flags.

10G and Direct Cache Access

As some of you might know, I currently work with a client doing 10G network stuff. 10G as in 10 gigabit/second Ethernet. That’s a lot of data. It’s actually so much data it’s hard to even generate network loads of this magnitude to be able to do good tests, as a typical server using SATA harddrives hardly fills a one gigabit pipe due to “slow” I/O: ordinary SATA drives don’t even reach 100MB/sec. You need RAID solutions or putting the entire thing in RAM first. So generating 10 gigabit network loads thus requires some extraordinary solutions.

Having a server that tries to “eat” a line speed 10G is a big challenge, and in fact we can’t do it as 1.25 GB/sec is just too much and yet we run a quad-core 3.00GHz Xeon thing here which is at least near the best “off-the-shelf” CPU/server you can get at the moment. Of course our software does a little bit more with the data than just receiving it as well.

Anyway, recently I’ve been experimenting with 10G cards from Myricom and when trying to maximize our performance with these beauties, I fell over the three-letter acronym DCA. Direct Cache Access. A terribly overused acronym consisting of often-used words make it hard to research and learn about! But here’s a great document describing some of the gory details:

Direct Cache Access for High Bandwidth Network I/O

Summary: it is an Intel technology for delivering data directly into the CPU’s cache, to reduce the bandwidth requirement to memory (note: it only decreases the bandwidth requirement at that moment, not the total requirement as it still needs to be read from memory into the cache, as noted in a comment below). Using this technique it should be possible to drastically reduce the time for getting the traffic. Support for this tech has been added to the Linux kernel as well since a while back.

It seems DCA is (only?) implemented in Intel’s 7300 chipset family which seems to only exist for Xeon 7300 and 7400. Too bad we don’t have one of these monsters so I haven’t been able to try this out for real yet…

Currently we can generate 10G network loads using two different approaches: one is uploading a specially crafted binary blob embedded with the FPGA image to a Xilinx-equipped board with a 10G MAC that then can do some fiddling with the packages (like increasing a counter) so that they aren’t all 100% identical. It makes a pretty good load test, even if the traffic isn’t at all shaped like the “real” traffic our product will receive. Our other approach has been even less good: upload a custom firmware to the network card and have that send the same Ethernet frame… This latter approach didn’t get better because it was a bit too complicated and badly documented on how to make a really good generator out of it. Even if I liked being able to upload custom code to my network card! 😉

Allow me to also mention that the problems with generating 10G is with small packet sizes, like 100 bytes or so as the main problem in the hardwares seem to the number of packets, not the payload part. Thus it is easier to do full line speed with 9000 bytes packets (jumbo frames) than the tiny ones we are likely to get when this product is in use by customers in the wild.

Update: this article was written in 2008. Please note that many things may have changed since then.

Avatars by gravatar

Daniel's gravatar avatar imageI’m using one of those fancy WordPress plugins on this blog that makes use of gravatar for the avatar images that appear next to your name when you post a comment. So if you comment here on daniel.haxx.se and want to see a fancy personal image next to your wise words, skip over to gravatar.com and put up a picture of you that then will be associated with your email address.

This system does not reveal your email address to any outsider, as the avatar is received from their service simply by sending a oneway hash of your address.

This isn’t really anything new here, it’s been like this for a while but I figured I should explain it better to the few who might not have realized this yet…

One less podcast to listen to

As I’ve mentioned in the past I do enjoy listening to podcasts while doing the dishes or shopping, and now I have one less show to monitor as Linux Action Show is giving up their “long” format and is going video and only doing short audio ones.

For me who likes listening while doing other things, video podcasts are totally wrong. And then doing audio streams based on a video podcast sounds like the wrong way forward, at least if you want to provide good a audio podcast. I think this is the end of my interest of their show.

Thankfully Randal and Leo have kept up the speed of FLOSS Weekly lately!

More libcurl adoption

Some recent news showing libcurl possibly widening its user-base:

Eugene V. Lyubimkin posted a suggestion that libcurl should be used by the upcoming APT release for all ftp and http accesses!

Mr Johansen at Sun told us libcurl is being considered (via the pycurl binding) for the new OpenSolaris package manager.

perl’s widely used module for HTTP/FTP etc, called LWP, has gotten a libcurl-powered sibling called LPW-curl, which if I understand things correctly makes transfers using the traditional LWP-style and API but is powered by libcurl underneath.

Someone (not being me) registered libcurl.org. The site actually contains rather accurate info but if I disable adblock it shows lots of ads on the page though so I guess that’s why the page exists… (googling for “libcurl” now shows this site among the 5-6 first hits, which surprises me…)

More?

c-ares 1.6.0

With a few bug fixes and general improvements, c-ares 1.6.0 was released just now with these new additions:

  • support for the glibc “rotate” resolv.conf option (or ARES_OPT_ROTATE)
  • ares_gethostbyname_file()
  • ares_dup()
  • ares_set_socket_callback()

(the man pages for the new functions are not yet available on the web site but I’m meaning to get to that soonish)