Development, Open Source, Web

Shared Dictionary Compression over HTTP

Wei-Hsin Lee of Google posted about their effort to create a dictionary-based compression scheme for HTTP. I find the idea rather interesting, and it’ll be fun to see what the actual browser and server vendors will say about this.

The idea is basically to use “cookie rules” (domain, path, port number, max-age etc) to make sure a client gets a dictionary and then the server can deliver responses that are diffs computed against the dictionary it has delivered before to the client. For repeated similar contents it should be able to achieve a lot better compression ratios than any other existing HTTP compression in use.

I figure it should be seen as a relative to the “Delta encoding in HTTP” idea, although the SDCH idea seems somewhat more generically applicable.

Since they seem to be using the VCDIFF algorithm for SDCH, the recent open-vcdiff announcement of course is interesting too.

Rockbox, Security, Web

A bad move. A really bad move.

So I wrote this little perl script to perform a lot of repeated binary Rockbox builds. It builds something like 35 builds and zips them up and gives them proper names in a dedicated output directory. Perfect to do things such as release builds.

Then I wrote a similar one to build manuals and offer them too. I then made the results available on the Rockbox 3.0RC (release candidate) page of mine.

Cool, me thinks, and since I’ll be away now for a week starting Wednesday I think I should make the scripts available in case someone else wants to play with them and possibly make a release while I’m gone.

I did

mv buildall.pl webdirectory/buildall.pl.txt

… thinking that I don’t want it to try to execute as a perl script on the server so I rename it to a .txt extension. But did this work? No. Did it cause total havoc? Yes.

First, Apache apparently still thinks these files are perl scripts (== cgi scripts) on my server, even if they got an additional extension. I really really didn’t expect this.

Then, my scripts are doing a command chain similar to “mkdir dir; cd dir; rm -rf *”. It works great when invoked in the correct directory. It works less fine when the web server invokes this because someone clicked on the file I just made available to the world.

Recursive deletion of all files the web server user was allowed to erase.

Did I immediately suspect foul play and evil doings by outsiders? Yes. Did it take quite a while to restore the damages from backups? Yes. Did it feel painful to realize that I myself was to blame for this entire incident and not at all any outside or evil perpetrator? Yes yes yes.

But honestly, in the end I felt good that it wasn’t a security hole somewhere that caused it since I hate spending all that time to track it down and fix it. And thanks to a very fine backup system, I had most of the site and things back up and running after roughly one hour off-line time.

Rockbox

cURL and libcurl, Security

Security and Open Source

OWASP Sweden is arranging an event on October 6th in Stockholm Sweden to talk about security in the open source process.

I will be there doing talk about security in open source projects, in particular then how we work with security in the curl project. If you think of anything particular you would like me to address or include, feel free to give be a clue already before the event!

Rockbox

DFU mode on 2nd gen Nanos

1 Comment

Some clever hackers in the Rockbox community wrote up a tool to access the Meizu players’ DFU mode (while running Linux – which I already mentioned), and using this we can upload and run code on several Meizu targets. The code is put and executed in SDRAM only. It makes it a perfect way to test new code on it.

The Meizu players have their SoC in common with Apple’s Nano 2nd gen and Shuffle 2nd gen.

There are indications that the Nanos have such a DFU mode as well, even though we don’t currently know of any way to trigger it by will. Possibly shorting the NAND chip or destroying the firmware or similar might do it.

If you have such a broken Nano or Shuffle, please get in touch and we can do some poking around!

Obviously, there’s a DFU mode on the iphone and iPod touch that can be triggered:

“Your phone must be off, but attached via USB to the PC. Then you hold the power and “home” buttons for 10 seconds. At the ten second mark, you release the power button, but keep the “home” button pressed for another 10 seconds. At the end of that process, the phone enters DFU mode (the only way to tell is windows will tell you a USB DFU device has connected)” (thanks to GodEater)

Although I’m convinced our limited DFU experiments will not be a lot of fun on those devices (yet).

It seems iPod Classics can also go into this mode.

For the iPod Nano 2nd gen:

“To access DFU mode, reset the iPod with MENU+SELECT, then press and hold BACK+PLAY. A picture of the dock connector should appear with the Apple support URL; according to lsusb, this is DFU mode…  it seems that you have to first trash the firmware before you can access it.” (thanks to LambdaCalculus37)

Since autumn 2009, Rockbox boots and runs on the iPod Nano 2nd generation!

Licensing, Open Source

Not so public file with GPL license header

1 Comment

Here’s a license dilemma for you:

Imagine company X hosting a tarball on their public web server. There’s no publicly available link to this tarball, but if you access the URL with your browser or download tool, you can download it with no restrictions from anywhere in the world.

The tarball contains GPL code. That is, the code in question has GPL license headers (in addition to Copyright (C) by Company X notices).

If you get your hands on said code, is it to be considered GPL and thus valid to be used by a GPL-compatible open source project?

Arguments against this include that the tarball, while being accessible, may not actually have been meant for distribution and thus the license may perhaps not be the one intended for the code in the end.

What if someone would publish the link on a totally unrelated site and say “get the code [here]” and link to the above mentioned code. Wouldn’t that cause at least some people to get the code in good faith and then would the GPL apply?

(Any resemblance to a real-life scenario is purely coincidental. Names have been changed to protect the innocent.)

cURL and libcurl

curl and libcurl 7.19.0

With almost 40 described bug fixes curl and libcurl 7.19.0 come flying with a range of new things, including the following:

  • curl_off_t gets its size/typedef somewhat differently than before. This may cause an ABI change for you. See lib/README.curl_off_t for a full explanation.
  • Added CURLINFO_PRIMARY_IP
  • Added CURLOPT_CRLFILE and CURLE_SSL_CRL_BADFILE
  • Added CURLOPT_ISSUERCERT and CURLE_SSL_ISSUER_ERROR
  • curl’s option parser for boolean options reworked
  • Added –remote-name-all
  • Now builds for the INTEGRITY operating system
  • Added CURLINFO_APPCONNECT_TIME
  • Added test selection by key word in runtests.pl
  • the curl tool’s -w option support the %{ssl_verify_result} variable
  • Added CURLOPT_ADDRESS_SCOPE and scope parsing of the URL according to RFC4007
  • Support –append on SFTP uploads (not with OpenSSH, though)
  • Added curlbuild.h and curlrules.h to the external library interface

We’ve worked really hard to get this to be a really solid and fine release. I hope it’ll show.

cURL and libcurl, Security, Web

Getting cacerts for your tools

As the primary curl author, I’m finding the comments here interesting. That blog entry “Teaching wget About Root Certificates” is about how you can get cacerts for wget by downloading them from curl’s web site, and people quickly point out how getting cacerts from an untrusted third party place of course is an ideal situation for an MITM “attack”.

Of course you can’t trust any files off a HTTP site or a HTTPS site without a “trusted” certificate, but thinking that the curl project would run one of those just to let random people load PEM files from our site seems a bit weird. Thus, we also provide the scripts we do all this with so that you can run them yourself with whatever input data you need, preferably something you trust. The more paranoid you are, the harder that gets of course.

On Fedora, curl does come with ca certs (at least I’m told recent Fedoras do) and even if it doesn’t, you can actually point curl to use whatever cacert you like and since most default installs of curl uses OpenSSL like wget does, you could tell curl to use the same cacert your wget install uses.

This last thing gets a little more complicated when one of the two gets compiled with a SSL library that doesn’t easily support PEM (read: NSS), but in the case of curl in recent Fedora they build it with NSS but with an additional patch that allows it to still be able to read PEM files.

tech, open source and networking