Category Archives: Network

Internet. Networking.

FTP vs HTTP, really!

Since I'm doing my share of both FTP and HTTP hacking in the curl project, I quite often see and sometimes get the questions about what the actual differences are between FTP and HTTP, which is the "best" and isn't it so that ... is the faster one?

FTP vs HTTP is my attempt at a write-up covering most differences to users of the protocols without going into too technical details. If you find flaws or have additional info you think should be included, please let me know!

The document includes comparisons between the protocols in these areas:

  • Age
  • Upload
  • ASCII/binary
  • Headers
  • Pipelining
  • FTP Command/Response
  • Two Connections
  • Active and Passive
  • Firewalls
  • Encrypted Control Connections
  • Authentications
  • Download
  • Ranges/resume
  • Persistent Connections
  • Chunked Encoding
  • Compression
  • FXP
  • IPv6
  • Name based virtual hosting
  • Proxy Support
  • Transfer Speed

With your help it could become a good resource to point curious minds to in the future...

HTTP implementations

I previously mentioned on the libcurl mailing list, that Mark Nottingham in the IETF HTTP Working Group has initiated the work on putting together an overview of all (interesting) existing HTTP implementations

Of course curl is included in the bunch, or rather libcurl, but I would also urge you all to step forward and provide further details on other implementations you worked on or know of!

Will 2008 become 1984?

Next week in Sweden (June 18th), as reported in several places lately including slashdot, the Swedish parliament is supposed to vote for the pretty far-going law allowing FRA (a swedish defence organization previously involved in radio-surveillance etc) to wire-tap phone calls and computer traffic that cross the Swedish borders. The majority in the parliament is for the law, while it seems most of the ordinary people are against it. The hope is now that a few people will vote against their parties, that they will have the guts to stand up and "do the right thing" instead of following the party line.

I won't go into how silly, stupid and bad such a law is but I'll instead just show this great video to all swedes:

(video snipped from here)

stopa FRAlagen nu

This banner says (roughly translated by me) "On June 18th the government will take away your personal integrety. All internet traffic, all phone calls, all email and SMS traffic will be wire-tapped starting January 1st 2009. Big brother sees you! ... and violates the Swedish Constitution."

public suffixes list

I noticed the new site publicsuffix.org that has been setup by the mozilla organization in an attempt to list public suffixes for all TLDs in the world, to basically know how to prevent sites from setting cookies that would span over just about all sites under that "public suffix".

While I can see what drives this effort and since we have the same underlying problem in curl as well, I have sympathy for the effort. Still, I dread "having to" import and support this entire list in curl only to be able to better work like the browsers in the cookie department. Also, it feels like a cat and mouse race where the list may never be complete anyway. It is doomed to lack entries, or in the worst case list "public suffixes" that aren't any such public suffixes anymore and thus it'll prevent sites using that suffix to properly use cookies...

There's no word on the site if IE or Opera etc are going to join this effort.

Update: there are several people expressing doubts about the virtues of this idea. Like Patrik Fältström on DNSOP.

The IPv6 failure being joined by DNSSEC?

In case you haven't read it before, Randy Bush's 55 page PDF slide show named "IPv6 Transition & Operational Reality" is a harsh (but quite accurate) description of how the IPv6 protocol was made, where some of its major problems lie and why the transition is going so slow etc.

I tried to find some official and recent figures or statements from some of the more IPv6-positive people and companies, but I failed to find much updates from after the year 2000 or so...

Speaking of network things that aren't so successfully deployed: DNSSEC. Apparently iis.se (runs the Swedish TLD) tested 10 broadband routers (article and PDF in Swedish only) how well they support this (I believe mainly because .se tries to be a pioneer in DNSSEC), and 7 of the tested ones failed... Personally I've never liked the fact that DNSSEC isn't really crafted to do it securely all the way.

Tunneling with libcurl

As I wrote a while ago, companies using http proxies make people feel a need to break out of their proxies.

Bryan is a friend who recently found out that his company is switching proxy to a different one and apparently both corkscrew and proxytunnel have problems with this new piece, and since libcurl offers quite a lot of functionality to accomplish almost this, a new project was born: curltunnel.libcurl

One immediate benefit of using libcurl is the support for multiple authentication methods, in fact more than any of the above mentioned tools.

However, it seems our first quick stab at making this tool (currently 278 lines of code), made it work for several common cases but... not for Bryan's new proxy.

The current theory is that the proxy actually checks for SSL traffic and only lets that through, and thus it prevents the ssh server banner to appear when we try to tunnel through the proxy to a remote ssh server on port 443. If further testing proves this correct, we will of course have to add a SSL layer to the mix.

download flv videos from youtube

My wife wants to keep some videos found on youtube, and I really can't recommend just keeping bookmarks to a random web site like that. Not if you want the content to be available in a few years ahead, or even ten or twenty years. Then downloading the files to keep the locally is the only sane way to make it somewhat more reliable.

To download the files you can do it with a browser or with a command line tool:

Browser StyleGreasemonkey

  1. Use Firefox
  2. Install Greasemonkey
  3. Within Greasemonkey there's concept of user scripts that customize it, and we want a certain customization for youtube pages. So we get the YouTube to me v2 script installed.
  4. Now, each youtube web page gets a red stripe on the top of the page that allows you to download the FLV.

Command Line Style

There exist several command line tools "out there" that do the job. I tried youtube-dl and it did the job splendidly by only proving the main HTTP URL on the command line.

The main lacking feature is that it names the output flv based on the 'v' variable in the URL so the downloads end up being named things like "f_8wuVEYMZ8.flv"...

Play the local FLV movies

For this, I can only recommend the lovely VLC media player, available on all modern platforms.

Form Submit Honeypot by Mistake

During the summer 2001, me and my wife toured Vietnam and we had a great time. For that occasion I set up a little online diary that would allow us to post entries while on the road, to allow our families and friends back home to be able to keep up with what we were doing.Boats in Vietnam

Fast forward to present day: the diary "submit new entry" form is still left on my site, and while it no longer works (it hasn't worked for many years) - it is still one of the most visited pages on my site! It seems the automated spam bots find it and submit crap to it... the crap doesn't end up anywhere to be seen nor is it even stored on the server, but it clearly identifies evil machines! Isn't that a honeypot as good as any?!

So far during September 2007, no less than 309 unique IP addresses have issued a POST on that page..