Back in 2003 I wrote up support for the HTTP NTLM authentication method for libcurl. Happy with my achievement, I later that year donated a GPL licensed version of my code to the Wget project (which also was my first contact with the signed paper stuff with the GNU/FSF to waive my copyright claims and instead hand them over). What was perhaps not so amusing with this code was when both curl and Wget 2005 were discovered to have the same security flaw due to my mistakes in this code shared by both projects!
Just recently, the neon project seems to be interested in taking on the version I adjusted somewhat for them, so possibly the third HTTP code is soon using this. Yeah I posted it on their mailing list back then so it has been sitting there in the archives maturing for some 6 years by now...
I also happened to fall over the SSH Tunnel Creator tool, which I've never used myself, that apparently snatched my neon donation (quite according to what the license allowed of course) and used it in their tool to do NTLM!
It's actually not until recent years I discovered libntlm, and while I don't know how good it was back in the days when I wrote my first NTLM stuff I generally think using existing libs is the better idea...
I've already previously expressed my deepest dislike with where the HTML5 work is going, and just yesterday two new internet-drafts appeared on ietf.org that spurred up discussions all around. They're claimed to be "part of our effort to remove from HTML5 sections that are more appropriate elsewhere" but I'm thinking they're rather inappropriate everywhere...
The first one named Content-Type Processing Model hits a subject that I've been over before, namely the stupidity of having web browsers guess the content based on what it looks like. IE introduced the "I really mean it property", the HTML5 team wants to standardize the way of the guessing. Personally, I think the world of web will become a better place if the browsers would instead become stricter and more closer follow what the servers actually say the contents is, and then all users would complain to the site admins if things are wrong and then things should be fixed.
Guessing content types allows for sloppy behaviors, it makes it harder to write browsers for the web and it still features a significant risk of guessing wrong.
The second draft propagates for the new HTTP header "Origin", which according to the authors would help to guard servers against CSRF ("Cross-Site Request Forgery"). The main author says 3% of users on the Internet gets their Referer header stripped while virtually none gets Origin stripped. I claim this is a bogus argument since they strip Referer beacause it is a known and established header and Origin is not. I also completely fail to see the goodness of this and based on several of the other responses on the ieth-http-wg mailing list I am not alone...
Over at the IETF another group was just created named http-state (with an associated mailing list) with the specific goal:
Ultimately, the purpose of this group is to create an updated HTTP State Management Mechanism RFC (aka cookies) that will supersede the Netscape spec, RFCs 2109, 2964, 2965 then add in real-world usage (e.g. HTTPOnly), and possibly add in additional features and possibly merge in draft-broyer-http-cookie-auth-00.txt and draft-pettersen-cookie-v2-03.txt.
I've joined the list and I hope to follow and participate in this, as I believe the current state of HTTP cookies is a rather sorry mess and the netscape spec is still what closest describes how cookies work in the wild. Of course I'll do it with my libcurl experience in my luggage.
While it perhaps would be cool to join the group in more formal way, there's no way for me to participate in that IETF meeting in San Francisco in March.
A few years ago I wrote up silly little perl script (let's call it script.pl) that would fetch a page from a site that returns a "random URL off the internet". I needed a range of URLs for a test program of mine and just making up a thousand or so URLs is tricky. Thus I wrote this script that I would run and allow to get a range of URLs on each invoke and then run it again later and append to the log file. It wasn't a fancy script, but it solved my task.
The script was part of a project I got funded to work on, that was improving libcurl back in 2005/2006 so I thought adding and committing the script to CVS felt only natural and served a good purpose. To allow others to repeat what I did.
Fast forward to late 2008. The script is now browsable via viewvc on a site that... eh, doesn't have ".pl" disabled as a cgi extension in its config! The result of course is that each time someone tries to view the script using the web interface, the web server invokes the script locally!
All of a sudden I get a mail from someone, who apparently is admin or something of the site this old script was using, and he mentions that a machine on our network is hammering his site with many requests per second (38 requests/second apparently) and asked me to stop this. It turns out a search engine crawler has indexed the viewvc output several times, and now some 8 processes or so were running this script.pl and they were all looping around getting a page, outputting the URL, getting another page...
While I think 38 requests second is a bit low to even be considered a DOS, it certainly wasn't intended nor friendly and I was greatly surprised when I slowly realized how it all came to end up like this! Man I suck! It reminds me of my other extension mess from just a few months ago...
Maybe I'll learn how to do things right in the future when I grow up!
Greg Dean posted an interesting idea on the ietf-http-wg mailing list, suggesting that a new response header would be added to HTTP (Estimated-Content-Length:) to allow servers to indicate a rough estimation of the content length in situation where it doesn't actually now the exact size before it starts sending data.
In the current world, HTTP servers can only report the exact size to the client or no size at all and then the client will have to just deal with the response becoming any size at all. It then has no way to know even roughly how large the data is or how long the transfer is going to take.
The discussions following Greg's post seem mostly positive thus far from several people.
Wei-Hsin Lee of Google posted about their effort to create a dictionary-based compression scheme for HTTP. I find the idea rather interesting, and it'll be fun to see what the actual browser and server vendors will say about this.
The idea is basically to use "cookie rules" (domain, path, port number, max-age etc) to make sure a client gets a dictionary and then the server can deliver responses that are diffs computed against the dictionary it has delivered before to the client. For repeated similar contents it should be able to achieve a lot better compression ratios than any other existing HTTP compression in use.
I figure it should be seen as a relative to the "Delta encoding in HTTP" idea, although the SDCH idea seems somewhat more generically applicable.
Since they seem to be using the VCDIFF algorithm for SDCH, the recent open-vcdiff announcement of course is interesting too.
Anthony Bryan and I had a talk the other day regarding FTP vs HTTP etc, and the outcome is available as this podcast.
Since I'm doing my share of both FTP and HTTP hacking in the curl project, I quite often see and sometimes get the questions about what the actual differences are between FTP and HTTP, which is the "best" and isn't it so that ... is the faster one?
FTP vs HTTP is my attempt at a write-up covering most differences to users of the protocols without going into too technical details. If you find flaws or have additional info you think should be included, please let me know!
The document includes comparisons between the protocols in these areas:
- FTP Command/Response
- Two Connections
- Active and Passive
- Encrypted Control Connections
- Persistent Connections
- Chunked Encoding
- Name based virtual hosting
- Proxy Support
- Transfer Speed
With your help it could become a good resource to point curious minds to in the future...
David M. Kristol is one of the authors of RFC2109 and RFC2965, "HTTP State Management Mechanism". RFC2109 is also known as the first attempt to standardize how cookies should be sent and received. Prior to that document, the only cookie spec was the very brief document released by Netscape in the old days and it certainly left many loose ends.
Mr Kristol has published a great and long document, HTTP Cookies: Standards, Privacy, and Politics, about the slow and dwindling story of how the work on the IETF with the cookie standard took place and how it proceeded.
Still today, none of those documents are used very much. The original Netscape way is still how cookies are done and even if a lot of good will and great efforts were spent on doing things right in these RFCs, I can't honestly say that I can see anything on the horizon that will push the web world towards changing the cookies to become compliant.
I previously mentioned on the libcurl mailing list, that Mark Nottingham in the IETF HTTP Working Group has initiated the work on putting together an overview of all (interesting) existing HTTP implementations
Of course curl is included in the bunch, or rather libcurl, but I would also urge you all to step forward and provide further details on other implementations you worked on or know of!