All posts by Daniel Stenberg

library for proxy detection

Only days after I wrote about the pacparser, another and in many ways more complete approach to detecting what proxy to use for accessing various internet resources emerge: libproxy.

One of the main authors of it, Alex Panit, already submitted a feature-request for libcurl to support this. but I’m not at all convinced that is a good idea. It seems the authors submit “please include support for this”-requests all over in similar and related projects – similar in style to how metalink did.

As usual, I value your input and feedback so please raise your voice and speak up!

So far this young project lacks docs on API and install process, so I haven’t yet even been able to build it for a test drive…

Jun-Ichiro

Today I just casually noticed the NetBSD 4.0 release, but what really caught my interest was the dedication of the release. It is dedicated to Jun-Ichiro “itojun” Hagino, who apparently passed away in October this year only 37 years old. See also this “memorial”.

Anyway, the reason I noticed is because I remember him and his contributions in the curl project back in 2001 and 2002 when he provided a set of very good and useful patches (for example this) that made curl take a huge step forward in its IPv6-support. It is of course with sadness that I get to hear about him again this way. I’m sure he will be missed in many camps.

curl icons

I got a question on the #curl IRC channel and I have to admit that the curl and libcurl icons are a bit hard to find on the curl web site, so I thought I’d paste them here too to put the lights on them a bit more.

If you use (lib)curl in your projects or site, please consider making this noticeable by mentioning it and possibly using an existing curl icon to show:

powered by libcurl powered by libcurl powered by libcurl powered by libcurl powered by libcurl

powered by libcurl powered by libcurl powered by libcurl

small curl icon small curl icon with border libcurl icon

cURL libcurl

… as you can see there are plenty to choose from! 😉

And please, if you are capable of drawing logos, bitmaps, icons or the likes and you feel like you can do better than these, please step forward and bring on your work!

Parsing those Dreaded PACs

Since a long time companies all over have found it suitable to use “PAC” (proxy auto-config) scripts to control browsers and tell them what proxy to use for what particular URL and more, and this has been a major nuisance for people who use other (automated) means to go through the proxy than one of them huge gui-based browsers.

For a long time, Ralph Mitchell’s take on this was the only half-decent way to get it working. Half-decent only because it was never packaged and provided in a convenient way to people, it was fully functional.

Now, Manu Garg seems to have stepped forward and introduces pacparser which is a C library for parsing PAC files. Period. It looks (very) similar to Ralph’s approach, in that it incorporates the Mozilla’s javascript engine Spidermonkey, with nothing really special added. So, it’s basically the same approach but this time it is packaged, shipped and documented how it is used!

Announced on the curl-users mailing list today!

The Bourne Nmap

Over at insecure.org we can read about nmap‘s appearance in The Bourne Ultimatum (IMDB) movie and they also show two screenshots, out of which I’ll show you one:

I couldn’t resist trying to resolve the host name in there, only to find that telservice.net is a Korean company/network (which kind of makes it less likely to have the address of the Guardian UK – supposedly the hacking target in the movie) and of course the specific host name in this shot doesn’t resolve and the IP address showing isn’t belonging to telservice.net… Wow, who could’ve guessed that? 😉

And yeah, I’m jealous. I want one of the projects I participate in to appear in movies too!

curl keeps connections alive

Just in the last few days we modified curl to enable the SO_KEEPALIVE option on connections it creates. It basically means that curl will now detect connections that are idle after a certain amount of time, even if that time is something around two hours by default and that’s what most systems will have it set to.cURL

The main problem that caused us to finally enable this (you can still disable this by using –no-keep-alive) is when people do (long-lasting) FTP transfers and they use a NAT, firewall or router that detects and removes what it considers are idle connections. An FTP transfer is using two connections, but the control one where the commands are sent over is completely quiet while the actual data transfer is in progress so when the transfer is done, the control connection has been nuked by the router/NAT. Of course curl survives this as good as possible, but it can’t do proper error-checking etc in this situation.

Funnily, there’s no really good fix for the FTP situation since the two hours SO_KEEPALIVE timeout will many times be too long to help (although most modern systems allows you to change the timeout or a system or application level), but the other “obvious” fix is to send a “NOOP” command on the control channel every once in a while during the transfer. But no, that doesn’t work fine either on most servers since it seems the servers often don’t listen on the control connection during the transfer, so all we’d get is curl sending commands that won’t be replied to until the end of the transfer, and thus it will end up causing problems.

Note: curl sets this option. libcurl still doesn’t, so if you want your app to set the option you can use the same CURLOPT_SOCKOPTFUNCTION callback that curl uses. This requires libcurl 7.16.0 or later.

The Mythical Man-Month

The Mythical Man-MonthFrederick Brooks wrote this classical book already back in 1975 and added a few extra chapters for the twenty years anniversary 1995…

Large portions of it feels of the age and there’s a lot of talk about Fortran, System/360 and PL-1 as if we should know about them (which made me fast forward over some chapters). But there are gems as well, and the most significant things people seem to remember Brooks’ book for are still pretty valid and fine.

Adding more people to a project leads to the need for more communication and thus it may slow down development rather than speed it up. Also known as Brooks’s law.

Given the complexity of software and software development, there’s no single method or concept that will lead to an improvement by an order of magnitude – within a decade. There’s No Silver Bullet. (This section was not in the original edition of the book.)

The risks involved when rewriting something and wants to fix everything that was wrong in the previous version so you over-work and over-design the successor. The so called Second system effect.

A lot of the book is spent on thoughts and theories around how to manage really really large software projects, like when you involve thousands of persons. Is it even possible to make such huge projects successful and if so, what does it take? The extra chapters do indeed add value since they offered Brooks a chance to re-evaluate his earlier claims and ideas and to check what seemed to be truths and what mistakes he did in the original edition.

A very interesting read that I’m glad I finally got time to get through!

A thankful Rockbox user

I just wanted to share with you some very friendly and encouraging words I received in a mail from an individual that shall remain unidentified. It is actually rather unusual to get this kind of cheerful words as people who get things and enjoy them rarely get back, but instead we hear more from the ones who get problems or are otherwise unhappy. sansa e200R - looks almost like a e200 v2

“From someone not nearly so gifted and talented, thanks for your efforts to develop Rockbox on the Sansa e200 v2.
Your efforts are deeply appreciated by many, and I’m glad that there is someone as clever and kind as you working on this project in your spare time (which I’m sure is precious to you.)
Work on the project when you can, but don’t deprive your family of your time and presence – they are your most important treasure. I’m just a private individual, and I know this little expression of thanks doesn’t mean much, but here it is, just the same. All the very best wishes to you and your family for Christmas and the coming new year!”

Thanks a lot. (linkified by me)

The Scandinavian Free Software Award went to…

skolelinux logoYou might remember that I mentioned that I was nominated for this award. The final winner of it was the Skolelinux project, as was announced on the fsf-europe press-release list just an hour ago (well the prize was handed out on Friday night but I forgot to ask yesterday so I didn’t know until now…). I guess the fscons site will be updated in due time as well…

As I said before, my own contributions to free software is rather tiny and insignificant compared to giants such as this. I think this project is a well-deserved winner of the award. Congratulations to all involved!