Category Archives: Technology

Really everything related to technology

Spammers now subscribe

During several years I’ve been setting mailing lists I admin to only accept posts from subscribers iA can with spamn order to avoid having to deal with very large amounts of spam posts.

While that is slightly awkward to users of the list, the huge benefit for me as admin has been the deciding factor.

Recently however, I’ve noticed how this way to prevent spam on the mailing lists have started to fail more and more frequently.

Now, I see a rapid growth in spam from users who actually subscribe first and then post their spam to the list. Of course, sometimes spammers happen to just fake the from address from a member of a list – like when a spammer fakes my address and sends spam to a list I am subscribed to, but it’s quite obvious that we also see the actual original spammer join lists and send spam as well.

It makes me sad, since I figure the next step I then need to take on the mailing lists I admin is to either spam check the incoming mails with a tool like spamassassin (and risk false positives or to not trap all spams) and/or start setting new members as moderated so that I have to acknowledge their first post to the list in order to make sure they’re not spammers.

Or is there any other good idea of what I can do that I haven’t thought of?

null-prefix domino

dominos

At the end of July 2009, Scott Cantor contacted us in the curl project and pointed out a security flaw in libcurl (in code that was using OpenSSL to verify server certificates). Having read his explanation I recalled that I had witnessed the discussion on the NSS list about this problem just a few days earlier (which resulted in their August 1st security advisory). The problem is basically that the cert can at times contain a name with an embedded zero in the middle, while most source code assumes plain C-style strings that ends with a zero. This turns out to be exploitable, and is explained in great detail in this document (PDF).

I started to work on a patch, and in the mean time I talked to Simon Josefsson of the GnuTLS team to see if GnuTLS was fine or not, only to get him confirm that GnuTLS did indeed have the same problem.

So I contacted vendor-sec, and then on the morning of August 5 I thought I’d just make a quick check how the other HTTPS client implementations do their cert checks.

Wget: vulnerable

neon: vulnerable

serf: vulnerable

So, Internet Explorer and Firefox were vulnerable. NSS and GnuTLS were. (OpenSSL wasn’t, but then it doesn’t provide this verifying feature by itself) (lib)curl, wget, neon, serf were all vulnerable. If that isn’t a large amount of the existing HTTPS clients then what is? I also think that this shows that it would be good for all of us if OpenSSL had this functionality, as even if it had been vulnerable we could’ve fixed a busload of different applications by repairing a single library. Now we instead need to hunt down all apps that use OpenSSL and that verify certificate names.

Quite clearly we (as implementers) have all had the same silly assumptions, and quite likely we’ve affected each other into doing these sloppy codes. SSL and certificates are over and over again getting hit by this kind of painful flaws and setbacks. Darn, getting things right really is very very hard…

(Disclaimer: I immediately notified the neon and serf projects but to my knowledge they have not yet released any fixed versions.)

Mini 2440 Lyre

On ebay there’s a fancy S3C244-based board named mini 2440 with a 3.5″ touch LCD attached on sale for 85 USD. 64MB ram, 400MHz CPU, a nand flash and more. Lots of stuff for the money.

mini2440

The guys in the lyre project seem to have adopted this as yet another hardware platform to attempt to run Rockbox on. After their Atmel AT91SAM target was ditched, they went the ARMopendous route and now this seems to have entered. This third hardware platform is called the Lyre prototype 2

You should note that this Mini 2440 board has no batteries or anything and thus is not really meant to be a portable device in this shape.

“Bob” seems to have initial Rockbox code running on this device, and well-established Rockbox hackers JdGordon and domonoky have both ordered their own kits so the future looks bright.

Sandisk: “our sound fidelity isn’t perfect”

Some owners of the SansSanDisk Sansa Clipa Clip player from SanDisk noticed that it does playback of all songs with a minor pitch. Due to a flawed HW setup they don’t do a proper 44,100 Hz but instead 43,791 Hz (0.993 times the target value) or something like that. According to some sources this problem might be fixed in the Sansa Fuze players now.

Bugs aren’t really so surprising, perhaps what is surprising is that this bug has been around now for almost two years. To make matters worse, SanDisk now decided that due to them making cheap players people shouldn’t expect them to be very good sound quality wise and therefore they can simply not fix the problems:

due to trade-off decisions that were made in engineering these products to deliver superior consumer value at what we believe are extremely attractive price points, our sound fidelity isn’t perfect. We have re-evaluated the possibility of reducing the pitch variation and due to the engineering trade-offs the decision was made to stay with the current design. Very few listeners, however, have noticed or complained about it as an issue in actual practice.  For those who can detect sound differences with their naked ears during actual use and not via frequency analysis, our products may not be the best choice for them.

Clip owners out there now put their hope even more on Rockbox for Clip.


Why top-posting annoys me

This is hardly any news to anyone who cares, and those who should care the most are either not understanding what top-posting is in the first place or they’re not aware of that people like me think top-posting is an evil decease we need to extinguish.

My primary reason to hate top-posting is that it is fast and easy for the single user who writes the mail reply, but it gives more work to the large amounts of people who read it. When someone posts to a mailing list, one should rather expect that the single user would be the one to put in a little extra effort to make the result readable for the masses who will read it.

Top-posting also most often involves the habit of including the entire previous conversation in a quoted manner below.

A sensible post and quote ethic, is to only quote as much as you need from the previous conversation to make your point clear, and to respond in a way so that it is clear to what parts of the quotes you are referring to. That more or less implies doing “interlaced” or “inlined” posting, where you show a few lines of quotes and then a few lines of comments over and over until the end of the mail.

The act of doing bottom-posting but keeping the entire thing quoted above the new text you add is almost as bad as top-posting. You remove the focus of what you write by providing far too much irrelevant text. Remove the irrelevant parts!

These days large portions of the modern world use broadband connections so the actual size of the mail is not a concern for bandwidth or speed reasons, but you probably still want the receivers to focus on your actual point. Also, a lot of mails these days end up in web archives or similar so they are then searchable by internet search engines and browsable by future people and then you even more want the mail to be on topic to become more relevant and less misleading to searches.

In case it isn’t obvious: this of course primarily concerns mails sent to (largish) mailing lists.

Making better advisories

A while ago yet another security flaw was discovered in curl (actually the tenth flaw in more than eleven years) by Scott Cantor. He reported it privately to us. We worked on the issue and in the end I posted an official project cURL security advisory about it. It wasn’t anything out of the ordinary really. Scott did great and we fixed the problem rather promptly and in coordination with vendor-sec etc.

After a security advisory and the accompanying release, something particular always happens. It’s the same every time I’ve done this and there’s really no surprise: one by one the different Linux distros and similar parties start to ship their security advisories and alerts about the same problem and they offer their upgrade paths for their users to get a corrected version of the package.

But I’ll tell you why I think those advisories tend to make me really sad. It’s not because of the flaws they fix or how fast or slow they are to appear. It’s entirely due to the contents of them or perhaps in many times the lack of contents.

When the first distro-based advisory comes out, they often tend not to use the phrasing used in the original advisory (which we’ve crafted on for weeks and coordinated with vendor-sec) but they instead offer their own interpretation. This isn’t necessarily a bad thing, but when the guys simplify matters they tend to blur out the actual cause and make the real issue hard to understand. Not to mention that when the first guy had done a mistake, most others just repeat that without thinking.

Credit to the doers

The craft of hunting down security problems in software and the art of then creating a fix for that problem is very time consuming and takes a fair amount of skill and patience. Yet some people do this. Some of those even track down problems in open source code bases and tell the projects about the issues to give them a chance to fix them before they’re made public.

Those people are good people that we need.

In the open source world, and in fact in a lot of other places too, the just about only reward we can offer guys who do outstanding work like this is with attribution. Give credit where credit is due. Mention the guy who did the job!

Distro advisories are not good

Very often the subsequent advisories go the lazy route and they borrow their advisory explanation from another distro’s advisory. Still not using the original explanation. They like short and not too detailed explanations. Factual errors seem to not be too important.

Very few distro-advisories give any credit to the original guy who found the error. The only one thing we can offer as payment is then neglected and this is more of an established practice than a mistake. All distros do this. At best they refer to a CVE number for the flaw, but CVE numbers have the great disadvantage that they very rarely reveal any particular details about the flaw until a long time after the advisory is made.

Not only do they often not credit the originator, they also rarely link back to the original advisory or even the advisory the originator sent out (sometimes they’re sent out independently) – so getting the full description from the actual upstream project is harder than it has to be. They do however generally  link to their own site, using their own issue number for the security problem. If things are good, you can find references to the original in that web page they link to. I’ve also seen several distro advisories that simply don’t at all mention what patches they’ve applied or what particular upstream changset they’ve backported.

In this latest advisory from curl, the common repeated mistake was that the certificate flaw concerned the Common Name field (and it implied that it was only about that field) which is wrong, and which is why the original advisory didn’t explicitly mention that field. It also affects the subjectAltName field and that’s at least – if not more – as important to address for this particular flaw. The flaw also only concerned curl built to use OpenSSL, which was a fact that was often not mentioned at all.

What I suggest!

That every vendor and Linux distro that ship security advisories do this:

  1. credit the original problem founder/researcher. This way the glory and fame goes to the person(s) who often did a lot of research and hard work.
  2. link to the original advisory so that everyone who wants to can get the info and details from the upstream project and their ideas of what the problems are and what the best fixes or work-arounds might be
  3. fact-check your error/solution description better and don’t just repeat what someone else wrote unless you know that’s an accurate description
  4. don’t repeat others’ simplifications and errors. The act of duplicating someone else’s description is pretty low in general and it would often only be a signal that you haven’t understood the issue in the first place. If you have a rule to not copy others you won’t risk copying their mistakes.

Slacka-fun!

A bunch of the local OpenBSD fans here in Stockholm run this one-day event every year, called Slackathon. I missed it last year, but in 2007 I was there (and I did a little talk about open source management) and this year I was eager to participate again.

This year, the event was scheduled to take place immediately after a bunch of core OpenBSD developers had had their “hackathon f2k9” in Stockholm, so they could now boast with a series of very well known and very knowledgeable OpenBSD kernel hackers. As I am really not more than a distant observer of the OpenBSD project this of course put the lights on a lot of dusty corners I had no previous idea about. I’m not really a stranger to kernels and kernel hacking in general, and I must confess I had a great time and the team who spoke of various very detailed kernel topics are charismatic and put on a great show.

So I learned about the terrors of the VFS layer and hacking it (and how they’re working on making all the involved caches dynamically sized). I learned how to do active-active syncing  of pf-based firewalls (basically using two independent firewalls in front of something), or at least how the guys made it work fairly well. Or how the pf firewall was optimized to double its forwarding performance. And I got to hear a few wise words from Theo de Raadt and learn not only about their 6 month release schedules but also their plans and ideas around solving problems with livelocking and more. Not to mention the talk about managing physical memory, or the work to get OpenBSD ported to sparc64s with hardware-based virtualization support.

Taken all the hardcore kernel talks into account, I think my own talk on libssh2 (just before dinner) felt like a very light snack to chew and possibly a tiny bit out of the general topic… Anyway, I gave a quick overview of the project, how it started, why it was started, what it is and a bit how it works etc.

The slides from my slackathon talk. I expect to re-use a fair bunch of that, with some improvements and additions, in my libssh2 talk at FSCONS later this year.

Looking forward to Slackathon 2010!

Pictures from Slackathon 2009 by Vladimir Bogodist.

Me in front of the projector screen doing libssh2 talk

Snaxx 21

HaxxYes!

It is now time to once again leave your dark and dusty corners of your office or closet, bring yourself up to speed on what currency we’re using in this country and then unite with fellow hackers and technologists in Stockholm City during a fine September evening. The entire Haxx team is delighted to inform that Snaxx-21 is about to happen…

Monday, September 28th 2009

Time: around 18:30

Where: see the snaxx site!

As usual we’re informal, and as our friends you’re of course allowed and encouraged to bring other friends who are similar in spirit and who you think would appreciate an event such as this.

When you’ve decided to show up, please email me and say so.

There might even be free t-shirts involved this time!

Oh, and if you are a Stockholmer and didn’t get this invite by mail already, let me know and I’ll add you to the list of people who get this notice by the old trusty RFC822 way.

curl fooled by null-prefix

We’ve just now released a security advisory on curl and libcurl regarding how a forger can trick libcurl to verify a forged site as having a fine certificate if you just had a CA create one for you with a carefully crafted embedded zero…

I think this flaw brings the light so greatly on the problems we deal with to maintain code to be safe and secure. When writing code, and as in this case using C, we might believe we’re mostly vulnerable to buffer overflows, pointer messups, memory leaks or similar. Then we see this fascinatingly imaginative “attack” creep up…

The theory in short and somewhat simplified:

A server certificate is always presented by a server when a client connects to it using SSL. The certificate contains the servers name. The client verifies that A) the cert is signed by the correct authority and B) that the cert has the correct name inside.

The A) thing works because servers buy their cert from a CA authority that has its public signature in all browsers, and thus we can be “cryptographically safe” when we see a match.

This last flaw was in the naming part (B). Apparently someone managed to trick a CA to hand out a cert to them using an embedded zero byte. Like if haxx.se would buy the cert, we’d get it with an embedded zero like:

“example.com\0.haxx.se”

Now, this works fine in certificates since they store the string and its length separately. In the language C we’re used to have strings that are terminated with a trailing zero… so, if we would take over the “example.com” HTTPS server we could put our legitimately purchased certificate on that server and clients would use strcmp() or the equivalent to check the name in the certificate against the host name they try to connect to.

The embedded zero makes strcmp(host, certname) return MATCH and the client was successfully fooled.

curl is no longer vulnerable to this trick since 7.19.6, and we have released a boatload of patches for older versions in case upgrading is not an option.

A view of a popular post

So I post frequently on this blog, but I’m not a particularly interesting person myself, I’m not really a master at writing and phrasing articles to make them thrilling and irresistible and I basically only deal with really geeky and technical subjects. It means there’s an average of perhaps 200 views per day.

The other day I wrote my multipath tcp post, and someone submitted it to reddit. It turned out to become my most read posting on my blog ever. By far. I think the “views per day” graph looks pretty cool:

visitor graph from daniel.haxx.se/blog