I maintain curl and lead the development there. This is how I spend my time an ordinary day in the project. Maybe I don’t do all of these things every single day, but sometimes I do and sometimes I just do a subset of them. I just want to give you a look into what I do and why I don’t add new stuff more often or faster… I spend about one to three hours on the project every day. Let me also stress that curl is a tiny little project in comparison with many other open source projects. I’m certainly not saying otherwise.
the new bug
Someone submits a new bug in the bug tracker or on one of the mailing lists. Most initial bug reports lack sufficient details so the first thing I do is ask for more info and possibly ask the submitter to try a recent version as very often we get bug reported on very old versions. Many bug reports take several demands for more info before the necessary details have been provided. I don’t really start to investigate a problem until I feel I have a sufficient amount of details. We’re a very small core team that acts on other people’s bugs.
the question by a newbie in the project
A new person shows up with a question. The question is usually similar to a FAQ entry or an example but not exactly. It deserves a proper response. This kind of question can often be answered by anyone, but also most people involved in the project don’t feel the need or “familiarity” to respond to such questions and therefore remain quiet.
the old mail I haven’t responded to yet
I want every serious email that reaches the mailing lists to get a response, so all mails that neither I nor anyone else responds to I keep around in my inbox and when I have idle time over I go back and catch up on old mails. Some of them can then of course result in a new bug or patch or whatever. Occasionally I have to resort to simply saving away the old mail without responding in order to catch up, just to cut the list of outstanding things to do a little.
the TODO list for my own sake, things I’d like to get working on
There are always things I really want to see done in the project, and I work on them far too little really. But every once in a while I ignore everything else in my life for a couple of hours and spend them on adding a new feature or fixing something I’ve been missing. Actual development of new features is a very small fraction of all time I spend on this project.
the list of open bug reports
I regularly revisit this list to see what I can do to push the open ones forward. Follow-up questions, deep dives into source code and specifications or just the sad realization that a particular issue won’t be fixed within the nearest time (year?) so that I close it as “future” and add the problem to our KNOWN_BUGS document. I strive to keep the bug list clean and only keep relevant bugs open. Those issues that are not reproducible, are left without the proper attention from the reporter or otherwise stall will get closed. In general I feel quite lonely as responder in the bug tracker…
the mailing list threads that are sort of dying but I do want some progress or feedback on
In my primary email inbox I usually keep ongoing threads around. Lots of discussions just silently stop getting more posts and thus slowly wither away further up the list to become forgotten and ignored. With some interval I go back to see if the posters are still around, if there’s any more feedback or whatever in order to figure out how to proceed with the subject. Very often this makes me get nothing at all back and instead I just save away the entire conversation thread, forget about it and move on.
the blog post I want to do about a recent change or fix I did I’d like to highlight
I try to explain some changes to the world in blog posts. Not all changes but the ones that are somehow noteworthy as they perhaps change the way things have been or introduce new fun features perhaps not that easily spotted. Of course all features are always documented etc, but sometimes I feel I need to put some extra attention on focus on things in a more free-form style. Or I just write about meta stuff, like this very posting.
the reviewing and merging of patches
One of the most important tasks I have is to review patches. I’m basically the only person in the project who volunteers to review patches against any angle or corner of the project. When people have spent time and effort and gallantly send the results of their labor our way in the best possible format (a patch!), the submitter deserves a good review and proper feedback. Also, paving the road for more patches is one of the best way to scale the project. Helping newcomers become productive is important.
Patches are preferably posted on the mailing lists but there’s also some coming in via pull requests on github and while I strongly discourage that (due to them not getting the same attention and possible scrutiny on the list like the others) I sometimes let them through anyway just to be smooth.
When the patch looks good (or sometimes good enough and I just edit some minor detail), I merge it.
the non-disclosed discussions about a potential security problem
We’re a small project with a wide reach and security problems can potentially have grave impact on users. We take security seriously, and we very often have at least one non-public discussion going on about a problem in curl that may have security implications. We then often work on phrasing security advisories, working down exactly which versions that are vulnerable, producing patches for at least the most recent ones of those affected versions and so on.
tame stackoverflow
stackoverflow.com has become almost like a wikipedia for source code and programming related issues (although it isn’t wiki), and that site is one of the primary referrers to curl’s web site these days. I tend to glance over the curl and libcurl related questions and offer my answers at times. If nothing else, it is good to help keeping the amount of disinformation at low levels.
I strongly disapprove of people filing bug reports on such places or even very detailed (lib)curl core questions that should’ve been asked on the curl-library list.
there are idle times too
Yeah. Not very often, but sometimes I actually just need a day off all this. Sometimes I just don’t find motivation or energy enough to dig into that terrible seldom-happening bug on a platform I’ve never seen personally. A project like this never ends. The same day we release a new release, we just reset our clocks and we’re back on improving curl, fixing bugs and cleaning up things for the next release. Forever and ever until the end of time.
I’ve been poking around the curl code for the last few days and what frightens me is the sheer scope of toggles in the code. Having choices is good in principle and so is support for a wide array of platforms, compilers, etc. but what I’m concerned about is whether this is healthy for the library.
If there’s a fatal flaw in curl (and with code this size, there most definitely is one), its exploitation will lead to shellshock/heartbleed level of panic. I don’t think it’s an exaggeration – curl is being used in a very wide variety of systems (and for a good reason).
So, yeah. Watcom C hasn’t been updated in 4 years now. DOS is dead. Symbian was put to rest. Who uses Gopher these days for reasons other than sentimental? And sentimental doesn’t seem to me like the right modus operandi for the library. Pragmatic on the other hand does. 😉
What I think I’m trying to ask is this: isn’t it time for libcurl2, with slightly fewer features and a little bit more modular design, so that e.g. features are neatly separated and CURL_DISABLE_foo isn’t sprinkled around? Old libcurl could still get critical securoty updates with the new one getting potentially interesting features like SPDY support.
Just my 2c. 🙂
Cheers!
DominikD: maybe you’re right. Maybe not.
I don’t see anyone volunteering on doing such a major rework and even you post the suggestion here where it will have zero effect (libcurl is discussed on the curl-library mailing list).
I don’t think you or other outsiders can properly say “those areas are unimportant – remove them” like you try here. Outdated compilers and old protocols don’t just evaporate and vanish, they are still around and we have large amount of users on platforms many random people think are dead. Who’s to say what to remove and what to keep? An interesting challenge.
I’m not a believer in “rewrite to fix the problems”. You’d only throw away over a dozen years of experiences and fixes of subtle problems in code, network and protocol logic. I believe in iterative and organic growth and fixing of problems.
I’m all for improving libcurl, and I believe we have proved through the years that we can do drastic changes of code and internals without rewriting everything. We just evolve. I believe that’s the key going forward as well.
I’m not saying these aren’t important. I’m sure support for, say, Watcom C wasn’t added for fun. It’s there because someone needed that, that’s pretty obvious. And I’m not claiming I know the exact set of things that should change or go completely. I’m simply suggesting that maybe some things should. I’m in no position to decide which ones are alive and which aren’t. But there are clearly people who are or could be. 😉
Don’t get me wrong, I’m not some hip youngster who’s come here with “duuude, these things are so obsolete, dude! you should write this thing in node.js, duuude!” attitude. I work mostly with legacy codebases in companies that are slow to adapt new stuff. So I understand the inertia of big software development. But there’s always a point, IMO, when one has to say: it’s been supported long enough, it’s time to cut it.
I’m also not suggesting rewrite (not sure where this came from, honestly), but rather an evolution. The great thing about curl is that docs/ contains a lot of useful info for people wanting to understand and contribute. So e.g. the number of ifdefs is massive but it’s still possible to grasp the complexity of the code. It’s also, I’d argue, possible to move to a slightly more modular code that would localize toggles.
But I guess it all boils down to “I don’t see anyone volunteering”. I’m still concerned and mildly worried. But I hope I’m just paranoid about the code in general. 🙂
Cheers!
I think you have a fair point. There probably is at least one pretty serious vulnerability hiding in there that will reveal itself one day in the future and then we can come back here and see that you said so already in September 2014… =)
I honestly hope this won’t happen. But if the glory comes, my body is ready! ;]
This post pretty much describes my work on Buildbot. I’ve been struggling for a while to broaden the group of people doing all of these things, so that I’m not a single point of failure (or delay) for the project. Are you doing anything similar, and if so, what have you found to be effective?
Dustin: yeah, I’ve always tried to make things in the project not be about me or rely on me personally, which is one of the reasons I so strongly urge people to not discuss open source matters with me privately but to use the mailing lists. I also try to document as much as possible and aid newcomers to find their way into the project etc.
I can’t say that I’ve managed particularly well. The core people in the project are but a handful of faithfuls that have been around for years.
I think we have a few things speaking against us: our stuff mostly just works (there’s usually no need to get involved), it isn’t a “sexy project” in any way just a core thing written in C89 and we’re pretty good already at fixing bugs and adding necessary features so if you just wait a while chances are that the few problems you’ve seen or features you’ve missed get fixed or added anyway.
I’ve mostly just come to terms with that this is the way it’ll be, and I’ll keep on pretending we can widen our developer group one day but I have no illusions that it will actually eventually be so…
I’m not complaining, this is my hobby and I love it and I even get to do it partly during paid work hours!