During the first few years of the cURL project I used to sum up something about the past year and what I expected of the year to come at every anniversary. I stopped that at some point as I felt I didn’t have much new to contribute with, it mostly got repetitive over the years. I’ve never been the guy with the eyes fixed at the horizon and a grand plan of what to do in a future or work towards on a long-term basis. I’m more the guy with my feet firmly on the ground solving today’s problems right now as good as possible so that it stays stable and fine tomorrow.
So here follows some thoughts and reasoning around stuff that may and may not be the future of curl and libcurl. I’m as always putting my trust in you my friends to help me out with the details…
Protocols
curl’s wide protocol support has surprised even me. While I think the amount of protocols that can be supported without bending over backwards far too much is shrinking rapidly, I also think that we’ve learned that there are always more protocols that can be supported and that people will like to use with curl and libcurl. I expect more to come.
Specific protocols that are on the watch list include:
- Gopher – I just had to mention it just because curl doesn’t currently support it, it was there once but got yanked out when we found out it didn’t work and hadn’t been doing so for a very long time without anyone noticing! The support is about to come back though thanks to a patch being worked on right now.
- SPDY is the Google HTTP replacement (experimental) protocol that if it turns out successful seems like a very important piece for curl to grok.
- Websockets will come to stay in your browser and it will happen soon. To remain a really powerful web scraper tool in the future with more and more sites switching to using Websockets for parts of its functionality, I believe at least some level of support might be required.
- SCTP. Long-shot.This transport layer protocol was standardized in 2007 but really has not taken off in any significant way, even though it features a lot of benefits compared to TCP in many aspects. The now dead HTTP over SCTP internet-draft shows that HTTP can indeed be moved over to use it and it would solve a bunch of the same problems that SPDY does (and MPTCP). SCTP also has its own share of problems that hamper its adoption, primarily the lack of support in middle-boxes like NATs and routers.
Do you think there’s any particular other protocol we should support in a future?
Stability
I’m not sure if curl is particular unstable. I don’t think it suffers from any unusual amounts of bugs or anything, but I figure a decent list of stuff to consider for the future is if we make things within the project in a good way. Do we use the proper procedures so that we reach stability and produce a good product? Should we fix, change or replace certain ways or practices so that our outputs get less flaws?
I suspect these questions are hard for someone as involved and inside the project as myself. Possibly it is also hard for someone coming from the outside to convince us old-timers about anything like that too…
Interface
The current libcurl API was basically introduced when libcurl was first made a reality back in the year 2000. While the multi interface and the subsequent multi_socket API were added later on, they are both heavily influenced and affected by the previous easy interface.
Maybe there’s a much better API we should have that would make it easier to make a better library and easier to write better application using this library? It would require some serious brain-storming to come up with something, or perhaps there’s someone out there who already have thought something out?
Contributors
We’re but a few people who push commits to the master branch. How should we proceed to attract more? Should we work differently, perhaps have sub-maintainers for parts of the code etc? Can we work differently or make anything better to encourage more people to send bug reports and/or patches?
We do in fact have a very low level to entry already so I’m not aware of many additional things we can carve out to streamline this further.
Sponsors
As friends of me know, I always feel a little bit envy for projects and developers who manage to get corporate sponsors or otherwise enough paid support contracts to make them able to work on their pet projects full-time or at least part-time. I certainly have never reached that level for more than just a few months at a time. I’m not sure this is very important, as lack of funding doesn’t stop us it just slows us down and really, in an open source project like ours there aren’t really many hard deadlines. If it doesn’t get done now, it’ll get done later if enough people want it to happen.
Getting funding isn’t always easy either, since there may very well appear a company that wants to pay for a feature to get added but we don’t agree that it is a feature suitable for the project…
Organization
At times I think of the long-term future of our little project. Like what’s gonna happen the day I want to give up my chieftainship or if a company would like to step up and sponsor the project, but won’t be able to because there’s no actual legal entity for the project to sponsor or pay.
Would it make sense to have a curl organization? We could (I presume) easily join an umbrella organization like ASF. Or perhaps even more suitable the Software Freedom Conservancy. I will certainly not advocate setting up our own non-profit or anything like that.
I don’t think FSF or GNU will ever be a serious alternative since a pretty solid design goal of mine has been to avoid the *GPL licenses for curl to keep it attractive for commercial interests in a higher degree than (L)GPL licensed projects are. (This is not a license debate, so please don’t lecture me about the existence of GPL’ed commercial stuff.) I will not change license.
Feedback
One thing I’m sure of though, is that we will continue to listen to our users and the general curl and libcurl community about what to work on and what isn’t good and what we should do next. Please tell me your opinions and views on these matters!
I’m curious to see how a tool like Curl would support WebSockets, at least as it’s currently envisioned. The analogous question would be — how does Curl support TCP?
curl does support TELNET, which is similar. But I quite agree, Websockets is wide open, fully duplex and flexible and might not at all fit curl’s model…
However, there might indeed still be reasons and ways, depending on how Websockets end up exactly and perhaps more importantly how the browsers and web apps end up using it…
Hi,
I’d really appreciate if curl would use the socket action interface internally, and make the rest of the multi/easy api just a wraper around the socket action interface.
While the socket action api is by far the most powerful and desirable interface, it is the most buggy interface too.
There are known problems with timeouts, downloads getting ‘stuck’ as timeouts are missing, and the usual approach is to call ‘socket_action’ periodically?
Timeouts are important, look at libev ev_timers, and learn from it, it is great, easy to use, and works.
If it helps, embed libev/…/… to get better timers, there are many, but I really like libev.
I’d love to see protocols in a source/filter/sink fashion.
Source, as in most cases, your socket/your application. Filter?, socks, openssl, rate-limiting … . Sink – the actual protocol.
So you can have a rate limited socks connection, put ssl on it, and apply rate limiting, for doing https.
For features, it is not that curl lacks features, but a map with features and notes would be great, simply look at http://www.postgresql.org/about/featurematrix.
Cluster by protocol, link the documentzation how to get it done, love it.
Markus
Markus, please read up on the facts before you state things about bugs and the lack of timeouts. Also, consider actually helping out in our project before you complain on our inability to fix bugs. The known bug #62 regarding timeouts was there A LONG TIME and yet nobody ever cared to fix it properly. I have now presented and committed a fix that I believe makes timeouts work fine with the multi interface. (Embedded libev is _not_ an option and in fact it wouldn’t even help.)
And yes, I’ve been wanting to use the multi interface internally for many years (see the code) and I’ll appreciate your help in making it reality.
Such a feature table looks like a great idea, but also a lot of work. Feel free to join in and help us making it happen!
I would urge you to join curl-library and continue these talks there if you’re serious about them!