Tag Archives: cURL and libcurl

cURL and libcurl

Heading towards curl eight

There’s a plan for version 8 being forged! Let me just take you back a bit in time first..

The early days

When we first created libcurl, we bumped the major version number of the project from the previous version 6 to version 7. In late summer of 2000 we shipped curl and libcurl 7.1 as the first ever release that featured a separate library for Internet transfer powers. Everything before version 7 was just the command line tool, curl. That was the moment in time we decided we should leave kindergarten and we were ready to take on some tougher loads.

I had the main approach to the API worked out already from the start. It would be transfer-oriented, it would be built up around URLs and shouldn’t necessarily require that the users are themselves protocol experts to use it. I used inspiration from ioctl and fcntl when I made curl_easy_setopt and curl_easy_getinfo. They’re fixed functions with flexible arguments. The idea was that by doing that, we wouldn’t have to add new functions for new features but we would “just” add options and new options would simply just not work with older libcurls.

The first API we shipped also only provided synchronous single transfers. The “easy” interface.

I had no anticipations or particular hopes on the library then. It would be cool if someone found good use for it and it would be even cooler if someone would help out to improve it further.

It grew, I learned

I had never before developed and shipped a library for the world to use. I hadn’t really fully grasped and considered the impact of APIs and ABI stability etc.

We gradually improved the library over time. We bumped the SONAME several times in the first few years as we modified internals. In the same time the library caught on a bigger and bigger audience and in September 2006 as I ripped out code for what is commonly referred to as “FTP third party transfers” I once again bumped the ABI number (to 4) since the older libcurl was no longer compatible with the new release.

People don’t like SONAME bumps

That bump was met with quite a lot of resistance and objections among users. Changing SONAME of a widely used library it turns out causes a lot of pain, agony and squeaking. Possibly this was one of the earlier signs that libcurl had grown up and I decided that we should try to avoid going through this again. We shall not break ABI compatibility again. Ever.

No bumps, no worries

In this world with no SONAME bumps I wanted to keep that solidity visible in the major number of the project so even though the version number of the releases aren’t strictly related to the SONAME we kept shipping curl version seven. In September 2021, we reach curl 7.79.0.

We’ve managed to stick to our goals and a binary libcurl using application built after September 2006 can run with the latest libcurl with no modification needed! This is one of the biggest edges and “selling points” we have in libcurl: We take compatibility and unmodified behavior very seriously.

In 2013 I even wrote blog post emphasizing this and in there I said there won’t be a curl version 8 “in a long time”. But read on, things have changed a little!

An ever-growing minor number

We bump the minor version number every time we “change” something in the project, or add features. If we only do bug-fixes we only bump the patch number. You know, in classic “major,minor.patch” style.

As we do curl releases at least every 8 weeks and most releases have changes added, we bump the minor number very frequently, up to 6 times a year or so.

We provide version number information for libcurl provided as a 24 bit number, using 8 bits for each field. This implies that none of the numbers can ever go above 255. We can’t ship a 7.256.0 for this practical reason.

But also, from what I’ve seen people do with and think about version numbers before, I’m concerned that increasing the minor number beyond 99 will cause confusions. Version 7.100.0 risks gettinged confused and mixed up with 7.1 or 7.10.0, two versions that are terribly old. And frankly that’s a very large minor number and it starts becoming many digits in that release version.

There exists a solution to this!

Reset minor, bump major, keep SONAME

The idea is simply to do “a Linux kernel”. We change to version 8.0.0 at a given point in time, but we stick to the same SONAME as before.

We don’t break any compatibility, there will no no API or functionality cleanups. There will just be a version number bump to lower the minor number and let us start over that journey. Reset the counter so to speak.

When do we do this? We have roughly 20 releases left before the minor counter can reach 100, 20 releases take at least 6*20 = 120 weeks. 120 weeks is 2.3 years.

Another event within this period

On March 20, 2023. About 18 months into the future, the curl project turns 25 years old. Here’s a golden opportunity! Let’s top off the 25 year celebrations with a major version number bump!

The plan

Independently of what version number we have reached to at that point, independently of if we add features or not in that release, independently of exactly how that date fits within the pre-determined release cycle and without changing any APIs, we ship curl version 8.0.0 on that day.

Turning 25 and bumping the major version number on the same day should be fun.

Version 8

I hope that a small side-effect of bumping the major number will make users still left on version 7 to slightly faster feel outdated and push for getting up to 8. It could work as a minor push to get users to catch up a bit.

The minor number will of course immediately start “climbing” again and in a worst-case scenario, we risk reaching minor number 100 again within another 17 years. Maybe we can plan another bump for the 40th birthday?

cURL and libcurl

Making world-class docs takes effort

6 Comments

Here are six requirements that I have on a project for it to reach my gold approval for stellar docs. Then something about what I’ve done recently to further improve the docs for curl.

Your docs belong in the code repository

It needs to be next to the code so that authors and contributors can update/read the docs while working on the code or docs. Providing it in a separate repository or otherwise separated will undoubtedly lead to discrepancies sooner or later. Similar to how all wikis are always wrong.

Your docs is not extracted from code

Admit it. If you browse around you will realize that the best documented projects you find never provide that docs generated directly from code. Such generated docs can still provide value, but you will not reach gold level without more effort.

Separated docs encourage more writing and writing by people who are otherwise scared of touching the code or thinking that fixing spelling errors in the docs isn’t worth patching the code for. It also allows for other and better formatting on the docs.

Your docs features examples

Users always ask for (more) examples. You can never go wrong by providing examples. If your docs don’t have enough examples, you’re not doing the docs good enough yet.

You document every API call you provide

Fewer things make me sigh more than when I have to dig up the source code and from that try to figure out exactly how an external and publicly provided API works. Yet this still happens regularly even for libraries that have been released and maintained for decades.

In one fairly recent instance I reported such an omission in a popular library. It took them over two years to add it.

Long-living libraries should also provide information about from which versions certain functionality or options exist or can be expected to work or not work.

Your docs is easily accessible and browsed

Good docs means that we can find what we’re looking for and that the documentation flows and is easily read and understood. Ideally, even simple google searches for API details in your library should lead us to suitable entry points.

Preferably, the documentation should also be provided for proper off-line reading, meaning man pages or something similar that can be browsed when disconnected from the Internet.

Your docs should be easy to contribute to

The docs should be easy for contributors to help out with (independently from the code if desired). That also includes that they should be easy for contributors to build and render locally so that they can test and view their updates while working on them.

Documentation in the curl project

I want the documentation for curl and libcurl to be known, recognized and widely admitted to be world-class.

I want the curl documentation to be of a quality and content to make users not able to find competitors or similar projects with better docs.

Documentation in curl is not an after-thought. It is not a second-tier component. It is a crucial and important foundation that allows users to use, trust and rely on our products. We require that new changes or improved functionality are provided with the corresponding updated and accurate documentation.

We also try to verify and check the docs as much as possible with scanners , tests and tools.

Non-stop iterating is key

I maintain that our documentation is as good as it is today a lot thanks to us very rigidly sticking to our guiding principles: compatibility and not breaking existing behavior. Documentation we wrote decades ago is still valid. It gives us plenty of time to keep refining and polishing the documentation of a feature that doesn’t change. No documentation was perfect already at the first attempt, but after numerous iterations and improvements chances are it is better. Time is on our side. And we are never done, documentation can always be improved.

I’m putting in the work

A few days ago I talked documentation with someone and when doing so I thought about what guiding principles I think we should put on project documentation. What I’ve listed above basically.

In then dawned on me that the current man curl.1 man page is actually not featuring that many examples, in spite of it being 3535 lines long. I pondered a little bit on how best add some, and then dove in and extended our system that generates it from the hundreds of individual files that each describe a single command line option. They should of course all offer at least one example!

Having 242 command line options (as of right now, it will be more soon), that sudden idea that seemed simple enough turned into a quite gruesome work and I spent many hours walking over the options to make up examples. I also made sure that our build system now returns an error if there’s a command option without an example in the documentation! This way, we can be sure that also all future command line options will have examples in the man page.

This made the curl.1 man page grew with over 1200 lines!

libcurl options too

A few years ago I did a manual effort and made sure most man pages for libcurl options include examples, but I never made that into a test or anything so there’s nothing that forces us to stick to this.

Having started this journey, I decided now was the time to add that requirement to the scripts. I extended test case 1173, which already scans all man pages to verify some basic syntax, to also check that man pages for options feature an EXAMPLE section.

There are at this moment 374 stand-alone man pages for libcurl options. Only ten of them were detected to not feature good enough examples and it wasn’t very hard to fix this.

Consistency is good

Having just extended the man page checker, another idea came up.

I made the script also check that each man page features the correct set of sections, in the required order! I’m a true believer in consistency and that using the same set in the same order will make the docs easier to read and find information in, and checking for these things will make sure that all future additions will be forced stick to the same.

The mandatory sections for libcurl option man pages are right now, in this order:

NAME
SYNOPSIS
DESCRIPTION
PROTOCOLS
EXAMPLE
AVAILABILITY
RETURN VALUE
SEE ALSO

These man pages are allowed to have other sections as well, and they can be placed anywhere among the mandatory ones, but the eight section headers that has to be there has to be in that order.

Cross-references

While at it, I also extended the man page scanner to check that all references in all curl man pages to libcurl options are verified to actually refer to existing options, to find typos. Ironically this extra check turned out finding exactly no such typos in the current 463 man pages!

Future

The outcome of the work I write about here will of course be merged asap and will be part of future releases and on the website.

We should keep thinking of more ways to improve the documentation and for more ways to verify and cross-reference things mentioned in the docs to increase its accuracy and detect typos.

If you find a problem, an inferior wording or just something you think we should improve in any curl documentation, file a bug or a PR! We also try to make this as easy as possible for users to do directly from the curl website by providing bug-reporting direct links from the documentation pages.

Updates

I added the sixth “rule” days after the initial post after Willy Tarreau’s feedback.

cURL and libcurl

curl 7.78.0 five in one

Welcome to another release! We did more bug-fixes than in any previous release (176). We paid more in bug-bounties than during any previous release cycle (4,200 USD) and we thank more contributors in the RELEASE-NOTES than ever before (83).

Release presentation

Numbers

the 201st release
6 changes
56 days (total: 8,524)
176 bug-fixes (total: 7,142)
263 commits (total: 27,465)
0 new public libcurl function (total: 85)
0 new curl_easy_setopt() option (total: 290)
0 new curl command line option (total: 242)
83 contributors, 49 new (total: 2,459)
56 authors, 32 new (total: 933)
5 security fixes (total: 108)
4,200 USD paid in Bug Bounties (total: 13,200 USD)

Security

This time we announce no less than 5 separate security advisories and we are once again setting a new bug-bounty record. This release cycle we spent 4,200 USD on rewarding security researchers.

Let’s do them in numerical order. Click the CVE links to get to the full and much more detailed advisories.

CVE-2021-22922: Wrong content via metalink not discarded

This was one of the problems we found that that all together made us take the drastic decision to completely remove metalink support.

The metalink format has a hash for the content so that a client can detect faulty contents. curl didn’t act properly if the has mismatched and it could easily make users not realize the bad content.

CVE-2021-22923: Metalink download sends credentials

If you download the metalink file using credentials, the subsequent download(s) of the file mentioned in that XML file will also get the same credentials passed to those servers, unexpectedly, thus potentially leaking sensitive information to other parties!

CVE-2021-22924: Bad connection reuse due to flawed path name checks

libcurl keeps previously used connections in a connection pool for subsequent transfers to reuse, if one of them matches the setup.

Due to errors in the logic, the config matching function did not take ‘issuer cert’ into account and it compared the involved paths case insensitively, which could lead to libcurl reusing wrong connections!

CVE-2021-22925: TELNET stack contents disclosure again

Possibly the most embarrassing security flaw in a long time.

When we shipped 7.77.0 we announced CVE-2021-22898, which was a flaw in the telnet code and an associated fix. Know what? The fix was incomplete and plain wrong so the original problem actually remained for a certain set of input.

This is thus the second advisory for the same problem and now we fix this again. Hopefully for real and for good this time…

CVE-2021-22926: CURLOPT_SSLCERT mixup with Secure Transport

When libcurl is built to use the macOS native TLS library Secure Transport, an application can ask for the client certificate by name or with a file name – using the same option. If the name exists as a file, it will be used instead of by name. This could be exploited in rare circumstances.

Changes

The six big changes this time around are:

curl_url_set now rejects spaces in the URL unless specifically asked to allow them.

CURLE_SETOPT_OPTION_SYNTAX is a brand new return code (name) for when libcurl detects an illegally formatted input passed to a setopt(), when it is detected later in the transfer.

localhost is now always local!

The mbedTLS backend now supports client certificate and key provided as “blob options” in memory instead of as files.

Metalink support was dropped.

Now username and password can be used for MQTT transfers.

Bug-fixes

I’m doing this release in the midst of my vacation so I’m doing this section a little shorter than usual. Here are some bug-fixes to highlight:

Lots of tiny fixes when built to use hyper for HTTP. Now curl built to use hyper can run many more test cases. There’s more to do and more will be done going forward.

Travis CI is gone. Zuul and Circle CI are in.

GnuTLS: set the preferred TLS versions in correct order. Previously the occasional TLS connection would fail because of the wrong way the code would instruct GnuTLS…

on macOs: free returned memory of SCDynamicStoreCopyProxies. A small memory leak on Apple operating systems, possibly as many as one per name resolve?

HSTS: not experimental anymore. It is now built and provided by default.

netrc: skip ‘macdef’ definitions. The netrc parser is ancient but it turned out this kind of macro use could threw it off.

OpenSSL: don’t remove session id entry in disassociate. We had a regression that basically killed session-id use and made subsequent TLS handshakes to the same host much slower.

Next

The plans says we ship the next release on September 15th 2021. See you then!

cURL and libcurl

curl reaches 100K raised

4 Comments

I’m proud and happy to mention that curl just passed the magic limit of 100,000 USD in raised sponsorship money. Or call it donations if you want. Since April 2018. That’s about 40 months.

Screenshot 06:50 UTC 2021-07-06

Donate?

Do it here!

Donations over time

A grand total of 440 awesome organizations and individuals have donated money to the curl project since we started our Open Collective fund, at almost 1300 separate occasions. It makes the averages to be about 77 USD per donation and 230 USD per sponsor. As usual, there’s a very long tail of single sponsors that donated a small amount and there’s a small set of sponsors who have donated lots of money many times.

We use donated money primarily for the bug-bounty, but recently we also spread sticker love across the world with the help of donated funds. The fund will also be used to pay for our annual developer meetups (that have been paused during covid) and potentially for some hardware and other infrastructure to aid the project and it’s core contributors.

Note: that we also have a set of sponsors who fund services and infrastructure directly for us without funneling the money through us. The shear value of those services are in several instances even greater in total than what the largest monetary contributors have given us.

Net vs gross details

This counts the 100K USD net amount that ended up in our fund. That is with the fees involved already deducted. Gross, that means we were given more than 100K already.

Before Open Collective you ask?

We never saw any serious donations to speak of before we started this collective. Before then we received the occasional donations to my PayPal account but they were very spurious and very far apart and never amounted to any “real money”.

Independent

I want to take this opportunity and remind readers that curl is a totally independent and stand-alone project. We’re not part of any larger/umbrella organization and we’re not run or owned by any company. It gives us total freedom to do whatever we want but it also means we need to fund things ourselves and find our own benefactors. Fortunately, we have many friends!

Top donors

  1. Backblaze
  2. Indeed
cURL and libcurl

curl user survey 2021

2 Comments

It is time to once again tell you that people responded very similarly to how they did last year…

curl user survey 2021 analysys

Not a lot changed this year compared to last year. Perhaps the biggest three changes this year were that

1. HTTP/3, Unix domain sockets and DNS-over-HTTPS increased significantly among “used features”

2. NSS and GnuTLS both had their usage shares among used TLS libraries fall significantly.

3. My twitter account and this blog are now top-voted as the two channels people follow mostly for participation in curl related topics.

The most used protocols are of course still HTTPS and HTTP, and the newest supported protocol (GOPHERS) checks in as the least used protocol this time around.

Much more details can be found in the linked PDF. Enjoy.

cURL and libcurl

Sending those stickers

This part 2. See Giving a away an insane amount of curl stickers for part 1.

As suspected already from the start, I ran out of stickers really fast. I ordered more from my trusted sticker guy on the corner, and he could even deliver stickers put into pre-printed envelopes. Envelopes that even got a curl logo on them. Around two hundred recipients got stickers that way.

It took me a while to complete this task. Getting all the addresses organized took time, getting all the materials restocked took time, packaging sticker of different sorts to almost a thousand people took time and then I also of course had to do occasional work in the mean time so I didn’t finish the delivery from my end until near the end of June.

When I write this, I’ve just sent off the last few parcels. 978 recipients are now close(r) to get curl stickers.

My daughter Agnes helped me get things organized.

A backpack full of sticker delivery. A portion of them.

The envelop and stickers Colin ‘t Hart got.

Redistributors

The bulk amount of stickers would still get sent in larger batches to individuals who would send them on.

Thicker envelopes with stickers for multiple final recipients,, going out to redistributors around the world.

One redistributor single-handedly took 174(!) addresses, but otherwise they typically were in the range of 20-40 recipients each. Each redistributor got 4 * [number of recipients] + 10 stickers in their envelops.

The final batch of some of the larger shipments just before they were mailed away.

Types

To spice up this game a little bit, most packages have gotten stickers of more than one kind. Hardly any package got the exact same setup (apart from the first batch). So whatever you might end up getting in the end, please cherish what you got and enjoy them. They are the result of quite a lot of work, sweat and devotion from a lot of people.

I’ve reimbursed most of my expenses for this from the curl fund, which means that this effort was paid for at least in part by our sponsors and Open Collective donors. Thanks for that!

Ending words

All the stickers have not yet arrived at their final destinations, so I’m writing this up a little premature. There will be disappointments involved because this whole process is very human-centric and error prone. Some of the addresses we got probably don’t even work or will be mis-interpreted down the line and more. I’ve done this to the best of my ability.

The plans for future sticker offers as discussed in part 1 are still just plans and have not materialized anything further yet.

cURL and libcurl

On the GitHub ReadMe podcast

On May 17, I joined the Kathy and Brian, the hosts of the GitHub ReadMe podcast on a video meeting from my home and we had a chat. Mostly about my work on curl. Today the episode “aired”.

“curl: 25 years and 200 releases later”

You find it here. Also: Spotify. Apple podcasts. RSS feed.

curl is one of the most widely used software component in the world. It is over twenty years old and I am the founder and I still work as lead developer and head honcho. It works!

We talked about how I got into computers and open source in general. How curl started and about how it works to drive such a project, do releases and how to work on it as a full-time job. I am far from alone in this project – I’m just the captain of this ship with a large about of contributors onboard!

Photographs

As a part of the promotion for this episode, I was photographed by a professional outside of my house and nearby on a very lovely summer’s evening. In a southern suburb of Stockholm, Sweden. So, not only does the GitHub material feature not previously seen images of me, since I’ve been given the photos I can now use them for various things going forward. Like for when I do presentations and organizers ask for photos etc.

Photos of Daniel

The photos I’ve used most commonly up until this point are the ones a professional photographer took of me when I spoke at the Velocity conference in New York in 2015. Of course I’m eternally young, but for some reason those past six years are visible on me…

Podcasts

I’ve participated in some podcasts before. If my count is correct, this is the 19th time. See the whole list.

Credits

The new set of photos of me were shot by Evia Photos. One of them is used on the top of this page.

cURL and libcurl

What goes into curl?

curl is a command line tool and library for doing Internet data transfers. It has been around for a loooong time (over 23 years) but there is still a flood of new things being added to it and development being made, to take it further and to keep it relevant today and in the future.

I’m the lead developer and head maintainer of the curl project.

How do we decide what goes into curl? And perhaps more importantly, what does not get accepted into curl?

Let’s look how this works in the curl factory.

Stick to our principles

curl has come this far by being reliable, trusted and familiar. We don’t rock the boat: curl does Internet transfers specified as URLs and it doesn’t parse or understand the content it transfers. That goes for libcurl too.

Whatever we add should stick to these constraints and core principles, at least. Then of course there are more things to consider.

A shortlist of things I personally want to see

I personally usually have a shortlist of a few features I personally want to work on in the coming months and maybe half year. Items I can grab when other things are slow or if I need a change or fun thing to work on a rainy day. These items are laid out in the ROADMAP document – which also tends to be updated a little too infrequently…

There’s also the TODO document that lists things we consider could be good to do and KNOWN_BUGS that lists known shortcomings we want to address.

Sponsored works have priority

I’m the lead developer of the curl project but I also offer commercial support and curl services to allow me to work on curl full-time. This means that paying customers can get a “priority lane” into landing new features or bug-fixes in future releases of curl. They still need to suit the project though, we don’t abandon our principles even for money. (Contact me to learn how I can help you get your proposed changes done!)

Keep up with where the world goes

All changes and improvements that help curl keep up with and follow where the Internet protocol community is moving, are considered good and necessary changes. The curl project has always been on the front-lines of protocols and that is where we want to remain. It takes a serious effort.

Asking the community

Every year around the May time frame we do a “user survey” that we try to get as many users as possible to respond to. It asks about user patterns, what’s missing and how things are working.

The results from that work provide good feedback on areas to improve and help us identify features our community think curl lacks etc. (The 2020 survey analysis)

Even outside of the annual survey, discussions on the mailing list is a good way for getting direct feedback on questions and ideas and users very often bring up their ideas and suggestions using those channels.

Ideas are easy, code is harder

Actually implementing and providing a feature is a lot harder than just providing an idea. We almost drown among all the good ideas people propose we might or could do one day. What someone might think is a great idea may therefore still not be implemented very soon. Because of the complexity of implementing it or because of lack of time or energy etc.

But at the same time: oftentimes, someone needs to bring the idea or crack the suggestion for it to happen.

It needs to exist to be considered

Related to the previous section. Code and changes that exist, that are provided are of course much more likely to actually end up in curl than abstract ideas. If a pull-request comes to curl and the change adheres to our standards and meet the requirements mentioned in this post, then the chances are very good that it will be accepted and merged.

As I am currently the only one working on curl professionally (ie I get paid to do it). I can rarely count on or assume work submissions from other team members. They usually show up more or less by surprise, which of course is awesome in itself but also makes such work and features very hard to plan for ahead of time. Sometimes people bring new features. Then we deal with them!

Half-baked is not good enough

A decent amount of all pull requests submitted to the project never get merged because they aren’t good enough and the person who submitted them doesn’t respond to feedback and improvement requests properly so that they never become good enough. Things like documentation and tests are typically just as important as the functionality itself.

Pull requests that are abandoned by the author can of course also get taken over by someone else but it cannot be expected or relied upon. A person giving up on the pull request is also a strong sign to the rest of us that obviously the desire to get that specific change landed wasn’t that big and that tells us something.

We don’t accept and merge partial changes that for example lack a crucial part like tests or documentation because we’ve learned the hard way many times over the years that it is just too common that the author then vanishes before completing the work – forcing others to do that work or we have to rip the change out again.

Standards and in-use are preferred properties

At times people suggest we support new protocols or experiments for new things. While that can be considered fun and useful, we typically want both the protocol and the associated URL syntax to already be in use and be somewhat established and preferably even standardized and properly documented in specifications. One of the fundamental core ideas with URLs is that they should mean the same thing for more than one application.

When no compass needle exists, maintain existing direction

Most changes are in line with what we already do and how the products work so no major considerations are necessary. Only once in a while do we get requests or suggestions that actually challenge the direction or forces us to consider what is the right and the wrong way.

If the reason and motivation provided is valid and holds up, then we might agree and go in that direction, If we don’t, we discuss the topic and see if we perhaps can change someone’s mind or “wiggle” the concepts and ideas to see whether we can change the suggestion or perhaps see it from n a different angle to reconsider. Sometimes we just have to decline and say no: that’s not something we think is in line with curl.

Who decides if its fine?

curl is not a democracy, we don’t vote about decisions or what to accept etc.

curl is also not a strict dictatorship where a single leader dictates all truths and facts from above for all subjects to accept and obey.

We’re somewhere in between. We discuss and try to find consensus of what and how to do things. The persons who bring the code or experience the actual problems of course will have more to say. Experienced and long-term maintainers’ opinions have more weight in discussions and they’re free and allowed to merge pull-requests they think are good.

I retain the right to veto stuff, but I very rarely exercise that right.

curl is still a small project. You’ll notice that you’ll quickly recognize the same handful of maintainers in all pull-requests and long tail of others chipping in here and there. There’s no massive crowd anywhere. That’s also the explanation why sometimes your pull-requests might not get reviewed instantly but you must rather wait for a while until you get someone’s attention.

If you’re curious to learn how the project is governed in more detail, then check out the governance docs.

How to land code in curl

I’ve done a previous presentation on how to work with the project get your code landed in curl. Check it out!

Your feedback helps!

Listening to what users want, miss and think are needed when going forward is very important to us. Even if it sometimes is hard to react immediately and often we have to bounce things a little back and forth before they can become “curl material”. So, please don’t expect us to immediately implement what you suggest, but please don’t let that stop you from bringing your grand ideas.

And bring your code. We love your code.

cURL and libcurl

Bye bye Travis CI

10 Comments

In the afternoon of October 17, 2013 we merged the first config file ever that would use Travis CI for the curl project using the nifty integration at GitHub. This was the actual introduction of the entire concept of building and testing the project on every commit and pull request for the curl project. Before this merge happened, we only had our autobuilds. They are systems run by volunteers that update the code from git maybe once per day, build and run the tests and then upload all the logs.

Don’t take this wrong: the autobuilds are awesome and have helped us make curl what it is. But they rely on individuals to host and admin the machines and to setup the specific configs that are tested.

With the introduction of “proper” CI, the configs that are tested are now also hosted in git and allows the project members to better control and adjust the tests and configs, plus that we can run them on already on pull-requests so that we can verify code before merge instead of having to first merge the code to master before the changes can get verified.

Seriously. Always.

Travis provided a free service with a great promise.

Promise from Travis website as recently as late 2020.

In 2017 we surpassed 10 jobs per commit, all still on Travis.

In early 2019 we reached 30 jobs per commit, and at that time we started to use and spread out the work on more CI services. Travis would still remain as the one we’d lean on the heaviest. It was there and we had custom-written a bunch of jobs for it and it performed well.

Travis even turned some levers for us so that we got more parallel processing powers than on the regular open source tier, and we listed them as sponsors of the curl project for their gracious help. This may or may not be related to the fact that I met Josh Kalderimis (co-founder of travis) in 2019 and we talked about curl’s use of it and they possibly helping us more.

Transition to death

This year, 2021, the curl project runs around 100 CI jobs per commit and PR. 33 of them ran on Travis when we were finally pushed over from travis-ci.org to their new travis-ci.com domain. A transition they’d been advertising for a while but wasn’t very clearly explained or motivated in my eyes.

The new domain also implied new rules and new tiers, we quickly learned. Now we would have to apply to be recognized as an open source project (after 7.5 years of using their services as an open source project). But also, in order to get to take advantage of their free tier being an open source project was no longer enough. Among the new requirements on the project was this:

Project must not be sponsored by a commercial company or
organization (monetary or with employees paid to work on the project)

We’re a small independent open source project, but yes I work on curl full-time thanks to companies paying for curl support. I’m paid to work on curl and therefore we cannot meet that requirement.

Not eligible but still there

I’m not sure why, but apparently we still got free “credits” for running CI on Travis. The CI jobs kept working and I think maybe I sighed a little from relief – of course I did it prematurely as it only took us a few days into the month of June until we had run out of the free credits. There’s no automatic refill but we can apparently ask for more. We asked, but many days after having asked we still had no more credits and no CI jobs could run on Travis anymore. CI on Travis at the same level as before would cost more than 249 USD/month. Maybe not so much “it will always be free”.

The 33 jobs on Travis were there for a purpose. They’re prerequisites for us to develop and ship a quality product. Without the CI jobs running, we risk landing bad code. This was not a sustainable situation.

We looked for alternative services and we quickly got several offers of help and assistance.

New service

Friends from both Zuul CI and Circle CI stepped up and helped us started to get CI jobs transitioned over from Travis over to their new homes.

At June 14th 2021, we officially had no more jobs running on Travis.

Visualized as a graph, we can see the Travis jobs “falling off a cliff” with Zuul rising to the challenge:

Services come and go. There’s no need to get hung up on that fact but instead keep moving forward with our eyes fixed on the horizon.

Thank you Travis CI for all those years of excellent service!

Pay up?

Lots of people have commented and think I’m “whining” about Travis CI charging for something that is useful and that I should rather just pay up. I could probably have gone with that but I dislike their broken promise and that they don’t consider us Open source anymore and I feel I have a responsibility to use the funds we get from gracious donors as wisely and economically as possible, and that includes using no-cost or cheap services rather than services charging thousands of dollars per year.

If there really were no other available and viable options, then paying could’ve been an alternative. Now, moving on to something else was the right choice for us.

Credits

Image by Gerd Altmann from Pixabay

cURL and libcurl

Bye bye metalink in curl

3 Comments

In 2012 I wrote a blog post titled curling the metalink, describing how we added support for metalink to curl.

Today, we remove that support again. This is a very drastic move, and I feel obliged to explain it so here it goes! curl 7.78.0 will ship without metalink support.

Metalink problems

There were several issues found that combined led us to this move.

Security problems

We’ve found several security problems and issues involving the metalink support in curl. The issues are not detailed here because they’ve not been made public yet.

When working on these issues, it become apparent to the curl security team that several of the problems are due to the system design, metalink library API and what the metalink RFC says. They are very hard to fix on the curl side only.

Unusual use pattern

Metalink usage with curl was only very briefly documented and was not following the “normal” curl usage pattern in several ways, making it surprising and non-intuitive which could lead to further security issues.

libmetalink is abandoned

The metalink library libmetalink was last updated 6 years ago and wasn’t very actively maintained the years before that either. An unmaintained library means there’s a security problem waiting to happen. This is probably reason enough.

XML is heavy

Metalink requires an XML parsing library, which is complex code (even the smaller alternatives) and to this day often gets security updates.

Not used much

Metalink is not a widely used curl feature. In the 2020 curl user survey, only 1.4% of the responders said that they’d are using it. In the just closed 2021 survey that number shrunk to 1.2%. Searching the web also show very few traces of it being used, even with other tools.

The torrent format and associated technology clearly won for downloading large files from multiple sources in parallel.

Violating a basic principle

This change unfortunately breaks command lines that uses --metalink. This move goes directly against one of our basic principles as it doesn’t maintain behavior with previous versions. We’re very sorry about this but we don’t see a way out of this pickle that also takes care of user’s security – which is another basic principle of ours. We think the security concern trumps the other concerns.

Possible to bring back?

The list above contains reasons for the removal. At least some of them can be addressed given enough efforts and work put into it. If someone is willing to do the necessary investment, I think we could entertain the possibility that support can be brought back in a future. I just don’t think it is very probable.

Credits

Image by Ron Porter from Pixabay