It is often said that Imitation is the Sincerest Form of Flattery.
Also, remember libcrurl? That was the name of the thing Google once planned to do: reimplement the libcurl API on top of their Chrome networking library. Flattery or not, it never went anywhere.
The other day I received an email asking me about details regarding something called xCurl. Not having a clue what that was, a quick search soon had me enlightened.
xCurl is, using their own words, a Microsoft Game Development Kit (GDK) compliant implementation of the libCurl API.
A Frankencurl
The article I link to above describes how xCurl differs from libcurl:
xCurl differs from libCurl in that xCurl is implemented on top of WinHttp and automatically follows all the extra Microsoft Game Development Kit (GDK) requirements and best practices. While libCurl itself doesn’t meet the security requirements for the Microsoft Game Development Kit (GDK) console platform, use xCurl to maintain your same libCurl HTTP implementation across all platforms while changing only one header include and library linkage.
I don’t know anything about WinHttp, but since that is an HTTP API I can only presume that making libcurl use that instead of plain old sockets has to mean a pretty large surgery and code change. I also have no idea what the mentioned “security requirements” might be. I’m not very familiar with Windows internals nor with their game development setups.
The article then goes on to describe with some detail exactly which libcurl options that work, and which don’t and what libcurl build options that were used when xCurl was built. No DoH, no proxy support, no cookies etc.
The provided functionality is certainly a very stripped down and limited version of the libcurl API. A fun detail is that they quite bluntly just link to the libcurl API documentation to describe how xCurl works. It is easy and convenient of course, and it will certainly make xCurl “forced” to stick to the libcurl behavior
With large invasive changes of this kind we can certainly hope that the team making it has invested time and spent serious effort on additional testing, both before release and ongoing.
Source code?
I have not been able to figure out how to download xCurl in any form, and since I can’t find the source code I cannot really get a grip of exactly how much and how invasive Microsoft has patched this. They have not been in touch or communicated about this work of theirs to anyone in the curl project.
Therefore, I also cannot say which libcurl version this is based on – as there is no telling of that on the page describing xCurl.
The email that triggered me to crawl down this rabbit hole included a copyright file that seems to originate from an xCurl package, and that includes the curl license. The curl license file has the specific detail that it shows the copyright year range at the top and this file said
Copyright (c) 1996 - 2020, Daniel Stenberg, daniel@haxx.se, and many contributors, see the THANKS file.
It might indicate that they use a libcurl from a few years back. Only might, because it is quite common among users of libcurl to “forget” (sometimes I’m sure on purpose) to update this copyright range even when they otherwise upgrade the source code. This makes the year range a rather weak evidence of the actual age of the libcurl code this is based on.
Updates
curl (including libcurl) ships a new version at least once every eight weeks. We merge bugfixes at a rate of around three bugfixes per day. Keeping a heavily modified libcurl version in sync with the latest curl releases is hard work.
Of course, since they deliberately limit the scope of the functionality of their clone, lots of upstream changes in curl will not affect xCurl users.
License
curl is licensed under… the curl license! It is an MIT license that I was unclever enough to slightly modify many years ago. The changes are enough for organizations such as SPDX to consider it a separate one: curl. I normally still say that curl is MIT licensed because the changes are minuscule and do not change the spirit of the license.
The curl license of course allows Microsoft or anyone else do to this kind of stunt and they don’t even have to provide the source code for their changes or the final product and they don’t have to ask or tell anyone:
Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.
I once picked this license for curl exactly because it allows this. Sure it might sometimes then make people do things in secret that they never contribute back, and we miss out on possible improvements because of that, but I think the more important property is that no company feels scared or restricted as to when and where they can use this code. A license designed for maximum adoption.
I have always had the belief that it is our relentless update scheme and never-ending flood of bugfixes that is what will keep users wanting to use the real thing and avoid maintaining long-running external patches. There will of course always be exceptions to that.
Follow-up
Forensics done by users who installed this indicate that this xCurl is based on libcurl 7.69..x. We removed a define from the headers in 7.70.0 (CURL_VERSION_ESNI) that this package still has. It also has the CURLOPT_MAIL_RCPT_ALLLOWFAILS define, added in 7.69.0.
curl 7.69.1 was released on March 11, 2020. It has 40 known vulnerabilities, and we have logged 3,566 bugfixes since then. Of course not all of any of those affect xCurl.
“lots of upstream changes in curl will not affect xCurl users”
I wonder if it’s a WinHttp backend for curl, or a totally new library with a curl interface. If it’s the latter, I wonder why they don’t make an attempt to upstream it.
@Matthew: I think wording implies that it is the first: a modification that uses lots of libcurl code but where they replaced underlying code to use WinHTTP instead of sockets. But this is just my guess.
There are some clear upsides to using WinHTTP instead of sockets. Like HTTP queries result in intermediate CA’s being automatically fetched and persisted through Windows Update rather than needing to worry about CA chasing schemes. But obvious downsides like you are limited to OS crypto stack for TLS versions and ciphers and Microsoft has historically not really been backporting these. I have seen organizations using both. I suppose compat API layer (xCurl?) is nice if you want to experiment which one works better for your use cases without touching functional code.
@Seppo: you can use the Windows CA store using vanilla curl already, when using the Schannel, OpenSSL, wolfSSL or GnuTLS backends. That would be a rather weak reason I think.
@Daniel: simply using CA store will *not* auto-fetch missing certificates. It does help if there are enterprise internal CA’s though. Schannel admittedly does have same CA behaviour as WinHTTP but its documentation is not great and it’s quite buggy. WinHTTP abstracts these away.
What are the potential implications of xCurl not supporting features like DoH, proxy support, and cookies, which are available in libCurl?
@JChrisJ: only that this range of features are missing. Users of xcurl can’t use them. It makes them less flexible than users of the “real” and fully featured libcurl. Presumably, many users will carry on just fine as those are features they don’t need. Some users will not.
Isn’t curl binary on Windows 10 also built on top of this (or is it only the SSL code?) (–version output below).
I was bit by this recently when trying to do tests with a public API that required client certificate files with full chains (ie one pub/priv pair and a bunch of public chain parts), the windows curl failed because it ignored the chain and only picked up one part whereas linux (and curl binaries from the homepage) used openssl that happily loaded the full chain.
curl 8.4.0 (Windows) libcurl/8.4.0 Schannel WinIDN
@Jonas: no, curl on Windows 10/11 is done using our unmodified upstream code, just built by Microsoft with a specific set of build options. In particular the Schannel TLS backend.
Yeah those corporate kindlings deeming themselves “senior developers” or whatnot, have those ideas all the time. In this case the schizophrenia is that on one hand Microsoft is using unmodified curl in Windows (used by “billions”), and it’s okay. But on the other hand for their game blahblah SDK the “requirements” are higher so they do need to have to maintain separate fork. That’s not curable I think.
Wait for Apple to adopt it, and redirect support requests to you 🙂
typo, ‘the’ should be ‘they’ in this line “A fun detail is that the quite bluntly just link to the libcurl API documentation”.
@Jesse: thanks, fixed