tldr: Apple thinks it is fine. I do not.
On December 28 2023, bugreport 12604 was filed in the curl issue tracker. We get a lot issues filed most days so this fact alone was hardly anything out of the ordinary. We read the reports, investigate, ask follow-up questions to see what we can learn and what we need to address.
The title stated of the problem in this case was quite clear: flag –cacert behavior isn’t consistent between macOS and Linux, and it was filed by Yuedong Wu.
The friendly reporter showed how the curl version bundled with macOS behaves differently than curl binaries built entirely from open source. Even when running the same curl version on the same macOS machine.
The curl command line option --cacert provides a way for the user to say to curl that this is the exact set of CA certificates to trust when doing the following transfer. If the TLS server cannot provide a certificate that can be verified with that set of certificates, it should fail and return error.
This particular behavior and functionality in curl has been established since many years (this option was added to curl in December 2000) and of course is provided to allow users to know that it communicates with a known and trusted server. A pretty fundamental part of what TLS does really.
When this command line option is used with curl on macOS, the version shipped by Apple, it seems to fall back and checks the system CA store in case the provided set of CA certs fail the verification. A secondary check that was not asked for, is not documented and plain frankly comes completely by surprise. Therefore, when a user runs the check with a trimmed and dedicated CA cert file, it will not fail if the system CA store contains a cert that can verify the server!
This is a security problem because now suddenly certificate checks pass that should not pass.
I reported this as a security problem in an email sent to Product Security at Apple on December 29 2023, 08:30 UTC. It’s not a major problem, but it is an issue.
Apple’s says it is fine
On March 8, 2024 Apple Product Security responded with their wisdom:
Hello, Thank you again for reporting this to us and allowing us time to investigate. Apple’s version of OpenSSL (LibreSSL) intentionally uses the built-in system trust store as a default source of trust. Because the server certificate can be validated successfully using the built-in system trust store, we don't consider this something that needs to be addressed in our platforms. Best regards, KC Apple Product Security
Case closed.
I disagree
Obviously I think differently. This undocumented feature makes CA cert verification with curl on macOS totally unreliable and inconsistent with documentation. It tricks users.
Be aware.
Since this is not a security vulnerability in the curl version we ship, we have not issued a CVE or anything for this problem. The problem is strictly speaking not even in curl code. It comes with the version of LibreSSL that Apple ships and builds curl to use on their platforms.
Nice of you guys to look into investigate these kinds of issues, tbh I would close the issue pretty quickly and advise the reporter to follow up with the OS vendor.
In this case I thought it was clearly a security related problem and I did not want to highlight that in the public issue so I decided to forward the issue myself.
It may be a security issue, but likely on the insecure user’s device, as it’s likely that anything new generated by the user will be rejected because it’s not secure.
Can this then be considered a backdoor?
I mean, if Apple gets appraoched by let’s say some government agency (hypothetical scenario) to look into the communication of an individual, they could simply “fake” a CA in the cert store and read along the communication.
Are you sure they understood the issue? The response says:
> as a default source of trust.
But the reported issue clearly isn’t about the default. It is about trusting the system CA list even when explicitly requested not to.
I’ve explained the issue to Apple Product Security to the best of my abilities. I have to assume that they are intelligent enough to understand the issue or that they would have asked me for clarifications of they did not. Of course I cannot guarantee that they understood it.
The dialogue with Apple has been sparse. They clearly don’t waste any words.
I work for ${giant SaaS company} and we very much had a similar experience with Apple. We needed clarification on their undocumented mass-enablement Apple Pay API, our use of which I’m certain has led to plenty of dollars in their pockets, and it was frustrating how much they simply don’t care.
> Obviously I think differently. This undocumented feature makes CA cert verification with curl on macOS totally unreliable and inconsistent with documentation. It tricks users.
You should add a note about it to the curl docs so that this issue does not get raised again
“It’s not a bug, a backdoor is a feature.
Best regards,
Apple”
–cacert
(TLS) Tells curl to use the specified
certificate file to verify the peer. The file
may contain multiple CA certificates. The
certificate(s) must be in PEM format. Normally
curl is built to use a default file for this, so
this option is typically used to alter that
default file.
…
(iOS and macOS only) If curl is built against
Secure Transport, then this option is supported
for backward compatibility with other SSL
engines, but it should not be set. If the option
is not set, then curl uses the certificates in
the system and user Keychain to verify the peer,
which is the preferred method of verifying the
peer’s certificate chain.
Ah, well at least they’ve documented their usual “we feel like doing things differently so we’re going to”, albeit obscurely (particularly if that note is a footer as your post implies; currently away from my machine or I’d check myself).
So Apple has done this in an attempt to enforce the use of the keychain as the CA source of truth. Surely there’s a better way to encourage that pattern.
Wow, Apple is definitely wrong on this. That means of you can’t script a cert-pinned app on MacOS. I do that all the time for intranet apps that use our corporate PKI.
Just make your own TLS implementation. We did it at our company because we want complete control like that. Took roughly 2 weeks in total with various extensions like false start implemented too.
Wow. What a lame response from Apple.
Since they’re hand waving the problem away to LibreSSL, have we any idea what their response to this is?
I’m not letting Apple off the hook, just curious.
Worth flashing a warning in future builds?
if macOS and cacert == true:
echo “-cacert works differently here, see: X”
Statement: I am not an Apple employee and I have no interest in Apple.
Checking the source code Apple has released, it seems we can disable the behavior by `OPENSSL_X509_TEA_DISABLE=1`.
I believe it would be great if Apple’s reply could point this out.
See https://github.com/curl/curl/issues/12604#issuecomment-1987645685
I don’t understand the big deal. Apple wants a walled garden of computing. So, anyone contributing to that is helping the creation of a dystopia.
Clearly, you just need to rename curl to danielhaxx and get a trademark for that name. Then, you give everyone the right to distribute *unmodified* versions under that trademark or versions that meet some specification.
This would stop Apple from doing that with the new danielhaxx program.
Since you do not own the trademark, all you can do is whine about it on your blog.
We established already that you do not care one bit about actual security (even the White House got the memo that C is not a programming language one should use anymore). As such, this is like the department of Swiss cheese being concerned about holes.
The *only* reason curl is popular, is because it is *free* and it might have been first. Using curl is a temporary cost cutting measure. curl is a tool for the poor.
curl is just the hobby of some guy that has no business providing a service to a billion people.
What a horrible thing to say to someone giving you a gift. Is this how you were raised? Shame on you.
As a response to the now-deleted reply by Art: no, when well-adjusted people receive a gift they don’t like, they shut up, take it, and deal with it on their own time.
@Rain: yes, I decided to not let Aart continue his crusade against me here. I think his initial post says everything about what kind of person he is. No point in walking any further down that road.
As Brett Cannon once said, “Open Source is a kindness”. Whether or not you think you’re too good for that kindness is your choice of course, but it’s also my choice to declare that your attitude is kinda lame.
Well at least he created something useful for billion of people for free, while you are ranting on his blog about the same thing. Who’s the poor now, salty kid?
Would be curious if the same behavior exists in capath. Based on their logic as articulated in the reply you got, I suspect it would.
This introduces a potential disaster scenario for applications that rely on certificate validation if additional checks are not made… I’ve personally considered macOS to be a less “enterprise ready” environment for many reasons, but this one is going to go into my top 10.
The idea that I could develop an application that relies on the validation of a particular set of certificates but an attacker who manages to get a cert signed by any system trusted root authority can have its data seen as trusted is obviously a huge risk that must be considered and controlled for.
This really goes against the spirit of cross platform compatibility. When they change the mechanics of how features work, what they are purporting to be “curl” is not curl. Going down the suggestion made (I presume in jest) by Aart de Vries, this really should be called macoscurl or something else.
And the MUCH bigger issue is, this is something that Daniel discovered entirely by accident. He found that Apple changed the way a security validation mechanisms works within a very popular service, told no one, and when asked, says “this kind of thing isn’t a problem.” That makes me wonder how many other security or other features that we all expect to behave a certain way are actually doing something different in the backend when run on macOS, and what this could mean to the security of our applications.
I just read the article a few moments ago, but I’m already starting to re-think the way I rely on any OS provided executables and libraries.
It’s not related to curl source directly, and not even the curl binary (unless statically compiled?). That takes “unit tests” out of equation for this..
But I believe, a platform integration test suite could be implemented (if not already).. which tests the expected behavior of the curl. So any system or any builds that cannot pass those tests can be assumed as “broken”.
Apple could keep shipping it “broken” if they want so. At least the situation would be correctly tagged as such on curl end.
@Furkan: Who would care if a test would fail because of this? Apple has obviously decided this is functionality they want. It is not a bug to them. Why would they care if we add a test that would fail for them? They have already decided to ship a broken curl.
Thanks for pointing out this ‘bug’ in curl as shipped in current MacOS. I wonder when this was introduced? Anyway, it’s important to be able to look up these hidden variants on different systems so you can bypass them if needed. For what it’s worth, you can side-step this by building curl yourself or installing it via homebrew or MacPorts and using that version in your scripts running on your Mac.
@Gordon: I presume it has worked like this more or less “forever”, but I don’t think anyone has done the necessary digging to actually tell.