I’ve grown a bit tired of the web filling up with curl command line examples showing use of superfluous -X’s. I’m putting code where my mouth is.
Starting with curl 7.45.0 (due to ship October 7th 2015), the tool will help users to understand that their use of the -X (or --request) is very often unnecessary or even downright wrong. If you specify the same method with -X that will be used anyway, and you have verbose mode enabled, curl will inform you about it and gently push you to stop doing it.
Example:
$ curl -I -XHEAD http://example.com --verbose
The option dash capital i means asking curl to issue a HEAD request. Adding -X HEAD to that command line asks for it again. This option sequence will now make curl say:
Note: Unnecessary use of -X or --request, HEAD is already inferred.
It’ll also inform the user similarly if you do -XGET on a normal fetch or -XPOST when using one of the -d options. Like this:
$ curl -v -d hello -XPOST http://example.com Note: Unnecessary use of
-Xor--request, POST is already inferred.
curl will still continue to work exactly like before though, these are only informational texts that won’t alter any behaviors. Again, it only says this if verbose mode is enabled.
What -X does
When doing HTTP with curl, the -X option changes the actual method string in the HTTP request. That’s all it does. It does not change behavior accordingly. It’s the perfect option when you want to send a DELETE method or TRACE or similar that curl has no native support for and you want to send easily. You can use it to make curl send a GET with a request-body or you can use it to have the -d option work even when you want to send a PUT. All good uses.
Why superfluous -X usage is bad
I know several users out there will disagree with this. That’s also why this is only shown in verbose mode and it only says “Note:” about it. For now.
There are a few problems with the superfluous uses of -X in curl:
One of most obvious problems is that if you also tell curl to follow HTTP redirects (using -L or --location), the -X option will also be used on the redirected-to requests which may not at all be what the server asks for and the user expected. Dropping the -X will make curl adhere to what the server asks for. And if you want to alter what method to use in a redirect, curl already have dedicated options for that named --post301, --post302 and --post303!
But even without following redirects, just throwing in an extra -X “to clarify” leads users into believing that -X has a function to serve there when it doesn’t. It leads the user to use that -X in his or her’s next command line too, which then may use redirects or something else that makes it unsuitable.
The perhaps biggest mistake you can do with -X, and one that now actually leads to curl showing a “warning”, is if you’d use -XHEAD on an ordinary command line (that isn’t using -I). Like this (I’ll display it crossed over to make it abundantly clear that this is a bad command line):
$ curl -XHEAD http://example.com/
… which will have curl act as if it sends a GET but it sends a HEAD. A response to a HEAD never has a body, although it sends the size of the body exactly like a GET response which thus mostly will lead to curl to sit there waiting for the response body to arrive when it simply won’t and it’ll hang.
Starting with this change, this is the warning it’ll show for the above command line:
Warning: Setting custom HTTP method to HEAD may not work the way you want.
If people misuse X to add clarity, why not offer flags that supports their goal?
I understand your point about -X being wrong because of its probably unwanted effects but claiming it the use is unnecessary misses the point.
The current flags -d and -I are not intuitive for anyone not using the tool frequently.
The last message “may not work the way expected” is very vague and not really helpful. Why not include a link to this post or at least include a suggestion for the -I flag.
Peter: curl has always abstracted the underlying protocol to some extent when doing normal operations. It downloads and uploads, you don’t need to know the command for fetching files over FTP nor HTTP, it just gets them. Specifying the HTTP method to use is not abstracting. I don’t think we should require that from users. I want to offer options that “performs transfers” without users having to know the protocol specs. GET and HEAD are not intuitive either unless the person has studied the protocol.
The “may not work as expected” is intentionally vague. I hope it’ll make the user reread the sections of the docs again and figure out what he/she wants.
If you have suggestions on how to improve things, please take them to the curl-users mailing list and we’ll discuss them!
I feel like this is a sign that the curl command line interface is counter-intuitive and not providing what users want. The people using -X know what they want at a HTTP level (because it’s a standard that applies to web browsers and things not-curl); they’re trying to construct raw HTTP requests on the command line, not fetch the resource it whatever way is appropriate. In my case, usually to figure out what I want so that I can duplicate it in a web browser / etc. So here curl trying to be smart is hindering my actual goal; it just happens to still be a better tool for it than wget 😉
So in my use case, having `-X HEAD` imply `-I` would be better, but that would be a backwards-incompatible change; so that second warning is the more useful of the two.
Mook: I’m convinced the command line interface isn’t ideal for any single individual since it is the result of almost twenty years of evolutionary development. But we’re open source and we discuss all uses open and we allow anyone to bring their ideas or objections to anything and everything we do.
curl doesn’t “hinder” this at all. It provides an option and it works as documented. It’s just that many people use the options found in examples without ever reading the documentation. This new text output also doesn’t change any behavior, it merely helps users understand how it actually works.
Perhaps theres a need for a souped-up version of -X (let’s call it -XX for now) whose value supersedes any conflicting command line options, and suppresses the new warnings about redundant parameters. So “curl -XX HEAD -d ‘some payload’ …” would actually drop the payload and perform a HEAD request, whilst “curl -XX POST -d ‘some payload’ …” would behave as it used to, without a warning.
This would simplify some scripts. For example I’ve been working with a RESTful API recently for which I need to send GET, POST and DELETE requests to the same URL. It’s easy to just switch the -X parameter (the small payload is ignored in the GET and DELETE cases, so I send it anyway to simplify the code), but now that will give me a warning in the POST case.
Hi Daniel
I love curl. However, as a casual user, I *hate* *hate* *hate* ‘warning’ messages that are a tease. Its like the developer is taunting me.
1. “I know something you don’t”.
2. Or “I am too l33t to bother telling you the ignorant peon luser, what is wrong. My documentation is so awwwwesooome”
3. “You are such a peon luser that I am sure that you have nothing more important than to read MY documentation.”
4. “Yes I know it is 3 am and you are trying to get your down server back up, and you really want to go to bed. But guess what, I, Daniel, know that 3 am with a down server and a boss that is calling every 10 minutes, is the perfect time to learn the gloriousness that is curl”
Vague error/warning messages are lazy and insulting and I typically run into them exactly when I am desperately trying to solve a time critical problem. Don’t make my stress level higher.
Pat: so you wrote all that but didn’t tell us what you think is the better solution? I’ve explained why I think using -X like this is wrong. What _else_ can I do than produce these informational messages?
Hi Daniel,
The messages are *not* informative. That was my whole point.
1. The messages provided no guidance of alternative ways of doing things. (i.e. suggest alternative option combinations)
2. The messages provided no suggestions of next steps.
3. New version of curl looks broken: previous versions of curl did not issue the message). Maybe something is broken in the new version of curl – because old version worked just fine?
4. The messages don’t provide additional resources/links for user to look up.
All the messages do is say “I don’t like something”.
———
Example improvements:
“curl -I -XHEAD http://example.com –verbose”
–> Note: -I option means -XHEAD or –request is unnecessary. Equivalent command is: “curl -I http://example.com –verbose” See http://daniel.haxx.se/blog/2015/09/11/unnecessary-use-of-curl-x
curl -XHEAD http://example.com/
–> Warning: -XHEAD is set on http message but curl will be expecting a body ( which may not be sent) – try using “curl xxxxxx http://example.com” to send just the HEAD request or “curl http://example.com” for a GET request.