Why curl defaults to stdout

(Recap: I founded the curl project, I am still the lead developer and maintainer)

When asking curl to get a URL it’ll send the output to stdout by default. You can of course easily change this behavior with options or just using your shell’s redirect feature, but without any option it’ll spew it out to stdout. If you’re invoking the command line on a shell prompt you’ll immediately get to see the response as soon as it arrives.

I decided curl should work like this, and it was a natural decision I made already when I worked on the predecessors during 1997 or so that later would turn into curl.

On Unix systems there’s a common mantra that “everything is a file” but also in fact that “everything is a pipe”. You accomplish things on Unix by piping the output of one program into the input of another program. Of course I wanted curl to work as good as the other components and I wanted it to blend in with the rest. I wanted curl to feel like cat but for a network resource. And cat is certainly not the only pre-curl command that writes to stdout by default; they are plentiful.

And then: once I had made that decision and I released curl for the first time on March 20, 1998: the call was made. The default was set. I will not change a default and hurt millions of users. I rather continue to be questioned by newcomers, but now at least I can point to this blog post! 🙂

About the wget rivalry

cURLAs I mention in my curl vs wget document, a very common comment to me about curl as compared to wget is that wget is “easier to use” because it needs no extra argument in order to download a single URL to a file on disk. I get that, if you type the full commands by hand you’ll use about three keys less to write “wget” instead of “curl -O”, but on the other hand if this is an operation you do often and you care so much about saving key presses I would suggest you make an alias anyway that is even shorter and then the amount of options for the command really doesn’t matter at all anymore.

I put that argument in the same category as the people who argue that wget is easier to use because you can type it with your left hand only on a qwerty keyboard. Sure, that is indeed true but I read it more like someone trying to come up with a reason when in reality there’s actually another one underneath. Sometimes that other reason is a philosophical one about preferring GNU software (which curl isn’t) or one that is licensed under the GPL (which wget is) or simply that wget is what they’re used to and they know its options and recognize or like its progress meter better.

I enjoy our friendly competition with wget and I seriously and honestly think it has made both our projects better and I like that users can throw arguments in our face like “but X can do Y”and X can alter between curl and wget depending on which camp you talk to. I also really like wget as a tool and I am the occasional user of it, just like most Unix users. I contribute to the wget project well, both with code and with general feedback. I consider myself a friend of the current wget maintainer as well as former ones.

27 thoughts on “Why curl defaults to stdout”

  1. I think it’s not just about the extra 3 characters of “-O “, it’s about remembering that you need them in the first place (which you remember only after you’ve started by having binary gunk spewed to your terminal, and (if you are unlucky) have to find some way to reset it) and then trying to remember what the name of the option is if you haven’t used curl for a while.

    Would it be possible to change to writing to a file by default if you could tell that output was going to a terminal rather than a pipe, and the MIME type of the resource was a binary type?

  2. @Gervase, there might be uses similar to pipe, but that appear to be terminal output. I think subshell execution is one of them.

  3. @Gervase, I think that also violates the principle of least surprise. “Why is curl behaving magically in this context?”

  4. Slightly wrong though. I was stating what many users have told me, I of course realize that if you download binary junk to your terminal it’ll require a few more keypresses and terminal reset and whatever.

    I could imagine having curl attempt to avoid binary junk sent to the terminal, as I suspect that would basically not hurt compatibility. It could for example output a question “are you sure?” and timeout after 3 seconds and continue. But I also have my TODO pretty filled up already so I won’t work on this personally in the foreseeable future.

  5. Let’s try something simple:

    $ curl github.com

    $ curl -v github.com
    * Rebuilt URL to: github.com/
    * Hostname was NOT found in DNS cache
    * Trying 192.30.252.129…
    * Connected to github.com (192.30.252.129) port 80 (#0)
    > GET / HTTP/1.1
    > User-Agent: curl/7.39.0
    > Host: github.com
    > Accept: */*
    >
    < HTTP/1.1 301 Moved Permanently
    < Content-length: 0
    < Location: https://github.com/
    < Connection: close
    <
    * Closing connection 0

    $ curl -L github.com

    $ curl -L -O github.com
    curl: Remote file name has no length!
    curl: try ‘curl –help’ or ‘curl –manual’ for more information

    $ curl -L -O -J github.com
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 –:–:– –:–:– –:–:– 0
    Warning: Remote filename has no length!
    curl: (23) Failed writing body (0 != 1365)

    Now let’s try the same with wget:

    $ wget github.com
    –2014-11-17 15:14:42– http://github.com/
    Resolving github.com… 192.30.252.131
    Connecting to github.com|192.30.252.131|:80… connected.
    HTTP request sent, awaiting response… 301 Moved Permanently
    Location: https://github.com/ [following]
    –2014-11-17 15:14:43– https://github.com/
    Connecting to github.com|192.30.252.131|:443… connected.
    HTTP request sent, awaiting response… 200 OK
    Length: unspecified [text/html]
    Saving to: ‘index.html’

    index.html [ ] 16.08K –.-KB/s in 0.1s

    2014-11-17 15:14:44 (114 KB/s) – ‘index.html’ saved [16470]

    Gee, I wonder where did wget gain the reputation of being easier to use than curl. 🙂

  6. “after you’ve started by having binary gunk spewed to your terminal, and (if you are unlucky) have to find some way to reset it)”

    Yes, it is a terrible burden to have to find some way to reset(1) your terminal.

  7. Meh, I use when I want to download a file, curl when I need stdout. So I use both frequently

  8. It may be a bit revisionist but if you tell people the “c” in “curl” stands for “cat”, then the output to stdout makes more mental sense.

    In fact, if you did want to change the behavior, you could make “caturl” the old behavior, you could do it in two steps. First, introduce “caturl” as the same behavior as the current command. However announce that in n years, curl will change behavior. That gives people an easy way to prepare, and makes converting old code easy to convert.

  9. @LCD: I was actually trying to explain why it defaults to stdout.

    So yeah, not doing so would’ve made it easier to download to a local file. I don’t think I disagree with that… Your sequence of commands mostly tell me that curl is consistent.

  10. I use curl all the time to get information from API’s. This is a ‘lot’ harder with wget, which will save the information to a file, when I just want to see it.

  11. @Daniel Stenberg: The point is wget tries to have meaningful defaults for the common use cases, while cURL has never made such attempts. The end result is that cURL is more flexible, but it has an user interface only a compiler would love.

  12. @LCD 47:

    I’m actually happy that curl doesn’t follow redirects. I normally use curl when I want to see what exactly the server says, and I use wget when I just want a file.

    I’m kind of in a different mindset when using the two tools. For me, curl is not a downloading tool, for me curl is a tool for debugging to see what hte server answers. But for me, wget is a download tool and not a tool that will help me to debug.

  13. First of all, I agree with “LCD 47” in that adding “-O” is not all that it takes to work the same as wget, due to the need to add “index.html” to any directory request that you want to save to disk.

    Also, the tone of your article is somewhat distasteful, especially the part where you say that wanting file output instead of a pipe is “in the same category as” people saying where you type a word on the keyboard matters. This is an absurd comparison and there are real reasons why it is not helpful to output to the console.

    Consider if you request a binary file that responds with headers saying that it should be downloaded ( attachment type ). Does it really make sense to output this to the screen when you aren’t redirecting the pipe? No. It doesn’t. I understand that the program does not know if the output is being redirected or not, but that doesn’t make it any less messy to output a binary file to the screen.

    Also, if you curl an “evil” binary file and don’t redirect, it is possible for your shell to become corrupted and/or attacked.

    Your post amounts to no more than “It outputs to the console because I decided that it should.” Great. Very helpful.

  14. @LCD I think the simple answer is that “common use cases” are different for curl and wget, in part because they make different things easy.

  15. You will need curl -OL in order to do the same as wget to download a file, because wget will follow redirects as default.

  16. @LCD
    cURL has meaningful defaults for *its* common use cases. I think the use cases that the authors of each program had in mind are different.

  17. cURL is built exactly right, and is really a far better tool than wget – it is much more powerful and flexible, and the fact that it *does* use stdout makes it even easier to use it as part of a pipeline of operations.

    I’m amazed at how many younger programmers, developers, and even admins don’t really grok the truly awesome leverage you get from pipes – one of the world’s best modular pluggable, reconfigurable software environments. Tools like the shell and utilities that act properly as filters (cURL, awk, sed, cut, paste, tee, etc.) take a *little* bit to learn, but allow good, relatively high performance apps to be built and tested in minutes. Being able to view the interim processed data at any time just by letting it go to the terminal is part of what makes good scripting such a low-friction development method.

    If you *really* want to see how powerful this “Stream/Operator” paradigm can be, read the classic paper, “The Unix Shell as a Fourth Generation Language” (www.rdb.com/lib/4gl.pdf) This technique can be amazing for doing things like building static websites for things like product catalogs and the like – since live updates are not usually important, there’s no reason you can’t do a quick “compile” to create the new static site structure when things change. This approach is simpler, more reliable, more secure, and considerably faster and cheaper than the “conventional” dynamic web app frameworks (with the backend DB being tagged for every request) popular today. Fortunately, there’s a rapidly growing awareness that static web apps have a real place going forward. If they’re a step “backward”, then they’re a return to good and proper application design – and the triumph of simplicity and modularity over bloat.

  18. I think you should take this one step more and put the alias here.

    alias wget=”curl -O”

  19. I love how I can directly use grep on curl and using wget always gets me looking at the man page to find the exact option to use.

  20. I’m perfectly fine with curl’s behaviour. It allowed me to have dead simple webserver availabilit scripts by just putting a ping.txt file with the content
    OK
    on the server and the just call
    if [ “x`curl http://mysite.org/ping.txt`” = “xOK” ]
    then
    echo all is well
    else
    echo get up and fix that
    fi

  21. Perhaps install a 2nd binary. curl for cat url, gurl for get url. (Though it seems too small a difference. Perhaps gurl should also do -L, and automatically pick index.html, and whatever shell-friendly modifications might make sense.)

  22. Yeah, a separate more wget-like binary is an option but I’m not convinced it is worth the effort. If someone wants to work on that I won’t stand in the way.

    In the wget camp they’re discussing a ‘wcat’ tool with more curl-like default behavior… 🙂

Comments are closed.