cURL and libcurl

curl ootw: –raw

2 Comments

(ootw is short for “option of the week“!)

--raw

Introduced back in April of 2007 in curl 7.16.2, the man page details for this option is very brief:

(HTTP) When used, it disables all internal HTTP decoding of content or transfer encodings and instead makes them passed on unaltered, raw.

This option is for HTTP(S) and it was brought to curl when someone wanted to use curl in a proxy solution. In that setup the user parsed the incoming headers and acted on them and in the case where for example chunked encoded data is received, which curl then automatically “decodes” so that it can deliver the pure clean data, the user would find that there were headers in the received response that says “chunked” but since libcurl had already decoded the body, it wasn’t actually still chunked when it landed!

In the libcurl side, an application can explicitly switch off this, by disabling transfer and content encoding with CURLOPT_HTTP_TRANSFER_DECODING and CURLOPT_HTTP_CONTENT_DECODING.

The --raw option is the command line version that disable both of those at once.

With --raw, no transfer or content decoding is done and the “raw” stream is instead delivered or saved. You really only do this if you for some reason want to handle those things yourself instead.

Content decoding includes automatice gzip compression, so --raw will also disable that, even if you use --compressed.

It should be noted that chunked encoding is a HTTP/1.1 thing. We don’t do that anymore in HTTP/2 and later – and curl will default to HTTP/2 over HTTPS if possible since a while back. Users can also often avoid chunked encoded responses by insisting on HTTP/1.0, like with the --http1.0 option (since chunked wasn’t included in 1.0).

Example command line

curl --raw https://example.com/dyn-content.cgi

Related options

--compressed asks the server to provide the response compressed and curl will then decompress it automatically. Thus reduce the amount of data that gets sent over the wire.

2 thoughts on “curl ootw: –raw”

  1. Hi Daniel,

    Note the title is wrong (using long dash instead two dashes).

    I think it make worth adding “–raw” and “–compressed” options at once may result on gzipped content, because ”–raw” avoids decompress operations. Probably it’s already known by readers…

    My suggestion for next OOTW: -v / –verbose
    If there is an option to avoid the flood of “* Expire in 0 ms for 1…” messages it may worth comment on that post.

    Best regards. Thanks for your time.

    1. That long dash conversion is a horrible wordpressism that’s very annoying to address so I think we’ll have to live with the fact that the titles will get the wrong prefix for long options! At least they’re correct within the post since I mark them specifically then.

      The “Expire in 0 ms” messages were removed a long time ago. They were mistakenly left in there in a single release.

Comments are closed.