{"id":14315,"date":"2020-07-29T00:08:00","date_gmt":"2020-07-28T22:08:00","guid":{"rendered":"https:\/\/daniel.haxx.se\/blog\/?p=14315"},"modified":"2020-07-29T00:08:00","modified_gmt":"2020-07-28T22:08:00","slug":"curl-ootw-path-as-is","status":"publish","type":"post","link":"https:\/\/daniel.haxx.se\/blog\/2020\/07\/29\/curl-ootw-path-as-is\/","title":{"rendered":"curl ootw: –path-as-is"},"content":{"rendered":"\n

Previous options of the week<\/a>.<\/p>\n\n\n\n

--path-as-is<\/code> is a boolean option that was added in curl 7.42.0.<\/p>\n\n\n\n

Path normalization in URLs<\/h2>\n\n\n\n

I hope it isn’t a surprise to you that curl works on URLs. It’s one of the fundamental pillars of curl. The “URLs” curl work with are actually called “URIs” in the IETF specs and the primary specification for them is RFC 3986<\/a>. (But also: my URL is not your URL<\/a>…)<\/p>\n\n\n\n

A URL can be split up into several different components, which is typically done by the “URL parser” in a program like curl. For example , we can identify a scheme, a host name and a path.<\/p>\n\n\n\n

When a program is given a URL, and the program has identified the path part of that URL – it is supposed to “Remove Dot Segments” (to use the wording from RFC 3986) before that path is used.<\/p>\n\n\n\n

Remove Dot Segments<\/h2>\n\n\n\n

Let me show you this with an example to make it clear. Ponder that you pass this URL to curl: \"https:\/\/example.org\/hello\/..\/to\/..\/your\/..\/file\"<\/code>. Those funny dot-dot sequences in there is traditional directory traversal speak for “one directory up”, while a single \".\/\"<\/code> means in the same directory.<\/p>\n\n\n\n

RFC 3986 says these sequences should be removed, so curl will iterate and remove them accordingly. A sequence like \"word\/..\/\"<\/code> will effectively evaluate to nothing. The example URL above will be massaged into the final version: \"https:\/\/example.org\/file\"<\/code> and so curl will ask the server for just \/file<\/code>.<\/p>\n\n\n\n

Compare the HTTP requests<\/h2>\n\n\n\n

Seen as pure HTTP 1.1, the result of the command line used without --path-as-is<\/code>:<\/p>\n\n\n\n

GET \/file HTTP\/1.1
Host: example.org
user-agent: curl\/7.71.0
accept: *\/<\/em>*<\/pre>\n\n\n\n
Same command line, with<\/em> --path-as-is<\/code>:<\/p>\n\n\n\n
GET \/hello\/..\/to\/..\/your\/..\/file HTTP\/1.1
Host: example.org
user-agent: curl\/7.71.1
accept: *\/*<\/em><\/pre>\n\n\n\n
Trick thy server<\/h2>\n\n\n\n
HTTP servers have over the years been found to have errors and mistakes in how they handle paths and a common way to exploit such flaws has been to pass on exactly this kind of dot-dot sequences to servers.<\/p>\n\n\n\n
The very minute curl started removing these sequences (as the spec tells us) security researcher objected and asked for ways to tell curl to not<\/em> do this. Enter --path-as-is<\/code>. Use this option to make curl send the path exactly as provided in the URL, without removing any dot segments.<\/p>\n\n\n\n
Related options<\/h2>\n\n\n\n
Other curl options that allow you to customize HTTP request details include --header<\/code>, --request<\/code> and --request-target<\/code>.<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Previous options of the week. –path-as-is is a boolean option that was added in curl 7.42.0. Path normalization in URLs I hope it isn’t a surprise to you that curl works on URLs. It’s one of the fundamental pillars of curl. The “URLs” curl work with are actually called “URIs” in the IETF specs and … Continue reading curl ootw: –path-as-is<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":5,"featured_media":13041,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[33,486],"class_list":["post-14315","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-curl","tag-curl-and-libcurl","tag-option-of-the-week"],"_links":{"self":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/14315","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/comments?post=14315"}],"version-history":[{"count":7,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/14315\/revisions"}],"predecessor-version":[{"id":14403,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/14315\/revisions\/14403"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/media\/13041"}],"wp:attachment":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/media?parent=14315"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/categories?post=14315"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/tags?post=14315"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}