{"id":22692,"date":"2023-07-31T11:53:44","date_gmt":"2023-07-31T09:53:44","guid":{"rendered":"https:\/\/daniel.haxx.se\/blog\/?p=22692"},"modified":"2023-08-02T11:41:08","modified_gmt":"2023-08-02T09:41:08","slug":"introducing-curl-command-line-variables","status":"publish","type":"post","link":"https:\/\/daniel.haxx.se\/blog\/2023\/07\/31\/introducing-curl-command-line-variables\/","title":{"rendered":"introducing curl command line variables"},"content":{"rendered":"\n<p>If you are anything like me, you appreciate solving your every day simple tasks directly from the command line. Creating crafty single shot command lines or a small shell script to solve that special task you figured out you needed and makes your day go a little smoother. A fellow <em>command line cowboy<\/em>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Video presentation<\/h2>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"curl command line variables with Daniel Stenberg\" width=\"474\" height=\"267\" src=\"https:\/\/www.youtube.com\/embed\/KzzrhLgCjik?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Background<\/h2>\n\n\n\n<p>To make life easier for curl users, the tool supports &#8220;<a href=\"https:\/\/everything.curl.dev\/cmdline\/configfile\">config files<\/a>&#8220;. They are a set of command line options written in a text file that you can point the curl tool to use. By default curl will check for and use such a config file named <code>.curlrc<\/code> if placed in your home directory.<\/p>\n\n\n\n<p>One day not too long ago, a user over in <a href=\"https:\/\/curl.se\/docs\/irc.html\">the curl IRC channel<\/a> asked me if it was possible to use environment variables in such config files to avoid having to actually store secrets directly in the file.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Variables<\/h2>\n\n\n\n<p>This new variable system that we introduce in <strong>curl 8.3.0<\/strong> (commit <a href=\"https:\/\/github.com\/curl\/curl\/commit\/2e160c9c652504e147f474ed920ae891481e299c\">2e160c9c65<\/a>) makes it possible to use environment variable in config files. But it does not stop there. It allows lots of other fun things.<\/p>\n\n\n\n<p>First off, you can <em>set<\/em> named variables on the command line. Like :<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">curl --variable name=content<\/pre>\n\n\n\n<p>or in the config file:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">variable=name=content<\/pre>\n\n\n\n<p>A variable name must only consist of a-z, A-Z, 0-9 or underscore (up to 128 characters). If you set the same name twice, the second set will overwrite the first.<\/p>\n\n\n\n<p>There can be an unlimited amount of variables. A variable can hold up to 10M of content. Variables are set in a left to right order as curl parses the command line or config file.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Assign<\/h2>\n\n\n\n<p>You can assign a variable a plain fixed string as shown above. Optionally, you can tell curl to populate it with the contents of a file:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">curl --variable name@filename<\/pre>\n\n\n\n<p>or straight from stdin:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">curl --variable name@-<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Environment variables<\/h2>\n\n\n\n<p>The variables mentioned above are only present in the curl command line. You can also opt to &#8220;import&#8221; an environment variable into this context. To import $HOME:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">curl --variable %HOME<\/pre>\n\n\n\n<p>In this case above, curl will exit if there is no environment variable by that name. Optionally, you can set a default value for the case where the variable does not exist:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">curl --variable %HOME=\/home\/nouser<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Expand variables<\/h2>\n\n\n\n<p>All variables that are set or &#8220;imported&#8221; as described above can be used in subsequent command line option arguments &#8211; or in config files.<\/p>\n\n\n\n<p>Variables must be explicitly asked for, to make sure they do not cause problems for older command lines or for users when they are not desired. To accomplish this, we introduce the <code>--expand-<\/code> option prefix.<\/p>\n\n\n\n<p>Only when you use  the <code>--expand-<\/code> prefix in front of an option will the argument get variables expanded.<\/p>\n\n\n\n<p>You reference (expand) a variable like <code>{{name}}<\/code>. That means two open braces, the variable name and then two closing braces. This sequence will then be replaced by the contents of the variable and  a non-existing variable will expand as blank\/nothing.<\/p>\n\n\n\n<p><strong>Trying to show a variable with a null byte causes error<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Examples<\/h2>\n\n\n\n<p>Use the variable named &#8216;content&#8217; in the argument to <code>--data<\/code>, telling curl what to send in a HTTP POST:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">--expand-data \u201c{{content}}\u201d<\/pre>\n\n\n\n<p>Create the URL to operate on by inserting the variables &#8216;host&#8217; and &#8216;user&#8217;.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">--expand-url \u201chttps:\/\/{{host}}\/user\/{{user}}\u201d<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Expand variables<\/h2>\n\n\n\n<p><code>--variable<\/code> itself can be expanded when you want to create a new variable that uses content from one or more other variables. Like:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">--expand-variable var1={{var2}}\n--expand-variable fullname=\u2019Mrs {{first}} {{last}}\u2019\n--expand-variable source@{{filename}}<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Expansion functions<\/h2>\n\n\n\n<p>When expanding a variable, functions can be applied. Like this: <code>{{name:function}}<\/code><\/p>\n\n\n\n<p>Such variable functions alter how the variable is expanded. How it gets output.<\/p>\n\n\n\n<p>Multiple functions can be applied in a left-to-right order:<br><code>{{name:func1:func2:func3}}<\/code>.<\/p>\n\n\n\n<p>curl offers four different <em>functions<\/em> to help you expand variables in the most productive way: trim, json, url and b64:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>trim<\/strong> &#8211; removes leading and trailing whitespace<\/li>\n\n\n\n<li><strong>json<\/strong> &#8211; outputs the variable JSON quoted (but without surrounding quotes)<\/li>\n\n\n\n<li><strong>url<\/strong> &#8211; shows the string URL encoded, also sometimes called percent encoding<\/li>\n\n\n\n<li><strong>b64<\/strong> &#8211; shows the variable base64 encoded<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Function examples<\/h2>\n\n\n\n<p>Expands the variable URL encoded. Also known as \u201cpercent encoded\u201d.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">--expand-data \u201cname={{name:url}}\u201d<\/pre>\n\n\n\n<p>To trim the variable first, apply both functions (in the right order):<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">--expand-data \u201cname={{name:trim:url}}\u201d<\/pre>\n\n\n\n<p>Send the HOME environment variable as part of a JSON object in a HTTP POST:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">--variable %HOME\n--expand-data \u201c{ \\\"homedir\\\": \\\"{{HOME:json}}\\\u201d \"<\/pre>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Discuss<\/h2>\n\n\n\n<p>On <a href=\"https:\/\/news.ycombinator.com\/item?id=36940525\">hacker news<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you are anything like me, you appreciate solving your every day simple tasks directly from the command line. Creating crafty single shot command lines or a small shell script to solve that special task you figured out you needed and makes your day go a little smoother. A fellow command line cowboy. Video presentation &hellip; <a href=\"https:\/\/daniel.haxx.se\/blog\/2023\/07\/31\/introducing-curl-command-line-variables\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">introducing curl command line variables<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":5,"featured_media":12225,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[214,33],"class_list":["post-22692","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-curl","tag-command-line","tag-curl-and-libcurl"],"_links":{"self":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/22692","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/comments?post=22692"}],"version-history":[{"count":22,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/22692\/revisions"}],"predecessor-version":[{"id":22886,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/22692\/revisions\/22886"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/media\/12225"}],"wp:attachment":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/media?parent=22692"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/categories?post=22692"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/tags?post=22692"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}