Output, Assign, Pipe and Redirect a Heredoc in a Shell Script
You may want to do this when working with multi-line strings.
A “Here Document” is a useful way to output text.
It’s a feature of most programming languages. It’s commonly used to output multiple lines of text In such a way that you don’t need to worry about using a convoluted way of breaking up lines or escaping quotes.
We’ll break down the first example in the most detail. The other examples incrementally combine using other shell features to handle specific use cases.
It’s worth pointing out all of these examples will work with sh, bash and
other compatible shells. There is 1 exception that only works with Bash or
Bash-compatible shells and that will be called out, however in that case we’ll
use an alternative method that works with sh too.
# Output
This could be useful to output a multi-line string.
With interpolation
numbers="123"
cat << EOF
a
b
c
${numbers}
EOF
# Here's the output of the above script:
a
b
c
123
We’re sending input to cat which then outputs it back. A heredoc expects us
to pass in a string delimiter too. Using EOF (end of file) is commonly used
but it’s not required, technically you can use any string as long as it
matches on the top and bottom.
It’s really important that the bottom EOF exists on its own line with no
extra indentation. If you deviate from that, it won’t be considered delimited.
Here we can see that variable interpolation works too because ${numbers} got
interpolated to its value of 123. All expressions will get interpolated.
If our multi-line string had single ' or double " quotes inside of it
everything will just work without needing to escape them. That’s quite nice and
also important to call out because you may choose to use a heredoc even for 1
line of output if you find yourself working with a string that has a lot of
mixed quotes.
Without interpolation
This is the same as above except variables and other expression won’t get interpolated.
numbers="123"
cat << 'EOF'
a
b
c
${numbers}
$(echo "wow")
EOF
# Here's the output of the above script:
a
b
c
${numbers}
$(echo "wow")
The only difference is we wrapped EOF in single quotes. Technically double
quotes works too but I prefer using single quotes because when working with
shell scripts if you wrap a string in single quotes it won’t interpolate your
value. This matches the same pattern.
It’s also worth pointing out if you didn’t use 'EOF' you can still output
something like ${numbers} literally by escaping the $ with \${numbers}.
# Assign
What if you want to assign the heredoc to a variable?
letters=$(cat << EOF
a
b
c
EOF
)
echo "${letters}"
# Here's the output of the above script:
a
b
c
This is pretty interesting because we’re doing the same cat strategy as before. The only difference is using command substitution to run it in a sub-shell and assign it to a variable.
If you Google around for this you might see this StackOverflow
answer that suggests using read instead
to “avoid a useless cat”.
That would look like this:
read -r -d "" letters << EOF
a
b
c
EOF
echo "${letters}"
The above produces the same output as the cat version, but it won’t work with
sh because POSIX compliant shell doesn’t support the -d flag with read.
This flag says that input lines should be delimited by a null byte.
There is a comment that I agree with which questions if it’s really worth avoiding the useless cat. I don’t mind the useless cat because in my opinion it’s more understandable given it piggy backs off prior knowledge of using a heredoc in other ways. It’s also more portable.
# Pipe
You can also pipe your heredoc into another command.
cat << EOF | sort
c
b
a
EOF
# Here's the output of the above script:
a
b
c
Notice how the output got sorted. It might seem weird to see the pipe at the
top instead of after the bottom EOF but if you think about the heredoc
“owning” the delimiter then it might make more sense.
The pipe is still operating like any other pipe sequence you might use on the
shell. The output of cat is being piped into sort as input.
Pipe (with file support)
Certain tools may accept input but only in the form of a file. We’re going to
cover 2 examples using kubectl and curl. It’s really not different than
the above example.
kubectl
cat << EOF | kubectl apply -f -
apiVersion: "v1"
# ...
EOF
I’ve omit the output here because it’s not important. You may have seen this pattern used in a bunch of spots within Kubernetes’ documentation. If not, that’s ok because none of this is specific to Kubernetes.
The main takeaway is kubectl apply expects you to pass in a file. A common
but not enforced Unix convention is to allow using - as a file name which
will read the file from STDIN instead of a real file on disk.
It’s what allows us to pass in a multi-line string file as input to kubectl
and other tools that use this pattern. It’s quite handy for writing scripts
that use these tools because it lets you avoid creating temporary files on disk
only to delete them after the command is run.
curl
curl localhost --data-binary @- << EOF
{
"name": "nick"
}
EOF
# Here's the output of the above script:
{
"name": "nick"
}
EOF
curl: (7) Failed to connect to localhost port 80 after 0 ms: Connection refused
The curl error makes sense because I don’t have a web server running on
localhost but this pattern does work for when you want to make a real HTTP
request to a site while sending your payload in as a multi-line string.
It uses the - trick as mentioned before, except in this case curl excepts
you to use @- because @ lets curl differentiate that we want to use a file.
By the way, --data-binary is being used instead of --data (-d) so that
when you send your payload with the request, new lines aren’t stripped.
# Redirect
Lastly we can redirect our multi-line string to a file. You can choose to use
>> instead of > to append to the file too.
cat << EOF > example_output
a
b
c
EOF
Technically the script doesn’t output anything but it did write an
example_output file out with the contents of a b c on new lines.
If you ran the above command you can run cat example_output && rm $_ to both
see the file and then delete it. I’ve made a post in the past about what $_
does.
The demo video below runs all of these commands and shows how they work.
# Demo Video
Timestamps
- 0:20 – A run down on heredocs
- 2:56 – Avoiding variable or expression interpolation
- 4:31 – Assigning a multi-line string to a variable with cat vs read
- 7:19 – Piping a heredoc to another command as input
- 8:20 – Piping things into kubectl where it expects a file as input
- 10:04 – Doing the same as before but with curl
- 11:14 – Redirecting a multi-line string to a file
What’s your preferred way to hide user input? Let me know below.