Updated on January 31, 2023 in #linux

Using curl with Multiline JSON Data

Occasionally I want to POST human formatted JSON with curl, here's how to do that without using shell hacks or temp files.

I tend to write a lot of glue code with shell scripts. Sometimes I find myself sending POST requests to APIs where it’s expected to send in a decent amount of JSON.

# Pitfalls

There’s a few pitfalls when it comes to doing the above.

By the way in all of the examples below I visit localhost:8008. If you want to follow along, I’ve created a zero dependency web server that echos your request back. I’ve written about it in the past. It comes in handy for testing scripts like this.

Single vs double quotes and shell variables:

curl \
  http://localhost:8008 \
  --request POST \
  --header "Content-Type: application/json" \
  --data '{"status": "ok"}'

Valid JSON needs to use double quotes so you may reach for using single quotes to wrap your --data to avoid having to escape double quotes in a bunch of spots.

That’s reasonable until you want to use a shell variable such as '"status": "${status}"' because single quotes will output literally ${status} instead of interpolating the variable.

Larger amounts of JSON quickly become unwieldy:

curl \
  http://localhost:8008 \
  --request POST \
  --header "Content-Type: application/json" \
  --data '[{"id": 123, "status": "pending"}, {"id": 456, "status": "complete"}]'

That’s only 2 items with a pretty simple object and it’s already getting pretty hard to read. Throw in a couple of nested objects and a few more properties and it’s going to get bad fast.

You may want to beak that out into a multiline string but it’s not immediately obvious how to do this in a clean way. At least it wasn’t for me initially.

I knew you could provide a file to curl’s --data flag with --data @example.json. Armed with that knowledge you may reach for a solution that involves writing a temporary file, running your curl command and then deleting that temp file.

That could look something like this:

TEMP_FILE="/tmp/example.json"

cat > "${TEMP_FILE}" << EOF
[
  {
    "id": 123,
    "status": "pending"
  },
  {
    "id": 456,
    "status": "complete"
  }
]
EOF

curl \
  http://localhost:8008 \
  --request POST \
  --header "Content-Type: application/json" \
  --data "@${TEMP_FILE}"

rm "${TEMP_FILE}"

At least the JSON is a bit more readable, although you could make a very strong case that having the objects on 1 line is more readable in this case because there’s only 2 properties. I fully agree, but when you have 8 properties it’s worse on 1 line.

Also, if you’re feeling fancy you may use mktemp instead of supplying a temp file path. If the path is safe to overwrite with new content, the manual file name isn’t bad because at least you know the file name to investigate it for troubleshooting purposes.

Anyways, this isn’t the most elegant solution. There’s a number of issues with the above approach in my opinion:

• There’s an awful lot of bookkeeping around creating and deleting the temp file
• The cat redirecting code is a nice way to write a heredoc to a file but it’s not super straight forward if you’ve never seen it before
• There’s a subtle detail with rm, as is the command will throw an error if the file doesn’t exist but rm -f won’t throw an error
  • You as the maintainer of the script need to be aware of that, maybe we want the script to crash with set -e if the file can’t be removed, etc.

Alternatively you may skip the temp file and set a heredoc style multiline string into a variable but this comes with a few implications of its own. This StackOverflow thread has a few answers and comments that go into detail.

The TL;DR is read will cause an error so it’s incompatible if you use set -e and when using cat you need to use a sub-shell and things get “interesting” with quotes and white space. These are things I don’t want to think about.

# A Clean Solution

It hits all the marks and more:

• Supports variable interpolation
• Avoids excess double quote escaping
• You can format your JSON in whatever way that works best for you
• Avoids creating a temp file
• Avoids a condition where the temp file doesn’t get deleted if curl fails
• Keeps everything together which is easier to take in conceptually
• There’s not much shell weirdness to be aware of
• It’s easy to copy / paste as a single command

curl \
  http://localhost:8008 \
  --request POST \
  --header "Content-Type: application/json" \
  --data @- << EOF
[
  {
    "id": 123,
    "status": "pending"
  },
  {
    "id": 456,
    "status": "complete"
  }
]
EOF

This takes advantage of a convention where tools and programs often allow - to be used as a file name where the “file” comes from stdin or stdout depending on how you’re using it. It’s not really a file, but it’s convenience that a lot of programs support.

With that in mind, we send the heredoc in as input to curl and we’re done, I like this method!

By the way --data will strip new lines which means your server will receive [ { "id": 123, "status": "pending" }, { "id": 456, "status": "complete" }]. If you need to keep new lines intact you can use --data-binary @- instead. This is in curl’s docs.

# Demo Video

Timestamps

• 0:19 – I’m using a local echo server if you want to follow along
• 0:37 – Creating a demo script
• 0:56 – A quick primer on the curl basics
• 1:53 – Using single quotes has issues with variables
• 2:46 – Using double quotes requires a lot of escaping
• 3:16 – A slightly more complex example is hard to read on 1 line
• 4:38 – You can pass a file to curl’s data flag with an at symbol
• 5:45 – Saving a multiline heredoc into a variable has a few caveats
• 7:51 – Creating and passing a file to curl then deleting it
• 11:15 – The solution, using a hyphen as a file with stdin
• 12:28 – Optionally retaining new lines with JSON

Reference Links

• https://github.com/nickjj/webserver

How do you prefer sending multiline strings into curl? Let me know below!