Updated on July 7, 2026 in #linux

Watch Out for Data Loss When Redirecting to a File in a Shell Script

There's a number of subtle things going on when you redirect output to a file, here's how to do it safely and verify it with strace.

Prefer video? Here it is on YouTube.

I’ve written 10,000+ lines of shell scripts over the years and I can’t believe I only learned about this recently.

Let’s say you have a normal shell script. I pretty much always set these options to protect myself. They’ll make your script exit if any commands you run encounter an error, if a pipeline fails or if a referenced variable is unset:

#!/usr/bin/env bash

set -o errexit
set -o pipefail
set -o nounset

echo "Hello"

Easy right? The expected output is “Hello” which is what we get.

What about echo "Hello" > results? Also easy, it will create a new results file in the directory where you ran this script with the contents of “Hello”. This is shell syntax to redirect output to a file.

# Showing the Problem

Without looking it up or trying it, take a guess what happens if you put this line at the bottom of the script. Don’t just run it in your terminal, since the same options aren’t set:

echo "Hello" | awk '{system("sleep 5"); print}' >results

To give you a hint, the awk command is just sleeping for 5 seconds. We’re using that instead of sleep 5 because the sleep command doesn’t accept STDIN where as awk does.

I don’t know about you but initially I thought this would happen:

• “Hello” would be printed to STDOUT
• It would get piped to awk which accepts STDIN
• awk would sleep for 5 seconds and print “Hello” to STDOUT
• A results file would be created with the contents “Hello”

The above mental model is how most pipelines are read. It flows from left to right.

What Really Happens

• The file descriptor for the results file is created immediately and a 0 byte file is created
  • You can verify this by running the command and catting the file in terminal #2
• echo prints “Hello” and pipes it into awk
• awk receives this input and begins executing its system command
• awk sleeps for 5 seconds
• awk prints “Hello” to STDOUT which is already connected to the open results file, this yields our expected text in the file

The important takeaway is as soon as this pipeline starts to run the file you’re redirecting to can be considered truncated to a 0 byte file and there’s 3 possible states this file can be in after the script is called:

• All commands finish successfully and it’s what you expect
• One of the commands fails and the scripts exits with a partially written file
• The script exits before processing has occurred (power outage) and the file is empty

What If You Use Tee Instead?

If you’re comfortable writing shell scripts, you might think about trying:

echo "Hello" | awk '{system("sleep 5"); print}' | tee results 1>/dev/null

This has the same problem as > since the results file will get immediately created.

Technically when you run the above, echo, awk and tee are being run in parallel. Then tee is like “oh, I need to write to the results file, let me open a file descriptor for that”. At this point awk can start streaming data into the file at its discretion.

Why This Is a Problem

It comes down to potential unexpected data loss. Here’s 1 of many examples:

Imagine you’re doing something like a database backup. It’s not unreasonable to think you’d run your database dump command (pg_dumpall, etc.), maybe pipe it into gzip and then redirect it to a file.

If you’re keeping 1 day’s worth of backups by overwriting the existing file, if your current day’s backup command fails to run for some reasons, you could end up with a partial backup that overwrote yesterday’s valid backup.

In a disaster scenario where you have to restore, you suddenly have data loss.

# Using strace to Pry Deeper

The more I use native Linux on the desktop, the more I find myself interested in lower level details. I wish I had unlimited time:

$ strace --follow-forks -e trace=openat ./demo

If you run the above with either version you’ll see this line early on which demonstrates a truncated 0 byte file is being created:

[pid 1964503] openat(AT_FDCWD, "results", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

Also, while no timestamps are shown, when you run the above command with the tee version you’ll see something like this pop out at about the same time:

strace: Process 1966073 attached
strace: Process 1966074 attached
strace: Process 1966075 attached

This demonstrates all parts of the pipeline are being running in parallel. Your shell forked them, that’s also why we set --follow-forks with strace, we want to be informed of what each command is doing.

# Writing Files Safely

What I like to do is write the output to a temporary file and in a separate command use mv to move that to the final destination:

echo "Hello" | awk '{system("sleep 5"); print}' >results.tmp
mv results.tmp results

As long as set -o errexit and set -o pipefail are set up top we can be confident our pipeline finished successfully by the time mv is called.

As long as the temp file and the real destination file exist on the same filesystem then this move command will be atomic. The easiest way to be sure of that is putting it in the same directory as the real file.

For example if you have 2 partitions or separate disks, mv will copy the file instead of doing an atomic rename. This is why I avoid /tmp because sometimes that can be mounted on a different filesystem (often tmpfs, etc.).

You can verify an atomic rename with strace with this 1 liner:

$ touch a && strace -e trace=renameat,renameat2 mv a a.tmp && rm a.tmp
renameat2(AT_FDCWD, "a", AT_FDCWD, "a.tmp", RENAME_NOREPLACE) = 0

Here’s an example from my machine where /tmp is mounted in RAM (different filesystem):

$ touch a && strace -e trace=renameat,renameat2 mv a /tmp/a && rm /tmp/a
renameat2(AT_FDCWD, "a", AT_FDCWD, "/tmp/a", RENAME_NOREPLACE) = -1 EXDEV (Invalid cross-device link)

The above makes sense because it didn’t do a renameat2 call. If you wanted to see some of the calls taken for a cross device copy, here you go:

# You can use "desc" instead of "read,write" to get a lot more information:
$ echo "cool" > a && strace -e trace=read,write mv a /tmp/a && rm /tmp/a
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200y\2\0\0\0\0\0"..., 832) = 832
read(3, "cool\n", 262144)               = 5
write(4, "cool\n", 5)                   = 5
read(3, "", 262144)                     = 0
+++ exited with 0 +++

In the above case it wrote a 5 byte file (“cool\n”) because a 0 byte file from touch wouldn’t get written in the same way a file with > 0 bytes would be.

Considerations

The upside here is it lets your pipeline operate normally. Data will be streamed into the file without needing to hold everything in memory. The amount of memory used will depend on the tools you’re using to process your data.

The downside or consideration is you temporarily need double the storage in our database backup example. You need the old / original backup and the new temp file backup to exist in parallel for a short period of time.

Personally this redirect to temp file + mv combo is what I use in most cases. Funny enough I’ve been using this approach to backup databases for a long time. I was recently doing client work and revisited some backup and restore scripts and wanted to justify my thought process so I did a deeper dive.

It’s always fun when you accidentally land on a safe solution based on intuition.

With backups, you could of course consider using S3 instead (or another object store) and that’s good too. This example focuses on backing up to a mounted drive. This drive could exist on a different filesystem than your database but the mv is still an atomic rename since that’s related to the temporary backup vs real backup file, not your database itself.

The demo video below shows running these commands.

# Demo Video

Timestamps

• 0:24 – Redirecting files unsafely
• 2:12 – Potential data loss
• 3:05 – What about using tee?
• 5:04 – Verifying it with strace
• 6:55 – Redirecting files safely
• 9:48 – Verifying atomic renames with mv
• 10:55 – Considerations of this temp file + mv approach

How do you safely redirect files? Let me know below.