Updated on February 1st, 2022 in #linux

Run Parallel Background Commands in Linux and Wait Until They Complete

Sometimes it's useful to run parallel jobs and wait until they all finish before moving on, we can easily do this on the command line.

Quick Jump: Demo Video

This could be handy when you have a few tasks that can run in isolation. In my case I wanted to run a few sql dump commands in parallel and then do something once all of them finished.

In this video we’ll go how to combine using & and the wait command.

Demo Video

Script

• Create the script: touch work && chmod +x work

• Copy / paste this into the above file:

#!/usr/bin/env sh

# & instructs this command to run in the background and immediately begin
# executing the next line in the script without waiting for it to finish.
printf "Doing something...\n"
sleep 3 &

printf "Doing something else...\n\n"
sleep 3 &

# List all of the jobs and their PIDs.
jobs -l

# Wait until all of the jobs finish before continuing.
wait

printf "\n...and we're done!\n\n"

• Run and time the script time ./work:

Doing something...
Doing something else...

[2] + 10254 Running
[1] - 10253 Running

...and we're done!

./work  0.00s user 0.00s system 0% cpu 3.003 total

Notice that it finishes in about 3 seconds instead of 6 because it ran both sleep commands in parallel instead of sequentially waiting for each one. It also only prints out the last line once both jobs are done.

Timestamps

• 0:24 – Running the demo script
• 1:04 – Running commands in the background with &
• 1:36 – The wait command will let you wait until jobs finish
• 2:45 – The jobs command lets us get the PID and status of our jobs
• 3:35 – You can also wait for specific PIDs to finish running
• 4:30 – Running the jobs command with different flags
• 5:09 – Getting the status of each job after they’re done running
• 5:50 – Running things in parallel can save a lot of time

What are some use cases where you’ve used this pattern? Let me know below.