Learn Docker With My Newest Course

Dive into Docker takes you from "What is Docker?" to confidently applying Docker to your own projects. It's packed with best practices and examples. Start Learning Docker →

Using Extended Globbing for Patterns in Bash

using-extended-globbing-for-patterns-in-bash.jpg

Sometimes you might want extra features like listing all files except for a specific one, we'll cover a few examples.

Quick Jump:

When filtering files on the command line, you may have encountered patterns like this:

  • cat * to output all files but ignore files that start with .
  • cat *.bak to output all files with a .bak file extension
  • cat a* to output all files that start with a
  • cat ??.txt to output all text files that are 2 characters long

If those concepts are new to you, you can experiment with this:

echo "hello" >hello.bak &&
  echo "hello" >hello.txt &&
  echo "apple" >apple &&
  echo "ab" >ab.txt

Now you can run the above commands and see what they output.

# Extended Globbing

What if you wanted to output all files except 1 specific file? You could write a custom script to loop over things and skip the one you want but it’s possible to do this out of the box.

You know how you can do set -o errexit to force scripts to exit on the first error? You can do something similar with shopt options such as shopt -s extglob. Keep in mind if you’re using zsh you’ll want to open a bash shell to allow that command to work.

To help remember the command think of it as “shell options” AKA. “shopt”.

Listing shopt Options

You can confirm it’s on by running shopt -s to list all enabled options:

$ shopt -s
checkwinsize    on
cmdhist         on
complete_fullquote      on
expand_aliases  on
extglob         on
extquote        on
force_fignore   on
globasciiranges on
hostcomplete    on
interactive_comments    on
progcomp        on
promptvars      on
sourcepath      on

Running shopt -u will list disabled options and shopt -p will list all options and their values.

All Files Except

You can do cat !(demo) to output all files except a file called demo. You can separate patterns with a | too such as cat !(demo|extras) to skip files named demo or extras.

To disable this behavior you can run shopt -u extglob to bring your shell back to normal.

Other Examples

Here’s the docs straight from man bash:

?(pattern-list)  Matches zero or one occurrence of the given patterns
*(pattern-list)  Matches zero or more occurrences of the given patterns
+(pattern-list)  Matches one or more occurrences of the given patterns
@(pattern-list)  Matches one of the given patterns
!(pattern-list)  Matches anything except one of the given patterns

If you’re wondering why it’s not enabled by default, there’s a good write up of that on StackOverflow.

If you did want to enable it by default you can put it into your shell’s rc file. You can also put it near the top of a shell script to have it apply to your whole script. You can optionally choose to turn it on and off on demand as needed.

# Edge Case with Line Parsing

Before going into the solution, it might help to understand a specific use case.

I had a bunch of shell scripts in a directory and I wanted to run shellcheck on all of them. Normally you can do shellcheck * to handle that but in my case I had a few Python scripts without a .py extension and the version of shellcheck I use tried to run against those Python scripts.

What I really wanted to do is “run ShellCheck against all files except these 3 Python scripts” so I used shellcheck !(a|b|c) where a|b|c would really be the Python script names. Problem solved!

As for the edge case, I was running shellcheck in a script so I put shopt -s extglob && shellcheck !(a|b|c) but I noticed it didn’t work. It was like the option wasn’t enabled.

It turns out that shopt -s extglob must exist on its own line before any lines that use it. All in all I just put it onto 2 separate lines and then it worked fine.

If you have a hard requirement where it must exist on 1 line you can launch Bash with extglob enabled for just that 1 line command with bash -O extglob -c "shellcheck !(a|b|c)".

The video below demos using extglob.

# Demo Video

Timestamps

  • 0:00 – Default patterns
  • 1:35 – All files except
  • 1:53 – Enabling extglob
  • 2:51 – Showing how it works
  • 3:56 – Checking shopt options
  • 5:00 – A specific use case
  • 6:25 – Edge case in a script
  • 8:42 – Toggling it in a script

When did you last use this pattern? Let me know below.

Never Miss a Tip, Trick or Tutorial

Like you, I'm super protective of my inbox, so don't worry about getting spammed. You can expect a few emails per year (at most), and you can 1-click unsubscribe at any time. See what else you'll get too.



Comments