Shell Scripting for Software Engineers

Quick, when it comes to software engineering in the corporate world, what language belongs here?

OK, sure, SQL is a great answer. May I suggest another? Shell scripts.

They’re everywhere - starting services, building Docker containers, setting up the new hire’s laptop, running inside that flaky CodeBuild job. Despite their ubiquity, I don’t think I’ve ever seen a company with established guidance or best practices for shell scripting, even when their engineering practices were otherwise high-quality. Let’s change that, OK? OK.

Fortunately, it has never been easier to get this right. I hope to convince you that:

• Shell scripts are just software.
• They are ubiquitous enough that it’s worth establishing some best practices for your business.
• Scripting can be done as safely and correctly as any software project.
• Getting to that point is not excessively difficult or time-consuming.

Shell scripting is software engineering, and it is worth doing correctly.

I’ll go over some general advice here. Note that this is all unix-specific; Windows is a completely different problem that I might discuss some other time.

If you’re in a hurry, I packaged up this article into a generator tool that will create projects conforming to the advice here:

shgen: A Generator for Shell Scripting Projects

$ git clone git@github.com:justinpratt/shgen.git
$ ./shgen/shgen.sh

Feel free to use it as a starting point for your own projects or guidelines.

Write POSIX-Compliant Scripts #

I’ll start with possibly the most controversial advice. If you’re writing a shell script, then portability should be a top concern. If you fully control the execution environment, then presumably you could bootstrap your favorite programming language or configuration management framework and enjoy better ergonomics.

This certainly goes against the current trend of using Bash everywhere, but I assert that you should be writing POSIX-compliant scripts that will work on any platform and Bourne Shell descendant.

The first line of your script should be:

#!/bin/sh

Standards are useful. If portability is a major motivating factor, why would you immediately throw that away?

There’s a second, more subjective reason to stick with POSIX. Trying to do something that’s impossible using only POSIX standards is a good sign that you should seek an alternative implementation. A lot of Bash and other shell-specific advice you’ll find online is terrible. Their solutions for POSIX shortcomings often aren’t much better, and some are actually traps that will bite you later. I’ll discuss alternatives later in this article.

Targeting a specific shell doesn’t really buy you much anyway. The available version or feature set may be different than you expect. And users with an alternative shell will be unable to use your script as a library. This is commonly an issue on macOS (where zsh is the default) and Alpine (busybox ash).

I’ll expand on all these points later, but let’s talk about that last one.

Write Libraries, Not Scripts #

Write all code inside functions, with a main function if the library provides script-like functionality.

This takes only a minimal amount of boilerplate. Let’s start a new project, which we’ll split into two files:

#!/bin/sh

if test -n "${FOO_VERSION:-}"; then
  return 0
fi
FOO_VERSION='0.0.1'

foo_main() {
  printf 'Hello, world!\n'
  return 0
}

#!/bin/sh

FOO_DIR="${FOO_DIR:-$(cd -P "$(dirname "$0")" && printf "$(pwd -P)")}"

. "${FOO_DIR}/foo_lib.sh"

foo_main

This arrangement has a number of benefits:

• Having all functionality inside library functions will greatly improve testability.
• Users can install anywhere they choose and run foo.sh to use the script-like functionality.
• They can also source the script as a library in the current shell by setting FOO_DIR and sourcing ${FOO_DIR}/foo_lib.sh.

# run as a script
$ ./foo.sh
Hello, world!

# source as a library
$ FOO_DIR=.
$ . "${FOO_DIR}/foo_lib.sh"
$ foo_main
Hello, world!

# already sourced; no-op
$ . "${FOO_DIR}/foo_lib.sh"

You will occasionally see attempts to accomplish this in a single file by trying to determine if the script was sourced or executed - don’t do this. It’s a perfect example of why I recommend sicking to POSIX. The shell-specific solutions are inelegant, ¹ and you’re better off keeping it simple with the two-file approach.

Know The Appropriate Use Cases #

This seems obvious, but shells are good at unix-y things: running programs, copying files, starting services, etc. They’re not so good at data structures, concurrency, and other things we typically associate with programming languages.

That said, there’s a lot of well-meaning advice out there trying to convince you that shell scripting is more dangerous than it really is. You’ll probably run into examples where someone has bootstrapped an entire language runtime to accomplish something that would be tens of lines of shell script.

The quintessential example is a Dockerfile. Containers typically have limited user-space tooling available, and so a shell script is an obvious choice to maximize portability without bloating the image size or requiring significant cleanup, especially if you want to use the same script across multiple images.

Here are some other use cases where a shell script might be a great idea:

• Build scripts which may invoke other tools.
  
• A laptop setup script for new hires.
  
• Handy command-line utilities to share with your team.
  
• Installation or bootstrapping procedures for other software.

Again, scripts are most desirable when portability is important. If you don’t own the execution environment and can’t assume much about available tooling, or just want to leave a minimal footprint, a shell script is a reasonable choice.

Format and Lint Your Code #

The excellent shellcheck and shfmt are available on nearly all platforms for your linting and formatting needs, respectively.

This can be a simple test runner included with the project:

#!/bin/sh 
 
FOO_DIR="${FOO_DIR:-$(cd -P "$(dirname "$0")" && printf "$(pwd -P)")}"

if ! shfmt -i 2 -ci -d "${FOO_DIR}"/*.sh; then 
  printf 'style check failed' 
  exit 1 
fi 
 
if ! shellcheck "${FOO_DIR}"/*.sh; then 
  printf 'shell check failed' 
  exit 1 
fi 

Shellcheck is smart enough to infer you want POSIX from the interpreter line. You can tweak the shfmt options to your liking, including automatically fixing any issues with the -w flag.

Unit Testing #

With the library file approach, testing your shell scripts is straightforward.

My recommendation is shunit2, which is itself a single-file, POSIX-compliant shell script that exemplifies the points I make here. It’s available via most package managers, or you can include it directly in your project.

Let’s add a test for the foo_main function we wrote previously:

#!/bin/sh

FOO_DIR="${FOO_DIR:-$(cd -P "$(dirname "$0")" && printf "$(pwd -P)")}"

test_foo_main() {
  msg=$(foo_main)
  assertEquals 0 $?
  assertEquals 'Hello, world!' "$msg"
}

oneTimeSetUp() {
  . "${FOO_DIR}/foo_lib.sh"
}

# shellcheck disable=all
. "$(command -v shunit2)"

Include your new test in the runner:

# ...
"${FOO_DIR}"/foo_test.sh

And confirm you can run the tests:

$ ./test.sh
test_hello

Ran 1 test.

OK

Side Effects #

This is a common gotcha for unsuspecting software engineers that perhaps aren’t as familiar with the command line. Shells are all about side effects. There aren’t really any functions when it comes to shell scripting. At least not like you’re thinking, with a scope and stack frame and such. Functions are simply a way to reuse a set of commands or defer their execution until called.

A function’s return code is a value between 0 and 255, with 0 indicating success. I like to make this explicit, even when it means just returning $? to indicate that the function returns the code of the last command executed.

Your unit tests should always verify the return code as well as all side effects:

• The output, including both stdout and stderr as appropriate.
• Defined variables, if any.
• Other externalities. Any external programs executed or filesystem modifications should be mocked for unit testing.

Because there is no function scope, you must always think about side effects, including variable definitions. Anything which is not part of the expected result should be cleaned up or reverted as appropriate.

Consider this alternate implementation, which still passes the test but would leak a variable definition:

foo_main() {
  _msg='Hello, world!'
  printf '%s\n' "$_msg"
  return 0
}

$ set | grep -E '^_msg'  # _msg is undefined

$ . ./foo_lib.sh         # source our library and call the function
$ foo_main
Hello, World!

$ set | grep -E '^_msg'  # oops, we leaked a variable
_msg='Hello, World!'

An easy fix here which is POSIX-compliant and avoids usage of local is to enclose the function body in (), which executes the function in a subshell:

 # note the () enclosing the function body
 hello() (
  _msg=$(echo 'Hello, World!')
  printf '%s\n' "$_msg"
  return 0
)

This pattern is so useful that I’d recommend using it whenever it makes sense. Using a subshell prevents modifying the caller’s environment. Otherwise, you can unset variables before returning an exit code.

Mocking #

If your function invokes an external program:

hello_with_date() (
  if ! _date=$(date); then
    return 1
  fi
  printf 'Hello, the current date is %s\n' "$_date"
  return 0
)

You should mock the call. Compared to many programming languages and test frameworks, this is trivial - just define a function:

NOW='Fri Apr 28 12:20:00 PDT 2023'
date() { printf '%s\n' "$NOW"; }

test_hello_with_date() {
  msg=$(hello_with_date)
  assertEquals 0 $?
  assertEquals 'Hello, the current date is Fri Apr 28 12:20:00 PDT 2023' "$msg"
}

Using function definitions for mocking avoids surprises you might encounter with alias and other alternatives.

Advanced Mocking #

To vary mock behavior between tests, you can utilize shunit2’s setUp() function, which runs before each individual test:

NOW='Fri Apr 28 12:20:00 PDT 2023'
date() {
  if [ "$__date_result" -eq 0 ]; then
    printf '%s\n' "$NOW"
  fi
  return "$__date_result"
}

test_hello_with_date_fail() {
  __date_result=1
  msg=$(hello_with_date)
  assertEquals 1 $?
  assertEquals '' "$msg"
}

test_hello_with_date() {
  msg=$(hello_with_date)
  assertEquals 0 $?
  assertEquals 'Hello, the current date is Fri Apr 28 12:20:00 PDT 2023' "$msg"
}

setUp() {
  export __date_result=0
}

If you prefer to keep the mock closer to the test, another option is the subshell trick we used earlier:

test_hello_with_date_fail() (
  date() {
    return 1
  }

  msg=$(hello_with_date)
  assertEquals 1 $?
  assertEquals '' "$msg"
)

test_hello_with_date() (
  date() {
    printf '%s\n' "$NOW"
    return 0
  }

  msg=$(hello_with_date)
  assertEquals 0 $?
  assertEquals 'Hello, the current date is Fri Apr 28 12:20:00 PDT 2023' "$msg"
)

To implement verifying mocks (“expectations”), you can make use of $SHUNIT_TMPDIR. Given a function that creates three files:

touch_files() (
  for f in one two three; do
    if ! touch "$f"; then
      return 1
    fi
  done
  return 0
)

You could test it as such:

touch() {                              # our mock touch
  echo "touch $*" >>"$__mocked_cmds_f" # record the call
}

test_touch_file() {
  touch_files
  assertEquals 0 $?
  assertEquals "touch one
touch two
touch three" "$(cat "${__mocked_cmds_f}")"
}

setUp() {
  export __mocked_cmds_f="${SHUNIT_TMPDIR}/mocked_cmds"
  cat /dev/null >"$__mocked_cmds_f"
}

Here we record all calls to touch in a temporary file, which is emptied before each test.

Testing Against Multiple Shells #

To ensure portability you’ll want to test on multiple platforms and shells.

Let’s address the “multiple shells” bit first and improve our test runner:

# ...
shells='ash bash dash ksh mksh pdksh sh zsh'
tests=$(find "${FOO_DIR}" -type f -maxdepth 1 -name "*_test.sh")

for shell in ${shells}; do
  printf '\n##############################\n# %s\n' "${shell}"
  if shell_bin=$(command -v "$shell"); then
    printf '# using %s\n' "${shell_bin}"
  else
    printf '# %s not found, skipping...\n' "${shell}"
    continue
  fi
  for t in ${tests}; do
    (exec ${shell_bin} "${t}" 2>&1)
  done
done

Now running ./test.sh will run our tests against every locally-available shell.

Integration Testing #

How you should perform integration testing largely depends on what your script actually does. It’s a bit beyond the scope of this article, but I’d like to at least demonstrate a simple setup using Docker.

Let’s add a runner that executes our unit tests on Alpine and Ubuntu:

FROM alpine:latest
RUN apk add --no-cache \
  shfmt \
  shellcheck \
  shunit2 \
  bash \
  dash \
  mksh \
  zsh

FROM ubuntu:latest
RUN apt-get update && apt-get -y install \
  shfmt \
  shellcheck \
  shunit2 \
  ash \
  bash \
  dash \
  ksh \
  mksh \
  zsh

#!/bin/sh -e

platforms="ubuntu alpine"

for platform in ${platforms}; do
  printf '\n##############################\n# %s\n' "${platform}"
  docker buildx build -f "Dockerfile.${platform}" -t "${platform}-foo" .
  docker buildx build -f- -t "${platform}-foo-test" . <<HERE
FROM ${platform}-foo
RUN mkdir /app
COPY ./ /app/
WORKDIR /app
CMD ./test.sh
HERE
  docker run -it --rm "${platform}-foo-test"
done

Now by simply executing ./test_it.sh we can run our unit tests against many different shells and platforms.

Summary #

I’ve wanted to write this article for some time after seeing so many untested scripts in the wild.

If you like the suggestions here, download the generator below and create your own project, or use it to develop your own shell scripting best practices.

shgen: A Generator for Shell Scripting Projects