-
Notifications
You must be signed in to change notification settings - Fork 210
Table driven tests
Table-driven tests (also known as keyword-driven testing) are one of the most powerful ways to test functionality. They enable a developer to test multiple cases easily, and also allow additional tests to be added with relatively little code.
Table-driven tests use a table of data to drive tests. They frequently include expected test behavior (e.g. an expected return code), and a keyword or description to include in output failure messages.
Let's say the following function was created to square the values passed into it.
square() { expr $1 \* $1; }If we wanted to test a few different values, say 1, 2, 3, and '', we'd need to write a test for each one of those. For small numbers of test cases, that isn't an issue, but when there are many possible ways that a test needs to be tested, the test code can become complicated and difficult to read. As test code is frequently used as API documentation, this means the code may lose its value as documentation.
test_square_direct() {
output=$(square 1 2>&1); rtrn=$?
assertTrue "one: square() unexpected error; return ${rtrn}" ${rtrn}
assertTrue "one: square() = '${output}', want 1" ${rtrn}
output=$(square 2 2>&1); rtrn=$?
assertTrue "two: square() unexpected error; return ${rtrn}" ${rtrn}
assertTrue "two: square() = '${output}', want 4" ${rtrn}
output=$(square 3 2>&1); rtrn=$?
assertTrue "three: square() unexpected error; return ${rtrn}" ${rtrn}
assertTrue "three: square() = '${output}', want 9" ${rtrn}
output=$(square '' 2>&1); rtrn=$?
assertFalse "empty: square() expected error, got '${output}'}" ${rtrn}
}The test above works fine and is readable, but adding new test cases means lots of copying-and-pasting, and value adjustment. Usually something gets forgotten, which requires addition time to troubleshoot and fix.
To test the above code efficiently with multiple cases, a table-driven test can be used.
A sample test table might look like this (0 == true in shell):
| desc | arg | want | ok |
|---|---|---|---|
| one | 1 | 1 | 0 |
| two | 2 | 4 | 0 |
| three | 3 | 9 | 0 |
| empty | '' | '' | 1 |
With it, we could test all four possibilities, and we test both succeeding and failing conditions. We could also add a new test case simply by adding it to the table.
The code below demonstrates a table-driven test. Note that the actual test code (the asserts) are only listed once per case. This makes the code much easier to read. Also note that the desc field is added to the output (in case of failure) making it immediately obvious which case failed.
test_square_table() {
while read desc arg want ok; do
got=$(square ${arg} 2>&1); rtrn=$?
if [ ${ok} -eq ${SHUNIT_TRUE} ]; then
assertTrue "${desc}: square() unexpected error; return ${rtrn}" ${rtrn}
assertEquals "${desc}: square() = '${got}', want ${want}" "${got}" "${want}"
else
assertFalse "${desc}: square() expected error, got '${got}'" ${rtrn}
fi
done <<EOF
one 1 1 ${SHUNIT_TRUE}
two 2 4 ${SHUNIT_TRUE}
three 3 9 ${SHUNIT_TRUE}
empty '' '' ${SHUNIT_FALSE}
EOF
}There are multiple ways to accomplish table-driven tests in shell code. In all the provided cases below, a while read loop will be used to read the table.
Here documents allow one to inline a file in the source code.
Quoting the manpage for bash:
This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input for a command.
The format of here-documents is:
<<[-]word here-document delimiter
The basic format for this type of test is:
while read desc arg want; do
got=$(fn ${arg})
rtrn=$?
assertTrue "${desc}: fn() unexpected error; return ${rtrn}" ${rtrn}
assertEquals "${desc}: fn() = ${got}, want ${want}" "${got}" "${want}"
done <<EOF
test#1 value1 output_one
test#2 value2 output_two
EOF💡 Frequently, the first variable name will be a description (desc) or keyword (kw), which is added to the output of each test.
read has the limitation that values are separated by spaces, which means individual table values cannot contain spaces. If spaces are needed, the best workaround is to place table values with spaces in the last column as the read command will append all content into the last variable name.
The following shell snippet demonstrates the workaround. Notice how ${b} holds the values "2 3 4".
$ read a b <<EOF
1 2 3 4
EOF
$ echo a:"${a}" b:"${b}"
a:"1" b:"2 3 4"The test_square_table() example above uses a here document.
Very similar to here documents, file descriptor redirection + here document allows one to inline a file in the code. The advantage of this method is that the tests precede the test code, rather than trailing it as in the basic here document example.
The basic format for this type of test is. It saves the STDIN file descriptor to file descriptor #9, and redirects STDIN to be the contents of the here document.
exec 9<&0 <<EOF
test#1 value1 output_one
test#2 value2 output_two
EOF
while read desc arg want; do
got=$(fn ${arg})
rtrn=$?
assertTrue "${desc}: fn() unexpected error; return ${rtrn}" ${rtrn}
assertEquals "${desc}: fn() = ${got}, want ${want}" "${got}" "${want}"
done
exec 0<&9 9<&-The shlib_relToAbsPath() example below uses this method.
Although generally discouraged in unit testing, the best option may be to read an external file that contains the test data. This method is just like the here document method, except that a file is redirected to STDIN instead of the here document.
This method is good if the test data is generated by some other source.
while read desc arg want; do
got=$(fn ${arg})
rtrn=$?
assertTrue "${desc}: fn() unexpected error; return ${rtrn}" ${rtrn}
assertEquals "${desc}: fn() = ${got}, want ${want}" "${got}" "${want}"
done </path/to/some/testfile/path/to/some/testfile
test#1 value1 output_one
test#2 value2 output_two
An example of how powerful table driven tests are can be seen by looking at the shlib_relToAbsPath() function from the https://github.com/kward/shlib repository.
Here is the shlib_relToAbsPath code (as of this writing).
# vim:et:ft=sh:sts=2:sw=2
# Copyright 2008 Kate Ward. All Rights Reserved.
# Released under the Apache 2.0 license.
#
# Author: [email protected] (Kate Ward)
# Repository: https://github.com/kward/shlib
SHLIB_PWD='pwd'
# Convert a relative path into it's absolute equivalent.
#
# This function will automatically prepend the current working directory if the
# path is not already absolute. It then removes all parent references (`../`) to
# reconstruct the proper absolute path.
#
# Args:
# shlib_path_: string: relative path
# Outputs:
# string: absolute path
shlib_relToAbsPath() {
shlib_path_=$1
# Prepend current directory to relative paths.
echo "${shlib_path_}" |grep '^/' >/dev/null 2>&1 \
|| shlib_path_="`${SHLIB_PWD}`/${shlib_path_}"
# Clean up the path. If all `sed` commands supported true regular expressions,
# then this is what they would do.
shlib_old_=${shlib_path_}
while true; do
shlib_new_=`echo "${shlib_old_}" |sed 's/[^/]*\/\.\.\/*//g;s/\/\.\//\//'`
[ "${shlib_old_}" = "${shlib_new_}" ] && break
shlib_old_=${shlib_new_}
done
echo "${shlib_new_}"
unset shlib_old_ shlib_new_ shlib_path_
}Here is the associated table driven unit test code shlib_relToAbsPath_test.sh (as of this writing). Notice the large number of test cases that are tested with a relatively small amount of test code.
#!/bin/sh
# Copyright 2008 Kate Ward. All Rights Reserved.
# Released under the Apache 2.0 license.
#
# Author: [email protected] (Kate Ward)
# Repository: https://github.com/kward/shlib
testRelToAbsPath() {
pwd='mock_pwd'
cwd='/path/to/cwd'
parent='/path/to'
grandparent='/path'
# Save STDIN and redirect it from an in-line file.
exec 9<&0 <<EOF
abc ${cwd}/abc
abc/def ${cwd}/abc/def
abc/def/ghi ${cwd}/abc/def/ghi
abc/./def ${cwd}/abc/def
abc/../def ${cwd}/def
abc/../def/../ghi ${cwd}/ghi
/abc /abc
/abc/def /abc/def
/abc/def/ghi /abc/def/ghi
/abc/def/../../ghi /ghi
/abc/def/ghi/../../jkl /abc/jkl
/abc/../def /def
/abc/../def/../ghi /ghi
./abc ${cwd}/abc
../abc ${parent}/abc
../abc/def ${parent}/abc/def
../../abc/def ${grandparent}/abc/def
${cwd}/../../abc/def ${grandparent}/abc/def
EOF
while read relPath absPath; do
# Ignore comment and blank lines.
echo "${relPath}" |egrep -v "^(#|$)" >/dev/null || continue
# Test the function.
newPath=`PWD=${cwd} shlib_relToAbsPath "${relPath}"`
assertEquals "${relPath}" "${absPath}" "${newPath}"
done
exec 0<&9 9<&-
}
mock_pwd() { echo '/path/to/cwd'; }
oneTimeSetUp() {
SHLIB_PWD_DEFAULT=${SHLIB_PWD}
# Load the function.
. './'`basename $0 |sed 's/_test.sh$//'`
}
setUp() {
SHLIB_PWD='mock_pwd'
}
tearDown() {
SHLIB_PWD=${SHLIB_PWD_DEFAULT}
}
# Run shunit2.
. ../lib/shunit2