Test Anything Protocol (TAP) with autotools and cmocka

Autotoole can handle test suits of several types. Usually, I store my tests in "test" directory in the root of the project and set up each test in "Makefile.am" file. Each test to run is a separate executable file listed in "TEST" variable in makefile. For example:

TESTS = foo_test bar_test xyz_test 

Automake will run the tests and generate report basing on the return value of each executed file. Does not matter how many assertions/tests you do in the test files (foo_test.c, bar_test.c and xyz_test.c), report will show summary only per test file:

PASS: foo_test
PASS: bar_test
PASS: xyz_test

============================================================================
Testsuite summary for foobar 0.1
============================================================================

# TOTAL: 3
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0

============================================================================

Fortunately, autotools supports Test Anything Protocol (TAP). TAP output should display number of expected tests in the beginning and per each test in file it will output line that begins with "ok" or "not ok". This is simplified explanation. For more information check TAP website.

To enable it with autotools the driver script should be installed "tap-driver.sh". To do that, add following line to "configure.ac":

AC_REQUIRE_AUX_FILE([tap-driver.sh])

And run command:

automake --foreign --add-missing

Finally, in Makefile.am in "test" directory add following:

LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) \
                  $(top_srcdir)/tap-driver.sh

After that, run "./configure" to regenerate Makefiles.

Last step is to make sure that unit test framework produces output that conforms TAP. This can be done manually or can be already implemented in testing library or framework. For example, cmocka supports TAP output format and can be done in two different ways:

1. Setting environment variable "CMOCKA_MESSAGE_OUTPUT=TAP" before running cmocka tests.
2. Using function "cmocka_set_message_output(CM_OUTPUT_TAP);" before running tests.

Here is example of main function with tests:

int main(void)
{
  const struct CMUnitTest tests[] = {
    cmocka_unit_test_setup_teardown (create_pack_with_no_header, setup_pack, teardown_pack),
    cmocka_unit_test_setup_teardown (create_pack_with_single_header, setup_pack, teardown_pack),
    cmocka_unit_test_setup_teardown (create_pack_with_two_headers, setup_pack, teardown_pack),
    cmocka_unit_test_setup_teardown (create_pack_with_three_headers, setup_pack, teardown_pack),
    cmocka_unit_test_setup_teardown (create_pack_with_multi_headers, setup_pack, teardown_pack),
    cmocka_unit_test_setup_teardown (create_pack_with_none_existing_header, setup_pack, teardown_pack),
    cmocka_unit_test_setup_teardown (create_pack_with_unknown_header, setup_pack, teardown_pack),
    cmocka_unit_test_setup_teardown (create_pack_with_empty_header, setup_pack, teardown_pack),
    cmocka_unit_test_setup_teardown (create_pack_with_empty_last_header, setup_pack, teardown_pack),
    cmocka_unit_test_setup_teardown (pack_find_headers, setup_pack, teardown_pack),
    cmocka_unit_test_setup_teardown (pack_find_header_by_name, setup_pack, teardown_pack),
  };

  cmocka_set_message_output(CM_OUTPUT_TAP);

  return cmocka_run_group_tests_name("Create AMI package tests.", tests, NULL, NULL);
}

Example of output:

The example of the project that is configured with autotools TAP can be found here.