| LTP Test Writing Guidelines |
| =========================== |
| |
| This document describes LTP guidelines and LTP test interface and is intended |
| for anybody who want to write or modify a LTP testcase. It's not a definitive |
| guide and it's not, by any means, a substitute for common sense. |
| |
| |
| 1. General Rules |
| ---------------- |
| |
| 1.1 Simplicity |
| ~~~~~~~~~~~~~~ |
| |
| For all it's worth keep the testcases simple or better as simple as possible. |
| The kernel and libc are tricky beasts and the complexity imposed by their |
| interfaces is quite high. Concentrate on the interface you want to test and |
| follow the UNIX philosophy. It's a good idea to make the test as |
| self-contained as possible too (it should not depend on tools or libraries |
| that are not widely available). |
| |
| Do not reinvent the wheel |
| |
| * Use LTP standard interface |
| * Do not add custom PASS/FAIL reporting functions |
| * Do not write Makefiles from scratch, |
| use LTP build system instead, etc. |
| * ... |
| |
| 1.2 Code duplication |
| ~~~~~~~~~~~~~~~~~~~~ |
| |
| Copy & paste is a good servant but very poor master. If you are about to copy a |
| large part of the code from one testcase to another, think what would happen if |
| you find bug in the code that has been copied all around the tree. What about |
| moving it to a library instead? |
| |
| The same goes for short but complicated parts, whenever you are about to copy & |
| paste a syscall wrapper that packs arguments accordingly to machine |
| architecture or similarly complicated code, put it into a header instead. |
| |
| 1.3 Coding style |
| ~~~~~~~~~~~~~~~~ |
| |
| 1.3.1 C coding style |
| ^^^^^^^^^^^^^^^^^^^^ |
| |
| LTP adopted Linux kernel coding style. If you aren't familiar with its rules |
| locate 'linux/Documentation/CodingStyle' in the kernel sources and read it, |
| it's a well written introduction. |
| |
| There is also a checkpatch (see 'linux/scripts/checkpatch.pl') script that can |
| be used to check your patches before the submission. |
| |
| NOTE: If checkpatch does not report any problems, the code still may be wrong |
| as the tool only looks for common mistakes. |
| |
| 1.3.2 Shell coding style |
| ^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| When writing testcases in shell write in portable shell only, it's a good idea |
| to try to run the test using alternative shell (alternative to bash, for |
| example dash) too. |
| |
| Here are some common sense style rules for shell |
| |
| * Keep lines under 80 chars |
| |
| * Use tabs for indentation |
| |
| * Keep things simple, avoid unnecessary subshells |
| |
| * Don't do confusing things (i.e. don't name your functions like common shell |
| commands, etc.) |
| |
| * Quote variables |
| |
| * Be consistent |
| |
| 1.4 Commenting code |
| ~~~~~~~~~~~~~~~~~~~ |
| |
| Comments can sometimes save you day but they can easily do more harm than |
| good. There has been several cases where comments and actual implementation |
| were drifting slowly apart which yielded into API misuses and hard to find |
| bugs. Remember there is only one thing worse than no documentation, wrong |
| documentation. |
| |
| Generally everybody should write code that is obvious (which unfortunately |
| isn't always possible). If there is a code that needs to be commented keep it |
| short and to the point. Never ever comment the obvious. |
| |
| In case of LTP testcases it's customary to add a paragraph with highlevel test |
| description somewhere at the beginning of the file (usually right under the GPL |
| header). This helps other people to understand the overall goal of the test |
| before they dive into the technical details. |
| |
| 1.5 Backwards compatibility |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| LTP test should be as backward compatible as possible. Think of an enterprise |
| distributions with long term support (more than five years since the initial |
| release) or of an embedded platform that needs to use several years old |
| toolchain supplied by the manufacturer. |
| |
| Therefore LTP test for more current features should be able to cope with older |
| systems. It should at least compile fine and if it's not appropriate for the |
| configuration it should return 'TCONF' (see test interface description below). |
| |
| There are several types of checks we use: |
| |
| The *configure script* is usually used to detect availability of a function |
| declarations in system headers. It's used to disable tests at compile time. |
| |
| The *tst_kvercmp()* which is runtime kernel version detection and comparison |
| and is used to disable tests at runtime. |
| |
| Checking the *errno* value is another type of runtime check. Most of the |
| syscalls returns either 'EINVAL' or 'ENOSYS' when syscall was not implemented |
| or was disabled upon kernel compilation. |
| |
| Sometimes it also makes sense to define a few macros instead of creating |
| configure test. One example are Linux specific POSIX clock ids in |
| 'include/lapi/posix_clocks.h'. |
| |
| 1.6 Dealing with messed up legacy code |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| LTP contains a lot of old and messy code and we are cleaning it up as fast as |
| we can but despite the efforts there is still a lot. If you start modifying |
| old or a messed up testcase and your changes are more complicated than simple |
| typo fixes you should do a cleanup first (in a separate patch). It's easier to |
| review the changes if you separate the formatting fixes from the changes that |
| affects the test behavior. |
| |
| The same goes for moving files. If you need a rename or move file do it in a |
| separate patch. |
| |
| 1.7 License |
| ~~~~~~~~~~~ |
| |
| Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or |
| any later version). |
| |
| 2. Writing a testcase |
| --------------------- |
| |
| 2.1 LTP Structure |
| ~~~~~~~~~~~~~~~~~ |
| |
| The structure of LTP is quite simple. Each test is a binary written either in |
| portable shell (no bashism) or C. The test gets a configuration via |
| environment variables and/or command line parameters, it prints additional |
| information into the stdout and reports overall success/failure via the exit |
| value. |
| |
| Tests are generally placed under the 'testcases/' directory. Everything that |
| is a syscall or (slightly confusingly) libc syscall wrapper goes under |
| 'testcases/kernel/syscalls/'. Then there is 'testcases/open_posix_testsuite' |
| which is a well maintained fork of the upstream project that has been dead |
| since 2005 and also a number of directories with tests for more specific |
| features. |
| |
| 2.1.1 Runtest Files |
| ^^^^^^^^^^^^^^^^^^^ |
| |
| The list of tests to be executed is stored in runtest files under the |
| 'runtest/' directory. The default set of runtest files to be executed is |
| stored in 'scenario_groups/default'. When you add a test you should add |
| corresponding entries into some runtest file(s) as well. |
| |
| For syscall tests (these placed under 'testcases/kernel/syscalls/') use |
| 'runtest/syscalls' file, for kernel related tests for memory management we |
| have 'runtest/mm', etc. |
| |
| IMPORTANT: The runtest files should have one entry per a test. Creating a |
| wrapper that runs all your tests and adding it as a single test |
| into runtest file is strongly discouraged. |
| |
| 2.1.2 Datafiles |
| ^^^^^^^^^^^^^^^ |
| |
| If your test needs datafiles to work, these should be put into a subdirectory |
| named 'datafiles' and installed into the 'testcases/data/$TCID' directory (to |
| do that you have to add 'INSTALL_DIR := testcases/data/TCID' into the |
| 'datafiles/Makefile'). |
| |
| You can obtain path to datafiles via $LTP_DATAROOT provided by test.sh |
| '$LTP_DATAROOT/...' |
| or via C function 'tst_dataroot()' provided by libltp: |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| const char *dataroot = tst_dataroot(); |
| ------------------------------------------------------------------------------- |
| |
| Datafiles can also be accessed as '$LTPROOT/testcases/data/$TCID/...', |
| but '$LTP_DATAROOT' and 'tst_dataroot()' are preffered as these can be used |
| when running testcases directly in git tree as well as from install |
| location. |
| |
| The path is constructed according to these rules: |
| |
| 1. if '$LTPROOT' is set, return '$LTPROOT/testcases/data/$TCID' |
| 2. else if 'tst_tmpdir()' was called return '$STARTWD/datafiles' |
| (where '$STARTWD' is initial working directory as recorded by 'tst_tmdir()') |
| 3. else return '$CWD/datafiles' |
| |
| See 'testcases/commands/ade/ldd/ldd01' for example. |
| |
| 2.1.3 Subexecutables |
| ^^^^^^^^^^^^^^^^^^^^ |
| |
| If you test needs to execute a binary, place it in the same directory as the |
| testcase and name the file starting with testname_ ('TCID_' see below). |
| Once the test is executed by the framework, the path to the directory with all |
| LTP binaries is added to the '$PATH' and you can execute it just by its name. |
| |
| TIP: If you need to execute such test from the LTP tree, you can add path to |
| current directory to '$PATH' manually with: 'PATH="$PATH:$PWD" ./foo01'. |
| |
| 2.2 Writing a test in C |
| ~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| 2.2.1 Basic test structure |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| Let's start with an example, following code is a simple test for a getenv(). |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| /* |
| * This is test for basic functionality of getenv(). |
| * |
| * - create an env variable and verify that getenv() can get get it |
| * |
| * - call getenv() with nonexisting variable name, check that it returns NULL |
| */ |
| |
| #include "test.h" |
| #include "usctest.h" |
| |
| char *TCID = "getenv01"; |
| int TST_TOTAL = 2; |
| |
| #define TEST_ENV "LTP_TEST_ENV" |
| #define TEST_NE_ENV "LTP_TEST_THIS_DOES_NOT_EXIST" |
| #define TEST_ENV_VAL "val" |
| |
| static void setup(void) |
| { |
| if (setenv(TEST_ENV, TEST_ENV_VAL, 1)) |
| tst_brkm(TBROK | TERRNO, NULL, "setenv() failed"); |
| } |
| |
| static void test(void) |
| { |
| char *ret; |
| |
| ret = getenv(TEST_ENV); |
| |
| if (ret) { |
| if (!strcmp(ret, TEST_ENV_VAL)) |
| tst_resm(TPASS, "getenv(" TEST_ENV ") = '" |
| TEST_ENV_VAL "'"); |
| else |
| tst_resm(TFAIL, "getenv(" TEST_ENV "} = '%s', " |
| "expected '" TEST_ENV_VAL "'", ret); |
| } else { |
| tst_resm(TFAIL, "getenv(" TEST_ENV ") = NULL"); |
| } |
| |
| ret = getenv(TEST_NE_ENV); |
| |
| if (ret) |
| tst_resm(TFAIL, "getenv(" TEST_NE_ENV ") = '%s'", ret); |
| else |
| tst_resm(TPASS, "getenv(" TEST_NE_ENV ") = NULL"); |
| } |
| |
| int main(int argc, char *argv[]) |
| { |
| const char *msg; |
| int lc; |
| |
| if ((msg = parse_opts(argc, argv, NULL, NULL))) |
| tst_brkm(TBROK, NULL, "OPTION PARSING ERROR - %s", msg); |
| |
| setup(); |
| |
| for (lc = 0; TEST_LOOPING(lc); lc++) |
| test(); |
| |
| tst_exit(); |
| } |
| ------------------------------------------------------------------------------- |
| |
| Each test must define 'TCID' and 'TST_TOTAL'. |
| |
| 'TCID' defines test name (usually syscall/libcall name + number) and is used |
| as a base for temporary directory name (if 'tst_tmpdir()' is used). In most of |
| the cases the 'TCID' is the same as test filename (without the extension). |
| |
| 'TST_TOTAL' defines total number of tests. |
| |
| NOTE: The test should report 'TST_TOTAL' PASSES/FAILS on each iteration. |
| |
| The overall test initialization is usually done in a 'setup()' function and |
| the overall cleanup is done in a 'cleanup()' function. Here 'cleanup()' is |
| omitted as the test don't have to clean anything up. Usually it's called right |
| before the 'tst_exit()' to do the cleanup and passed to library functions that |
| can exit the test execution, for example 'tst_brkm()', so that the test can do |
| a cleanup before it exits. |
| |
| WARNING: Never pass a cleanup function to functions called from cleanup. |
| If you don't see why this is a problem you are not ready to write |
| a testcase yet. |
| |
| WARNING: Don't use 'tst_brkm()' in 'cleanup()' unless you really want to exit |
| the cleanup prematurely. See discussion below for further |
| information. |
| |
| The 'parse_opts()' parses the test command line arguments, it's important to |
| use it even if the test has no (other than default) parameters. |
| |
| The last important thing is the 'TEST_LOOPING()' macro, each test has standard |
| options so that it can be executed N times or for M seconds. |
| |
| NOTE: The 'test()' function must work correctly even if executed several times |
| from test main loop. |
| |
| A word about the cleanup() callback |
| +++++++++++++++++++++++++++++++++++ |
| |
| There are a few rules that needs to be followed in order to write correct |
| cleanup() callback. |
| |
| 1. Don't call callback() from within the callback(). |
| (This includes passing callback pointer to library 'tst_' functions.) |
| |
| 2. Make sure to free resources in the reverse order they were |
| initialized. (Some of the steps may not depend on others and everything |
| will work if there were swapped but let's keep it in order.) |
| |
| 3. Free only resources that were initialized. |
| |
| The third rule needs a bit more detailed discussion. Consider, for example, |
| test setup below which is example of a setup that prepares temporary |
| directory, two file descriptors and allocates a buffer. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| static int fd0, fd1; |
| static void *buf; |
| #define BUFSIZE (1024 * 1024) |
| |
| static void setup(void) |
| { |
| tst_require_root(); |
| tst_tmpdir(); |
| |
| fd0 = SAFE_OPEN(cleanup, "test_file1", O_CREAT | O_RDWR, 0666); |
| SAFE_UNLINK(cleanup, "test_file1"); |
| fd1 = SAFE_OPEN(cleanup, "test_file2", O_CREAT | O_RDWR, 0666); |
| SAFE_UNLINK(cleanup, "test_file2"); |
| |
| buf = SAFE_MALLOC(cleanup, BUFSIZE); |
| } |
| ------------------------------------------------------------------------------- |
| |
| In this case the 'cleanup()' function may be entered in five different states: |
| |
| * The first 'SAFE_OPEN()' has failed, temporary directory is created |
| no files are open and +fd0 == -1+, +fd1 == 0+ and +buf == NULL+. |
| |
| * The first 'SAFE_UNLINK()' has failed, +fd0+ holds file descriptor and |
| +fd1 == 0+. |
| |
| * The second 'SAFE_OPEN()' has failed, +fd0+ holds file descriptor and |
| +fd1 == -1+. |
| |
| * The second 'SAFE_UNLINK()' or 'SAFE_MALLOC()' has failed and both 'fd0' and |
| 'fd1' holds file descriptors, +buf+ is still +NULL+. |
| |
| * The 'cleanup()' was called at the end of the test, all +fd0+, +fd1+ and |
| +buf+ are initialized. |
| |
| The 'cleanup()' functions should be able to cope with all scenarios. In this |
| case following code will do: |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| static void cleanup(void) |
| { |
| free(buf); |
| |
| if (fd1 > 0) |
| close(fd1); |
| |
| if (fd0 > 0) |
| close(fd0); |
| |
| tst_rmdir(); |
| } |
| ------------------------------------------------------------------------------- |
| |
| 2.2.2 Basic test interface |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| void tst_resm(int ttype, char *arg_fmt, ...); |
| ------------------------------------------------------------------------------- |
| |
| Printf-like function to report test result, it's mostly used with ttype: |
| |
| |============================== |
| | 'TPASS' | Test has passed. |
| | 'TFAIL' | Test has failed. |
| | 'TINFO' | General message. |
| |============================== |
| |
| The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print |
| 'errno', 'TEST_ERRNO' respectively. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| void tst_brkm(int ttype, void (*cleanup)(void), char *arg_fmt, ...); |
| ------------------------------------------------------------------------------- |
| |
| Printf-like function to report error and exit the test, it can be used with ttype: |
| |
| |============================================================ |
| | 'TBROK' | Something has failed in test preparation phase. |
| | 'TCONF' | test is not appropriate for current configuration |
| (syscall not implemented, unsupported arch, ...) |
| | 'TFAIL' | test has failed |
| |============================================================ |
| |
| The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print |
| 'errno', 'TEST_ERRNO' respectively. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| void tst_require_root(void (*cleanup)(void)); |
| ------------------------------------------------------------------------------- |
| |
| Abort the test if it's not executed with root privileges. If needed this should |
| be one of the first checks in the test 'setup()'. |
| |
| WARNING: The cleanup parameter is deprecated and should always be 'NULL'. |
| |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| void tst_exit(void); |
| ------------------------------------------------------------------------------- |
| |
| Exits the tests, note that this function has no parameters, the PASSES/FAILS |
| reported by the 'tst_resm()' interfaces were stored and are used for the exit |
| value. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| const char *tst_strsig(int sig); |
| ------------------------------------------------------------------------------- |
| |
| Return the given signal number's corresponding string. |
| |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| const char *tst_strerrno(int err); |
| ------------------------------------------------------------------------------- |
| |
| Return the given errno number's corresponding string. |
| |
| |
| 2.2.3 Test temporary directory |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| If the test needs to create files (which is common case) it must create a test |
| temporary directory in LTP temp directory and work with files only inside of |
| this directory. Happily there are easy to use library functions exactly for |
| this purpose. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| void tst_tmpdir(void); |
| ------------------------------------------------------------------------------- |
| |
| Creates a directory with a unique name (based on the test 'TCID') under the |
| LTP temporary directory (defaults to '/tmp/') and changes the test working |
| directory to it. It's usually called from the test 'setup()'. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| void tst_rmdir(void); |
| ------------------------------------------------------------------------------- |
| |
| Removes the directory recursively. It's usually called from test 'cleanup()'. |
| |
| It's important to close all file descriptors (that point to files in test |
| temporary directory, even the unlinked ones) before the test calls |
| 'tst_rmdir()' otherwise the test may break on NFS mounted temp dir (look for |
| "NFS silly rename"). |
| |
| [[2.2.4]] |
| 2.2.4 Safe macros |
| ^^^^^^^^^^^^^^^^^ |
| |
| Safe macros aim to simplify error checking in test preparation. Instead of |
| calling system API functions, checking for their return value and aborting the |
| test if the operation has failed, you just use corresponding safe macros. |
| |
| Use them whenever it's possible. |
| |
| NOTE: Have a look at the the xref:2.2.7[doing real test in the child process] |
| if you want to use safe macros from within a child process. |
| |
| Instead of writing: |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| fd = open("/dev/null", O_RDONLY); |
| if (fd < 0) |
| tst_brkm(TBROK | TERRNO, cleanup, "opening /dev/null failed"); |
| ------------------------------------------------------------------------------- |
| |
| You write just: |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| fd = SAFE_OPEN(cleanup, "/dev/null", O_RDONLY); |
| ------------------------------------------------------------------------------- |
| |
| They can also simplify reading and writing of sysfs files, you can, for |
| example, do: |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| SAFE_FILE_SCANF(cleanup, "/proc/sys/kernel/pid_max", "%lu", &pid_max); |
| ------------------------------------------------------------------------------- |
| |
| See 'include/safe_macros.h', 'include/safe_stdio.h' and |
| 'include/safe_file_ops.h' for a complete list. |
| |
| NOTE: It's wise NOT to use safe macros in test cleanup(). This is because |
| all safe macros call tst_brkm(), which exits the test immediately, making |
| the cleanup() exit prematurely. (Actually, this is hacked around in |
| the test library at the moment so that the cleanup() will finish, but |
| the hack will be removed in the future). |
| |
| 2.2.5 Runtime kernel version detection |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| int tst_kvercmp(int r1, int r2, int r3); |
| |
| struct tst_kern_exv { |
| char *dist_name; |
| char *extra_ver; |
| }; |
| |
| int tst_kvercmp2(int r1, int r2, int r3, struct tst_kern_exv *vers); |
| ------------------------------------------------------------------------------- |
| |
| These two functions are intended for runtime kernel version detection. They |
| parse the output from 'uname()' and compare it to the passed values. |
| |
| The return value is similar to the 'strcmp()' function, i.e. zero means equal, |
| negative value means that the kernel is older than than the expected value and |
| positive means that it's newer. |
| |
| The second function 'tst_kvercmp2()' allows for specifying per-vendor table of |
| kernel versions as vendors typically backport fixes to their kernels and the |
| test may be relevant even if the kernel version does not suggests so. See |
| 'testcases/kernel/syscalls/inotify/inotify04.c' for example usage. |
| |
| 2.2.6 Fork()-ing |
| ^^^^^^^^^^^^^^^^ |
| |
| Be wary that if the test forks and there were messages printed by the tst_* |
| interfaces, the data may still be in kernel buffers and these are NOT flushed |
| automatically. |
| |
| This happens when 'stdout' gets redirected to a file. In this case, the |
| 'stdout' is not line buffered, but block buffered, and buffered messages will be |
| printed by the parent and each of the children. |
| |
| To avoid that, you should either call 'tst_flush()' right before the 'fork()', |
| or even better - use 'tst_fork()' instead. |
| |
| [[2.2.7]] |
| 2.2.7 Doing real test in the child process |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| If you have to do the test in a child process you have two possibilities: |
| |
| * You can use test interface ('tst_resm()', 'tst_brkm()', 'tst_exit()', ...) |
| but there are some rules to obey: |
| |
| ** If you use 'tst_resm()' to record test the results in child process, you |
| *MUST* use either 'tst_exit()' or 'tst_brkm()' to exit the child process. |
| |
| ** If you use 'tst_brkm()' with a non-NULL cleanup argument in the child |
| process, you're strongly recommended to create a separate cleanup function |
| for child process that will only clean up what has been allocated in child |
| process. Otherwise the 'cleanup()' will be called both from the parent and |
| child and the test will likely fail (for example in 'tst_rmdir()' because |
| both parent and child will attempt to remove the directory and one of them |
| will fail). |
| |
| ** The same applies to xref:2.2.4[safe macros] because they call 'tst_brkm()' |
| when anything goes wrong. |
| |
| * Or you can stick to plain old 'exit()' with 'TPASS', 'TBROK', 'TFAIL' and |
| 'TCONF' constants. |
| |
| Then you call 'tst_record_childstatus()' to records the result of the test |
| (done in child process) which propagates the child result (child exit status) |
| to the parent process correctly. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "test.h" |
| |
| void tst_record_childstatus(void (*cleanup)(void), pid_t child); |
| ------------------------------------------------------------------------------- |
| |
| This function does a 'waitpid()' on child process and record child process's |
| return value into the overall test result. |
| |
| 2.2.8 Fork() and Parent-child synchronization |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| As LTP tests are written for Linux, most of the test involves fork()-ing and |
| parent-child process synchronization. We have a checkpoint library code |
| that works even for two different processes, all they need to is to run with |
| the same working directory (they use FIFO for synchronization). The checkpoint |
| interface provides two pairs of signal and wait functions. One pair to be used |
| to signal child from parent and second to signal parent from child. |
| |
| Checkpoint is represented by 'struct tst_checkpoint', which has to be |
| initialized before first usage and FIFO has to be created. This is |
| usually done in parent process in single step by calling |
| 'tst_checkpoint_create()'. |
| |
| Child processes created via fork() are ready to use tst_checkpoint_* |
| synchronization functions, as they inherited already initialized |
| 'struct tst_checkpoint'. |
| |
| Child processes started via exec*, or any other process can use already |
| created FIFO, provided they initialize 'struct tst_checkpoint' first by |
| call to 'tst_checkpoint_init()'. This function does not create any FIFO, |
| it relies on fact, that it was already created by some other process. |
| Note, that if you use multiple FIFOs in this scenario, these should be |
| initialized in same order as in process you are sychronizing against. |
| |
| IMPORTANT: Be aware, that following scenario is _NOT_ safe when using |
| only single checkpoint. You are advised to use two checkpoints |
| in this case. |
| |
| * tst_checkpoint_signal_child() followed by tst_checkpoint_parent_wait() |
| |
| parent | child |
| -------------------------------+------------------------------- |
| | tst_checkpoint_child_wait() |
| | blocking read, waits until |
| | parent opens write side |
| tst_checkpoint_signal_child() | |
| NONBLOCK write | |
| | child is now able to read |
| | from FIFO |
| tst_checkpoint_parent_wait() | |
| NONBLOCK read parent able to | |
| read from FIFO and can race | |
| with read in child | |
| | tst_checkpoint_signal_parent() |
| -------------------------------+-------------------------------- |
| |
| For the details of the interface, look into the 'include/tst_checkpoint.h' and |
| 'lib/tests/tst_checkpoint_*'. |
| |
| There is also an interface that allows parent to wait until child is blocked |
| in kernel (for example waits in 'pause()'), see 'include/tst_process_state.h' |
| for more. |
| |
| 2.2.9 Signal handlers |
| ^^^^^^^^^^^^^^^^^^^^^ |
| |
| If you need to use signal handlers, keep the code short and simple. Don't |
| forget that the signal handler is called asynchronously and can interrupt the |
| code execution at any place. |
| |
| This means that problems arise when global state is changed both from the test |
| code and signal handler, which will occasionally lead to: |
| |
| * Data corruption (data gets into inconsistent state), this may happen, for |
| example, for any operations on 'FILE' objects. |
| |
| * Deadlock, this happens, for example, if you call 'malloc(2)', 'free(2)', |
| etc. from both the test code and the signal handler at the same time because |
| 'malloc' has global lock for it's internal data structures. (Be wary that |
| 'malloc(2)' is used by the libc functions internally too.) |
| |
| * Any other unreproducible and unexpected behavior. |
| |
| Quite common mistake is to call 'exit(3)' from a signal handler. Note that this |
| function is not signal-async-safe as it flushes buffers, etc. If you need to |
| exit a test immediately from a signal handler use '_exit(2)' instead. |
| |
| TIP: See 'man 7 signal' for the list of signal-async-safe functions. |
| |
| If a signal handler sets a variable, its declaration must be 'volatile', |
| otherwise compiler may misoptimize the code, because the variable may not be |
| changed in the code flow analysis. There is 'sig_atomic_t' type defined in C99 |
| but this one DOES NOT imply volatile (it's just a typedef to int). So the |
| correct type for a flag that is changed from a signal handler is either |
| 'volatile int' or 'volatile sig_atomic_t'. |
| |
| 2.2.10 Kernel Modules |
| ^^^^^^^^^^^^^^^^^^^^^ |
| |
| There are certain cases where the test needs a kernel part and userspace part, |
| happily, LTP can build a kernel module and then insert it to the kernel on test |
| start for you. See 'testcases/kernel/device-drivers/block' for details. |
| |
| 2.2.11 Usefull macros |
| ^^^^^^^^^^^^^^^^^^^^^ |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| ARRAY_SIZE(arr) |
| ------------------------------------------------------------------------------- |
| |
| Returns the size of statically defined array, i.e. |
| '(sizeof(arr) / sizeof(*arr))' |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| LTP_ALIGN(x, a) |
| ------------------------------------------------------------------------------- |
| |
| Aligns the x to be next multiple of a. The a must be power of 2. |
| |
| 2.2.12 Filesystem type detection |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| Some tests are known to fail on certain filesytems (you cannot swap on TMPFS, |
| there are unimplemented fcntl() etc.). |
| |
| If your test needs to be skipped on certain filesystems, use the interface |
| below: |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "tst_fs_type.h" |
| |
| /* |
| * Unsupported only on NFS. |
| */ |
| if (tst_fs_type(cleanup, ".") == TST_NFS_MAGIC) { |
| tst_brkm(TCONF, cleanup, |
| "Test not supported on NFS filesystem"); |
| } |
| |
| /* |
| * Unsupported on NFS, TMPFS and RAMFS |
| */ |
| long type; |
| |
| switch ((type = tst_fs_type(cleanup, "."))) { |
| case TST_NFS_MAGIC: |
| case TST_TMPFS_MAGIC: |
| case TST_RAMFS_MAGIC: |
| tst_brkm(TCONF, cleanup, "Test not supported on %s filesytem", |
| tst_fs_type_name(type)); |
| break; |
| } |
| ------------------------------------------------------------------------------- |
| |
| 2.2.13 Thread-safety in the LTP library |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| It is safe to use library 'tst_' functions in multi-threaded tests. |
| Synchronization mechanism is enabled only if the test is linked with pthread |
| library, otherwise no-op stubs (defined in libc) are used. |
| |
| Also, LTP has a helper "TST_DECLARE_ONCE_FN" macro to declare a function which |
| must be run only once (e.g. init or cleanup function), but might be called by |
| multiple threads at the same time. See example below: |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "test.h" |
| |
| ... |
| |
| void do_cleanup(void) |
| { |
| /* cleanup code */ |
| ... |
| } |
| TST_DECLARE_ONCE_FN(cleanup, do_cleanup); |
| |
| ... |
| |
| void test01(void) |
| { |
| sfd = socket(AF_INET, SOCK_STREAM, 0); |
| if (sfd == -1) |
| tst_brkm(TBROK, cleanup, "Failed to create a socket"); |
| } |
| ------------------------------------------------------------------------------- |
| |
| 2.2.14 Acquiring a block device |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| Some tests needs a block device (inotify tests, syscall 'EROFS' failures, |
| etc.). For your convenience LTP library includes a function to acquire and |
| release a testing device. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "test.h" |
| |
| const char *tst_acquire_device(void (cleanup_fn)(void)); |
| ------------------------------------------------------------------------------- |
| |
| This function returns a path to small block device suitable for formatting and |
| mounting. |
| |
| WARNING: You can acquire only one device, calling it for second time without |
| releasing the device first will abort the test. |
| |
| In case that 'LTP_DEV' is passed to the test in environment, the function just |
| checks that the file exists and that it's a block device. |
| |
| Otherwise a temporary file is created and attached to a free loop device. |
| |
| In case that that there are no unused loop devices, 'NULL' is returned. The |
| test should skip the particular part of the test that needs the device with |
| 'TCONF'. |
| |
| WARNING: Because 'tst_acquire_device()' may create temporary file, you must |
| call 'tst_tmpdir()' before you call 'tst_acquire_device()' |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "test.h" |
| |
| void tst_release_device(void (cleanup_fn)(void), const char *dev); |
| ------------------------------------------------------------------------------- |
| |
| Releases the device acquired by 'tst_acquire_device()'. |
| |
| 2.2.15 Formatting a device with a filesystem |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "test.h" |
| |
| const char *tst_dev_fs_type(void); |
| ------------------------------------------------------------------------------- |
| |
| Returns name of filesystem that should be used for tests. Unless is your test |
| designed to test specific filesystem you should use this function to select |
| filesystem for the 'tst_mkfs()'. |
| |
| NOTE: The default filesytem is hardcoded to 'ext2' in the sources and can be |
| overridden by setting 'LTP_DEV_FS_TYPE' environment variable. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "test.h" |
| |
| void tst_mkfs(void (cleanup_fn)(void), const char *dev, |
| const char *fs_type, const char *const fs_opts[]); |
| ------------------------------------------------------------------------------- |
| |
| This function takes a path to a device, filesystem type and an array of extra |
| options passed to mkfs. |
| |
| The extra options 'fs_opts' should either be 'NULL' if there are none, or a |
| 'NULL' terminated array of strings such as '{"-b", "1024", NULL}'. |
| |
| 2.2.16 Verifying a filesystem's free space |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| Some tests have size requirements for the filesystem's free space. If these |
| requirements are not satisfied, the tests are not appropriate to run. |
| Especially when you run runltp with "-z" option to specify a big block device, |
| different tests have different requirements for the free space of this block |
| device. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "test.h" |
| |
| int tst_fs_has_free(void (*cleanup)(void), const char *path, |
| unsigned int size, unsigned int mult); |
| ------------------------------------------------------------------------------- |
| |
| The 'tst_fs_has_free()' function returns 1 if there is enough space and 0 if |
| there is not. |
| |
| The 'path' is the pathname of any directory/file within a filesystem you want to |
| verify its free space of. |
| |
| The 'mult' is a multiplier, one of 'TST_BYTES', 'TST_KB', 'TST_MB' or 'TST_GB'. |
| |
| The required free space is calculated by 'size * mult', e.g. |
| 'tst_fs_has_free(cleanup, "/tmp/testfile", 64, TST_MB)' will return 1 if the |
| filesystem, which '"/tmp/testfile"' is in, has 64MB free space at least, and 0 |
| if not. |
| |
| 2.2.17 Getting the maximum number of links to a regular file or directory |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| Some tests need to know the maximum count of links to a regular file or |
| directory, such as 'rename(2)' or 'linkat(2)' to test 'EMLINK' error. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "test.h" |
| |
| int tst_fs_fill_hardlinks(void (*cleanup)(void), const char *dir); |
| ------------------------------------------------------------------------------- |
| |
| Try to get maximum count of hard links to a regular file inside the 'dir'. |
| |
| NOTE: This number depends on the filesystem 'dir' is on. |
| |
| This function uses 'link(2)' to create hard links to a single file until it |
| gets 'EMLINK' or creates 65535 links. If the limit is hit, the maximum number of |
| hardlinks is returned and the 'dir' is filled with hardlinks in format |
| "testfile%i", where i belongs to [0, limit) interval. If no limit is hit or if |
| 'link(2)' failed with 'ENOSPC' or 'EDQUOT', zero is returned and previously |
| created files are removed. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "test.h" |
| |
| int tst_fs_fill_subdirs(void (*cleanup)(void), const char *dir); |
| ------------------------------------------------------------------------------- |
| |
| Try to get maximum number of subdirectories in directory. |
| |
| NOTE: This number depends on the filesystem 'dir' is on. |
| |
| This function uses 'mkdir(2)' to create directories in 'dir' until it gets |
| 'EMLINK' or creates 65535 directories. If the limit is hit, the maximum number |
| of subdirectories is returned and the 'dir' is filled with subdirectories in |
| format "testdir%i", where i belongs to [0, limit - 2) interval (because each |
| newly created dir has two links already - the '.' and the link from parent |
| dir). If no limit is hit or if 'mkdir(2)' failed with 'ENOSPC' or 'EDQUOT', |
| zero is returned and previously created directories are removed. |
| |
| 2.2.18 Getting an unused PID number |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| Some tests require a 'PID', which is not used by the OS (does not belong to |
| any process within it). For example, kill(2) should set errno to 'ESRCH' if |
| it's passed such 'PID'. |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "test.h" |
| |
| pid_t tst_get_unused_pid(void (*cleanup)(void)); |
| ------------------------------------------------------------------------------- |
| |
| Return a 'PID' value not used by the OS or any process within it. |
| |
| 2.2.19 Umounting devices |
| ^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| [source,c] |
| ------------------------------------------------------------------------------- |
| #include "test.h" |
| |
| int tst_umount(const char *path); |
| ------------------------------------------------------------------------------- |
| |
| Since various desktop deamons (gvfsd-trash is known for that) may be stupid |
| enough to probe all newly mounted filesystem which results in 'umount(2)' |
| failing with 'EBUSY' LTP library includes 'tst_umount()' function that behaves |
| exactly as 'umount(2)' but retries several times on a failure. |
| |
| IMPORTANT: All testcases should use 'tst_umount()' instead of 'umount(2)' to |
| umount filesystems. |
| |
| 2.3 Writing a testcase in shell |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| LTP supports testcases to be written in a portable shell too. |
| |
| There is a shell library modeled closely to the C interface (the source is |
| located at 'testcases/lib/test.sh') and is installed to the same directory as |
| the rest of the LTP test binaries. |
| |
| WARNING: All identifiers starting with TST_ or tst_ are reserved for the |
| 'test.sh' library. |
| |
| 2.3.1 Basic shell test structure |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| [source,sh] |
| ------------------------------------------------------------------------------- |
| #!/bin/sh |
| # |
| # This is a basic test for true shell buildin |
| # |
| |
| TCID=true01 |
| TST_TOTAL=1 |
| . test.sh |
| |
| true |
| ret=$? |
| |
| if [ $ret -eq 0 ]; then |
| tst_resm TPASS "true returned 0" |
| else |
| tst_resm TFAIL "true rturned $ret" |
| fi |
| |
| tst_exit |
| ------------------------------------------------------------------------------- |
| |
| TIP: To execute this test the 'test.sh' library must be in '$PATH'. If you are |
| executing the test from a git checkout you can run it as |
| 'PATH="$PATH:../../lib" ./foo01.sh' |
| |
| WARNING: Do not forget to add the 'tst_exit' at the end of the test, |
| otherwise the test return value would be the return value of last |
| executed command. |
| |
| 2.3.2 Basic test interface |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| Following functions similar to the LTP C intearface are available. |
| |
| * tst_resm() |
| * tst_brkm() |
| * tst_exit() |
| * tst_require_root() |
| * tst_tmpdir() |
| * tst_rmdir() |
| * tst_fs_has_free() |
| |
| There is one more function called 'tst_check_cmds()' that gets unspecified |
| number of parameters and asserts that each parameter is a name of an |
| executable in '$PATH' and exits the test with 'TCONF' on first missing. |
| |
| .tst_fs_has_free |
| [source,sh] |
| ------------------------------------------------------------------------------- |
| #!/bin/sh |
| |
| ... |
| |
| # whether current directory has 100MB free space at least. |
| if ! tst_fs_has_free . 100MB; then |
| tst_brkm TCONF "Not enough free space" |
| fi |
| |
| ... |
| ------------------------------------------------------------------------------- |
| |
| The tst_fs_has_free shell interface returns 0 if the specified free space is |
| satisfied, 1 if not, and 2 on error. |
| |
| The second argument supports suffixes kB, MB and GB, the default unit is Byte. |
| |
| 2.3.3 Cleanup |
| ^^^^^^^^^^^^^ |
| |
| Due to differences between C and shell, the cleanup callback is done using a |
| 'TST_CLEANUP' shell variable that, if not empty, is evaluated before the test |
| exits (either after calling 'tst_exit()' or 'tst_brkm()'). See example below. |
| |
| [source,sh] |
| ------------------------------------------------------------------------------- |
| #!/bin/sh |
| # |
| # Test cleanup example |
| # |
| |
| TCID=true01 |
| TST_TOTAL=1 |
| . test.sh |
| |
| cleanup() |
| { |
| tst_rmdir |
| } |
| |
| tst_tmpdir |
| TST_CLEANUP=cleanup |
| |
| # Do the test here |
| |
| tst_exit |
| ------------------------------------------------------------------------------- |
| |
| 2.3.4 Restarting daemons |
| ^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| Restarting system daemons is a complicated task for two reasons. |
| |
| * There are different init systems |
| (SysV init, systemd, etc...) |
| |
| * Daemon names are not unified between distributions |
| (apache vs httpd, cron vs crond, various syslog variations) |
| |
| To solve these problems LTP has 'testcases/lib/daemonlib.sh' library that |
| provides functions to start/stop/query daemons as well as variables that store |
| correct daemon name. |
| |
| .Supported operations |
| |============================================================================== |
| | start_daemon() | Starts daemon, name is passed as first parameter. |
| | stop_daemon() | Stops daemon, name is passed as first parameter. |
| | restart_daemon() | Restarts daemon, name is passed as first parameter. |
| | status_daemon() | Returns daemon status, TODO: what is return value? |
| |============================================================================== |
| |
| .Variables with detected names |
| |============================================================================== |
| | CROND_DAEMON | Cron daemon name (cron, crond). |
| | SYSLOG_DAEMON | Syslog daemon name (syslog, syslog-ng, rsyslog). |
| |============================================================================== |
| |
| .Cron daemon restart example |
| [source,sh] |
| ------------------------------------------------------------------------------- |
| #!/bin/sh |
| # |
| # Cron daemon restart example |
| # |
| TCID=cron01 |
| TST_COUNT=1 |
| . test.sh |
| . daemonlib.sh |
| |
| ... |
| |
| restart_daemon $CROND_DAEMON |
| |
| ... |
| |
| tst_exit |
| ------------------------------------------------------------------------------- |
| |
| |
| 3. Common problems |
| ------------------ |
| |
| This chapter describes common problems/misuses and less obvious design patters |
| (quirks) in UNIX interfaces. Read it carefully :) |
| |
| 3.1 umask() |
| ~~~~~~~~~~~ |
| |
| I've been hit by this one several times already... When you create files |
| with 'open()' or 'creat()' etc, the mode specified as the last parameter *is |
| not* the mode the file is created with. The mode depends on current 'umask()' |
| settings which may clear some of the bits. If your test depends on specific |
| file permissions you need either to change umask to 0 or 'chmod()' the file |
| afterwards or use SAFE_TOUCH() that does the 'chmod()' for you. |
| |
| 3.2 access() |
| ~~~~~~~~~~~ |
| |
| If 'access(some_file, W_OK)' is executed by root, it will return success even |
| if the file doesn't have write permission bits set (the same holds for R_OK |
| too). For sysfs files you can use 'open()' as a workaround to check file |
| read/write permissions. It might not work for other filesystems, for these you |
| have to use 'stat()', 'lstat()' or 'fstat()'. |
| |
| 3.3 umount() EBUSY |
| ~~~~~~~~~~~~~~~~~~ |
| |
| Various desktop deamons (gvfsd-trash is known for that) may be stupid enough |
| to probe all newly mounted filesystem which results in 'umount(2)' failing |
| with 'EBUSY'; use 'tst_umount()' described in 2.2.19 that retries in this case |
| instead of plain 'umount(2)'. |
| |
| |
| 4. Test Contribution Checklist |
| ------------------------------ |
| |
| 1. Test compiles and runs fine (check with -i 10 too) |
| 2. Checkpatch does not report any errors |
| 3. The runtest entires are in place |
| 4. Test files are added into corresponding .gitignore files |
| 5. Patches apply over the latest git |
| |
| |
| 4.1 About .gitignore files |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| There are numerous '.gitignore' files in the LTP tree. Usually there is a |
| '.gitignore' file per a group of tests. The reason for this setup is simple. |
| It's easier to maintain a '.gitignore' file per directory with tests, rather |
| than having single file in the project root directory. This way, we don't have |
| to update all the gitignore files when moving directories, and they get deleted |
| automatically when a directory with tests is removed. |