Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 1 | ============== |
| 2 | Testing libc++ |
| 3 | ============== |
| 4 | |
| 5 | .. contents:: |
| 6 | :local: |
| 7 | |
| 8 | Getting Started |
| 9 | =============== |
| 10 | |
| 11 | libc++ uses LIT to configure and run its tests. The primary way to run the |
| 12 | libc++ tests is by using make check-libcxx. However since libc++ can be used |
| 13 | in any number of possible configurations it is important to customize the way |
| 14 | LIT builds and runs the tests. This guide provides information on how to use |
| 15 | LIT directly to test libc++. |
| 16 | |
| 17 | Please see the `Lit Command Guide`_ for more information about LIT. |
| 18 | |
| 19 | .. _LIT Command Guide: http://llvm.org/docs/CommandGuide/lit.html |
| 20 | |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 21 | Setting up the Environment |
| 22 | -------------------------- |
| 23 | |
| 24 | After building libc++ you must setup your environment to test libc++ using |
| 25 | LIT. |
| 26 | |
| 27 | #. Create a shortcut to the actual lit executable so that you can invoke it |
| 28 | easily from the command line. |
| 29 | |
| 30 | .. code-block:: bash |
| 31 | |
| 32 | $ alias lit='python path/to/llvm/utils/lit/lit.py' |
| 33 | |
| 34 | #. Tell LIT where to find your build configuration. |
| 35 | |
| 36 | .. code-block:: bash |
| 37 | |
| 38 | $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg |
| 39 | |
Eric Fiselier | 2bc8f6c | 2015-10-14 20:44:44 +0000 | [diff] [blame] | 40 | Example Usage |
| 41 | ------------- |
| 42 | |
| 43 | Once you have your environment set up and you have built libc++ you can run |
| 44 | parts of the libc++ test suite by simply running `lit` on a specified test or |
| 45 | directory. For example: |
| 46 | |
| 47 | .. code-block:: bash |
| 48 | |
| 49 | $ cd path/to/src/libcxx |
| 50 | $ lit -sv test/std/re # Run all of the std::regex tests |
| 51 | $ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test |
| 52 | $ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic |
| 53 | |
| 54 | Sometimes you'll want to change the way LIT is running the tests. Custom options |
| 55 | can be specified using the `--param=<name>=<val>` flag. The most common option |
| 56 | you'll want to change is the standard dialect (ie -std=c++XX). By default the |
| 57 | test suite will select the newest C++ dialect supported by the compiler and use |
| 58 | that. However if you want to manually specify the option like so: |
| 59 | |
| 60 | .. code-block:: bash |
| 61 | |
| 62 | $ lit -sv test/std/containers # Run the tests with the newest -std |
| 63 | $ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03 |
| 64 | |
| 65 | Occasionally you'll want to add extra compile or link flags when testing. |
| 66 | You can do this as follows: |
| 67 | |
| 68 | .. code-block:: bash |
| 69 | |
| 70 | $ lit -sv --param=compile_flags='-Wcustom-warning' |
| 71 | $ lit -sv --param=link_flags='-L/custom/library/path' |
| 72 | |
| 73 | Some other common examples include: |
| 74 | |
| 75 | .. code-block:: bash |
| 76 | |
| 77 | # Specify a custom compiler. |
| 78 | $ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std |
| 79 | |
| 80 | # Enable warnings in the test suite |
| 81 | $ lit -sv --param=enable_warnings=true test/std |
| 82 | |
| 83 | # Use UBSAN when running the tests. |
| 84 | $ lit -sv --param=use_sanitizer=Undefined |
| 85 | |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 86 | |
| 87 | LIT Options |
| 88 | =========== |
| 89 | |
| 90 | :program:`lit` [*options*...] [*filenames*...] |
| 91 | |
| 92 | Command Line Options |
| 93 | -------------------- |
| 94 | |
| 95 | To use these options you pass them on the LIT command line as --param NAME or |
| 96 | --param NAME=VALUE. Some options have default values specified during CMake's |
| 97 | configuration. Passing the option on the command line will override the default. |
| 98 | |
| 99 | .. program:: lit |
| 100 | |
Eric Fiselier | 4289b6e | 2016-10-14 06:15:27 +0000 | [diff] [blame] | 101 | .. option:: cxx_under_test=<path/to/compiler> |
Eric Fiselier | 2bc8f6c | 2015-10-14 20:44:44 +0000 | [diff] [blame] | 102 | |
| 103 | Specify the compiler used to build the tests. |
| 104 | |
Eric Fiselier | 4289b6e | 2016-10-14 06:15:27 +0000 | [diff] [blame] | 105 | .. option:: cxx_stdlib_under_test=<stdlib name> |
| 106 | |
| 107 | **Values**: libc++, libstdc++ |
Eric Fiselier | 5300359 | 2016-10-12 00:00:37 +0000 | [diff] [blame] | 108 | |
| 109 | Specify the C++ standard library being tested. Unless otherwise specified |
| 110 | libc++ is used. This option is intended to allow running the libc++ test |
| 111 | suite against other standard library implementations. |
| 112 | |
Eric Fiselier | 2bc8f6c | 2015-10-14 20:44:44 +0000 | [diff] [blame] | 113 | .. option:: std=<standard version> |
| 114 | |
| 115 | **Values**: c++98, c++03, c++11, c++14, c++1z |
| 116 | |
| 117 | Change the standard version used when building the tests. |
| 118 | |
Eric Fiselier | 4289b6e | 2016-10-14 06:15:27 +0000 | [diff] [blame] | 119 | .. option:: libcxx_site_config=<path/to/lit.site.cfg> |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 120 | |
| 121 | Specify the site configuration to use when running the tests. This option |
Eric Fiselier | 01c2bb3 | 2017-05-09 23:47:20 +0000 | [diff] [blame] | 122 | overrides the environment variable LIBCXX_SITE_CONFIG. |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 123 | |
Eric Fiselier | 4289b6e | 2016-10-14 06:15:27 +0000 | [diff] [blame] | 124 | .. option:: cxx_headers=<path/to/headers> |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 125 | |
Eric Fiselier | 5300359 | 2016-10-12 00:00:37 +0000 | [diff] [blame] | 126 | Specify the c++ standard library headers that are tested. By default the |
| 127 | headers in the source tree are used. |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 128 | |
Eric Fiselier | 4289b6e | 2016-10-14 06:15:27 +0000 | [diff] [blame] | 129 | .. option:: cxx_library_root=<path/to/lib/> |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 130 | |
Eric Fiselier | cf4550d | 2016-04-20 04:17:39 +0000 | [diff] [blame] | 131 | Specify the directory of the libc++ library to be tested. By default the |
| 132 | library folder of the build directory is used. This option cannot be used |
Eric Fiselier | 904a9cb | 2016-12-23 19:09:14 +0000 | [diff] [blame] | 133 | when use_system_cxx_lib is provided. |
Eric Fiselier | cf4550d | 2016-04-20 04:17:39 +0000 | [diff] [blame] | 134 | |
| 135 | |
Eric Fiselier | 4289b6e | 2016-10-14 06:15:27 +0000 | [diff] [blame] | 136 | .. option:: cxx_runtime_root=<path/to/lib/> |
Eric Fiselier | cf4550d | 2016-04-20 04:17:39 +0000 | [diff] [blame] | 137 | |
| 138 | Specify the directory of the libc++ library to use at runtime. This directory |
| 139 | is not added to the linkers search path. This can be used to compile tests |
| 140 | against one version of libc++ and run them using another. The default value |
| 141 | for this option is `cxx_library_root`. This option cannot be used |
Eric Fiselier | 904a9cb | 2016-12-23 19:09:14 +0000 | [diff] [blame] | 142 | when use_system_cxx_lib is provided. |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 143 | |
Eric Fiselier | 904a9cb | 2016-12-23 19:09:14 +0000 | [diff] [blame] | 144 | .. option:: use_system_cxx_lib=<bool> |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 145 | |
| 146 | **Default**: False |
| 147 | |
| 148 | Enable or disable testing against the installed version of libc++ library. |
| 149 | Note: This does not use the installed headers. |
| 150 | |
Eric Fiselier | 4289b6e | 2016-10-14 06:15:27 +0000 | [diff] [blame] | 151 | .. option:: use_lit_shell=<bool> |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 152 | |
| 153 | Enable or disable the use of LIT's internal shell in ShTests. If the |
| 154 | environment variable LIT_USE_INTERNAL_SHELL is present then that is used as |
| 155 | the default value. Otherwise the default value is True on Windows and False |
| 156 | on every other platform. |
| 157 | |
Eric Fiselier | 4289b6e | 2016-10-14 06:15:27 +0000 | [diff] [blame] | 158 | .. option:: no_default_flags=<bool> |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 159 | |
| 160 | **Default**: False |
| 161 | |
| 162 | Disable all default compile and link flags from being added. When this |
| 163 | option is used only flags specified using the compile_flags and link_flags |
| 164 | will be used. |
| 165 | |
Eric Fiselier | 4289b6e | 2016-10-14 06:15:27 +0000 | [diff] [blame] | 166 | .. option:: compile_flags="<list-of-args>" |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 167 | |
| 168 | Specify additional compile flags as a space delimited string. |
| 169 | Note: This options should not be used to change the standard version used. |
| 170 | |
Eric Fiselier | 4289b6e | 2016-10-14 06:15:27 +0000 | [diff] [blame] | 171 | .. option:: link_flags="<list-of-args>" |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 172 | |
| 173 | Specify additional link flags as a space delimited string. |
| 174 | |
Eric Fiselier | 4289b6e | 2016-10-14 06:15:27 +0000 | [diff] [blame] | 175 | .. option:: debug_level=<level> |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 176 | |
| 177 | **Values**: 0, 1 |
| 178 | |
| 179 | Enable the use of debug mode. Level 0 enables assertions and level 1 enables |
| 180 | assertions and debugging of iterator misuse. |
| 181 | |
| 182 | .. option:: use_sanitizer=<sanitizer name> |
| 183 | |
| 184 | **Values**: Memory, MemoryWithOrigins, Address, Undefined |
| 185 | |
| 186 | Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when |
| 187 | building libc++ then that sanitizer will be used by default. |
| 188 | |
| 189 | .. option:: color_diagnostics |
| 190 | |
| 191 | Enable the use of colorized compile diagnostics. If the color_diagnostics |
| 192 | option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is |
| 193 | present then color diagnostics will be enabled. |
| 194 | |
| 195 | |
| 196 | Environment Variables |
| 197 | --------------------- |
| 198 | |
Eric Fiselier | 953f34f | 2016-05-05 08:12:25 +0000 | [diff] [blame] | 199 | .. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg> |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 200 | |
| 201 | Specify the site configuration to use when running the tests. |
Eric Fiselier | f05ac13 | 2016-09-16 03:47:53 +0000 | [diff] [blame] | 202 | Also see `libcxx_site_config`. |
Eric Fiselier | b9f425a | 2015-08-22 19:40:49 +0000 | [diff] [blame] | 203 | |
| 204 | .. envvar:: LIBCXX_COLOR_DIAGNOSTICS |
| 205 | |
| 206 | If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt |
| 207 | to use color diagnostic outputs from the compiler. |
Eric Fiselier | f05ac13 | 2016-09-16 03:47:53 +0000 | [diff] [blame] | 208 | Also see `color_diagnostics`. |
Eric Fiselier | d9b9ef7 | 2016-07-19 23:07:03 +0000 | [diff] [blame] | 209 | |
| 210 | Benchmarks |
| 211 | ========== |
| 212 | |
| 213 | Libc++ contains benchmark tests separately from the test of the test suite. |
| 214 | The benchmarks are written using the `Google Benchmark`_ library, a copy of which |
| 215 | is stored in the libc++ repository. |
| 216 | |
| 217 | For more information about using the Google Benchmark library see the |
| 218 | `official documentation <https://github.com/google/benchmark>`_. |
| 219 | |
| 220 | .. _`Google Benchmark`: https://github.com/google/benchmark |
| 221 | |
| 222 | Building Benchmarks |
| 223 | ------------------- |
| 224 | |
Eric Fiselier | 0a799bd | 2016-08-29 19:50:49 +0000 | [diff] [blame] | 225 | The benchmark tests are not built by default. The benchmarks can be built using |
| 226 | the ``cxx-benchmarks`` target. |
Eric Fiselier | d9b9ef7 | 2016-07-19 23:07:03 +0000 | [diff] [blame] | 227 | |
| 228 | An example build would look like: |
| 229 | |
| 230 | .. code-block:: bash |
| 231 | |
| 232 | $ cd build |
Eric Fiselier | 0a799bd | 2016-08-29 19:50:49 +0000 | [diff] [blame] | 233 | $ cmake [options] <path to libcxx sources> |
| 234 | $ make cxx-benchmarks |
Eric Fiselier | d9b9ef7 | 2016-07-19 23:07:03 +0000 | [diff] [blame] | 235 | |
| 236 | This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be |
| 237 | built against the just-built libc++. The compiled tests are output into |
| 238 | ``build/benchmarks``. |
| 239 | |
| 240 | The benchmarks can also be built against the platforms native standard library |
| 241 | using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This |
| 242 | is useful for comparing the performance of libc++ to other standard libraries. |
| 243 | The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and |
| 244 | ``<test>.native.out`` otherwise. |
| 245 | |
| 246 | Also See: |
| 247 | |
| 248 | * :ref:`Building Libc++ <build instructions>` |
| 249 | * :ref:`CMake Options` |
| 250 | |
| 251 | Running Benchmarks |
| 252 | ------------------ |
| 253 | |
| 254 | The benchmarks must be run manually by the user. Currently there is no way |
| 255 | to run them as part of the build. |
| 256 | |
| 257 | For example: |
| 258 | |
| 259 | .. code-block:: bash |
| 260 | |
| 261 | $ cd build/benchmarks |
Eric Fiselier | 0a799bd | 2016-08-29 19:50:49 +0000 | [diff] [blame] | 262 | $ make cxx-benchmarks |
Eric Fiselier | d9b9ef7 | 2016-07-19 23:07:03 +0000 | [diff] [blame] | 263 | $ ./algorithms.libcxx.out # Runs all the benchmarks |
| 264 | $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks |
| 265 | |
| 266 | For more information about running benchmarks see `Google Benchmark`_. |