Revamp test-suite documentation
- Remove duplication: Both TestingGuide and TestSuiteMakefileGuide
would give a similar overview over the test-suite.
- Present cmake/lit as the default/normal way of running the test-suite:
- Move information about the cmake/lit testsuite into the new
TestSuiteGuide.rst file. Mark the remaining information in
TestSuiteMakefilesGuide.rst as deprecated.
- General simplification and shorting of language.
- Remove paragraphs about tests known to fail as everything should pass
nowadays.
- Remove paragraph about zlib requirement; it's not required anymore
since we copied a zlib source snapshot into the test-suite.
- Remove paragraph about comparison with "native compiler". Correctness is
always checked against reference outputs nowadays.
- Change cmake/lit quickstart section to recommend `pip` for installing
lit and use `CMAKE_C_COMPILER` and a cache file in the example as that
is what most people will end up doing anyway. Also a section about
compare.py to quickstart.
- Document `Bitcode` and `MicroBenchmarks` directories.
- Add section with commonly used cmake configuration options.
- Add section about showing and comparing result files via compare.py.
- Add section about using external benchmark suites.
- Add section about using custom benchmark suites.
- Add section about profile guided optimization.
- Add section about cross-compilation and running on external devices.
Differential Revision: https://reviews.llvm.org/D51465
llvm-svn: 341260
diff --git a/llvm/docs/TestingGuide.rst b/llvm/docs/TestingGuide.rst
index 3cc8cf4..e3db60d 100644
--- a/llvm/docs/TestingGuide.rst
+++ b/llvm/docs/TestingGuide.rst
@@ -8,6 +8,7 @@
.. toctree::
:hidden:
+ TestSuiteGuide
TestSuiteMakefileGuide
Overview
@@ -25,11 +26,7 @@
software required to build LLVM, as well as `Python <http://python.org>`_ 2.7 or
later.
-If you intend to run the :ref:`test-suite <test-suite-overview>`, you will also
-need a development version of zlib (zlib1g-dev is known to work on several Linux
-distributions).
-
-LLVM testing infrastructure organization
+LLVM Testing Infrastructure Organization
========================================
The LLVM testing infrastructure contains two major categories of tests:
@@ -77,6 +74,8 @@
The test-suite is located in the ``test-suite`` Subversion module.
+See the :doc:`TestSuiteGuide` for details.
+
Debugging Information tests
---------------------------
@@ -96,9 +95,8 @@
``llvm/test`` (so you get these tests for free with the main LLVM tree).
Use ``make check-all`` to run the regression tests after building LLVM.
-The more comprehensive test suite that includes whole programs in C and C++
-is in the ``test-suite`` module. See :ref:`test-suite Quickstart
-<test-suite-quickstart>` for more information on running these tests.
+The ``test-suite`` module contains more comprehensive tests including whole C
+and C++ programs. See the :doc:`TestSuiteGuide` for details.
Regression tests
----------------
@@ -585,65 +583,3 @@
(b) it speeds things up for really big test cases by avoiding
interpretation of the remainder of the file.
-
-.. _test-suite-overview:
-
-``test-suite`` Overview
-=======================
-
-The ``test-suite`` module contains a number of programs that can be
-compiled and executed. The ``test-suite`` includes reference outputs for
-all of the programs, so that the output of the executed program can be
-checked for correctness.
-
-``test-suite`` tests are divided into three types of tests: MultiSource,
-SingleSource, and External.
-
-- ``test-suite/SingleSource``
-
- The SingleSource directory contains test programs that are only a
- single source file in size. These are usually small benchmark
- programs or small programs that calculate a particular value. Several
- such programs are grouped together in each directory.
-
-- ``test-suite/MultiSource``
-
- The MultiSource directory contains subdirectories which contain
- entire programs with multiple source files. Large benchmarks and
- whole applications go here.
-
-- ``test-suite/External``
-
- The External directory contains Makefiles for building code that is
- external to (i.e., not distributed with) LLVM. The most prominent
- members of this directory are the SPEC 95 and SPEC 2000 benchmark
- suites. The ``External`` directory does not contain these actual
- tests, but only the Makefiles that know how to properly compile these
- programs from somewhere else. When using ``LNT``, use the
- ``--test-externals`` option to include these tests in the results.
-
-.. _test-suite-quickstart:
-
-``test-suite`` Quickstart
--------------------------
-
-The modern way of running the ``test-suite`` is focused on testing and
-benchmarking complete compilers using the
-`LNT <http://llvm.org/docs/lnt>`_ testing infrastructure.
-
-For more information on using LNT to execute the ``test-suite``, please
-see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_
-documentation.
-
-``test-suite`` Makefiles
-------------------------
-
-Historically, the ``test-suite`` was executed using a complicated setup
-of Makefiles. The LNT based approach above is recommended for most
-users, but there are some testing scenarios which are not supported by
-the LNT approach. In addition, LNT currently uses the Makefile setup
-under the covers and so developers who are interested in how LNT works
-under the hood may want to understand the Makefile based setup.
-
-For more information on the ``test-suite`` Makefile setup, please see
-the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`.