doc/ltp-howto.lyx

#LyX 1.1 created this file. For more info see http://www.lyx.org/
\lyxformat 2.16
\textclass docbook
\begin_preamble
<!entity header system "header.sgml">
\end_preamble
\language default
\inputencoding default
\fontscheme default
\graphics default
\paperfontsize default
\spacing single 
\papersize Default
\paperpackage a4
\use_geometry 0
\use_amsmath 0
\paperorientation portrait
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\defskip medskip
\quotes_language english
\quotes_times 2
\papercolumns 1
\papersides 1
\paperpagestyle default

\layout Title
\added_space_top vfill \added_space_bottom vfill 
Linux Test Project HOWTO
\layout Date

4 October 2000 
\layout Author

Nate Straz
\layout Abstract

This document explains some of the more in depth topics of the Linux Test
 Project and related testing issues.
 It does not cover basic installation procedures.
 See the INSTALL and README files in the tarball for that information.
\layout Section

Preface
\layout Standard

This document was written to help bring the community up to speed on the
 ins and outs of the Linux Test Project.
\layout Subsection

Copyright 
\layout Standard

Copyright (c) 2000 by SGI, Inc.
 
\layout Standard

Please freely copy and distribute (sell or give away) this document in any
 format.
  It's requested that corrections and/or comments be fowarded to the document
 maintainer.
 You may create a derivative work and distribute it provided that you: 
\layout Itemize

Send your derivative work (in the most suitable format such as sgml) to
 the LDP (Linux Documentation Project) or the like for posting on the Internet.
  If not the LDP, then let the LDP know where it is available.
 
\layout Itemize

License the derivative work with this same license or use GPL.
 Include a copyright notice and at least a pointer to the license used.
 
\layout Itemize

Give due credit to previous authors and major contributors.
 
\layout Standard

If you're considering making a derived work other than a translation, it's
 requested that you discuss your plans with the current maintainer.
 
\layout Subsection

Disclaimer 
\layout Standard

Use the information in this document at your own risk.
 I disavow any potential liability for the contents of this document.
 Use of the concepts, examples, and/or other content of this document is
 entirely at your own risk.
 
\layout Standard

All copyrights are owned by their owners, unless specifically noted otherwise.
  Use of a term in this document should not be regarded as affecting the
 validity of any trademark or service mark.
 
\layout Standard

Naming of particular products or brands should not be seen as endorsements.
 
\layout Standard

You are strongly recommended to take a backup of your system before major
 installation and backups at regular intervals.
 
\layout Section

Introduction 
\layout Subsection

What is the Linux Test Project? 
\layout Standard

The Linux Test Project (LTP) is an effort to create a set of tools and tests
 to verify the functionality and stability of the Linux kernel.
 We hope this will support Linux development by making unit testing more
 complete and minimizing user impact by building a barrier to keep bugs
 from making it to the user.
 
\layout Subsection

What is wrong with the current testing model? 
\layout Standard

The Linux development community utilizes two important (some out argue most
 important) testing techniques in its normal operations: Design and Code
 Inspections.
 The intent of LTP is to support this by giving developers an ever growing
 set of tools to help identify any operational problems in their code that
 may be missed by human review.
 One of the toughest categories of problems to catch with inspection is
 that of interaction of features.
 With a continuously improving set of tests and tools, developers can get
 an indication of whether their changes may have broken some other functionality.
 
\layout Standard

There is no such thing as a perfect test base.
  It is only useful it if keeps up with new and changing functionality,
 and if it actually gets used.
 
\layout Subsection

Are you doing benchmarking? 
\layout Standard

Not at  this time.
 We are more interested in functional, regression, and stress testing the
 Linux kernel.
 Benchmarking may be workable to compare the performance among kernel versions.
 
\layout Subsection

Are you doing standards testing? 
\layout Standard

No, we are leaving that to the Linux Standards Base (LSB).
  See the Linux Standards Base 
\begin_inset LatexCommand \htmlurl[web site]{http://www.linuxbase.org/}

\end_inset 

 for more information.
 
\layout Section

Structure 
\layout Standard

The basic building block of the test project is a 
\series bold 
test case
\series default 
 that consists of a single action and a verification that the action worked.
  The result of the test case is usually restricted to PASS/FAIL.
 
\layout Standard

A 
\series bold 
test program
\series default 
 is a runnable program that contains one or more test cases.
 Test programs often understand command line options which alter their behavior.
 The options could determine the amount of memory tested, the location of
 temporary files, the type of network packet used, or any other useful parameter.
\layout Standard


\series bold 
Test tags
\series default 
 are used to pair a unique identifier with a test program and a set of command
 line options.
 Test tags are the basis for test suites.
\layout Section

Writing Tests 
\layout Standard

Writing a test case is a lot easier than most people think.
  Any code that you write to examine how a part of the kernel works can
 be adapted into a test case.
  All that is needed is a way to report the result of the action to the
 rest of the world.
  There are several ways of doing this, some more involved than others.
 
\layout Subsection

Exit Style Tests 
\layout Standard

Probably the simplest way of reporting the results of a test case is the
 exit status of your program.
  If your test program encounters unexpected or incorrect results, exit
 the test program with a non-zero exit status, i.e.
 
\family typewriter 
exit(1)
\family default 
.
 Conversely, if your program completes as expected, return a zero exit status,
 i.e.
 
\family typewriter 
exit(0)
\family default 
.
 Any test driver should be able to handle this type of error reporting.
 If a test program has multiple test cases you won't know which test case
 failed, but you will know the program that failed.
 
\layout Subsection

Formatted Output Tests 
\layout Standard

The next easiest way of reporting the results is to write the results of
 each test case to standard output.
 This allows for the testing results to be more understandable to both the
 tester and the analysis tools.
 When the results are written in a standard way, tools can be used to analyze
 the results.
 
\layout Section

Testing Tools
\layout Standard

The Linux Test Project has not yet decided on a "final" test harness.
  We have provided a simple solution with 
\family typewriter 
pan
\family default 
 to make due until a complete solution has been found/created that compliments
 the Linux kernel development process.
  Several people have said we should use such and such a test harness.
 Until we find we need a large complex test harness, we will apply the KISS
 concept.
   
\layout Subsection

Pan and Scanner 
\layout Standard


\family typewriter 
pan
\family default 
 is a simple test driver.
  It will take a list of test tags and command lines and run them.
  
\family typewriter 
pan
\family default 
 has the ability to run the test sequentially or randomly and in parallel
 while capturing test output and cleaning up orphanned processes.
  
\family typewriter 
pan
\family default 
 can also be nested to create very complex test environments.
   
\layout Standard


\family typewriter 
scanner
\family default 
 is a results analysis tool that understands the 
\emph on 
rts
\emph default 
 style output which 
\family typewriter 
pan
\family default 
 generates by default.
 It will produce a table summarizing which tests passed and which failed.
   
\layout Subsubsection

Example with Pan 
\layout Standard

This would be a Good Thing(tm) to have.
 Show off the nesting that can be done with Pan
\layout Subsection

The Quickhitter Package
\layout Standard

Many of the tests released use the Quickhitter test package to perform tasks
 like create and move to a temporary directory, handle some common command
 line parameters, loop, run in parallel, handle signals, and clean up.
 
\layout Section

To Do 
\layout Standard

There are a lot of things that still need to be doine to make this a complete
 kernel testing system.
 The following sections will discuss some of the to do items in detail.
  
\layout Subsection

Configuration Analysis 
\layout Standard

While the number of configuration options for the Linux kernel is seen as
 a strength to developers and users alike, it is a curse to testers.
  To create a powerful automated testing system, we need to be able to determine
 what the configuration on the booted box is and then determine which tests
 should be run on that box.
 
\layout Standard

The Linux kernel has hundreds of configuration options that can be set to
 compile the kernel.
  There are more options that can be set when you boot the kernel and while
 it is running.
  There are also many patches that can be applied to the kernel to add functiona
lity or change behavior.
   
\layout Subsection

Result Comparison 
\layout Standard

A lot of testing will be done in the life of the Linux Test Project.
 Keeping track of the results from all the testing will require some infrastruct
ure.
 It would be nice to take that output from a test machine, feed it to a
 program and receive a list of items that broke since the last run on that
 machine, or were fixed, or work on another test machine but not on this
 one.
   
\layout Section

Contact information and updates 
\layout Literal

URL: http://oss.sgi.com/projects/ltp/ 
\layout Literal

email: owners-ltp@oss.sgi.com 
\layout Literal

mailing list: ltp@oss.sgi.com 
\layout Literal

list archive: http://oss.sgi.com/projects/ltp/mail-threaded/ 
\layout Standard

Questions and comments should be sent to the LTP mailing list at ltp@oss.sgi.com.
  To subscribe, send mail to majordomo@oss.sgi.com with "subscribe ltp" in
 the body of the message.
 
\layout Standard

The source is also available via CVS.
  See the web site for a web interface and check out instructions.
 
\layout Section

Glossary 
\layout Description

Test IEEE/ANSI
\begin_float footnote 
\layout Standard

Kit, Edward, Software Testing in the Real World: Improving the Process.
 P.
 82.
 ACM Press, 1995.
\end_float 
: 
\shape italic 

\newline 

\shape default 
 
\shape italic 
(i)
\shape default 
 An activity in which a system or component is executed under specified
 conditions, the results are observed or record, and an evaluation is made
 of some aspect of the system or component.
 
\shape italic 

\newline 

\shape default 
 
\shape italic 
(ii)
\shape default 
 A set of one or more test cases.
 
\layout Description

Test\SpecialChar ~
Case A test assertion with a single result that is being verified.
 This allows designations such as PASS or FAIL to be applied to a single
 bit of functionality.
  A single test case may be one of many test cases for testing the complete
 functionality of a system.
 
\newline 
IEEE/ANSI: 
\shape italic 

\newline 
(i)
\shape default 
A set of test inputs, execution conditions, and expected results developed
 for a particular objective.
 
\shape italic 

\newline 
(ii)
\shape default 
 The smallest entity that is always executed as a unit, from beginning to
 end.
 
\layout Description

Test\SpecialChar ~
Driver A program that handles the execution of test programs.
 It is responsible for starting the test programs, capturing their output,
 and recording their results.
 Pan is an example of a test driver.
\layout Description

Test\SpecialChar ~
Framework A mechanism for organising a group of tests.
  Frameworks may have complex or very simple API's, drivers and result logging
 mechanisms.
 Examples of frameworks are TETware and DejaGnu.
 
\layout Description

Test\SpecialChar ~
Harness A Test harness is the mechanism that connects a test program
 to a test framework.
  It may be a specification of exit codes,  or a set of libraries for formatting
 messages and determining exit codes.
  In TETware, the tet_result() API is the test harness.
 
\layout Description

Test\SpecialChar ~
Program A single invokable program.
  A test program can contain one or more test cases.
 The test harness's API allows for reporting/analysis of the individual
 test cases.
 
\layout Description

Test\SpecialChar ~
Suite A collection of tests programs, assertions, cases grouped together
 under a framework.
   
\layout Description

Test\SpecialChar ~
Tag An identifier that corresponds to a command line which runs a test.
  The tag is a single word that matches a test program with a set of command
 line arguments.
 
\the_end