blob: a24f46a805f4f117fcc3a8fdc3311caae5915191 [file] [log] [blame]
nstraz7638a532000-10-09 20:35:46 +00001#LyX 1.1 created this file. For more info see http://www.lyx.org/
2\lyxformat 2.16
3\textclass docbook
4\begin_preamble
5<!entity header system "header.sgml">
6\end_preamble
7\language default
8\inputencoding default
9\fontscheme default
10\graphics default
11\paperfontsize default
12\spacing single
13\papersize Default
14\paperpackage a4
15\use_geometry 0
16\use_amsmath 0
17\paperorientation portrait
18\secnumdepth 3
19\tocdepth 3
20\paragraph_separation indent
21\defskip medskip
22\quotes_language english
23\quotes_times 2
24\papercolumns 1
25\papersides 1
26\paperpagestyle default
27
28\layout Title
29\added_space_top vfill \added_space_bottom vfill
30Linux Test Project HOWTO
31\layout Date
32
nstraz562c7372000-10-10 21:57:51 +00003310 October 2000
nstraz7638a532000-10-09 20:35:46 +000034\layout Author
35
36Nate Straz
37\layout Abstract
38
39This document explains some of the more in depth topics of the Linux Test
40 Project and related testing issues.
41 It does not cover basic installation procedures.
42 See the INSTALL and README files in the tarball for that information.
43\layout Section
44
45Preface
46\layout Standard
47
48This document was written to help bring the community up to speed on the
49 ins and outs of the Linux Test Project.
50\layout Subsection
51
52Copyright
53\layout Standard
54
55Copyright (c) 2000 by SGI, Inc.
56
57\layout Standard
58
59Please freely copy and distribute (sell or give away) this document in any
60 format.
61 It's requested that corrections and/or comments be fowarded to the document
62 maintainer.
63 You may create a derivative work and distribute it provided that you:
64\layout Itemize
65
66Send your derivative work (in the most suitable format such as sgml) to
67 the LDP (Linux Documentation Project) or the like for posting on the Internet.
68 If not the LDP, then let the LDP know where it is available.
69
70\layout Itemize
71
72License the derivative work with this same license or use GPL.
73 Include a copyright notice and at least a pointer to the license used.
74
75\layout Itemize
76
77Give due credit to previous authors and major contributors.
78
79\layout Standard
80
81If you're considering making a derived work other than a translation, it's
82 requested that you discuss your plans with the current maintainer.
83
84\layout Subsection
85
86Disclaimer
87\layout Standard
88
89Use the information in this document at your own risk.
90 I disavow any potential liability for the contents of this document.
91 Use of the concepts, examples, and/or other content of this document is
92 entirely at your own risk.
93
94\layout Standard
95
96All copyrights are owned by their owners, unless specifically noted otherwise.
97 Use of a term in this document should not be regarded as affecting the
98 validity of any trademark or service mark.
99
100\layout Standard
101
102Naming of particular products or brands should not be seen as endorsements.
103
104\layout Standard
105
106You are strongly recommended to take a backup of your system before major
107 installation and backups at regular intervals.
108
109\layout Section
110
111Introduction
112\layout Subsection
113
114What is the Linux Test Project?
115\layout Standard
116
117The Linux Test Project (LTP) is an effort to create a set of tools and tests
118 to verify the functionality and stability of the Linux kernel.
119 We hope this will support Linux development by making unit testing more
120 complete and minimizing user impact by building a barrier to keep bugs
121 from making it to the user.
122
123\layout Subsection
124
125What is wrong with the current testing model?
126\layout Standard
127
128The Linux development community utilizes two important (some out argue most
129 important) testing techniques in its normal operations: Design and Code
130 Inspections.
131 The intent of LTP is to support this by giving developers an ever growing
132 set of tools to help identify any operational problems in their code that
133 may be missed by human review.
134 One of the toughest categories of problems to catch with inspection is
135 that of interaction of features.
136 With a continuously improving set of tests and tools, developers can get
137 an indication of whether their changes may have broken some other functionality.
138
139\layout Standard
140
141There is no such thing as a perfect test base.
142 It is only useful it if keeps up with new and changing functionality,
143 and if it actually gets used.
144
145\layout Subsection
146
147Are you doing benchmarking?
148\layout Standard
149
150Not at this time.
151 We are more interested in functional, regression, and stress testing the
152 Linux kernel.
153 Benchmarking may be workable to compare the performance among kernel versions.
154
155\layout Subsection
156
157Are you doing standards testing?
158\layout Standard
159
160No, we are leaving that to the Linux Standards Base (LSB).
161 See the Linux Standards Base
162\begin_inset LatexCommand \htmlurl[web site]{http://www.linuxbase.org/}
163
164\end_inset
165
166 for more information.
167
168\layout Section
169
170Structure
171\layout Standard
172
173The basic building block of the test project is a
174\series bold
175test case
176\series default
177 that consists of a single action and a verification that the action worked.
178 The result of the test case is usually restricted to PASS/FAIL.
179
180\layout Standard
181
182A
183\series bold
184test program
185\series default
186 is a runnable program that contains one or more test cases.
187 Test programs often understand command line options which alter their behavior.
188 The options could determine the amount of memory tested, the location of
189 temporary files, the type of network packet used, or any other useful parameter.
190\layout Standard
191
192
193\series bold
194Test tags
195\series default
196 are used to pair a unique identifier with a test program and a set of command
197 line options.
198 Test tags are the basis for test suites.
199\layout Section
200
201Writing Tests
202\layout Standard
203
204Writing a test case is a lot easier than most people think.
205 Any code that you write to examine how a part of the kernel works can
206 be adapted into a test case.
207 All that is needed is a way to report the result of the action to the
208 rest of the world.
209 There are several ways of doing this, some more involved than others.
210
211\layout Subsection
212
213Exit Style Tests
214\layout Standard
215
216Probably the simplest way of reporting the results of a test case is the
217 exit status of your program.
218 If your test program encounters unexpected or incorrect results, exit
219 the test program with a non-zero exit status, i.e.
220
221\family typewriter
222exit(1)
223\family default
224.
225 Conversely, if your program completes as expected, return a zero exit status,
226 i.e.
227
228\family typewriter
229exit(0)
230\family default
231.
232 Any test driver should be able to handle this type of error reporting.
233 If a test program has multiple test cases you won't know which test case
234 failed, but you will know the program that failed.
235
236\layout Subsection
237
238Formatted Output Tests
239\layout Standard
240
241The next easiest way of reporting the results is to write the results of
242 each test case to standard output.
243 This allows for the testing results to be more understandable to both the
244 tester and the analysis tools.
245 When the results are written in a standard way, tools can be used to analyze
246 the results.
247
248\layout Section
249
250Testing Tools
251\layout Standard
252
253The Linux Test Project has not yet decided on a "final" test harness.
254 We have provided a simple solution with
255\family typewriter
256pan
257\family default
258 to make due until a complete solution has been found/created that compliments
259 the Linux kernel development process.
260 Several people have said we should use such and such a test harness.
261 Until we find we need a large complex test harness, we will apply the KISS
262 concept.
263
264\layout Subsection
265
nstraz562c7372000-10-10 21:57:51 +0000266Pan
nstraz7638a532000-10-09 20:35:46 +0000267\layout Standard
268
269
270\family typewriter
271pan
272\family default
nstraz03d32f42000-10-11 22:46:16 +0000273 is a simple test driver with the ability to keep track of orphaned processes
274 and capture test output.
275 It works by reading a list of test tags and command lines and runs them.
276 By default pan will select a command randomly from the list of test tags,
277 wait for it to finish.
278 Through command line options you can run through the entire list sequentially,
279 run n tests, keep n test running at all times, and buffer test output.
280 Pan can be nested to create very complex test environments.
nstraz7638a532000-10-09 20:35:46 +0000281\layout Standard
282
nstraz03d32f42000-10-11 22:46:16 +0000283Pan uses an
284\emph on
285active file
286\emph default
287, also called a
288\emph on
289zoo file
290\emph default
291 to keep track of which tests are currently running.
292 This file holds the pid, tag, and a portion of the command line.
293 When you start pan it becomes a test tag in itself, thus it requires a
294 name for itself.
295 Pan updates the active file to show which test tags are currently running.
296 When a test tag exits, pan will overwrite the first character with a '#'.
297 The active file can be shared between multiple instances of pan so you
298 know which tests were running when the system crashes by looking at one
299 file.
300
301\layout Standard
302
303A
304\emph on
305pan file
306\emph default
307 contains a list of test tags for pan to run.
nstraz562c7372000-10-10 21:57:51 +0000308 The format of a pan file is as follows:
309\layout Code
310
311testtag testprogram -o one -p two other command line options
312\layout Code
313
314# This is a comment.
315 It is a good idea to describe the test
316\layout Code
317
318# tags in your pan file.
319 Tests programs can have different
320\layout Code
321
322# behaviors depending on the command line options so it is
323\layout Code
324
nstraz03d32f42000-10-11 22:46:16 +0000325# helpful to describe what each test tag is meant to verify or # provoke.
nstraz562c7372000-10-10 21:57:51 +0000326\layout Code
327
328# Some more test cases
329\layout Code
330
331mm01 mmap001 -m 10000
332\layout Code
333
334# 40 Mb mmap() test.
335\layout Code
336
337# Creates a 10000 page mmap, touches all of the map, sync's
338\layout Code
339
340# it, and munmap()s it.
341\layout Code
342
343mm03 mmap001 -i 0 -I 1 -m 100
344\layout Code
345
346# repetitive mmapping test.
347\layout Code
348
349# Creates a one page map repetitively for one minute.
350\layout Code
351
352dup02 dup02
353\layout Code
354
355# Negative test for dup(2) with bad fd
356\layout Code
357
358kill09 kill09
359\layout Code
360
361# Basic test for kill(2)
362\layout Code
363
364fs-suite01 pan -e -a fs-suite01.zoo -n fs-suite01 -f runtest/fs
365\layout Code
366
367# run the entire set of file system tests
368\layout Standard
369
nstraz03d32f42000-10-11 22:46:16 +0000370The test tags are simple identifiers, no spaces are allowed.
371 The test of the line is the program to run, which is done using execvp(3).
372 Lines starting with '#' are comments and ignored by pan.
373 It is a good practice to include descriptions with your test tags so you
374 can have a reminder what a certain obscure test tag tries to do.
375\layout Subsubsection
376
377Examples
378\layout Standard
379
380Test true
381\layout Standard
382
nstraz562c7372000-10-10 21:57:51 +0000383For more information on pan see the man page
384\family typewriter
385doc/man1/pan.1
386\family default
387.
388\layout Subsection
389
390Scanner
391\layout Standard
392
nstraz7638a532000-10-09 20:35:46 +0000393
394\family typewriter
395scanner
396\family default
397 is a results analysis tool that understands the
398\emph on
399rts
400\emph default
401 style output which
402\family typewriter
403pan
404\family default
405 generates by default.
406 It will produce a table summarizing which tests passed and which failed.
407
nstraz7638a532000-10-09 20:35:46 +0000408\layout Subsection
409
nstraz562c7372000-10-10 21:57:51 +0000410The Quick-hitter Package
nstraz7638a532000-10-09 20:35:46 +0000411\layout Standard
412
nstraz562c7372000-10-10 21:57:51 +0000413Many of the tests released use the Quick-hitter test package to perform
414 tasks like create and move to a temporary directory, handle some common
415 command line parameters, loop, run in parallel, handle signals, and clean
416 up.
nstraz7638a532000-10-09 20:35:46 +0000417
nstraz562c7372000-10-10 21:57:51 +0000418\layout Standard
419
420There is an example test case,
421\family typewriter
422doc/examples/quickhit.c
423\family default
424, which shows how the quick-hitter package can be used.
425 The file is meant to be a supplement to the documentation, not a working
426 test case.
427 Use any of the tests in
428\family typewriter
429tests/
430\family default
431 as a template.
nstraz7638a532000-10-09 20:35:46 +0000432\layout Section
433
434To Do
435\layout Standard
436
nstraz562c7372000-10-10 21:57:51 +0000437There are a lot of things that still need to be done to make this a complete
nstraz7638a532000-10-09 20:35:46 +0000438 kernel testing system.
439 The following sections will discuss some of the to do items in detail.
440
441\layout Subsection
442
443Configuration Analysis
444\layout Standard
445
446While the number of configuration options for the Linux kernel is seen as
447 a strength to developers and users alike, it is a curse to testers.
448 To create a powerful automated testing system, we need to be able to determine
449 what the configuration on the booted box is and then determine which tests
450 should be run on that box.
451
452\layout Standard
453
454The Linux kernel has hundreds of configuration options that can be set to
455 compile the kernel.
456 There are more options that can be set when you boot the kernel and while
457 it is running.
458 There are also many patches that can be applied to the kernel to add functiona
459lity or change behavior.
460
461\layout Subsection
462
463Result Comparison
464\layout Standard
465
466A lot of testing will be done in the life of the Linux Test Project.
467 Keeping track of the results from all the testing will require some infrastruct
468ure.
469 It would be nice to take that output from a test machine, feed it to a
470 program and receive a list of items that broke since the last run on that
471 machine, or were fixed, or work on another test machine but not on this
472 one.
473
474\layout Section
475
476Contact information and updates
477\layout Literal
478
479URL: http://oss.sgi.com/projects/ltp/
480\layout Literal
481
482email: owners-ltp@oss.sgi.com
483\layout Literal
484
485mailing list: ltp@oss.sgi.com
486\layout Literal
487
488list archive: http://oss.sgi.com/projects/ltp/mail-threaded/
489\layout Standard
490
491Questions and comments should be sent to the LTP mailing list at ltp@oss.sgi.com.
492 To subscribe, send mail to majordomo@oss.sgi.com with "subscribe ltp" in
493 the body of the message.
494
495\layout Standard
496
497The source is also available via CVS.
498 See the web site for a web interface and check out instructions.
499
500\layout Section
501
502Glossary
503\layout Description
504
505Test IEEE/ANSI
506\begin_float footnote
507\layout Standard
508
509Kit, Edward, Software Testing in the Real World: Improving the Process.
510 P.
511 82.
512 ACM Press, 1995.
513\end_float
514:
515\shape italic
516
517\newline
518
519\shape default
520
521\shape italic
522(i)
523\shape default
524 An activity in which a system or component is executed under specified
525 conditions, the results are observed or record, and an evaluation is made
526 of some aspect of the system or component.
527
528\shape italic
529
530\newline
531
532\shape default
533
534\shape italic
535(ii)
536\shape default
537 A set of one or more test cases.
538
539\layout Description
540
541Test\SpecialChar ~
542Case A test assertion with a single result that is being verified.
543 This allows designations such as PASS or FAIL to be applied to a single
544 bit of functionality.
545 A single test case may be one of many test cases for testing the complete
546 functionality of a system.
547
548\newline
549IEEE/ANSI:
550\shape italic
551
552\newline
553(i)
554\shape default
555A set of test inputs, execution conditions, and expected results developed
556 for a particular objective.
557
558\shape italic
559
560\newline
561(ii)
562\shape default
563 The smallest entity that is always executed as a unit, from beginning to
564 end.
565
566\layout Description
567
568Test\SpecialChar ~
569Driver A program that handles the execution of test programs.
570 It is responsible for starting the test programs, capturing their output,
571 and recording their results.
572 Pan is an example of a test driver.
573\layout Description
574
575Test\SpecialChar ~
nstraz562c7372000-10-10 21:57:51 +0000576Framework A mechanism for organizing a group of tests.
nstraz7638a532000-10-09 20:35:46 +0000577 Frameworks may have complex or very simple API's, drivers and result logging
578 mechanisms.
579 Examples of frameworks are TETware and DejaGnu.
580
581\layout Description
582
583Test\SpecialChar ~
584Harness A Test harness is the mechanism that connects a test program
585 to a test framework.
586 It may be a specification of exit codes, or a set of libraries for formatting
587 messages and determining exit codes.
588 In TETware, the tet_result() API is the test harness.
589
590\layout Description
591
592Test\SpecialChar ~
593Program A single invokable program.
594 A test program can contain one or more test cases.
595 The test harness's API allows for reporting/analysis of the individual
596 test cases.
597
598\layout Description
599
600Test\SpecialChar ~
601Suite A collection of tests programs, assertions, cases grouped together
602 under a framework.
603
604\layout Description
605
606Test\SpecialChar ~
607Tag An identifier that corresponds to a command line which runs a test.
608 The tag is a single word that matches a test program with a set of command
609 line arguments.
610
611\the_end