blob: cc2df9b72421486d96d47e2d72220a0a5848c55a [file] [log] [blame]
Jens Axboe71bfa162006-10-25 11:08:19 +02001Table of contents
2-----------------
3
41. Overview
52. How fio works
63. Running fio
74. Job file format
85. Detailed list of parameters
96. Normal output
107. Terse output
Paul Dubs25c8b9d2011-07-21 17:26:02 +0200118. Trace file format
Jens Axboe71bfa162006-10-25 11:08:19 +020012
131.0 Overview and history
14------------------------
15fio was originally written to save me the hassle of writing special test
16case programs when I wanted to test a specific workload, either for
17performance reasons or to find/reproduce a bug. The process of writing
18such a test app can be tiresome, especially if you have to do it often.
19Hence I needed a tool that would be able to simulate a given io workload
20without resorting to writing a tailored test case again and again.
21
22A test work load is difficult to define, though. There can be any number
23of processes or threads involved, and they can each be using their own
24way of generating io. You could have someone dirtying large amounts of
25memory in an memory mapped file, or maybe several threads issuing
26reads using asynchronous io. fio needed to be flexible enough to
27simulate both of these cases, and many more.
28
292.0 How fio works
30-----------------
31The first step in getting fio to simulate a desired io workload, is
32writing a job file describing that specific setup. A job file may contain
33any number of threads and/or files - the typical contents of the job file
34is a global section defining shared parameters, and one or more job
35sections describing the jobs involved. When run, fio parses this file
36and sets everything up as described. If we break down a job from top to
37bottom, it contains the following basic parameters:
38
39 IO type Defines the io pattern issued to the file(s).
40 We may only be reading sequentially from this
41 file(s), or we may be writing randomly. Or even
42 mixing reads and writes, sequentially or randomly.
43
44 Block size In how large chunks are we issuing io? This may be
45 a single value, or it may describe a range of
46 block sizes.
47
48 IO size How much data are we going to be reading/writing.
49
50 IO engine How do we issue io? We could be memory mapping the
51 file, we could be using regular read/write, we
Jens Axboed0ff85d2007-02-14 01:19:41 +010052 could be using splice, async io, syslet, or even
Jens Axboe71bfa162006-10-25 11:08:19 +020053 SG (SCSI generic sg).
54
Jens Axboe6c219762006-11-03 15:51:45 +010055 IO depth If the io engine is async, how large a queuing
Jens Axboe71bfa162006-10-25 11:08:19 +020056 depth do we want to maintain?
57
58 IO type Should we be doing buffered io, or direct/raw io?
59
60 Num files How many files are we spreading the workload over.
61
62 Num threads How many threads or processes should we spread
63 this workload over.
64
65The above are the basic parameters defined for a workload, in addition
66there's a multitude of parameters that modify other aspects of how this
67job behaves.
68
69
703.0 Running fio
71---------------
72See the README file for command line parameters, there are only a few
73of them.
74
75Running fio is normally the easiest part - you just give it the job file
76(or job files) as parameters:
77
78$ fio job_file
79
80and it will start doing what the job_file tells it to do. You can give
81more than one job file on the command line, fio will serialize the running
82of those files. Internally that is the same as using the 'stonewall'
83parameter described the the parameter section.
84
Jens Axboeb4692822006-10-27 13:43:22 +020085If the job file contains only one job, you may as well just give the
86parameters on the command line. The command line parameters are identical
87to the job parameters, with a few extra that control global parameters
88(see README). For example, for the job file parameter iodepth=2, the
Jens Axboec2b1e752006-10-30 09:03:13 +010089mirror command line option would be --iodepth 2 or --iodepth=2. You can
90also use the command line for giving more than one job entry. For each
91--name option that fio sees, it will start a new job with that name.
92Command line entries following a --name entry will apply to that job,
93until there are no more entries or a new --name entry is seen. This is
94similar to the job file options, where each option applies to the current
95job until a new [] job entry is seen.
Jens Axboeb4692822006-10-27 13:43:22 +020096
Jens Axboe71bfa162006-10-25 11:08:19 +020097fio does not need to run as root, except if the files or devices specified
98in the job section requires that. Some other options may also be restricted,
Jens Axboe6c219762006-11-03 15:51:45 +010099such as memory locking, io scheduler switching, and decreasing the nice value.
Jens Axboe71bfa162006-10-25 11:08:19 +0200100
101
1024.0 Job file format
103-------------------
104As previously described, fio accepts one or more job files describing
105what it is supposed to do. The job file format is the classic ini file,
106where the names enclosed in [] brackets define the job name. You are free
107to use any ascii name you want, except 'global' which has special meaning.
108A global section sets defaults for the jobs described in that file. A job
109may override a global section parameter, and a job file may even have
110several global sections if so desired. A job is only affected by a global
Jens Axboe65db0852007-02-20 10:22:01 +0100111section residing above it. If the first character in a line is a ';' or a
112'#', the entire line is discarded as a comment.
Jens Axboe71bfa162006-10-25 11:08:19 +0200113
Aaron Carroll3c54bc42008-10-07 11:25:38 +0200114So let's look at a really simple job file that defines two processes, each
Jens Axboeb22989b2009-07-17 22:29:23 +0200115randomly reading from a 128MB file.
Jens Axboe71bfa162006-10-25 11:08:19 +0200116
117; -- start job file --
118[global]
119rw=randread
120size=128m
121
122[job1]
123
124[job2]
125
126; -- end job file --
127
128As you can see, the job file sections themselves are empty as all the
129described parameters are shared. As no filename= option is given, fio
Jens Axboec2b1e752006-10-30 09:03:13 +0100130makes up a filename for each of the jobs as it sees fit. On the command
131line, this job would look as follows:
132
133$ fio --name=global --rw=randread --size=128m --name=job1 --name=job2
134
Jens Axboe71bfa162006-10-25 11:08:19 +0200135
Aaron Carroll3c54bc42008-10-07 11:25:38 +0200136Let's look at an example that has a number of processes writing randomly
Jens Axboe71bfa162006-10-25 11:08:19 +0200137to files.
138
139; -- start job file --
140[random-writers]
141ioengine=libaio
142iodepth=4
143rw=randwrite
144bs=32k
145direct=0
146size=64m
147numjobs=4
148
149; -- end job file --
150
151Here we have no global section, as we only have one job defined anyway.
152We want to use async io here, with a depth of 4 for each file. We also
Jens Axboeb22989b2009-07-17 22:29:23 +0200153increased the buffer size used to 32KB and define numjobs to 4 to
Jens Axboe71bfa162006-10-25 11:08:19 +0200154fork 4 identical jobs. The result is 4 processes each randomly writing
Jens Axboeb22989b2009-07-17 22:29:23 +0200155to their own 64MB file. Instead of using the above job file, you could
Jens Axboeb4692822006-10-27 13:43:22 +0200156have given the parameters on the command line. For this case, you would
157specify:
158
159$ fio --name=random-writers --ioengine=libaio --iodepth=4 --rw=randwrite --bs=32k --direct=0 --size=64m --numjobs=4
Jens Axboe71bfa162006-10-25 11:08:19 +0200160
Jens Axboe74929ac2009-08-05 11:42:37 +02001614.1 Environment variables
162-------------------------
163
Aaron Carroll3c54bc42008-10-07 11:25:38 +0200164fio also supports environment variable expansion in job files. Any
165substring of the form "${VARNAME}" as part of an option value (in other
166words, on the right of the `='), will be expanded to the value of the
167environment variable called VARNAME. If no such environment variable
168is defined, or VARNAME is the empty string, the empty string will be
169substituted.
170
171As an example, let's look at a sample fio invocation and job file:
172
173$ SIZE=64m NUMJOBS=4 fio jobfile.fio
174
175; -- start job file --
176[random-writers]
177rw=randwrite
178size=${SIZE}
179numjobs=${NUMJOBS}
180; -- end job file --
181
182This will expand to the following equivalent job file at runtime:
183
184; -- start job file --
185[random-writers]
186rw=randwrite
187size=64m
188numjobs=4
189; -- end job file --
190
Jens Axboe71bfa162006-10-25 11:08:19 +0200191fio ships with a few example job files, you can also look there for
192inspiration.
193
Jens Axboe74929ac2009-08-05 11:42:37 +02001944.2 Reserved keywords
195---------------------
196
197Additionally, fio has a set of reserved keywords that will be replaced
198internally with the appropriate value. Those keywords are:
199
200$pagesize The architecture page size of the running system
201$mb_memory Megabytes of total memory in the system
202$ncpus Number of online available CPUs
203
204These can be used on the command line or in the job file, and will be
205automatically substituted with the current system values when the job
Jens Axboe892a6ff2009-11-13 12:19:49 +0100206is run. Simple math is also supported on these keywords, so you can
207perform actions like:
208
209size=8*$mb_memory
210
211and get that properly expanded to 8 times the size of memory in the
212machine.
Jens Axboe74929ac2009-08-05 11:42:37 +0200213
Jens Axboe71bfa162006-10-25 11:08:19 +0200214
2155.0 Detailed list of parameters
216-------------------------------
217
218This section describes in details each parameter associated with a job.
219Some parameters take an option of a given type, such as an integer or
220a string. The following types are used:
221
222str String. This is a sequence of alpha characters.
Jens Axboeb09da8f2009-07-17 23:16:17 +0200223time Integer with possible time suffix. In seconds unless otherwise
Jens Axboee417fd62008-09-11 09:27:15 +0200224 specified, use eg 10m for 10 minutes. Accepts s/m/h for seconds,
225 minutes, and hours.
Jens Axboeb09da8f2009-07-17 23:16:17 +0200226int SI integer. A whole number value, which may contain a suffix
227 describing the base of the number. Accepted suffixes are k/m/g/t/p,
228 meaning kilo, mega, giga, tera, and peta. The suffix is not case
Jens Axboe57fc29f2010-06-23 22:24:07 +0200229 sensitive, and you may also include trailing 'b' (eg 'kb' is the same
230 as 'k'). So if you want to specify 4096, you could either write
Jens Axboeb09da8f2009-07-17 23:16:17 +0200231 out '4096' or just give 4k. The suffixes signify base 2 values, so
Jens Axboe57fc29f2010-06-23 22:24:07 +0200232 1024 is 1k and 1024k is 1m and so on, unless the suffix is explicitly
233 set to a base 10 value using 'kib', 'mib', 'gib', etc. If that is the
234 case, then 1000 is used as the multiplier. This can be handy for
235 disks, since manufacturers generally use base 10 values when listing
236 the capacity of a drive. If the option accepts an upper and lower
237 range, use a colon ':' or minus '-' to separate such values. May also
238 include a prefix to indicate numbers base. If 0x is used, the number
239 is assumed to be hexadecimal. See irange.
Jens Axboe71bfa162006-10-25 11:08:19 +0200240bool Boolean. Usually parsed as an integer, however only defined for
241 true and false (1 and 0).
Jens Axboeb09da8f2009-07-17 23:16:17 +0200242irange Integer range with suffix. Allows value range to be given, such
Jens Axboebf9a3ed2008-06-05 11:53:08 +0200243 as 1024-4096. A colon may also be used as the separator, eg
Jens Axboe0c9baf92007-01-11 15:59:26 +0100244 1k:4k. If the option allows two sets of ranges, they can be
245 specified with a ',' or '/' delimiter: 1k-4k/8k-32k. Also see
Jens Axboef7fa2652009-03-09 14:20:20 +0100246 int.
Yu-ju Hong83349192011-08-13 00:53:44 +0200247float_list A list of floating numbers, separated by a ':' character.
Jens Axboe71bfa162006-10-25 11:08:19 +0200248
249With the above in mind, here follows the complete list of fio job
250parameters.
251
252name=str ASCII name of the job. This may be used to override the
253 name printed by fio for this job. Otherwise the job
Jens Axboec2b1e752006-10-30 09:03:13 +0100254 name is used. On the command line this parameter has the
Jens Axboe6c219762006-11-03 15:51:45 +0100255 special purpose of also signaling the start of a new
Jens Axboec2b1e752006-10-30 09:03:13 +0100256 job.
Jens Axboe71bfa162006-10-25 11:08:19 +0200257
Jens Axboe61697c32007-02-05 15:04:46 +0100258description=str Text description of the job. Doesn't do anything except
259 dump this text description when this job is run. It's
260 not parsed.
261
Randy Dunlap37760412009-05-13 07:51:05 +0200262directory=str Prefix filenames with this directory. Used to place files
Jens Axboe71bfa162006-10-25 11:08:19 +0200263 in a different location than "./".
264
265filename=str Fio normally makes up a filename based on the job name,
266 thread number, and file number. If you want to share
267 files between threads in a job or several jobs, specify
Jens Axboeed92ac02007-02-06 14:43:52 +0100268 a filename for each of them to override the default. If
Jens Axboe414c2a32009-01-16 13:21:15 +0100269 the ioengine used is 'net', the filename is the host, port,
270 and protocol to use in the format of =host/port/protocol.
271 See ioengine=net for more. If the ioengine is file based, you
272 can specify a number of files by separating the names with a
273 ':' colon. So if you wanted a job to open /dev/sda and /dev/sdb
274 as the two working files, you would use
Bruce Cran03e20d62011-01-02 20:14:54 +0100275 filename=/dev/sda:/dev/sdb. On Windows, disk devices are accessed
Bruce Cranecc314b2011-01-04 10:59:30 +0100276 as \\.\PhysicalDrive0 for the first device, \\.\PhysicalDrive1
277 for the second etc. If the wanted filename does need to
278 include a colon, then escape that with a '\' character.
279 For instance, if the filename is "/dev/dsk/foo@3,0:c",
280 then you would use filename="/dev/dsk/foo@3,0\:c".
Bruce Cran03e20d62011-01-02 20:14:54 +0100281 '-' is a reserved name, meaning stdin or stdout. Which of the
282 two depends on the read/write direction set.
Jens Axboe71bfa162006-10-25 11:08:19 +0200283
Jens Axboebbf6b542007-03-13 15:28:55 +0100284opendir=str Tell fio to recursively add any file it can find in this
285 directory and down the file system tree.
286
Randy Dunlap37760412009-05-13 07:51:05 +0200287lockfile=str Fio defaults to not locking any files before it does
Jens Axboe4d4e80f2008-03-04 10:18:56 +0100288 IO to them. If a file or file descriptor is shared, fio
289 can serialize IO to that file to make the end result
290 consistent. This is usual for emulating real workloads that
291 share files. The lock modes are:
Jens Axboe29c13492008-03-01 19:25:20 +0100292
Jens Axboe4d4e80f2008-03-04 10:18:56 +0100293 none No locking. The default.
294 exclusive Only one thread/process may do IO,
295 excluding all others.
296 readwrite Read-write locking on the file. Many
297 readers may access the file at the
298 same time, but writes get exclusive
299 access.
300
301 The option may be post-fixed with a lock batch number. If
302 set, then each thread/process may do that amount of IOs to
Jens Axboebf9a3ed2008-06-05 11:53:08 +0200303 the file before giving up the lock. Since lock acquisition is
Jens Axboe4d4e80f2008-03-04 10:18:56 +0100304 expensive, batching the lock/unlocks will speed up IO.
Jens Axboe29c13492008-03-01 19:25:20 +0100305
Jens Axboed3aad8f2007-03-15 14:12:05 +0100306readwrite=str
Jens Axboe71bfa162006-10-25 11:08:19 +0200307rw=str Type of io pattern. Accepted values are:
308
309 read Sequential reads
310 write Sequential writes
311 randwrite Random writes
312 randread Random reads
313 rw Sequential mixed reads and writes
314 randrw Random mixed reads and writes
315
316 For the mixed io types, the default is to split them 50/50.
317 For certain types of io the result may still be skewed a bit,
Jens Axboe211097b2007-03-22 18:56:45 +0100318 since the speed may be different. It is possible to specify
Jens Axboe38dad622010-07-20 14:46:00 -0600319 a number of IO's to do before getting a new offset, this is
320 one by appending a ':<nr>' to the end of the string given.
321 For a random read, it would look like 'rw=randread:8' for
Jens Axboe059b0802011-08-25 09:09:37 +0200322 passing in an offset modifier with a value of 8. If the
323 postfix is used with a sequential IO pattern, then the value
324 specified will be added to the generated offset for each IO.
325 For instance, using rw=write:4k will skip 4k for every
326 write. It turns sequential IO into sequential IO with holes.
327 See the 'rw_sequencer' option.
Jens Axboe38dad622010-07-20 14:46:00 -0600328
329rw_sequencer=str If an offset modifier is given by appending a number to
330 the rw=<str> line, then this option controls how that
331 number modifies the IO offset being generated. Accepted
332 values are:
333
334 sequential Generate sequential offset
335 identical Generate the same offset
336
337 'sequential' is only useful for random IO, where fio would
338 normally generate a new random offset for every IO. If you
339 append eg 8 to randread, you would get a new random offset for
Jens Axboe211097b2007-03-22 18:56:45 +0100340 every 8 IO's. The result would be a seek for only every 8
341 IO's, instead of for every IO. Use rw=randread:8 to specify
Jens Axboe38dad622010-07-20 14:46:00 -0600342 that. As sequential IO is already sequential, setting
343 'sequential' for that would not result in any differences.
344 'identical' behaves in a similar fashion, except it sends
345 the same offset 8 number of times before generating a new
346 offset.
Jens Axboe71bfa162006-10-25 11:08:19 +0200347
Jens Axboe90fef2d2009-07-17 22:33:32 +0200348kb_base=int The base unit for a kilobyte. The defacto base is 2^10, 1024.
349 Storage manufacturers like to use 10^3 or 1000 as a base
350 ten unit instead, for obvious reasons. Allow values are
351 1024 or 1000, with 1024 being the default.
352
Jens Axboeee738492007-01-10 11:23:16 +0100353randrepeat=bool For random IO workloads, seed the generator in a predictable
354 way so that results are repeatable across repetitions.
355
Jens Axboe2615cc42011-03-28 09:35:09 +0200356use_os_rand=bool Fio can either use the random generator supplied by the OS
357 to generator random offsets, or it can use it's own internal
358 generator (based on Tausworthe). Default is to use the
359 internal generator, which is often of better quality and
360 faster.
361
Eric Gourioua596f042011-06-17 09:11:45 +0200362fallocate=str Whether pre-allocation is performed when laying down files.
363 Accepted values are:
364
365 none Do not pre-allocate space
366 posix Pre-allocate via posix_fallocate()
367 keep Pre-allocate via fallocate() with
368 FALLOC_FL_KEEP_SIZE set
369 0 Backward-compatible alias for 'none'
370 1 Backward-compatible alias for 'posix'
371
372 May not be available on all supported platforms. 'keep' is only
373 available on Linux.If using ZFS on Solaris this must be set to
374 'none' because ZFS doesn't support it. Default: 'posix'.
Jens Axboe7bc8c2c2010-01-28 11:31:31 +0100375
Jens Axboed2f3ac32007-03-22 19:24:09 +0100376fadvise_hint=bool By default, fio will use fadvise() to advise the kernel
377 on what IO patterns it is likely to issue. Sometimes you
378 want to test specific IO patterns without telling the
379 kernel about it, in which case you can disable this option.
380 If set, fio will use POSIX_FADV_SEQUENTIAL for sequential
381 IO and POSIX_FADV_RANDOM for random IO.
382
Jens Axboef7fa2652009-03-09 14:20:20 +0100383size=int The total size of file io for this job. Fio will run until
Jens Axboe7616caf2007-05-25 09:26:05 +0200384 this many bytes has been transferred, unless runtime is
385 limited by other options (such as 'runtime', for instance).
Randy Dunlap37760412009-05-13 07:51:05 +0200386 Unless specific nrfiles and filesize options are given,
Jens Axboe7616caf2007-05-25 09:26:05 +0200387 fio will divide this size between the available files
Jens Axboed6667262010-06-25 11:32:48 +0200388 specified by the job. If not set, fio will use the full
389 size of the given files or devices. If the the files
Jens Axboe7bb59102011-07-12 19:47:03 +0200390 do not exist, size must be given. It is also possible to
391 give size as a percentage between 1 and 100. If size=20%
392 is given, fio will use 20% of the full size of the given
393 files or devices.
Jens Axboe71bfa162006-10-25 11:08:19 +0200394
Jens Axboef7fa2652009-03-09 14:20:20 +0100395filesize=int Individual file sizes. May be a range, in which case fio
Jens Axboe9c60ce62007-03-15 09:14:47 +0100396 will select sizes for files at random within the given range
397 and limited to 'size' in total (if that is given). If not
398 given, each created file is the same size.
399
Jens Axboe74586c12011-01-20 10:16:03 -0700400fill_device=bool
401fill_fs=bool Sets size to something really large and waits for ENOSPC (no
Shawn Lewisaa31f1f2008-01-11 09:45:11 +0100402 space left on device) as the terminating condition. Only makes
Jens Axboe3ce9dca2009-06-10 08:55:21 +0200403 sense with sequential write. For a read workload, the mount
Jens Axboe4f124322011-01-19 15:35:26 -0700404 point will be filled first then IO started on the result. This
405 option doesn't make sense if operating on a raw device node,
406 since the size of that is already known by the file system.
407 Additionally, writing beyond end-of-device will not return
408 ENOSPC there.
Shawn Lewisaa31f1f2008-01-11 09:45:11 +0100409
Jens Axboef7fa2652009-03-09 14:20:20 +0100410blocksize=int
411bs=int The block size used for the io units. Defaults to 4k. Values
412 can be given for both read and writes. If a single int is
413 given, it will apply to both. If a second int is specified
Jens Axboef90eff52006-11-06 11:08:21 +0100414 after a comma, it will apply to writes only. In other words,
415 the format is either bs=read_and_write or bs=read,write.
416 bs=4k,8k will thus use 4k blocks for reads, and 8k blocks
Jens Axboe787f7e92006-11-06 13:26:29 +0100417 for writes. If you only wish to set the write size, you
418 can do so by passing an empty read size - bs=,8k will set
419 8k for writes and leave the read default value.
Jens Axboea00735e2006-11-03 08:58:08 +0100420
Jens Axboe2b7a01d2009-03-11 11:00:13 +0100421blockalign=int
422ba=int At what boundary to align random IO offsets. Defaults to
423 the same as 'blocksize' the minimum blocksize given.
424 Minimum alignment is typically 512b for using direct IO,
425 though it usually depends on the hardware block size. This
426 option is mutually exclusive with using a random map for
427 files, so it will turn off that option.
428
Jens Axboed3aad8f2007-03-15 14:12:05 +0100429blocksize_range=irange
Jens Axboe71bfa162006-10-25 11:08:19 +0200430bsrange=irange Instead of giving a single block size, specify a range
431 and fio will mix the issued io block sizes. The issued
432 io unit will always be a multiple of the minimum value
Jens Axboef90eff52006-11-06 11:08:21 +0100433 given (also see bs_unaligned). Applies to both reads and
434 writes, however a second range can be given after a comma.
435 See bs=.
Jens Axboea00735e2006-11-03 08:58:08 +0100436
Jens Axboe564ca972007-12-14 12:21:19 +0100437bssplit=str Sometimes you want even finer grained control of the
438 block sizes issued, not just an even split between them.
439 This option allows you to weight various block sizes,
440 so that you are able to define a specific amount of
441 block sizes issued. The format for this option is:
442
443 bssplit=blocksize/percentage:blocksize/percentage
444
445 for as many block sizes as needed. So if you want to define
446 a workload that has 50% 64k blocks, 10% 4k blocks, and
447 40% 32k blocks, you would write:
448
449 bssplit=4k/10:64k/50:32k/40
450
451 Ordering does not matter. If the percentage is left blank,
452 fio will fill in the remaining values evenly. So a bssplit
453 option like this one:
454
455 bssplit=4k/50:1k/:32k/
456
457 would have 50% 4k ios, and 25% 1k and 32k ios. The percentages
458 always add up to 100, if bssplit is given a range that adds
459 up to more, it will error out.
460
Jens Axboe720e84a2009-04-21 08:29:55 +0200461 bssplit also supports giving separate splits to reads and
462 writes. The format is identical to what bs= accepts. You
463 have to separate the read and write parts with a comma. So
464 if you want a workload that has 50% 2k reads and 50% 4k reads,
465 while having 90% 4k writes and 10% 8k writes, you would
466 specify:
467
468 bssplit=2k/50:4k/50,4k/90,8k/10
469
Jens Axboed3aad8f2007-03-15 14:12:05 +0100470blocksize_unaligned
Jens Axboe690adba2006-10-30 15:25:09 +0100471bs_unaligned If this option is given, any byte size value within bsrange
472 may be used as a block range. This typically wont work with
473 direct IO, as that normally requires sector alignment.
Jens Axboe71bfa162006-10-25 11:08:19 +0200474
Jens Axboee9459e52007-04-17 15:46:32 +0200475zero_buffers If this option is given, fio will init the IO buffers to
476 all zeroes. The default is to fill them with random data.
477
Jens Axboe5973caf2008-05-21 19:52:35 +0200478refill_buffers If this option is given, fio will refill the IO buffers
479 on every submit. The default is to only fill it at init
480 time and reuse that data. Only makes sense if zero_buffers
Jens Axboe41ccd842008-05-22 09:17:33 +0200481 isn't specified, naturally. If data verification is enabled,
482 refill_buffers is also automatically enabled.
Jens Axboe5973caf2008-05-21 19:52:35 +0200483
Jens Axboefd684182011-09-19 09:24:44 +0200484scramble_buffers=bool If refill_buffers is too costly and the target is
485 using data deduplication, then setting this option will
486 slightly modify the IO buffer contents to defeat normal
487 de-dupe attempts. This is not enough to defeat more clever
488 block compression attempts, but it will stop naive dedupe of
489 blocks. Default: true.
490
Jens Axboe71bfa162006-10-25 11:08:19 +0200491nrfiles=int Number of files to use for this job. Defaults to 1.
492
Jens Axboe390b1532007-03-09 13:03:00 +0100493openfiles=int Number of files to keep open at the same time. Defaults to
494 the same as nrfiles, can be set smaller to limit the number
495 simultaneous opens.
496
Jens Axboe5af1c6f2007-03-01 10:06:10 +0100497file_service_type=str Defines how fio decides which file from a job to
498 service next. The following types are defined:
499
500 random Just choose a file at random.
501
502 roundrobin Round robin over open files. This
503 is the default.
504
Jens Axboea086c252009-03-04 08:27:37 +0100505 sequential Finish one file before moving on to
506 the next. Multiple files can still be
507 open depending on 'openfiles'.
508
Jens Axboe1907dbc2007-03-12 11:44:28 +0100509 The string can have a number appended, indicating how
510 often to switch to a new file. So if option random:4 is
511 given, fio will switch to a new random file after 4 ios
512 have been issued.
513
Jens Axboe71bfa162006-10-25 11:08:19 +0200514ioengine=str Defines how the job issues io to the file. The following
515 types are defined:
516
517 sync Basic read(2) or write(2) io. lseek(2) is
518 used to position the io location.
519
gurudas paia31041e2007-10-23 15:12:30 +0200520 psync Basic pread(2) or pwrite(2) io.
521
Gurudas Paie05af9e2008-02-06 11:16:15 +0100522 vsync Basic readv(2) or writev(2) IO.
Jens Axboe1d2af022008-02-04 10:59:07 +0100523
Jens Axboe15d182a2009-01-16 19:15:07 +0100524 libaio Linux native asynchronous io. Note that Linux
525 may only support queued behaviour with
526 non-buffered IO (set direct=1 or buffered=0).
Jens Axboec44b1ff2011-08-30 20:43:53 -0600527 This engine also has a sub-option,
528 userspace_reap. To set it, use
529 ioengine=libaio:userspace_reap. Normally, with
530 the libaio engine in use, fio will use the
531 io_getevents system call to reap newly returned
532 events. With this flag turned on, the AIO ring
533 will be read directly from user-space to reap
534 events. The reaping mode is only enabled when
535 polling for a minimum of 0 events (eg when
536 iodepth_batch_complete=0).
Jens Axboe71bfa162006-10-25 11:08:19 +0200537
538 posixaio glibc posix asynchronous io.
539
Jens Axboe417f0062008-06-02 11:59:30 +0200540 solarisaio Solaris native asynchronous io.
541
Bruce Cran03e20d62011-01-02 20:14:54 +0100542 windowsaio Windows native asynchronous io.
543
Jens Axboe71bfa162006-10-25 11:08:19 +0200544 mmap File is memory mapped and data copied
545 to/from using memcpy(3).
546
547 splice splice(2) is used to transfer the data and
548 vmsplice(2) to transfer data from user
549 space to the kernel.
550
Jens Axboed0ff85d2007-02-14 01:19:41 +0100551 syslet-rw Use the syslet system calls to make
552 regular read/write async.
553
Jens Axboe71bfa162006-10-25 11:08:19 +0200554 sg SCSI generic sg v3 io. May either be
Jens Axboe6c219762006-11-03 15:51:45 +0100555 synchronous using the SG_IO ioctl, or if
Jens Axboe71bfa162006-10-25 11:08:19 +0200556 the target is an sg character device
557 we use read(2) and write(2) for asynchronous
558 io.
559
Jens Axboea94ea282006-11-24 12:37:34 +0100560 null Doesn't transfer any data, just pretends
561 to. This is mainly used to exercise fio
562 itself and for debugging/testing purposes.
563
Jens Axboeed92ac02007-02-06 14:43:52 +0100564 net Transfer over the network to given host:port.
565 'filename' must be set appropriately to
Jens Axboe414c2a32009-01-16 13:21:15 +0100566 filename=host/port/protocol regardless of send
Jens Axboeed92ac02007-02-06 14:43:52 +0100567 or receive, if the latter only the port
Jens Axboe414c2a32009-01-16 13:21:15 +0100568 argument is used. 'host' may be an IP address
569 or hostname, port is the port number to be used,
570 and protocol may be 'udp' or 'tcp'. If no
571 protocol is given, TCP is used.
Jens Axboeed92ac02007-02-06 14:43:52 +0100572
Jens Axboe9cce02e2007-06-22 15:42:21 +0200573 netsplice Like net, but uses splice/vmsplice to
574 map data and send/receive.
575
gurudas pai53aec0a2007-10-05 13:20:18 +0200576 cpuio Doesn't transfer any data, but burns CPU
Jens Axboeba0fbe12007-03-09 14:34:23 +0100577 cycles according to the cpuload= and
578 cpucycle= options. Setting cpuload=85
579 will cause that job to do nothing but burn
Gurudas Pai36ecec82008-02-08 08:50:14 +0100580 85% of the CPU. In case of SMP machines,
581 use numjobs=<no_of_cpu> to get desired CPU
582 usage, as the cpuload only loads a single
583 CPU at the desired rate.
Jens Axboeba0fbe12007-03-09 14:34:23 +0100584
Jens Axboee9a18062007-03-21 08:51:56 +0100585 guasi The GUASI IO engine is the Generic Userspace
586 Asyncronous Syscall Interface approach
587 to async IO. See
588
589 http://www.xmailserver.org/guasi-lib.html
590
591 for more info on GUASI.
592
ren yufei21b8aee2011-08-01 10:01:57 +0200593 rdma The RDMA I/O engine supports both RDMA
Bart Van Asscheeb52fa32011-08-15 09:01:05 +0200594 memory semantics (RDMA_WRITE/RDMA_READ) and
595 channel semantics (Send/Recv) for the
596 InfiniBand, RoCE and iWARP protocols.
ren yufei21b8aee2011-08-01 10:01:57 +0200597
Jens Axboe8a7bd872007-02-28 11:12:25 +0100598 external Prefix to specify loading an external
599 IO engine object file. Append the engine
600 filename, eg ioengine=external:/tmp/foo.o
601 to load ioengine foo.o in /tmp.
602
Jens Axboe71bfa162006-10-25 11:08:19 +0200603iodepth=int This defines how many io units to keep in flight against
604 the file. The default is 1 for each file defined in this
605 job, can be overridden with a larger value for higher
Jens Axboeee72ca02010-12-02 20:05:37 +0100606 concurrency. Note that increasing iodepth beyond 1 will not
607 affect synchronous ioengines (except for small degress when
Bruce Cran9b836562011-01-08 19:49:54 +0100608 verify_async is in use). Even async engines may impose OS
Jens Axboeee72ca02010-12-02 20:05:37 +0100609 restrictions causing the desired depth not to be achieved.
610 This may happen on Linux when using libaio and not setting
611 direct=1, since buffered IO is not async on that OS. Keep an
612 eye on the IO depth distribution in the fio output to verify
613 that the achieved depth is as expected. Default: 1.
Jens Axboe71bfa162006-10-25 11:08:19 +0200614
Jens Axboe49504212008-06-05 09:03:30 +0200615iodepth_batch_submit=int
Jens Axboecb5ab512007-02-26 12:57:09 +0100616iodepth_batch=int This defines how many pieces of IO to submit at once.
Jens Axboe89e820f2008-01-18 10:30:07 +0100617 It defaults to 1 which means that we submit each IO
618 as soon as it is available, but can be raised to submit
619 bigger batches of IO at the time.
Jens Axboecb5ab512007-02-26 12:57:09 +0100620
Jens Axboe49504212008-06-05 09:03:30 +0200621iodepth_batch_complete=int This defines how many pieces of IO to retrieve
622 at once. It defaults to 1 which means that we'll ask
623 for a minimum of 1 IO in the retrieval process from
624 the kernel. The IO retrieval will go on until we
625 hit the limit set by iodepth_low. If this variable is
626 set to 0, then fio will always check for completed
627 events before queuing more IO. This helps reduce
628 IO latency, at the cost of more retrieval system calls.
629
Jens Axboee916b392007-02-20 14:37:26 +0100630iodepth_low=int The low water mark indicating when to start filling
631 the queue again. Defaults to the same as iodepth, meaning
632 that fio will attempt to keep the queue full at all times.
633 If iodepth is set to eg 16 and iodepth_low is set to 4, then
634 after fio has filled the queue of 16 requests, it will let
635 the depth drain down to 4 before starting to fill it again.
636
Jens Axboe71bfa162006-10-25 11:08:19 +0200637direct=bool If value is true, use non-buffered io. This is usually
Bruce Cran9b836562011-01-08 19:49:54 +0100638 O_DIRECT. Note that ZFS on Solaris doesn't support direct io.
Jens Axboe76a43db2007-01-11 13:24:44 +0100639
640buffered=bool If value is true, use buffered io. This is the opposite
641 of the 'direct' option. Defaults to true.
Jens Axboe71bfa162006-10-25 11:08:19 +0200642
Jens Axboef7fa2652009-03-09 14:20:20 +0100643offset=int Start io at the given offset in the file. The data before
Jens Axboe71bfa162006-10-25 11:08:19 +0200644 the given offset will not be touched. This effectively
645 caps the file size at real_size - offset.
646
647fsync=int If writing to a file, issue a sync of the dirty data
648 for every number of blocks given. For example, if you give
649 32 as a parameter, fio will sync the file for every 32
650 writes issued. If fio is using non-buffered io, we may
651 not sync the file. The exception is the sg io engine, which
Jens Axboe6c219762006-11-03 15:51:45 +0100652 synchronizes the disk cache anyway.
Jens Axboe71bfa162006-10-25 11:08:19 +0200653
Jens Axboee76b1da2010-03-09 20:49:54 +0100654fdatasync=int Like fsync= but uses fdatasync() to only sync data and not
Jens Axboe5f9099e2009-06-16 22:40:26 +0200655 metadata blocks.
Joshua Aunee72fa4d2010-02-11 00:59:18 -0700656 In FreeBSD there is no fdatasync(), this falls back to
657 using fsync()
Jens Axboe5f9099e2009-06-16 22:40:26 +0200658
Jens Axboee76b1da2010-03-09 20:49:54 +0100659sync_file_range=str:val Use sync_file_range() for every 'val' number of
660 write operations. Fio will track range of writes that
661 have happened since the last sync_file_range() call. 'str'
662 can currently be one or more of:
663
664 wait_before SYNC_FILE_RANGE_WAIT_BEFORE
665 write SYNC_FILE_RANGE_WRITE
666 wait_after SYNC_FILE_RANGE_WAIT_AFTER
667
668 So if you do sync_file_range=wait_before,write:8, fio would
669 use SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE for
670 every 8 writes. Also see the sync_file_range(2) man page.
671 This option is Linux specific.
672
Jens Axboe5036fc12008-04-15 09:20:46 +0200673overwrite=bool If true, writes to a file will always overwrite existing
674 data. If the file doesn't already exist, it will be
675 created before the write phase begins. If the file exists
676 and is large enough for the specified write phase, nothing
677 will be done.
Jens Axboe71bfa162006-10-25 11:08:19 +0200678
679end_fsync=bool If true, fsync file contents when the job exits.
680
Jens Axboeebb14152007-03-13 14:42:15 +0100681fsync_on_close=bool If true, fio will fsync() a dirty file on close.
682 This differs from end_fsync in that it will happen on every
683 file close, not just at the end of the job.
684
Jens Axboe71bfa162006-10-25 11:08:19 +0200685rwmixread=int How large a percentage of the mix should be reads.
686
687rwmixwrite=int How large a percentage of the mix should be writes. If both
688 rwmixread and rwmixwrite is given and the values do not add
689 up to 100%, the latter of the two will be used to override
Jens Axboec35dd7a2009-06-10 08:39:16 +0200690 the first. This may interfere with a given rate setting,
691 if fio is asked to limit reads or writes to a certain rate.
692 If that is the case, then the distribution may be skewed.
Jens Axboe71bfa162006-10-25 11:08:19 +0200693
Jens Axboebb8895e2006-10-30 15:14:48 +0100694norandommap Normally fio will cover every block of the file when doing
695 random IO. If this option is given, fio will just get a
696 new random offset without looking at past io history. This
697 means that some blocks may not be read or written, and that
698 some blocks may be read/written more than once. This option
Jens Axboe83472392009-02-19 21:32:12 +0100699 is mutually exclusive with verify= if and only if multiple
700 blocksizes (via bsrange=) are used, since fio only tracks
701 complete rewrites of blocks.
Jens Axboebb8895e2006-10-30 15:14:48 +0100702
Jens Axboe0408c202011-08-08 09:07:28 +0200703softrandommap=bool See norandommap. If fio runs with the random block map
704 enabled and it fails to allocate the map, if this option is
705 set it will continue without a random block map. As coverage
706 will not be as complete as with random maps, this option is
Jens Axboe2b386d22008-03-26 10:32:57 +0100707 disabled by default.
708
Jens Axboe71bfa162006-10-25 11:08:19 +0200709nice=int Run the job with the given nice value. See man nice(2).
710
711prio=int Set the io priority value of this job. Linux limits us to
712 a positive value between 0 and 7, with 0 being the highest.
713 See man ionice(1).
714
715prioclass=int Set the io priority class. See man ionice(1).
716
717thinktime=int Stall the job x microseconds after an io has completed before
718 issuing the next. May be used to simulate processing being
Jens Axboe48097d52007-02-17 06:30:44 +0100719 done by an application. See thinktime_blocks and
720 thinktime_spin.
721
722thinktime_spin=int
723 Only valid if thinktime is set - pretend to spend CPU time
724 doing something with the data received, before falling back
725 to sleeping for the rest of the period specified by
726 thinktime.
Jens Axboe9c1f7432007-01-03 20:43:19 +0100727
728thinktime_blocks
729 Only valid if thinktime is set - control how many blocks
730 to issue, before waiting 'thinktime' usecs. If not set,
731 defaults to 1 which will make fio wait 'thinktime' usecs
732 after every block.
Jens Axboe71bfa162006-10-25 11:08:19 +0200733
Jens Axboe581e7142009-06-09 12:47:16 +0200734rate=int Cap the bandwidth used by this job. The number is in bytes/sec,
Jens Axboeb09da8f2009-07-17 23:16:17 +0200735 the normal suffix rules apply. You can use rate=500k to limit
Jens Axboe581e7142009-06-09 12:47:16 +0200736 reads and writes to 500k each, or you can specify read and
737 writes separately. Using rate=1m,500k would limit reads to
738 1MB/sec and writes to 500KB/sec. Capping only reads or
739 writes can be done with rate=,500k or rate=500k,. The former
740 will only limit writes (to 500KB/sec), the latter will only
741 limit reads.
Jens Axboe71bfa162006-10-25 11:08:19 +0200742
743ratemin=int Tell fio to do whatever it can to maintain at least this
Jens Axboe4e991c22007-03-15 11:41:11 +0100744 bandwidth. Failing to meet this requirement, will cause
Jens Axboe581e7142009-06-09 12:47:16 +0200745 the job to exit. The same format as rate is used for
746 read vs write separation.
Jens Axboe4e991c22007-03-15 11:41:11 +0100747
748rate_iops=int Cap the bandwidth to this number of IOPS. Basically the same
749 as rate, just specified independently of bandwidth. If the
750 job is given a block size range instead of a fixed value,
Jens Axboe581e7142009-06-09 12:47:16 +0200751 the smallest block size is used as the metric. The same format
752 as rate is used for read vs write seperation.
Jens Axboe4e991c22007-03-15 11:41:11 +0100753
754rate_iops_min=int If fio doesn't meet this rate of IO, it will cause
Jens Axboe581e7142009-06-09 12:47:16 +0200755 the job to exit. The same format as rate is used for read vs
756 write seperation.
Jens Axboe71bfa162006-10-25 11:08:19 +0200757
758ratecycle=int Average bandwidth for 'rate' and 'ratemin' over this number
Jens Axboe6c219762006-11-03 15:51:45 +0100759 of milliseconds.
Jens Axboe71bfa162006-10-25 11:08:19 +0200760
761cpumask=int Set the CPU affinity of this job. The parameter given is a
Jens Axboea08bc172007-06-13 21:00:46 +0200762 bitmask of allowed CPU's the job may run on. So if you want
763 the allowed CPUs to be 1 and 5, you would pass the decimal
764 value of (1 << 1 | 1 << 5), or 34. See man
Jens Axboe7dbb6eb2007-05-22 09:13:31 +0200765 sched_setaffinity(2). This may not work on all supported
Jens Axboeb0ea08c2008-12-05 12:57:11 +0100766 operating systems or kernel versions. This option doesn't
767 work well for a higher CPU count than what you can store in
768 an integer mask, so it can only control cpus 1-32. For
769 boxes with larger CPU counts, use cpus_allowed.
Jens Axboe71bfa162006-10-25 11:08:19 +0200770
Jens Axboed2e268b2007-06-15 10:33:49 +0200771cpus_allowed=str Controls the same options as cpumask, but it allows a text
772 setting of the permitted CPUs instead. So to use CPUs 1 and
Jens Axboe62a72732008-12-08 11:37:01 +0100773 5, you would specify cpus_allowed=1,5. This options also
774 allows a range of CPUs. Say you wanted a binding to CPUs
775 1, 5, and 8-15, you would set cpus_allowed=1,5,8-15.
Jens Axboed2e268b2007-06-15 10:33:49 +0200776
Jens Axboee417fd62008-09-11 09:27:15 +0200777startdelay=time Start this job the specified number of seconds after fio
Jens Axboe71bfa162006-10-25 11:08:19 +0200778 has started. Only useful if the job file contains several
779 jobs, and you want to delay starting some jobs to a certain
780 time.
781
Jens Axboee417fd62008-09-11 09:27:15 +0200782runtime=time Tell fio to terminate processing after the specified number
Jens Axboe71bfa162006-10-25 11:08:19 +0200783 of seconds. It can be quite hard to determine for how long
784 a specified job will run, so this parameter is handy to
785 cap the total runtime to a given time.
786
Jens Axboecf4464c2007-04-17 20:14:42 +0200787time_based If set, fio will run for the duration of the runtime
Jens Axboebf9a3ed2008-06-05 11:53:08 +0200788 specified even if the file(s) are completely read or
Jens Axboecf4464c2007-04-17 20:14:42 +0200789 written. It will simply loop over the same workload
790 as many times as the runtime allows.
791
Jens Axboee417fd62008-09-11 09:27:15 +0200792ramp_time=time If set, fio will run the specified workload for this amount
Jens Axboe721938a2008-09-10 09:46:16 +0200793 of time before logging any performance numbers. Useful for
794 letting performance settle before logging results, thus
Jens Axboeb29ee5b2008-09-11 10:17:26 +0200795 minimizing the runtime required for stable results. Note
796 that the ramp_time is considered lead in time for a job,
797 thus it will increase the total runtime if a special timeout
798 or runtime is specified.
Jens Axboe721938a2008-09-10 09:46:16 +0200799
Jens Axboe71bfa162006-10-25 11:08:19 +0200800invalidate=bool Invalidate the buffer/page cache parts for this file prior
801 to starting io. Defaults to true.
802
803sync=bool Use sync io for buffered writes. For the majority of the
804 io engines, this means using O_SYNC.
805
Jens Axboed3aad8f2007-03-15 14:12:05 +0100806iomem=str
Jens Axboe71bfa162006-10-25 11:08:19 +0200807mem=str Fio can use various types of memory as the io unit buffer.
808 The allowed values are:
809
810 malloc Use memory from malloc(3) as the buffers.
811
812 shm Use shared memory as the buffers. Allocated
813 through shmget(2).
814
Jens Axboe74b025b2006-12-19 15:18:14 +0100815 shmhuge Same as shm, but use huge pages as backing.
816
Jens Axboe313cb202006-12-21 09:50:00 +0100817 mmap Use mmap to allocate buffers. May either be
818 anonymous memory, or can be file backed if
819 a filename is given after the option. The
820 format is mem=mmap:/path/to/file.
Jens Axboe71bfa162006-10-25 11:08:19 +0200821
Jens Axboed0bdaf42006-12-20 14:40:44 +0100822 mmaphuge Use a memory mapped huge file as the buffer
823 backing. Append filename after mmaphuge, ala
824 mem=mmaphuge:/hugetlbfs/file
825
Jens Axboe71bfa162006-10-25 11:08:19 +0200826 The area allocated is a function of the maximum allowed
Jens Axboe5394ae52006-12-20 20:15:41 +0100827 bs size for the job, multiplied by the io depth given. Note
828 that for shmhuge and mmaphuge to work, the system must have
829 free huge pages allocated. This can normally be checked
830 and set by reading/writing /proc/sys/vm/nr_hugepages on a
Jens Axboeb22989b2009-07-17 22:29:23 +0200831 Linux system. Fio assumes a huge page is 4MB in size. So
Jens Axboe5394ae52006-12-20 20:15:41 +0100832 to calculate the number of huge pages you need for a given
833 job file, add up the io depth of all jobs (normally one unless
834 iodepth= is used) and multiply by the maximum bs set. Then
835 divide that number by the huge page size. You can see the
836 size of the huge pages in /proc/meminfo. If no huge pages
837 are allocated by having a non-zero number in nr_hugepages,
Jens Axboe56bb17f2006-12-20 20:27:36 +0100838 using mmaphuge or shmhuge will fail. Also see hugepage-size.
Jens Axboe5394ae52006-12-20 20:15:41 +0100839
840 mmaphuge also needs to have hugetlbfs mounted and the file
841 location should point there. So if it's mounted in /huge,
842 you would use mem=mmaphuge:/huge/somefile.
Jens Axboe71bfa162006-10-25 11:08:19 +0200843
Jens Axboed529ee12009-07-01 10:33:03 +0200844iomem_align=int This indiciates the memory alignment of the IO memory buffers.
845 Note that the given alignment is applied to the first IO unit
846 buffer, if using iodepth the alignment of the following buffers
847 are given by the bs used. In other words, if using a bs that is
848 a multiple of the page sized in the system, all buffers will
849 be aligned to this value. If using a bs that is not page
850 aligned, the alignment of subsequent IO memory buffers is the
851 sum of the iomem_align and bs used.
852
Jens Axboef7fa2652009-03-09 14:20:20 +0100853hugepage-size=int
Jens Axboe56bb17f2006-12-20 20:27:36 +0100854 Defines the size of a huge page. Must at least be equal
Jens Axboeb22989b2009-07-17 22:29:23 +0200855 to the system setting, see /proc/meminfo. Defaults to 4MB.
Jens Axboec51074e2006-12-20 20:28:33 +0100856 Should probably always be a multiple of megabytes, so using
857 hugepage-size=Xm is the preferred way to set this to avoid
858 setting a non-pow-2 bad value.
Jens Axboe56bb17f2006-12-20 20:27:36 +0100859
Jens Axboe71bfa162006-10-25 11:08:19 +0200860exitall When one job finishes, terminate the rest. The default is
861 to wait for each job to finish, sometimes that is not the
862 desired action.
863
864bwavgtime=int Average the calculated bandwidth over the given time. Value
Jens Axboe6c219762006-11-03 15:51:45 +0100865 is specified in milliseconds.
Jens Axboe71bfa162006-10-25 11:08:19 +0200866
867create_serialize=bool If true, serialize the file creating for the jobs.
868 This may be handy to avoid interleaving of data
869 files, which may greatly depend on the filesystem
870 used and even the number of processors in the system.
871
872create_fsync=bool fsync the data file after creation. This is the
873 default.
874
Jens Axboe814452b2009-03-04 12:53:13 +0100875create_on_open=bool Don't pre-setup the files for IO, just create open()
876 when it's time to do IO to that file.
877
Zhang, Yanminafad68f2009-05-20 11:30:55 +0200878pre_read=bool If this is given, files will be pre-read into memory before
Jens Axboe34f1c042009-06-02 14:19:25 +0200879 starting the given IO operation. This will also clear
880 the 'invalidate' flag, since it is pointless to pre-read
Jens Axboe9c0d2242009-07-01 12:26:28 +0200881 and then drop the cache. This will only work for IO engines
882 that are seekable, since they allow you to read the same data
883 multiple times. Thus it will not work on eg network or splice
884 IO.
Zhang, Yanminafad68f2009-05-20 11:30:55 +0200885
Jens Axboee545a6c2007-01-14 00:00:29 +0100886unlink=bool Unlink the job files when done. Not the default, as repeated
Jens Axboebf9a3ed2008-06-05 11:53:08 +0200887 runs of that job would then waste time recreating the file
888 set again and again.
Jens Axboe71bfa162006-10-25 11:08:19 +0200889
890loops=int Run the specified number of iterations of this job. Used
891 to repeat the same workload a given number of times. Defaults
892 to 1.
893
Jens Axboe68e1f292007-08-10 10:32:14 +0200894do_verify=bool Run the verify phase after a write phase. Only makes sense if
Shawn Lewise84c73a2007-08-02 22:19:32 +0200895 verify is set. Defaults to 1.
896
Jens Axboe71bfa162006-10-25 11:08:19 +0200897verify=str If writing to a file, fio can verify the file contents
898 after each iteration of the job. The allowed values are:
899
900 md5 Use an md5 sum of the data area and store
901 it in the header of each block.
902
Jens Axboe17dc34d2007-07-27 15:36:02 +0200903 crc64 Use an experimental crc64 sum of the data
904 area and store it in the header of each
905 block.
906
Jens Axboebac39e02008-06-11 20:46:19 +0200907 crc32c Use a crc32c sum of the data area and store
908 it in the header of each block.
909
Jens Axboe38455912008-08-04 15:35:26 +0200910 crc32c-intel Use hardware assisted crc32c calcuation
Jens Axboe0539d752010-06-21 15:22:56 +0200911 provided on SSE4.2 enabled processors. Falls
912 back to regular software crc32c, if not
913 supported by the system.
Jens Axboe38455912008-08-04 15:35:26 +0200914
Jens Axboe71bfa162006-10-25 11:08:19 +0200915 crc32 Use a crc32 sum of the data area and store
916 it in the header of each block.
917
Jens Axboe969f7ed2007-07-27 09:07:17 +0200918 crc16 Use a crc16 sum of the data area and store
919 it in the header of each block.
920
Jens Axboe17dc34d2007-07-27 15:36:02 +0200921 crc7 Use a crc7 sum of the data area and store
922 it in the header of each block.
923
Jens Axboecd14cc12007-07-30 10:59:33 +0200924 sha512 Use sha512 as the checksum function.
925
926 sha256 Use sha256 as the checksum function.
927
Jens Axboe7c353ce2009-08-09 22:40:33 +0200928 sha1 Use optimized sha1 as the checksum function.
929
Shawn Lewis7437ee82007-08-02 21:05:58 +0200930 meta Write extra information about each io
931 (timestamp, block number etc.). The block
Jens Axboe996093b2010-06-24 08:37:13 +0200932 number is verified. See also verify_pattern.
Shawn Lewis7437ee82007-08-02 21:05:58 +0200933
Jens Axboe36690c92007-03-26 10:23:34 +0200934 null Only pretend to verify. Useful for testing
935 internals with ioengine=null, not for much
936 else.
937
Jens Axboe6c219762006-11-03 15:51:45 +0100938 This option can be used for repeated burn-in tests of a
Jens Axboe71bfa162006-10-25 11:08:19 +0200939 system to make sure that the written data is also
Jens Axboeb892dc02009-09-05 20:37:35 +0200940 correctly read back. If the data direction given is
941 a read or random read, fio will assume that it should
942 verify a previously written file. If the data direction
943 includes any form of write, the verify will be of the
944 newly written data.
Jens Axboe71bfa162006-10-25 11:08:19 +0200945
Jens Axboe160b9662007-03-27 10:59:49 +0200946verifysort=bool If set, fio will sort written verify blocks when it deems
947 it faster to read them back in a sorted manner. This is
948 often the case when overwriting an existing file, since
949 the blocks are already laid out in the file system. You
950 can ignore this option unless doing huge amounts of really
951 fast IO where the red-black tree sorting CPU time becomes
952 significant.
Shawn Lewis3f9f4e22007-07-28 21:10:37 +0200953
Jens Axboef7fa2652009-03-09 14:20:20 +0100954verify_offset=int Swap the verification header with data somewhere else
Shawn Lewis546a9142007-07-28 21:11:37 +0200955 in the block before writing. Its swapped back before
956 verifying.
957
Jens Axboef7fa2652009-03-09 14:20:20 +0100958verify_interval=int Write the verification header at a finer granularity
Shawn Lewis3f9f4e22007-07-28 21:10:37 +0200959 than the blocksize. It will be written for chunks the
960 size of header_interval. blocksize should divide this
961 evenly.
Jens Axboe90059d62007-07-30 09:33:12 +0200962
Radha Ramachandran0e92f872009-10-27 20:14:27 +0100963verify_pattern=str If set, fio will fill the io buffers with this
Shawn Lewise28218f2008-01-16 11:01:33 +0100964 pattern. Fio defaults to filling with totally random
965 bytes, but sometimes it's interesting to fill with a known
966 pattern for io verification purposes. Depending on the
967 width of the pattern, fio will fill 1/2/3/4 bytes of the
Radha Ramachandran0e92f872009-10-27 20:14:27 +0100968 buffer at the time(it can be either a decimal or a hex number).
969 The verify_pattern if larger than a 32-bit quantity has to
Jens Axboe996093b2010-06-24 08:37:13 +0200970 be a hex number that starts with either "0x" or "0X". Use
971 with verify=meta.
Shawn Lewise28218f2008-01-16 11:01:33 +0100972
Jens Axboe68e1f292007-08-10 10:32:14 +0200973verify_fatal=bool Normally fio will keep checking the entire contents
Jens Axboea12a3b42007-08-09 10:20:54 +0200974 before quitting on a block verification failure. If this
975 option is set, fio will exit the job on the first observed
976 failure.
Jens Axboee8462bd2009-07-06 12:59:04 +0200977
Jens Axboeb463e932011-01-12 09:03:23 +0100978verify_dump=bool If set, dump the contents of both the original data
979 block and the data block we read off disk to files. This
980 allows later analysis to inspect just what kind of data
981 corruption occurred. On by default.
982
Jens Axboee8462bd2009-07-06 12:59:04 +0200983verify_async=int Fio will normally verify IO inline from the submitting
984 thread. This option takes an integer describing how many
985 async offload threads to create for IO verification instead,
986 causing fio to offload the duty of verifying IO contents
Jens Axboec85c3242009-07-06 14:12:57 +0200987 to one or more separate threads. If using this offload
988 option, even sync IO engines can benefit from using an
989 iodepth setting higher than 1, as it allows them to have
990 IO in flight while verifies are running.
Jens Axboee8462bd2009-07-06 12:59:04 +0200991
992verify_async_cpus=str Tell fio to set the given CPU affinity on the
993 async IO verification threads. See cpus_allowed for the
994 format used.
Jens Axboe6f874182010-06-21 12:53:26 +0200995
996verify_backlog=int Fio will normally verify the written contents of a
997 job that utilizes verify once that job has completed. In
998 other words, everything is written then everything is read
999 back and verified. You may want to verify continually
1000 instead for a variety of reasons. Fio stores the meta data
1001 associated with an IO block in memory, so for large
1002 verify workloads, quite a bit of memory would be used up
1003 holding this meta data. If this option is enabled, fio
Jens Axboef42195a2010-10-26 08:10:58 -06001004 will write only N blocks before verifying these blocks.
1005
Jens Axboe6f874182010-06-21 12:53:26 +02001006 will verify the previously written blocks before continuing
1007 to write new ones.
1008
1009verify_backlog_batch=int Control how many blocks fio will verify
1010 if verify_backlog is set. If not set, will default to
1011 the value of verify_backlog (meaning the entire queue
Jens Axboef42195a2010-10-26 08:10:58 -06001012 is read back and verified). If verify_backlog_batch is
1013 less than verify_backlog then not all blocks will be verified,
1014 if verify_backlog_batch is larger than verify_backlog, some
1015 blocks will be verified more than once.
Jens Axboe160b9662007-03-27 10:59:49 +02001016
Jens Axboed3923652011-08-03 12:38:39 +02001017stonewall
1018wait_for_previous Wait for preceeding jobs in the job file to exit, before
Jens Axboe71bfa162006-10-25 11:08:19 +02001019 starting this one. Can be used to insert serialization
Jens Axboeb3d62a72007-03-20 14:23:26 +01001020 points in the job file. A stone wall also implies starting
1021 a new reporting group.
1022
1023new_group Start a new reporting group. If this option isn't given,
1024 jobs in a file will be part of the same reporting group
Jens Axboebf9a3ed2008-06-05 11:53:08 +02001025 unless separated by a stone wall (or if it's a group
Jens Axboeb3d62a72007-03-20 14:23:26 +01001026 by itself, with the numjobs option).
Jens Axboe71bfa162006-10-25 11:08:19 +02001027
1028numjobs=int Create the specified number of clones of this job. May be
1029 used to setup a larger number of threads/processes doing
Jens Axboefa28c852007-03-06 15:40:49 +01001030 the same thing. We regard that grouping of jobs as a
1031 specific group.
1032
1033group_reporting If 'numjobs' is set, it may be interesting to display
1034 statistics for the group as a whole instead of for each
1035 individual job. This is especially true of 'numjobs' is
1036 large, looking at individual thread/process output quickly
1037 becomes unwieldy. If 'group_reporting' is specified, fio
1038 will show the final report per-group instead of per-job.
Jens Axboe71bfa162006-10-25 11:08:19 +02001039
1040thread fio defaults to forking jobs, however if this option is
1041 given, fio will use pthread_create(3) to create threads
1042 instead.
1043
Jens Axboef7fa2652009-03-09 14:20:20 +01001044zonesize=int Divide a file into zones of the specified size. See zoneskip.
Jens Axboe71bfa162006-10-25 11:08:19 +02001045
Jens Axboef7fa2652009-03-09 14:20:20 +01001046zoneskip=int Skip the specified number of bytes when zonesize data has
Jens Axboe71bfa162006-10-25 11:08:19 +02001047 been read. The two zone options can be used to only do
1048 io on zones of a file.
1049
Jens Axboe076efc72006-10-27 11:24:25 +02001050write_iolog=str Write the issued io patterns to the specified file. See
Stefan Hajnoczi5b42a482011-01-08 20:28:41 +01001051 read_iolog. Specify a separate file for each job, otherwise
1052 the iologs will be interspersed and the file may be corrupt.
Jens Axboe71bfa162006-10-25 11:08:19 +02001053
Jens Axboe076efc72006-10-27 11:24:25 +02001054read_iolog=str Open an iolog with the specified file name and replay the
Jens Axboe71bfa162006-10-25 11:08:19 +02001055 io patterns it contains. This can be used to store a
Jens Axboe6df8ada2007-05-15 13:23:19 +02001056 workload and replay it sometime later. The iolog given
1057 may also be a blktrace binary file, which allows fio
1058 to replay a workload captured by blktrace. See blktrace
1059 for how to capture such logging data. For blktrace replay,
1060 the file needs to be turned into a blkparse binary data
Jens Axboeea3e51c2010-05-17 19:51:45 +02001061 file first (blkparse <device> -o /dev/null -d file_for_fio.bin).
David Nellans64bbb862010-08-24 22:13:30 +02001062
1063replay_no_stall=int When replaying I/O with read_iolog the default behavior
Jens Axboe62776222010-09-02 15:30:16 +02001064 is to attempt to respect the time stamps within the log and
1065 replay them with the appropriate delay between IOPS. By
1066 setting this variable fio will not respect the timestamps and
1067 attempt to replay them as fast as possible while still
1068 respecting ordering. The result is the same I/O pattern to a
1069 given device, but different timings.
Jens Axboe71bfa162006-10-25 11:08:19 +02001070
David Nellansd1c46c02010-08-31 21:20:47 +02001071replay_redirect=str While replaying I/O patterns using read_iolog the
1072 default behavior is to replay the IOPS onto the major/minor
1073 device that each IOP was recorded from. This is sometimes
1074 undesireable because on a different machine those major/minor
1075 numbers can map to a different device. Changing hardware on
1076 the same system can also result in a different major/minor
1077 mapping. Replay_redirect causes all IOPS to be replayed onto
1078 the single specified device regardless of the device it was
1079 recorded from. i.e. replay_redirect=/dev/sdc would cause all
1080 IO in the blktrace to be replayed onto /dev/sdc. This means
1081 multiple devices will be replayed onto a single, if the trace
1082 contains multiple devices. If you want multiple devices to be
1083 replayed concurrently to multiple redirected devices you must
1084 blkparse your trace into separate traces and replay them with
1085 independent fio invocations. Unfortuantely this also breaks
1086 the strict time ordering between multiple device accesses.
1087
Jens Axboee3cedca2008-11-19 19:57:52 +01001088write_bw_log=str If given, write a bandwidth log of the jobs in this job
Jens Axboe71bfa162006-10-25 11:08:19 +02001089 file. Can be used to store data of the bandwidth of the
Jens Axboee0da9bc2006-10-25 13:08:57 +02001090 jobs in their lifetime. The included fio_generate_plots
1091 script uses gnuplot to turn these text files into nice
Jens Axboee3cedca2008-11-19 19:57:52 +01001092 graphs. See write_log_log for behaviour of given
1093 filename. For this option, the postfix is _bw.log.
Jens Axboe71bfa162006-10-25 11:08:19 +02001094
Jens Axboee3cedca2008-11-19 19:57:52 +01001095write_lat_log=str Same as write_bw_log, except that this option stores io
Jens Axboe02af0982010-06-24 09:59:34 +02001096 submission, completion, and total latencies instead. If no
1097 filename is given with this option, the default filename of
1098 "jobname_type.log" is used. Even if the filename is given,
1099 fio will still append the type of log. So if one specifies
Jens Axboee3cedca2008-11-19 19:57:52 +01001100
1101 write_lat_log=foo
1102
Jens Axboe02af0982010-06-24 09:59:34 +02001103 The actual log names will be foo_slat.log, foo_slat.log,
1104 and foo_lat.log. This helps fio_generate_plot fine the logs
1105 automatically.
Jens Axboe71bfa162006-10-25 11:08:19 +02001106
Jens Axboef7fa2652009-03-09 14:20:20 +01001107lockmem=int Pin down the specified amount of memory with mlock(2). Can
Jens Axboe71bfa162006-10-25 11:08:19 +02001108 potentially be used instead of removing memory or booting
1109 with less memory to simulate a smaller amount of memory.
1110
1111exec_prerun=str Before running this job, issue the command specified
1112 through system(3).
1113
1114exec_postrun=str After the job completes, issue the command specified
1115 though system(3).
1116
1117ioscheduler=str Attempt to switch the device hosting the file to the specified
1118 io scheduler before running.
1119
1120cpuload=int If the job is a CPU cycle eater, attempt to use the specified
1121 percentage of CPU cycles.
1122
1123cpuchunks=int If the job is a CPU cycle eater, split the load into
Randy Dunlap26eca2d2009-05-13 07:50:38 +02001124 cycles of the given time. In microseconds.
Jens Axboe71bfa162006-10-25 11:08:19 +02001125
Jens Axboe0a839f32007-04-26 09:02:34 +02001126disk_util=bool Generate disk utilization statistics, if the platform
1127 supports it. Defaults to on.
1128
Jens Axboe02af0982010-06-24 09:59:34 +02001129disable_lat=bool Disable measurements of total latency numbers. Useful
Jens Axboe9520ebb2008-10-16 21:03:27 +02001130 only for cutting back the number of calls to gettimeofday,
1131 as that does impact performance at really high IOPS rates.
1132 Note that to really get rid of a large amount of these
1133 calls, this option must be used with disable_slat and
1134 disable_bw as well.
1135
Jens Axboe02af0982010-06-24 09:59:34 +02001136disable_clat=bool Disable measurements of completion latency numbers. See
1137 disable_lat.
1138
Jens Axboe9520ebb2008-10-16 21:03:27 +02001139disable_slat=bool Disable measurements of submission latency numbers. See
Jens Axboe02af0982010-06-24 09:59:34 +02001140 disable_slat.
Jens Axboe9520ebb2008-10-16 21:03:27 +02001141
1142disable_bw=bool Disable measurements of throughput/bandwidth numbers. See
Jens Axboe02af0982010-06-24 09:59:34 +02001143 disable_lat.
Jens Axboe9520ebb2008-10-16 21:03:27 +02001144
Yu-ju Hong83349192011-08-13 00:53:44 +02001145clat_percentiles=bool Enable the reporting of percentiles of
1146 completion latencies.
1147
1148percentile_list=float_list Overwrite the default list of percentiles
1149 for completion latencies. Each number is a floating
1150 number in the range (0,100], and the maximum length of
1151 the list is 20. Use ':' to separate the numbers, and
1152 list the numbers in ascending order. For example,
1153 --percentile_list=99.5:99.9 will cause fio to report
1154 the values of completion latency below which 99.5% and
1155 99.9% of the observed latencies fell, respectively.
1156
Jens Axboe993bf482008-11-14 13:04:53 +01001157gtod_reduce=bool Enable all of the gettimeofday() reducing options
1158 (disable_clat, disable_slat, disable_bw) plus reduce
1159 precision of the timeout somewhat to really shrink
1160 the gettimeofday() call count. With this option enabled,
1161 we only do about 0.4% of the gtod() calls we would have
1162 done if all time keeping was enabled.
1163
Jens Axboebe4ecfd2008-12-08 14:10:52 +01001164gtod_cpu=int Sometimes it's cheaper to dedicate a single thread of
1165 execution to just getting the current time. Fio (and
1166 databases, for instance) are very intensive on gettimeofday()
1167 calls. With this option, you can set one CPU aside for
1168 doing nothing but logging current time to a shared memory
1169 location. Then the other threads/processes that run IO
1170 workloads need only copy that segment, instead of entering
1171 the kernel with a gettimeofday() call. The CPU set aside
1172 for doing these time calls will be excluded from other
1173 uses. Fio will manually clear it from the CPU mask of other
1174 jobs.
Jens Axboea696fa22009-12-04 10:05:02 +01001175
Radha Ramachandranf2bba182009-06-15 08:40:16 +02001176continue_on_error=bool Normally fio will exit the job on the first observed
1177 failure. If this option is set, fio will continue the job when
1178 there is a 'non-fatal error' (EIO or EILSEQ) until the runtime
1179 is exceeded or the I/O size specified is completed. If this
1180 option is used, there are two more stats that are appended,
1181 the total error count and the first error. The error field
1182 given in the stats is the first error that was hit during the
1183 run.
Jens Axboebe4ecfd2008-12-08 14:10:52 +01001184
Jens Axboe6adb38a2009-12-07 08:01:26 +01001185cgroup=str Add job to this control group. If it doesn't exist, it will
1186 be created. The system must have a mounted cgroup blkio
1187 mount point for this to work. If your system doesn't have it
1188 mounted, you can do so with:
Jens Axboea696fa22009-12-04 10:05:02 +01001189
1190 # mount -t cgroup -o blkio none /cgroup
1191
Jens Axboea696fa22009-12-04 10:05:02 +01001192cgroup_weight=int Set the weight of the cgroup to this value. See
1193 the documentation that comes with the kernel, allowed values
1194 are in the range of 100..1000.
Jens Axboe71bfa162006-10-25 11:08:19 +02001195
Vivek Goyal7de87092010-03-31 22:55:15 +02001196cgroup_nodelete=bool Normally fio will delete the cgroups it has created after
1197 the job completion. To override this behavior and to leave
1198 cgroups around after the job completion, set cgroup_nodelete=1.
1199 This can be useful if one wants to inspect various cgroup
1200 files after job completion. Default: false
1201
Jens Axboee0b0d892009-12-08 10:10:14 +01001202uid=int Instead of running as the invoking user, set the user ID to
1203 this value before the thread/process does any work.
1204
1205gid=int Set group ID, see uid.
1206
Jens Axboe71bfa162006-10-25 11:08:19 +020012076.0 Interpreting the output
1208---------------------------
1209
1210fio spits out a lot of output. While running, fio will display the
1211status of the jobs created. An example of that would be:
1212
Jens Axboe73c8b082007-01-11 19:25:52 +01001213Threads: 1: [_r] [24.8% done] [ 13509/ 8334 kb/s] [eta 00h:01m:31s]
Jens Axboe71bfa162006-10-25 11:08:19 +02001214
1215The characters inside the square brackets denote the current status of
1216each thread. The possible values (in typical life cycle order) are:
1217
1218Idle Run
1219---- ---
1220P Thread setup, but not started.
1221C Thread created.
1222I Thread initialized, waiting.
Jens Axboeb0f65862009-05-20 11:52:15 +02001223 p Thread running pre-reading file(s).
Jens Axboe71bfa162006-10-25 11:08:19 +02001224 R Running, doing sequential reads.
1225 r Running, doing random reads.
1226 W Running, doing sequential writes.
1227 w Running, doing random writes.
1228 M Running, doing mixed sequential reads/writes.
1229 m Running, doing mixed random reads/writes.
1230 F Running, currently waiting for fsync()
Jens Axboefc6bd432009-04-29 09:52:10 +02001231 V Running, doing verification of written data.
Jens Axboe71bfa162006-10-25 11:08:19 +02001232E Thread exited, not reaped by main thread yet.
1233_ Thread reaped.
1234
1235The other values are fairly self explanatory - number of threads
Jens Axboec9f60302007-07-20 12:43:05 +02001236currently running and doing io, rate of io since last check (read speed
1237listed first, then write speed), and the estimated completion percentage
1238and time for the running group. It's impossible to estimate runtime of
1239the following groups (if any).
Jens Axboe71bfa162006-10-25 11:08:19 +02001240
1241When fio is done (or interrupted by ctrl-c), it will show the data for
1242each thread, group of threads, and disks in that order. For each data
1243direction, the output looks like:
1244
1245Client1 (g=0): err= 0:
Paul Dubs35649e52011-07-21 16:04:52 +02001246 write: io= 32MB, bw= 666KB/s, iops=89 , runt= 50320msec
Jens Axboe6104ddb2007-01-11 14:24:29 +01001247 slat (msec): min= 0, max= 136, avg= 0.03, stdev= 1.92
1248 clat (msec): min= 0, max= 631, avg=48.50, stdev=86.82
Jens Axboeb22989b2009-07-17 22:29:23 +02001249 bw (KB/s) : min= 0, max= 1196, per=51.00%, avg=664.02, stdev=681.68
Jens Axboee7823a92007-09-07 20:33:33 +02001250 cpu : usr=1.49%, sys=0.25%, ctx=7969, majf=0, minf=17
Jens Axboe71619dc2007-01-13 23:56:33 +01001251 IO depths : 1=0.1%, 2=0.3%, 4=0.5%, 8=99.0%, 16=0.0%, 32=0.0%, >32=0.0%
Jens Axboe838bc702008-05-22 13:08:23 +02001252 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
1253 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
Jens Axboe30061b92007-04-17 13:31:34 +02001254 issued r/w: total=0/32768, short=0/0
Jens Axboe8abdce62007-02-21 10:22:55 +01001255 lat (msec): 2=1.6%, 4=0.0%, 10=3.2%, 20=12.8%, 50=38.4%, 100=24.8%,
1256 lat (msec): 250=15.2%, 500=0.0%, 750=0.0%, 1000=0.0%, >=2048=0.0%
Jens Axboe71bfa162006-10-25 11:08:19 +02001257
1258The client number is printed, along with the group id and error of that
1259thread. Below is the io statistics, here for writes. In the order listed,
1260they denote:
1261
1262io= Number of megabytes io performed
1263bw= Average bandwidth rate
Paul Dubs35649e52011-07-21 16:04:52 +02001264iops= Average IOs performed per second
Jens Axboe71bfa162006-10-25 11:08:19 +02001265runt= The runtime of that thread
Jens Axboe72fbda22007-03-20 10:02:06 +01001266 slat= Submission latency (avg being the average, stdev being the
Jens Axboe71bfa162006-10-25 11:08:19 +02001267 standard deviation). This is the time it took to submit
1268 the io. For sync io, the slat is really the completion
Jens Axboe8a35c712007-06-19 09:53:31 +02001269 latency, since queue/complete is one operation there. This
Jens Axboebf9a3ed2008-06-05 11:53:08 +02001270 value can be in milliseconds or microseconds, fio will choose
Jens Axboe8a35c712007-06-19 09:53:31 +02001271 the most appropriate base and print that. In the example
Jens Axboebf9a3ed2008-06-05 11:53:08 +02001272 above, milliseconds is the best scale.
Jens Axboe71bfa162006-10-25 11:08:19 +02001273 clat= Completion latency. Same names as slat, this denotes the
1274 time from submission to completion of the io pieces. For
1275 sync io, clat will usually be equal (or very close) to 0,
1276 as the time from submit to complete is basically just
1277 CPU time (io has already been done, see slat explanation).
1278 bw= Bandwidth. Same names as the xlat stats, but also includes
1279 an approximate percentage of total aggregate bandwidth
1280 this thread received in this group. This last value is
1281 only really useful if the threads in this group are on the
1282 same disk, since they are then competing for disk access.
1283cpu= CPU usage. User and system time, along with the number
Jens Axboee7823a92007-09-07 20:33:33 +02001284 of context switches this thread went through, usage of
1285 system and user time, and finally the number of major
1286 and minor page faults.
Jens Axboe71619dc2007-01-13 23:56:33 +01001287IO depths= The distribution of io depths over the job life time. The
1288 numbers are divided into powers of 2, so for example the
1289 16= entries includes depths up to that value but higher
1290 than the previous entry. In other words, it covers the
1291 range from 16 to 31.
Jens Axboe838bc702008-05-22 13:08:23 +02001292IO submit= How many pieces of IO were submitting in a single submit
1293 call. Each entry denotes that amount and below, until
1294 the previous entry - eg, 8=100% mean that we submitted
1295 anywhere in between 5-8 ios per submit call.
1296IO complete= Like the above submit number, but for completions instead.
Jens Axboe30061b92007-04-17 13:31:34 +02001297IO issued= The number of read/write requests issued, and how many
1298 of them were short.
Jens Axboeec118302007-02-17 04:38:20 +01001299IO latencies= The distribution of IO completion latencies. This is the
1300 time from when IO leaves fio and when it gets completed.
1301 The numbers follow the same pattern as the IO depths,
1302 meaning that 2=1.6% means that 1.6% of the IO completed
Jens Axboe8abdce62007-02-21 10:22:55 +01001303 within 2 msecs, 20=12.8% means that 12.8% of the IO
1304 took more than 10 msecs, but less than (or equal to) 20 msecs.
Jens Axboe71bfa162006-10-25 11:08:19 +02001305
1306After each client has been listed, the group statistics are printed. They
1307will look like this:
1308
1309Run status group 0 (all jobs):
Jens Axboeb22989b2009-07-17 22:29:23 +02001310 READ: io=64MB, aggrb=22178, minb=11355, maxb=11814, mint=2840msec, maxt=2955msec
1311 WRITE: io=64MB, aggrb=1302, minb=666, maxb=669, mint=50093msec, maxt=50320msec
Jens Axboe71bfa162006-10-25 11:08:19 +02001312
1313For each data direction, it prints:
1314
1315io= Number of megabytes io performed.
1316aggrb= Aggregate bandwidth of threads in this group.
1317minb= The minimum average bandwidth a thread saw.
1318maxb= The maximum average bandwidth a thread saw.
1319mint= The smallest runtime of the threads in that group.
1320maxt= The longest runtime of the threads in that group.
1321
1322And finally, the disk statistics are printed. They will look like this:
1323
1324Disk stats (read/write):
1325 sda: ios=16398/16511, merge=30/162, ticks=6853/819634, in_queue=826487, util=100.00%
1326
1327Each value is printed for both reads and writes, with reads first. The
1328numbers denote:
1329
1330ios= Number of ios performed by all groups.
1331merge= Number of merges io the io scheduler.
1332ticks= Number of ticks we kept the disk busy.
1333io_queue= Total time spent in the disk queue.
1334util= The disk utilization. A value of 100% means we kept the disk
1335 busy constantly, 50% would be a disk idling half of the time.
1336
1337
13387.0 Terse output
1339----------------
1340
1341For scripted usage where you typically want to generate tables or graphs
Jens Axboe6af019c2007-03-06 19:50:58 +01001342of the results, fio can output the results in a semicolon separated format.
Jens Axboe71bfa162006-10-25 11:08:19 +02001343The format is one long line of values, such as:
1344
David Nellans562c2d22010-09-23 08:38:17 +020013452;card0;0;0;7139336;121836;60004;1;10109;27.932460;116.933948;220;126861;3495.446807;1085.368601;226;126864;3523.635629;1089.012448;24063;99944;50.275485%;59818.274627;5540.657370;7155060;122104;60004;1;8338;29.086342;117.839068;388;128077;5032.488518;1234.785715;391;128085;5061.839412;1236.909129;23436;100928;50.287926%;59964.832030;5644.844189;14.595833%;19.394167%;123706;0;7313;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.02%;0.05%;0.16%;6.04%;40.40%;52.68%;0.64%;0.01%;0.00%;0.01%;0.00%;0.00%;0.00%;0.00%;0.00%
1346A description of this job goes here.
1347
1348The job description (if provided) follows on a second line.
Jens Axboe71bfa162006-10-25 11:08:19 +02001349
Jens Axboe525c2bf2010-06-30 15:22:21 +02001350To enable terse output, use the --minimal command line option. The first
1351value is the version of the terse output format. If the output has to
1352be changed for some reason, this number will be incremented by 1 to
1353signify that change.
Jens Axboe6820cb32008-09-27 12:33:53 +02001354
Jens Axboe71bfa162006-10-25 11:08:19 +02001355Split up, the format is as follows:
1356
Jens Axboe525c2bf2010-06-30 15:22:21 +02001357 version, jobname, groupid, error
Jens Axboe71bfa162006-10-25 11:08:19 +02001358 READ status:
Jens Axboe85549ba2011-09-13 10:12:12 +02001359 Total IO (KB), bandwidth (KB/sec), runtime (msec)
Jens Axboe71bfa162006-10-25 11:08:19 +02001360 Submission latency: min, max, mean, deviation
1361 Completion latency: min, max, mean, deviation
Jens Axboe525c2bf2010-06-30 15:22:21 +02001362 Total latency: min, max, mean, deviation
Jens Axboe6c219762006-11-03 15:51:45 +01001363 Bw: min, max, aggregate percentage of total, mean, deviation
Jens Axboe71bfa162006-10-25 11:08:19 +02001364 WRITE status:
Jens Axboe85549ba2011-09-13 10:12:12 +02001365 Total IO (KB), bandwidth (KB/sec), runtime (msec)
Jens Axboe71bfa162006-10-25 11:08:19 +02001366 Submission latency: min, max, mean, deviation
1367 Completion latency: min, max, mean, deviation
Jens Axboe525c2bf2010-06-30 15:22:21 +02001368 Total latency: min, max, mean, deviation
Jens Axboe6c219762006-11-03 15:51:45 +01001369 Bw: min, max, aggregate percentage of total, mean, deviation
Shawn Lewis046ee302007-11-21 09:38:34 +01001370 CPU usage: user, system, context switches, major faults, minor faults
Jens Axboe22708902007-03-06 17:05:32 +01001371 IO depths: <=1, 2, 4, 8, 16, 32, >=64
David Nellans562c2d22010-09-23 08:38:17 +02001372 IO latencies microseconds: <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000
1373 IO latencies milliseconds: <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, 2000, >=2000
1374 Additional Info (dependant on continue_on_error, default off): total # errors, first error code
1375
Jens Axboef42195a2010-10-26 08:10:58 -06001376 Additional Info (dependant on description being set): Text description
Paul Dubs25c8b9d2011-07-21 17:26:02 +02001377
1378
13798.0 Trace file format
1380---------------------
1381There are two trace file format that you can encounter. The older (v1) format
1382is unsupported since version 1.20-rc3 (March 2008). It will still be described
1383below in case that you get an old trace and want to understand it.
1384
1385In any case the trace is a simple text file with a single action per line.
1386
1387
13888.1 Trace file format v1
1389------------------------
1390Each line represents a single io action in the following format:
1391
1392rw, offset, length
1393
1394where rw=0/1 for read/write, and the offset and length entries being in bytes.
1395
1396This format is not supported in Fio versions => 1.20-rc3.
1397
1398
13998.2 Trace file format v2
1400------------------------
1401The second version of the trace file format was added in Fio version 1.17.
1402It allows to access more then one file per trace and has a bigger set of
1403possible file actions.
1404
1405The first line of the trace file has to be:
1406
1407fio version 2 iolog
1408
1409Following this can be lines in two different formats, which are described below.
1410
1411The file management format:
1412
1413filename action
1414
1415The filename is given as an absolute path. The action can be one of these:
1416
1417add Add the given filename to the trace
1418open Open the file with the given filename. The filename has to have
1419 been added with the add action before.
1420close Close the file with the given filename. The file has to have been
1421 opened before.
1422
1423
1424The file io action format:
1425
1426filename action offset length
1427
1428The filename is given as an absolute path, and has to have been added and opened
1429before it can be used with this format. The offset and length are given in
1430bytes. The action can be one of these:
1431
1432wait Wait for 'offset' microseconds. Everything below 100 is discarded.
1433read Read 'length' bytes beginning from 'offset'
1434write Write 'length' bytes beginning from 'offset'
1435sync fsync() the file
1436datasync fdatasync() the file
1437trim trim the given file from the given 'offset' for 'length' bytes