blob: 09bd8e9029892e4e1d48078de4d076e24eff3dd2 [file] [log] [blame]
Theodore Ts'oabd41442009-04-11 15:51:18 -04001 Event Tracing
2
3 Documentation written by Theodore Ts'o
Tom Zanussi95b69602009-09-10 23:13:51 -05004 Updated by Li Zefan and Tom Zanussi
Theodore Ts'oabd41442009-04-11 15:51:18 -04005
Li Zefan143c1452009-05-19 14:43:15 +080061. Introduction
7===============
Theodore Ts'oabd41442009-04-11 15:51:18 -04008
9Tracepoints (see Documentation/trace/tracepoints.txt) can be used
10without creating custom kernel modules to register probe functions
11using the event tracing infrastructure.
12
13Not all tracepoints can be traced using the event tracing system;
14the kernel developer must provide code snippets which define how the
15tracing information is saved into the tracing buffer, and how the
Li Zefan143c1452009-05-19 14:43:15 +080016tracing information should be printed.
Theodore Ts'oabd41442009-04-11 15:51:18 -040017
Li Zefan143c1452009-05-19 14:43:15 +0800182. Using Event Tracing
19======================
20
212.1 Via the 'set_event' interface
22---------------------------------
Theodore Ts'oabd41442009-04-11 15:51:18 -040023
24The events which are available for tracing can be found in the file
GeunSik Lim52ad51e2009-09-07 21:37:17 +090025/sys/kernel/debug/tracing/available_events.
Theodore Ts'oabd41442009-04-11 15:51:18 -040026
27To enable a particular event, such as 'sched_wakeup', simply echo it
GeunSik Lim52ad51e2009-09-07 21:37:17 +090028to /sys/kernel/debug/tracing/set_event. For example:
Theodore Ts'oabd41442009-04-11 15:51:18 -040029
GeunSik Lim52ad51e2009-09-07 21:37:17 +090030 # echo sched_wakeup >> /sys/kernel/debug/tracing/set_event
Theodore Ts'oabd41442009-04-11 15:51:18 -040031
Li Zefan143c1452009-05-19 14:43:15 +080032[ Note: '>>' is necessary, otherwise it will firstly disable
33 all the events. ]
Theodore Ts'oabd41442009-04-11 15:51:18 -040034
35To disable an event, echo the event name to the set_event file prefixed
36with an exclamation point:
37
GeunSik Lim52ad51e2009-09-07 21:37:17 +090038 # echo '!sched_wakeup' >> /sys/kernel/debug/tracing/set_event
Theodore Ts'oabd41442009-04-11 15:51:18 -040039
Li Zefan143c1452009-05-19 14:43:15 +080040To disable all events, echo an empty line to the set_event file:
Theodore Ts'oabd41442009-04-11 15:51:18 -040041
GeunSik Lim52ad51e2009-09-07 21:37:17 +090042 # echo > /sys/kernel/debug/tracing/set_event
Li Zefan143c1452009-05-19 14:43:15 +080043
44To enable all events, echo '*:*' or '*:' to the set_event file:
45
GeunSik Lim52ad51e2009-09-07 21:37:17 +090046 # echo *:* > /sys/kernel/debug/tracing/set_event
Theodore Ts'oabd41442009-04-11 15:51:18 -040047
48The events are organized into subsystems, such as ext4, irq, sched,
49etc., and a full event name looks like this: <subsystem>:<event>. The
50subsystem name is optional, but it is displayed in the available_events
51file. All of the events in a subsystem can be specified via the syntax
52"<subsystem>:*"; for example, to enable all irq events, you can use the
53command:
54
GeunSik Lim52ad51e2009-09-07 21:37:17 +090055 # echo 'irq:*' > /sys/kernel/debug/tracing/set_event
Theodore Ts'oabd41442009-04-11 15:51:18 -040056
Li Zefan143c1452009-05-19 14:43:15 +0800572.2 Via the 'enable' toggle
58---------------------------
Theodore Ts'oabd41442009-04-11 15:51:18 -040059
GeunSik Lim52ad51e2009-09-07 21:37:17 +090060The events available are also listed in /sys/kernel/debug/tracing/events/ hierarchy
Li Zefan143c1452009-05-19 14:43:15 +080061of directories.
Theodore Ts'oabd41442009-04-11 15:51:18 -040062
Li Zefan143c1452009-05-19 14:43:15 +080063To enable event 'sched_wakeup':
Theodore Ts'oabd41442009-04-11 15:51:18 -040064
GeunSik Lim52ad51e2009-09-07 21:37:17 +090065 # echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
Theodore Ts'oabd41442009-04-11 15:51:18 -040066
Li Zefan143c1452009-05-19 14:43:15 +080067To disable it:
Theodore Ts'oabd41442009-04-11 15:51:18 -040068
GeunSik Lim52ad51e2009-09-07 21:37:17 +090069 # echo 0 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
Theodore Ts'oabd41442009-04-11 15:51:18 -040070
Li Zefan143c1452009-05-19 14:43:15 +080071To enable all events in sched subsystem:
Theodore Ts'oabd41442009-04-11 15:51:18 -040072
GeunSik Lim52ad51e2009-09-07 21:37:17 +090073 # echo 1 > /sys/kernel/debug/tracing/events/sched/enable
Theodore Ts'oabd41442009-04-11 15:51:18 -040074
Michal Sojka6afb1c62009-09-10 08:02:21 +020075To enable all events:
Theodore Ts'oabd41442009-04-11 15:51:18 -040076
GeunSik Lim52ad51e2009-09-07 21:37:17 +090077 # echo 1 > /sys/kernel/debug/tracing/events/enable
Theodore Ts'oabd41442009-04-11 15:51:18 -040078
Li Zefan143c1452009-05-19 14:43:15 +080079When reading one of these enable files, there are four results:
Theodore Ts'oabd41442009-04-11 15:51:18 -040080
Li Zefan143c1452009-05-19 14:43:15 +080081 0 - all events this file affects are disabled
82 1 - all events this file affects are enabled
83 X - there is a mixture of events enabled and disabled
84 ? - this file does not affect any event
Theodore Ts'oabd41442009-04-11 15:51:18 -040085
Li Zefan020e5f82009-07-01 10:47:05 +0800862.3 Boot option
87---------------
88
89In order to facilitate early boot debugging, use boot option:
90
91 trace_event=[event-list]
92
Li Zefan03d646e2009-12-21 14:27:24 +080093event-list is a comma separated list of events. See section 2.1 for event
94format.
Li Zefan020e5f82009-07-01 10:47:05 +080095
Li Zefan143c1452009-05-19 14:43:15 +0800963. Defining an event-enabled tracepoint
97=======================================
Theodore Ts'oabd41442009-04-11 15:51:18 -040098
Li Zefan143c1452009-05-19 14:43:15 +080099See The example provided in samples/trace_events
Theodore Ts'oabd41442009-04-11 15:51:18 -0400100
Tom Zanussi95b69602009-09-10 23:13:51 -05001014. Event formats
102================
103
104Each trace event has a 'format' file associated with it that contains
105a description of each field in a logged event. This information can
106be used to parse the binary trace stream, and is also the place to
107find the field names that can be used in event filters (see section 5).
108
109It also displays the format string that will be used to print the
110event in text mode, along with the event name and ID used for
111profiling.
112
113Every event has a set of 'common' fields associated with it; these are
114the fields prefixed with 'common_'. The other fields vary between
115events and correspond to the fields defined in the TRACE_EVENT
116definition for that event.
117
118Each field in the format has the form:
119
120 field:field-type field-name; offset:N; size:N;
121
122where offset is the offset of the field in the trace record and size
123is the size of the data item, in bytes.
124
125For example, here's the information displayed for the 'sched_wakeup'
126event:
127
128# cat /debug/tracing/events/sched/sched_wakeup/format
129
130name: sched_wakeup
131ID: 60
132format:
133 field:unsigned short common_type; offset:0; size:2;
134 field:unsigned char common_flags; offset:2; size:1;
135 field:unsigned char common_preempt_count; offset:3; size:1;
136 field:int common_pid; offset:4; size:4;
137 field:int common_tgid; offset:8; size:4;
138
139 field:char comm[TASK_COMM_LEN]; offset:12; size:16;
140 field:pid_t pid; offset:28; size:4;
141 field:int prio; offset:32; size:4;
142 field:int success; offset:36; size:4;
143 field:int cpu; offset:40; size:4;
144
145print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid,
146 REC->prio, REC->success, REC->cpu
147
148This event contains 10 fields, the first 5 common and the remaining 5
149event-specific. All the fields for this event are numeric, except for
150'comm' which is a string, a distinction important for event filtering.
151
1525. Event filtering
153==================
154
155Trace events can be filtered in the kernel by associating boolean
156'filter expressions' with them. As soon as an event is logged into
157the trace buffer, its fields are checked against the filter expression
158associated with that event type. An event with field values that
159'match' the filter will appear in the trace output, and an event whose
160values don't match will be discarded. An event with no filter
161associated with it matches everything, and is the default when no
162filter has been set for an event.
163
1645.1 Expression syntax
165---------------------
166
167A filter expression consists of one or more 'predicates' that can be
168combined using the logical operators '&&' and '||'. A predicate is
169simply a clause that compares the value of a field contained within a
170logged event with a constant value and returns either 0 or 1 depending
171on whether the field value matched (1) or didn't match (0):
172
173 field-name relational-operator value
174
175Parentheses can be used to provide arbitrary logical groupings and
176double-quotes can be used to prevent the shell from interpreting
177operators as shell metacharacters.
178
179The field-names available for use in filters can be found in the
180'format' files for trace events (see section 4).
181
182The relational-operators depend on the type of the field being tested:
183
184The operators available for numeric fields are:
185
186==, !=, <, <=, >, >=
187
188And for string fields they are:
189
190==, !=
191
192Currently, only exact string matches are supported.
193
194Currently, the maximum number of predicates in a filter is 16.
195
1965.2 Setting filters
197-------------------
198
199A filter for an individual event is set by writing a filter expression
200to the 'filter' file for the given event.
201
202For example:
203
204# cd /debug/tracing/events/sched/sched_wakeup
205# echo "common_preempt_count > 4" > filter
206
207A slightly more involved example:
208
209# cd /debug/tracing/events/sched/sched_signal_send
210# echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter
211
212If there is an error in the expression, you'll get an 'Invalid
213argument' error when setting it, and the erroneous string along with
214an error message can be seen by looking at the filter e.g.:
215
216# cd /debug/tracing/events/sched/sched_signal_send
217# echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter
218-bash: echo: write error: Invalid argument
219# cat filter
220((sig >= 10 && sig < 15) || dsig == 17) && comm != bash
221^
222parse_error: Field not found
223
224Currently the caret ('^') for an error always appears at the beginning of
225the filter string; the error message should still be useful though
226even without more accurate position info.
227
2285.3 Clearing filters
229--------------------
230
231To clear the filter for an event, write a '0' to the event's filter
232file.
233
234To clear the filters for all events in a subsystem, write a '0' to the
235subsystem's filter file.
236
2375.3 Subsystem filters
238---------------------
239
240For convenience, filters for every event in a subsystem can be set or
241cleared as a group by writing a filter expression into the filter file
Thomas Weber88393162010-03-16 11:47:56 +0100242at the root of the subsystem. Note however, that if a filter for any
Tom Zanussi95b69602009-09-10 23:13:51 -0500243event within the subsystem lacks a field specified in the subsystem
244filter, or if the filter can't be applied for any other reason, the
245filter for that event will retain its previous setting. This can
246result in an unintended mixture of filters which could lead to
247confusing (to the user who might think different filters are in
248effect) trace output. Only filters that reference just the common
249fields can be guaranteed to propagate successfully to all events.
250
251Here are a few subsystem filter examples that also illustrate the
252above points:
253
Thomas Weber88393162010-03-16 11:47:56 +0100254Clear the filters on all events in the sched subsystem:
Tom Zanussi95b69602009-09-10 23:13:51 -0500255
256# cd /sys/kernel/debug/tracing/events/sched
257# echo 0 > filter
258# cat sched_switch/filter
259none
260# cat sched_wakeup/filter
261none
262
263Set a filter using only common fields for all events in the sched
Thomas Weber88393162010-03-16 11:47:56 +0100264subsystem (all events end up with the same filter):
Tom Zanussi95b69602009-09-10 23:13:51 -0500265
266# cd /sys/kernel/debug/tracing/events/sched
267# echo common_pid == 0 > filter
268# cat sched_switch/filter
269common_pid == 0
270# cat sched_wakeup/filter
271common_pid == 0
272
273Attempt to set a filter using a non-common field for all events in the
Thomas Weber88393162010-03-16 11:47:56 +0100274sched subsystem (all events but those that have a prev_pid field retain
Tom Zanussi95b69602009-09-10 23:13:51 -0500275their old filters):
276
277# cd /sys/kernel/debug/tracing/events/sched
278# echo prev_pid == 0 > filter
279# cat sched_switch/filter
280prev_pid == 0
281# cat sched_wakeup/filter
282common_pid == 0