blob: 4a64b4c69d3f9590733b8b5e23899449748fcabf [file] [log] [blame]
Jonathan Corbet4b9033a2016-08-08 16:03:14 -06001.. Copyright 2010 Nicolas Palix <npalix@diku.dk>
2.. Copyright 2010 Julia Lawall <julia@diku.dk>
3.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr>
4
5.. highlight:: none
6
7Coccinelle
8==========
9
10Coccinelle is a tool for pattern matching and text transformation that has
11many uses in kernel development, including the application of complex,
12tree-wide patches and detection of problematic programming patterns.
13
14Getting Coccinelle
15-------------------
16
17The semantic patches included in the kernel use features and options
18which are provided by Coccinelle version 1.0.0-rc11 and above.
19Using earlier versions will fail as the option names used by
20the Coccinelle files and coccicheck have been updated.
21
22Coccinelle is available through the package manager
23of many distributions, e.g. :
24
25 - Debian
26 - Fedora
27 - Ubuntu
28 - OpenSUSE
29 - Arch Linux
30 - NetBSD
31 - FreeBSD
32
33You can get the latest version released from the Coccinelle homepage at
34http://coccinelle.lip6.fr/
35
36Information and tips about Coccinelle are also provided on the wiki
37pages at http://cocci.ekstranet.diku.dk/wiki/doku.php
38
39Once you have it, run the following command::
40
41 ./configure
42 make
43
44as a regular user, and install it with::
45
46 sudo make install
47
48Supplemental documentation
49---------------------------
50
51For supplemental documentation refer to the wiki:
52
53https://bottest.wiki.kernel.org/coccicheck
54
55The wiki documentation always refers to the linux-next version of the script.
56
57Using Coccinelle on the Linux kernel
58------------------------------------
59
60A Coccinelle-specific target is defined in the top level
61Makefile. This target is named ``coccicheck`` and calls the ``coccicheck``
62front-end in the ``scripts`` directory.
63
64Four basic modes are defined: ``patch``, ``report``, ``context``, and
65``org``. The mode to use is specified by setting the MODE variable with
66``MODE=<mode>``.
67
68- ``patch`` proposes a fix, when possible.
69
70- ``report`` generates a list in the following format:
71 file:line:column-column: message
72
73- ``context`` highlights lines of interest and their context in a
74 diff-like style.Lines of interest are indicated with ``-``.
75
76- ``org`` generates a report in the Org mode format of Emacs.
77
78Note that not all semantic patches implement all modes. For easy use
79of Coccinelle, the default mode is "report".
80
81Two other modes provide some common combinations of these modes.
82
83- ``chain`` tries the previous modes in the order above until one succeeds.
84
85- ``rep+ctxt`` runs successively the report mode and the context mode.
86 It should be used with the C option (described later)
87 which checks the code on a file basis.
88
89Examples
90~~~~~~~~
91
92To make a report for every semantic patch, run the following command::
93
94 make coccicheck MODE=report
95
96To produce patches, run::
97
98 make coccicheck MODE=patch
99
100
101The coccicheck target applies every semantic patch available in the
102sub-directories of ``scripts/coccinelle`` to the entire Linux kernel.
103
104For each semantic patch, a commit message is proposed. It gives a
105description of the problem being checked by the semantic patch, and
106includes a reference to Coccinelle.
107
108As any static code analyzer, Coccinelle produces false
109positives. Thus, reports must be carefully checked, and patches
110reviewed.
111
112To enable verbose messages set the V= variable, for example::
113
114 make coccicheck MODE=report V=1
115
116Coccinelle parallelization
117---------------------------
118
119By default, coccicheck tries to run as parallel as possible. To change
120the parallelism, set the J= variable. For example, to run across 4 CPUs::
121
122 make coccicheck MODE=report J=4
123
124As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization,
125if support for this is detected you will benefit from parmap parallelization.
126
127When parmap is enabled coccicheck will enable dynamic load balancing by using
128``--chunksize 1`` argument, this ensures we keep feeding threads with work
129one by one, so that we avoid the situation where most work gets done by only
130a few threads. With dynamic load balancing, if a thread finishes early we keep
131feeding it more work.
132
133When parmap is enabled, if an error occurs in Coccinelle, this error
134value is propagated back, the return value of the ``make coccicheck``
135captures this return value.
136
137Using Coccinelle with a single semantic patch
138---------------------------------------------
139
140The optional make variable COCCI can be used to check a single
141semantic patch. In that case, the variable must be initialized with
142the name of the semantic patch to apply.
143
144For instance::
145
146 make coccicheck COCCI=<my_SP.cocci> MODE=patch
147
148or::
149
150 make coccicheck COCCI=<my_SP.cocci> MODE=report
151
152
153Controlling Which Files are Processed by Coccinelle
154---------------------------------------------------
155
156By default the entire kernel source tree is checked.
157
158To apply Coccinelle to a specific directory, ``M=`` can be used.
159For example, to check drivers/net/wireless/ one may write::
160
161 make coccicheck M=drivers/net/wireless/
162
163To apply Coccinelle on a file basis, instead of a directory basis, the
164following command may be used::
165
166 make C=1 CHECK="scripts/coccicheck"
167
168To check only newly edited code, use the value 2 for the C flag, i.e.::
169
170 make C=2 CHECK="scripts/coccicheck"
171
172In these modes, which works on a file basis, there is no information
173about semantic patches displayed, and no commit message proposed.
174
175This runs every semantic patch in scripts/coccinelle by default. The
176COCCI variable may additionally be used to only apply a single
177semantic patch as shown in the previous section.
178
179The "report" mode is the default. You can select another one with the
180MODE variable explained above.
181
182Debugging Coccinelle SmPL patches
183---------------------------------
184
185Using coccicheck is best as it provides in the spatch command line
186include options matching the options used when we compile the kernel.
187You can learn what these options are by using V=1, you could then
188manually run Coccinelle with debug options added.
189
190Alternatively you can debug running Coccinelle against SmPL patches
191by asking for stderr to be redirected to stderr, by default stderr
192is redirected to /dev/null, if you'd like to capture stderr you
193can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For
194instance::
195
196 rm -f cocci.err
197 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err
198 cat cocci.err
199
200You can use SPFLAGS to add debugging flags, for instance you may want to
201add both --profile --show-trying to SPFLAGS when debugging. For instance
202you may want to use::
203
204 rm -f err.log
205 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
206 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c
207
208err.log will now have the profiling information, while stdout will
209provide some progress information as Coccinelle moves forward with
210work.
211
212DEBUG_FILE support is only supported when using coccinelle >= 1.2.
213
214.cocciconfig support
215--------------------
216
217Coccinelle supports reading .cocciconfig for default Coccinelle options that
218should be used every time spatch is spawned, the order of precedence for
219variables for .cocciconfig is as follows:
220
221- Your current user's home directory is processed first
222- Your directory from which spatch is called is processed next
223- The directory provided with the --dir option is processed last, if used
224
225Since coccicheck runs through make, it naturally runs from the kernel
226proper dir, as such the second rule above would be implied for picking up a
227.cocciconfig when using ``make coccicheck``.
228
229``make coccicheck`` also supports using M= targets.If you do not supply
230any M= target, it is assumed you want to target the entire kernel.
231The kernel coccicheck script has::
232
233 if [ "$KBUILD_EXTMOD" = "" ] ; then
234 OPTIONS="--dir $srctree $COCCIINCLUDE"
235 else
236 OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE"
237 fi
238
239KBUILD_EXTMOD is set when an explicit target with M= is used. For both cases
240the spatch --dir argument is used, as such third rule applies when whether M=
241is used or not, and when M= is used the target directory can have its own
242.cocciconfig file. When M= is not passed as an argument to coccicheck the
243target directory is the same as the directory from where spatch was called.
244
245If not using the kernel's coccicheck target, keep the above precedence
246order logic of .cocciconfig reading. If using the kernel's coccicheck target,
247override any of the kernel's .coccicheck's settings using SPFLAGS.
248
249We help Coccinelle when used against Linux with a set of sensible defaults
250options for Linux with our own Linux .cocciconfig. This hints to coccinelle
251git can be used for ``git grep`` queries over coccigrep. A timeout of 200
252seconds should suffice for now.
253
254The options picked up by coccinelle when reading a .cocciconfig do not appear
255as arguments to spatch processes running on your system, to confirm what
256options will be used by Coccinelle run::
257
258 spatch --print-options-only
259
260You can override with your own preferred index option by using SPFLAGS. Take
261note that when there are conflicting options Coccinelle takes precedence for
262the last options passed. Using .cocciconfig is possible to use idutils, however
263given the order of precedence followed by Coccinelle, since the kernel now
264carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if
265desired. See below section "Additional flags" for more details on how to use
266idutils.
267
268Additional flags
269----------------
270
271Additional flags can be passed to spatch through the SPFLAGS
272variable. This works as Coccinelle respects the last flags
273given to it when options are in conflict. ::
274
275 make SPFLAGS=--use-glimpse coccicheck
276
277Coccinelle supports idutils as well but requires coccinelle >= 1.0.6.
278When no ID file is specified coccinelle assumes your ID database file
279is in the file .id-utils.index on the top level of the kernel, coccinelle
280carries a script scripts/idutils_index.sh which creates the database with::
281
282 mkid -i C --output .id-utils.index
283
284If you have another database filename you can also just symlink with this
285name. ::
286
287 make SPFLAGS=--use-idutils coccicheck
288
289Alternatively you can specify the database filename explicitly, for
290instance::
291
292 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck
293
294See ``spatch --help`` to learn more about spatch options.
295
296Note that the ``--use-glimpse`` and ``--use-idutils`` options
297require external tools for indexing the code. None of them is
298thus active by default. However, by indexing the code with
299one of these tools, and according to the cocci file used,
300spatch could proceed the entire code base more quickly.
301
302SmPL patch specific options
303---------------------------
304
305SmPL patches can have their own requirements for options passed
306to Coccinelle. SmPL patch specific options can be provided by
307providing them at the top of the SmPL patch, for instance::
308
309 // Options: --no-includes --include-headers
310
311SmPL patch Coccinelle requirements
312----------------------------------
313
314As Coccinelle features get added some more advanced SmPL patches
315may require newer versions of Coccinelle. If an SmPL patch requires
316at least a version of Coccinelle, this can be specified as follows,
317as an example if requiring at least Coccinelle >= 1.0.5::
318
319 // Requires: 1.0.5
320
321Proposing new semantic patches
322-------------------------------
323
324New semantic patches can be proposed and submitted by kernel
325developers. For sake of clarity, they should be organized in the
326sub-directories of ``scripts/coccinelle/``.
327
328
329Detailed description of the ``report`` mode
330-------------------------------------------
331
332``report`` generates a list in the following format::
333
334 file:line:column-column: message
335
336Example
337~~~~~~~
338
339Running::
340
341 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci
342
343will execute the following part of the SmPL script::
344
345 <smpl>
346 @r depends on !context && !patch && (org || report)@
347 expression x;
348 position p;
349 @@
350
351 ERR_PTR@p(PTR_ERR(x))
352
353 @script:python depends on report@
354 p << r.p;
355 x << r.x;
356 @@
357
358 msg="ERR_CAST can be used with %s" % (x)
359 coccilib.report.print_report(p[0], msg)
360 </smpl>
361
362This SmPL excerpt generates entries on the standard output, as
363illustrated below::
364
365 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg
366 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth
367 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg
368
369
370Detailed description of the ``patch`` mode
371------------------------------------------
372
373When the ``patch`` mode is available, it proposes a fix for each problem
374identified.
375
376Example
377~~~~~~~
378
379Running::
380
381 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci
382
383will execute the following part of the SmPL script::
384
385 <smpl>
386 @ depends on !context && patch && !org && !report @
387 expression x;
388 @@
389
390 - ERR_PTR(PTR_ERR(x))
391 + ERR_CAST(x)
392 </smpl>
393
394This SmPL excerpt generates patch hunks on the standard output, as
395illustrated below::
396
397 diff -u -p a/crypto/ctr.c b/crypto/ctr.c
398 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200
399 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200
400 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct
401 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER,
402 CRYPTO_ALG_TYPE_MASK);
403 if (IS_ERR(alg))
404 - return ERR_PTR(PTR_ERR(alg));
405 + return ERR_CAST(alg);
406
407 /* Block size must be >= 4 bytes. */
408 err = -EINVAL;
409
410Detailed description of the ``context`` mode
411--------------------------------------------
412
413``context`` highlights lines of interest and their context
414in a diff-like style.
415
416 **NOTE**: The diff-like output generated is NOT an applicable patch. The
417 intent of the ``context`` mode is to highlight the important lines
418 (annotated with minus, ``-``) and gives some surrounding context
419 lines around. This output can be used with the diff mode of
420 Emacs to review the code.
421
422Example
423~~~~~~~
424
425Running::
426
427 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci
428
429will execute the following part of the SmPL script::
430
431 <smpl>
432 @ depends on context && !patch && !org && !report@
433 expression x;
434 @@
435
436 * ERR_PTR(PTR_ERR(x))
437 </smpl>
438
439This SmPL excerpt generates diff hunks on the standard output, as
440illustrated below::
441
442 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing
443 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200
444 +++ /tmp/nothing
445 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct
446 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER,
447 CRYPTO_ALG_TYPE_MASK);
448 if (IS_ERR(alg))
449 - return ERR_PTR(PTR_ERR(alg));
450
451 /* Block size must be >= 4 bytes. */
452 err = -EINVAL;
453
454Detailed description of the ``org`` mode
455----------------------------------------
456
457``org`` generates a report in the Org mode format of Emacs.
458
459Example
460~~~~~~~
461
462Running::
463
464 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci
465
466will execute the following part of the SmPL script::
467
468 <smpl>
469 @r depends on !context && !patch && (org || report)@
470 expression x;
471 position p;
472 @@
473
474 ERR_PTR@p(PTR_ERR(x))
475
476 @script:python depends on org@
477 p << r.p;
478 x << r.x;
479 @@
480
481 msg="ERR_CAST can be used with %s" % (x)
482 msg_safe=msg.replace("[","@(").replace("]",")")
483 coccilib.org.print_todo(p[0], msg_safe)
484 </smpl>
485
486This SmPL excerpt generates Org entries on the standard output, as
487illustrated below::
488
489 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]]
490 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]]
491 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]]