blob: 8dd36d0414b43972df303175cef3938de2ae2911 [file] [log] [blame]
Armin Rigoa871ef22006-02-08 12:53:56 +00001\chapter{The Python Profilers \label{profile}}
Fred Drakeea003fc1999-04-05 21:59:15 +00002
3\sectionauthor{James Roskind}{}
Guido van Rossumdf804f81995-03-02 12:38:39 +00004
Fred Drake4b3f0311996-12-13 22:04:31 +00005Copyright \copyright{} 1994, by InfoSeek Corporation, all rights reserved.
Fred Drake5dabeed1998-04-03 07:02:35 +00006\index{InfoSeek Corporation}
Guido van Rossumdf804f81995-03-02 12:38:39 +00007
Fred Drakeea003fc1999-04-05 21:59:15 +00008Written by James Roskind.\footnote{
Armin Rigoa871ef22006-02-08 12:53:56 +00009 Updated and converted to \LaTeX\ by Guido van Rossum.
10 Further updated by Armin Rigo to integrate the documentation for the new
11 \module{cProfile} module of Python 2.5.}
Guido van Rossumdf804f81995-03-02 12:38:39 +000012
13Permission to use, copy, modify, and distribute this Python software
14and its associated documentation for any purpose (subject to the
15restriction in the following sentence) without fee is hereby granted,
16provided that the above copyright notice appears in all copies, and
17that both that copyright notice and this permission notice appear in
18supporting documentation, and that the name of InfoSeek not be used in
19advertising or publicity pertaining to distribution of the software
20without specific, written prior permission. This permission is
21explicitly restricted to the copying and modification of the software
22to remain in Python, compiled Python, or other languages (such as C)
23wherein the modified or derived code is exclusively imported into a
24Python module.
25
26INFOSEEK CORPORATION DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
27SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
28FITNESS. IN NO EVENT SHALL INFOSEEK CORPORATION BE LIABLE FOR ANY
29SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
30RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
31CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
32CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
33
34
35The profiler was written after only programming in Python for 3 weeks.
36As a result, it is probably clumsy code, but I don't know for sure yet
37'cause I'm a beginner :-). I did work hard to make the code run fast,
38so that profiling would be a reasonable thing to do. I tried not to
39repeat code fragments, but I'm sure I did some stuff in really awkward
40ways at times. Please send suggestions for improvements to:
Fred Drake8fa5eb81998-02-27 05:23:37 +000041\email{jar@netscape.com}. I won't promise \emph{any} support. ...but
Guido van Rossumdf804f81995-03-02 12:38:39 +000042I'd appreciate the feedback.
43
44
Armin Rigoa871ef22006-02-08 12:53:56 +000045\section{Introduction to the profilers}
Guido van Rossum86cb0921995-03-20 12:59:56 +000046\nodename{Profiler Introduction}
Guido van Rossumdf804f81995-03-02 12:38:39 +000047
48A \dfn{profiler} is a program that describes the run time performance
49of a program, providing a variety of statistics. This documentation
50describes the profiler functionality provided in the modules
Fred Drake8fa5eb81998-02-27 05:23:37 +000051\module{profile} and \module{pstats}. This profiler provides
Guido van Rossumdf804f81995-03-02 12:38:39 +000052\dfn{deterministic profiling} of any Python programs. It also
53provides a series of report generation tools to allow users to rapidly
54examine the results of a profile operation.
Fred Drake8fa5eb81998-02-27 05:23:37 +000055\index{deterministic profiling}
56\index{profiling, deterministic}
Guido van Rossumdf804f81995-03-02 12:38:39 +000057
Armin Rigoa871ef22006-02-08 12:53:56 +000058The Python standard library provides three different profilers:
59
60\begin{enumerate}
61\item \module{profile}, a pure Python module, described in the sequel.
62 Copyright \copyright{} 1994, by InfoSeek Corporation.
63 \versionchanged[also reports the time spent in calls to built-in
64 functions and methods]{2.4}
65
66\item \module{cProfile}, a module written in C, with a reasonable
67 overhead that makes it suitable for profiling long-running programs.
68 Based on \module{lsprof}, contributed by Brett Rosen and Ted Czotter.
69 \versionadded{2.5}
70
71\item \module{hotshot}, a C module focusing on minimizing the overhead
72 while profiling, at the expense of long data post-processing times.
73 \versionchanged[the results should be more meaningful than in the
74 past: the timing core contained a critical bug]{2.5}
75\end{enumerate}
76
77The \module{profile} and \module{cProfile} modules export the same
78interface, so they are mostly interchangeables; \module{cProfile} has a
79much lower overhead but is not so far as well-tested and might not be
80available on all systems. \module{cProfile} is really a compatibility
81layer on top of the internal \module{_lsprof} module. The
82\module{hotshot} module is reserved to specialized usages.
Guido van Rossumdf804f81995-03-02 12:38:39 +000083
Georg Brandl6c1908d2005-12-26 23:44:29 +000084%\section{How Is This Profiler Different From The Old Profiler?}
85%\nodename{Profiler Changes}
86%
87%(This section is of historical importance only; the old profiler
88%discussed here was last seen in Python 1.1.)
89%
90%The big changes from old profiling module are that you get more
91%information, and you pay less CPU time. It's not a trade-off, it's a
92%trade-up.
93%
94%To be specific:
95%
96%\begin{description}
97%
98%\item[Bugs removed:]
99%Local stack frame is no longer molested, execution time is now charged
100%to correct functions.
101%
102%\item[Accuracy increased:]
103%Profiler execution time is no longer charged to user's code,
104%calibration for platform is supported, file reads are not done \emph{by}
105%profiler \emph{during} profiling (and charged to user's code!).
106%
107%\item[Speed increased:]
108%Overhead CPU cost was reduced by more than a factor of two (perhaps a
109%factor of five), lightweight profiler module is all that must be
110%loaded, and the report generating module (\module{pstats}) is not needed
111%during profiling.
112%
113%\item[Recursive functions support:]
114%Cumulative times in recursive functions are correctly calculated;
115%recursive entries are counted.
116%
117%\item[Large growth in report generating UI:]
118%Distinct profiles runs can be added together forming a comprehensive
119%report; functions that import statistics take arbitrary lists of
120%files; sorting criteria is now based on keywords (instead of 4 integer
121%options); reports shows what functions were profiled as well as what
122%profile file was referenced; output format has been improved.
123%
124%\end{description}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000125
126
Andrew M. Kuchling984bdd72006-04-14 12:07:41 +0000127\section{Instant User's Manual \label{profile-instant}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000128
129This section is provided for users that ``don't want to read the
130manual.'' It provides a very brief overview, and allows a user to
131rapidly perform profiling on an existing application.
132
Fred Drakefee6f332004-03-23 21:40:07 +0000133To profile an application with a main entry point of \function{foo()},
134you would add the following to your module:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000135
Fred Drake19479911998-02-13 06:58:54 +0000136\begin{verbatim}
Armin Rigoa871ef22006-02-08 12:53:56 +0000137import cProfile
138cProfile.run('foo()')
Fred Drake19479911998-02-13 06:58:54 +0000139\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000140
Armin Rigoa871ef22006-02-08 12:53:56 +0000141(Use \module{profile} instead of \module{cProfile} if the latter is not
142available on your system.)
143
Fred Drakefee6f332004-03-23 21:40:07 +0000144The above action would cause \function{foo()} to be run, and a series of
Guido van Rossumdf804f81995-03-02 12:38:39 +0000145informative lines (the profile) to be printed. The above approach is
146most useful when working with the interpreter. If you would like to
147save the results of a profile into a file for later examination, you
Fred Drake8fa5eb81998-02-27 05:23:37 +0000148can supply a file name as the second argument to the \function{run()}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000149function:
150
Fred Drake19479911998-02-13 06:58:54 +0000151\begin{verbatim}
Armin Rigoa871ef22006-02-08 12:53:56 +0000152import cProfile
153cProfile.run('foo()', 'fooprof')
Fred Drake19479911998-02-13 06:58:54 +0000154\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000155
Armin Rigoa871ef22006-02-08 12:53:56 +0000156The file \file{cProfile.py} can also be invoked as
Guido van Rossumbac80021997-06-02 17:29:12 +0000157a script to profile another script. For example:
Fred Drake8fa5eb81998-02-27 05:23:37 +0000158
159\begin{verbatim}
Armin Rigoa871ef22006-02-08 12:53:56 +0000160python -m cProfile myscript.py
Fred Drake8fa5eb81998-02-27 05:23:37 +0000161\end{verbatim}
Guido van Rossumbac80021997-06-02 17:29:12 +0000162
Armin Rigoa871ef22006-02-08 12:53:56 +0000163\file{cProfile.py} accepts two optional arguments on the command line:
Nicholas Bastin824b1b22004-03-23 18:44:39 +0000164
165\begin{verbatim}
Armin Rigoa871ef22006-02-08 12:53:56 +0000166cProfile.py [-o output_file] [-s sort_order]
Nicholas Bastin824b1b22004-03-23 18:44:39 +0000167\end{verbatim}
168
Fred Drakefee6f332004-03-23 21:40:07 +0000169\programopt{-s} only applies to standard output (\programopt{-o} is
170not supplied). Look in the \class{Stats} documentation for valid sort
171values.
Nicholas Bastin824b1b22004-03-23 18:44:39 +0000172
Guido van Rossumdf804f81995-03-02 12:38:39 +0000173When you wish to review the profile, you should use the methods in the
Fred Drake8fa5eb81998-02-27 05:23:37 +0000174\module{pstats} module. Typically you would load the statistics data as
Guido van Rossumdf804f81995-03-02 12:38:39 +0000175follows:
176
Fred Drake19479911998-02-13 06:58:54 +0000177\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000178import pstats
179p = pstats.Stats('fooprof')
Fred Drake19479911998-02-13 06:58:54 +0000180\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000181
Fred Drake8fa5eb81998-02-27 05:23:37 +0000182The class \class{Stats} (the above code just created an instance of
Guido van Rossumdf804f81995-03-02 12:38:39 +0000183this class) has a variety of methods for manipulating and printing the
Fred Drakefee6f332004-03-23 21:40:07 +0000184data that was just read into \code{p}. When you ran
Armin Rigoa871ef22006-02-08 12:53:56 +0000185\function{cProfile.run()} above, what was printed was the result of three
Guido van Rossumdf804f81995-03-02 12:38:39 +0000186method calls:
187
Fred Drake19479911998-02-13 06:58:54 +0000188\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000189p.strip_dirs().sort_stats(-1).print_stats()
Fred Drake19479911998-02-13 06:58:54 +0000190\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000191
Guido van Rossumdf804f81995-03-02 12:38:39 +0000192The first method removed the extraneous path from all the module
193names. The second method sorted all the entries according to the
Armin Rigoa871ef22006-02-08 12:53:56 +0000194standard module/line/name string that is printed.
195%(this is to comply with the semantics of the old profiler).
196The third method printed out
Guido van Rossumdf804f81995-03-02 12:38:39 +0000197all the statistics. You might try the following sort calls:
198
Fred Drake19479911998-02-13 06:58:54 +0000199\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000200p.sort_stats('name')
201p.print_stats()
Fred Drake19479911998-02-13 06:58:54 +0000202\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000203
Guido van Rossumdf804f81995-03-02 12:38:39 +0000204The first call will actually sort the list by function name, and the
205second call will print out the statistics. The following are some
206interesting calls to experiment with:
207
Fred Drake19479911998-02-13 06:58:54 +0000208\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000209p.sort_stats('cumulative').print_stats(10)
Fred Drake19479911998-02-13 06:58:54 +0000210\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000211
Guido van Rossumdf804f81995-03-02 12:38:39 +0000212This sorts the profile by cumulative time in a function, and then only
213prints the ten most significant lines. If you want to understand what
214algorithms are taking time, the above line is what you would use.
215
216If you were looking to see what functions were looping a lot, and
217taking a lot of time, you would do:
218
Fred Drake19479911998-02-13 06:58:54 +0000219\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000220p.sort_stats('time').print_stats(10)
Fred Drake19479911998-02-13 06:58:54 +0000221\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000222
Guido van Rossumdf804f81995-03-02 12:38:39 +0000223to sort according to time spent within each function, and then print
224the statistics for the top ten functions.
225
226You might also try:
227
Fred Drake19479911998-02-13 06:58:54 +0000228\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000229p.sort_stats('file').print_stats('__init__')
Fred Drake19479911998-02-13 06:58:54 +0000230\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000231
Guido van Rossumdf804f81995-03-02 12:38:39 +0000232This will sort all the statistics by file name, and then print out
Fred Drakefee6f332004-03-23 21:40:07 +0000233statistics for only the class init methods (since they are spelled
234with \code{__init__} in them). As one final example, you could try:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000235
Fred Drake19479911998-02-13 06:58:54 +0000236\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000237p.sort_stats('time', 'cum').print_stats(.5, 'init')
Fred Drake19479911998-02-13 06:58:54 +0000238\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000239
Guido van Rossumdf804f81995-03-02 12:38:39 +0000240This line sorts statistics with a primary key of time, and a secondary
241key of cumulative time, and then prints out some of the statistics.
242To be specific, the list is first culled down to 50\% (re: \samp{.5})
243of its original size, then only lines containing \code{init} are
244maintained, and that sub-sub-list is printed.
245
246If you wondered what functions called the above functions, you could
Fred Drakefee6f332004-03-23 21:40:07 +0000247now (\code{p} is still sorted according to the last criteria) do:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000248
Fred Drake19479911998-02-13 06:58:54 +0000249\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000250p.print_callers(.5, 'init')
Fred Drake19479911998-02-13 06:58:54 +0000251\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000252
Tim Peters0a1fc4e2001-10-07 03:12:08 +0000253and you would get a list of callers for each of the listed functions.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000254
255If you want more functionality, you're going to have to read the
256manual, or guess what the following functions do:
257
Fred Drake19479911998-02-13 06:58:54 +0000258\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000259p.print_callees()
260p.add('fooprof')
Fred Drake19479911998-02-13 06:58:54 +0000261\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000262
Eric S. Raymond4f3980d2001-04-13 00:23:01 +0000263Invoked as a script, the \module{pstats} module is a statistics
264browser for reading and examining profile dumps. It has a simple
Fred Drakea3e56a62001-04-13 14:34:58 +0000265line-oriented interface (implemented using \refmodule{cmd}) and
Eric S. Raymond4f3980d2001-04-13 00:23:01 +0000266interactive help.
267
Guido van Rossumdf804f81995-03-02 12:38:39 +0000268\section{What Is Deterministic Profiling?}
Guido van Rossum86cb0921995-03-20 12:59:56 +0000269\nodename{Deterministic Profiling}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000270
271\dfn{Deterministic profiling} is meant to reflect the fact that all
Fred Drakea3e56a62001-04-13 14:34:58 +0000272\emph{function call}, \emph{function return}, and \emph{exception} events
Guido van Rossumdf804f81995-03-02 12:38:39 +0000273are monitored, and precise timings are made for the intervals between
274these events (during which time the user's code is executing). In
275contrast, \dfn{statistical profiling} (which is not done by this
276module) randomly samples the effective instruction pointer, and
277deduces where time is being spent. The latter technique traditionally
278involves less overhead (as the code does not need to be instrumented),
279but provides only relative indications of where time is being spent.
280
281In Python, since there is an interpreter active during execution, the
282presence of instrumented code is not required to do deterministic
283profiling. Python automatically provides a \dfn{hook} (optional
284callback) for each event. In addition, the interpreted nature of
285Python tends to add so much overhead to execution, that deterministic
286profiling tends to only add small processing overhead in typical
287applications. The result is that deterministic profiling is not that
288expensive, yet provides extensive run time statistics about the
289execution of a Python program.
290
291Call count statistics can be used to identify bugs in code (surprising
292counts), and to identify possible inline-expansion points (high call
293counts). Internal time statistics can be used to identify ``hot
294loops'' that should be carefully optimized. Cumulative time
295statistics should be used to identify high level errors in the
296selection of algorithms. Note that the unusual handling of cumulative
297times in this profiler allows statistics for recursive implementations
298of algorithms to be directly compared to iterative implementations.
299
300
Armin Rigoa871ef22006-02-08 12:53:56 +0000301\section{Reference Manual -- \module{profile} and \module{cProfile}}
Fred Drakeb91e9341998-07-23 17:59:49 +0000302
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000303\declaremodule{standard}{profile}
Armin Rigoa871ef22006-02-08 12:53:56 +0000304\declaremodule{standard}{cProfile}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000305\modulesynopsis{Python profiler}
Fred Drakeb91e9341998-07-23 17:59:49 +0000306
Guido van Rossumdf804f81995-03-02 12:38:39 +0000307
Guido van Rossumdf804f81995-03-02 12:38:39 +0000308
309The primary entry point for the profiler is the global function
Armin Rigoa871ef22006-02-08 12:53:56 +0000310\function{profile.run()} (resp. \function{cProfile.run()}).
311It is typically used to create any profile
Guido van Rossumdf804f81995-03-02 12:38:39 +0000312information. The reports are formatted and printed using methods of
Fred Drake8fa5eb81998-02-27 05:23:37 +0000313the class \class{pstats.Stats}. The following is a description of all
Guido van Rossumdf804f81995-03-02 12:38:39 +0000314of these standard entry points and functions. For a more in-depth
315view of some of the code, consider reading the later section on
316Profiler Extensions, which includes discussion of how to derive
317``better'' profilers from the classes presented, or reading the source
318code for these modules.
319
Nicholas Bastin1eb4bfc2004-03-22 20:12:56 +0000320\begin{funcdesc}{run}{command\optional{, filename}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000321
322This function takes a single argument that has can be passed to the
Fred Drake8fa5eb81998-02-27 05:23:37 +0000323\keyword{exec} statement, and an optional file name. In all cases this
324routine attempts to \keyword{exec} its first argument, and gather profiling
Guido van Rossumdf804f81995-03-02 12:38:39 +0000325statistics from the execution. If no file name is present, then this
326function automatically prints a simple profiling report, sorted by the
327standard name string (file/line/function-name) that is presented in
328each line. The following is a typical output from such a call:
329
Fred Drake19479911998-02-13 06:58:54 +0000330\begin{verbatim}
Guido van Rossum96628a91995-04-10 11:34:00 +0000331 2706 function calls (2004 primitive calls) in 4.504 CPU seconds
Guido van Rossumdf804f81995-03-02 12:38:39 +0000332
Guido van Rossum96628a91995-04-10 11:34:00 +0000333Ordered by: standard name
Guido van Rossumdf804f81995-03-02 12:38:39 +0000334
Guido van Rossum96628a91995-04-10 11:34:00 +0000335ncalls tottime percall cumtime percall filename:lineno(function)
336 2 0.006 0.003 0.953 0.477 pobject.py:75(save_objects)
337 43/3 0.533 0.012 0.749 0.250 pobject.py:99(evaluate)
338 ...
Fred Drake19479911998-02-13 06:58:54 +0000339\end{verbatim}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000340
Armin Rigoa871ef22006-02-08 12:53:56 +0000341The first line indicates that 2706 calls were
Guido van Rossumdf804f81995-03-02 12:38:39 +0000342monitored. Of those calls, 2004 were \dfn{primitive}. We define
343\dfn{primitive} to mean that the call was not induced via recursion.
344The next line: \code{Ordered by:\ standard name}, indicates that
345the text string in the far right column was used to sort the output.
346The column headings include:
347
348\begin{description}
349
350\item[ncalls ]
Tim Peters0a1fc4e2001-10-07 03:12:08 +0000351for the number of calls,
Guido van Rossumdf804f81995-03-02 12:38:39 +0000352
353\item[tottime ]
354for the total time spent in the given function (and excluding time
355made in calls to sub-functions),
356
357\item[percall ]
358is the quotient of \code{tottime} divided by \code{ncalls}
359
360\item[cumtime ]
Fred Drake907e76b2001-07-06 20:30:11 +0000361is the total time spent in this and all subfunctions (from invocation
362till exit). This figure is accurate \emph{even} for recursive
Guido van Rossumdf804f81995-03-02 12:38:39 +0000363functions.
364
365\item[percall ]
366is the quotient of \code{cumtime} divided by primitive calls
367
368\item[filename:lineno(function) ]
369provides the respective data of each function
370
371\end{description}
372
Fred Drake907e76b2001-07-06 20:30:11 +0000373When there are two numbers in the first column (for example,
374\samp{43/3}), then the latter is the number of primitive calls, and
375the former is the actual number of calls. Note that when the function
376does not recurse, these two values are the same, and only the single
377figure is printed.
Guido van Rossum96628a91995-04-10 11:34:00 +0000378
Guido van Rossumdf804f81995-03-02 12:38:39 +0000379\end{funcdesc}
380
Nicholas Bastin1eb4bfc2004-03-22 20:12:56 +0000381\begin{funcdesc}{runctx}{command, globals, locals\optional{, filename}}
Armin Rigoa871ef22006-02-08 12:53:56 +0000382This function is similar to \function{run()}, with added
Nicholas Bastin1eb4bfc2004-03-22 20:12:56 +0000383arguments to supply the globals and locals dictionaries for the
384\var{command} string.
385\end{funcdesc}
386
Fred Drake8fa5eb81998-02-27 05:23:37 +0000387Analysis of the profiler data is done using this class from the
388\module{pstats} module:
389
Fred Drake8fe533e1998-03-27 05:27:08 +0000390% now switch modules....
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000391% (This \stmodindex use may be hard to change ;-( )
Fred Drake8fe533e1998-03-27 05:27:08 +0000392\stmodindex{pstats}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000393
George Yoshida51a23fe2006-04-22 15:27:14 +0000394\begin{classdesc}{Stats}{filename\optional{, stream=sys.stdout\optional{, \moreargs}}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000395This class constructor creates an instance of a ``statistics object''
Fred Drake8fa5eb81998-02-27 05:23:37 +0000396from a \var{filename} (or set of filenames). \class{Stats} objects are
Skip Montanaro262fb922006-04-21 02:31:07 +0000397manipulated by methods, in order to print useful reports. You may specify
398an alternate output stream by giving the keyword argument, \code{stream}.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000399
Skip Montanaro262fb922006-04-21 02:31:07 +0000400The file selected by the above constructor must have been created by the
401corresponding version of \module{profile} or \module{cProfile}. To be
402specific, there is \emph{no} file compatibility guaranteed with future
403versions of this profiler, and there is no compatibility with files produced
404by other profilers.
Armin Rigoa871ef22006-02-08 12:53:56 +0000405%(such as the old system profiler).
Guido van Rossumdf804f81995-03-02 12:38:39 +0000406
407If several files are provided, all the statistics for identical
408functions will be coalesced, so that an overall view of several
409processes can be considered in a single report. If additional files
Fred Drake8fa5eb81998-02-27 05:23:37 +0000410need to be combined with data in an existing \class{Stats} object, the
411\method{add()} method can be used.
George Yoshida51a23fe2006-04-22 15:27:14 +0000412
413\versionchanged[The \var{stream} parameter was added]{2.5}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000414\end{classdesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000415
416
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000417\subsection{The \class{Stats} Class \label{profile-stats}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000418
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000419\class{Stats} objects have the following methods:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000420
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000421\begin{methoddesc}[Stats]{strip_dirs}{}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000422This method for the \class{Stats} class removes all leading path
423information from file names. It is very useful in reducing the size
424of the printout to fit within (close to) 80 columns. This method
425modifies the object, and the stripped information is lost. After
426performing a strip operation, the object is considered to have its
427entries in a ``random'' order, as it was just after object
428initialization and loading. If \method{strip_dirs()} causes two
Fred Drake907e76b2001-07-06 20:30:11 +0000429function names to be indistinguishable (they are on the same
Fred Drake8fa5eb81998-02-27 05:23:37 +0000430line of the same filename, and have the same function name), then the
431statistics for these two entries are accumulated into a single entry.
Fred Drake8fe533e1998-03-27 05:27:08 +0000432\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000433
434
Fred Drakee05c3e02004-03-23 20:30:59 +0000435\begin{methoddesc}[Stats]{add}{filename\optional{, \moreargs}}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000436This method of the \class{Stats} class accumulates additional
437profiling information into the current profiling object. Its
438arguments should refer to filenames created by the corresponding
Armin Rigoa871ef22006-02-08 12:53:56 +0000439version of \function{profile.run()} or \function{cProfile.run()}.
440Statistics for identically named
Fred Drake8fa5eb81998-02-27 05:23:37 +0000441(re: file, line, name) functions are automatically accumulated into
442single function statistics.
Fred Drake8fe533e1998-03-27 05:27:08 +0000443\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000444
Fred Drake126d3662003-05-14 14:29:27 +0000445\begin{methoddesc}[Stats]{dump_stats}{filename}
446Save the data loaded into the \class{Stats} object to a file named
447\var{filename}. The file is created if it does not exist, and is
448overwritten if it already exists. This is equivalent to the method of
Armin Rigoa871ef22006-02-08 12:53:56 +0000449the same name on the \class{profile.Profile} and
450\class{cProfile.Profile} classes.
Fred Drake126d3662003-05-14 14:29:27 +0000451\versionadded{2.3}
452\end{methoddesc}
453
Fred Drakee05c3e02004-03-23 20:30:59 +0000454\begin{methoddesc}[Stats]{sort_stats}{key\optional{, \moreargs}}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000455This method modifies the \class{Stats} object by sorting it according
456to the supplied criteria. The argument is typically a string
Fred Drake2cb824c1998-04-09 18:10:35 +0000457identifying the basis of a sort (example: \code{'time'} or
458\code{'name'}).
Guido van Rossumdf804f81995-03-02 12:38:39 +0000459
460When more than one key is provided, then additional keys are used as
Walter Dörwaldf0dfc7a2003-10-20 14:01:56 +0000461secondary criteria when there is equality in all keys selected
Fred Drakefee6f332004-03-23 21:40:07 +0000462before them. For example, \code{sort_stats('name', 'file')} will sort
Fred Drake8fa5eb81998-02-27 05:23:37 +0000463all the entries according to their function name, and resolve all ties
Guido van Rossumdf804f81995-03-02 12:38:39 +0000464(identical function names) by sorting by file name.
465
466Abbreviations can be used for any key names, as long as the
467abbreviation is unambiguous. The following are the keys currently
Tim Peters0a1fc4e2001-10-07 03:12:08 +0000468defined:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000469
Fred Drakeee601911998-04-11 20:53:03 +0000470\begin{tableii}{l|l}{code}{Valid Arg}{Meaning}
Fred Drake5dabeed1998-04-03 07:02:35 +0000471 \lineii{'calls'}{call count}
472 \lineii{'cumulative'}{cumulative time}
473 \lineii{'file'}{file name}
474 \lineii{'module'}{file name}
475 \lineii{'pcalls'}{primitive call count}
476 \lineii{'line'}{line number}
477 \lineii{'name'}{function name}
478 \lineii{'nfl'}{name/file/line}
479 \lineii{'stdname'}{standard name}
480 \lineii{'time'}{internal time}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000481\end{tableii}
482
483Note that all sorts on statistics are in descending order (placing
484most time consuming items first), where as name, file, and line number
Fred Drake907e76b2001-07-06 20:30:11 +0000485searches are in ascending order (alphabetical). The subtle
Fred Drake2cb824c1998-04-09 18:10:35 +0000486distinction between \code{'nfl'} and \code{'stdname'} is that the
Guido van Rossumdf804f81995-03-02 12:38:39 +0000487standard name is a sort of the name as printed, which means that the
488embedded line numbers get compared in an odd way. For example, lines
4893, 20, and 40 would (if the file names were the same) appear in the
Fred Drake2cb824c1998-04-09 18:10:35 +0000490string order 20, 3 and 40. In contrast, \code{'nfl'} does a numeric
491compare of the line numbers. In fact, \code{sort_stats('nfl')} is the
492same as \code{sort_stats('name', 'file', 'line')}.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000493
Armin Rigoa871ef22006-02-08 12:53:56 +0000494%For compatibility with the old profiler,
495For backward-compatibility reasons, the numeric arguments
Fred Drake2cb824c1998-04-09 18:10:35 +0000496\code{-1}, \code{0}, \code{1}, and \code{2} are permitted. They are
497interpreted as \code{'stdname'}, \code{'calls'}, \code{'time'}, and
498\code{'cumulative'} respectively. If this old style format (numeric)
Guido van Rossumdf804f81995-03-02 12:38:39 +0000499is used, only one sort key (the numeric key) will be used, and
500additional arguments will be silently ignored.
Fred Drake8fe533e1998-03-27 05:27:08 +0000501\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000502
503
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000504\begin{methoddesc}[Stats]{reverse_order}{}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000505This method for the \class{Stats} class reverses the ordering of the basic
Armin Rigoa871ef22006-02-08 12:53:56 +0000506list within the object. %This method is provided primarily for
507%compatibility with the old profiler.
508Note that by default ascending vs descending order is properly selected
509based on the sort key of choice.
Fred Drake8fe533e1998-03-27 05:27:08 +0000510\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000511
Fred Drake20006b22001-07-02 21:22:39 +0000512\begin{methoddesc}[Stats]{print_stats}{\optional{restriction, \moreargs}}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000513This method for the \class{Stats} class prints out a report as described
514in the \function{profile.run()} definition.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000515
Fred Drake8fa5eb81998-02-27 05:23:37 +0000516The order of the printing is based on the last \method{sort_stats()}
517operation done on the object (subject to caveats in \method{add()} and
Raymond Hettinger0e53d232003-07-14 18:24:26 +0000518\method{strip_dirs()}).
Guido van Rossumdf804f81995-03-02 12:38:39 +0000519
520The arguments provided (if any) can be used to limit the list down to
521the significant entries. Initially, the list is taken to be the
522complete set of profiled functions. Each restriction is either an
523integer (to select a count of lines), or a decimal fraction between
5240.0 and 1.0 inclusive (to select a percentage of lines), or a regular
Guido van Rossum364e6431997-11-18 15:28:46 +0000525expression (to pattern match the standard name that is printed; as of
526Python 1.5b1, this uses the Perl-style regular expression syntax
Fred Drakeffbe6871999-04-22 21:23:22 +0000527defined by the \refmodule{re} module). If several restrictions are
Guido van Rossum364e6431997-11-18 15:28:46 +0000528provided, then they are applied sequentially. For example:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000529
Fred Drake19479911998-02-13 06:58:54 +0000530\begin{verbatim}
Fred Drake2cb824c1998-04-09 18:10:35 +0000531print_stats(.1, 'foo:')
Fred Drake19479911998-02-13 06:58:54 +0000532\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000533
Guido van Rossumdf804f81995-03-02 12:38:39 +0000534would first limit the printing to first 10\% of list, and then only
Fred Drakefee6f332004-03-23 21:40:07 +0000535print functions that were part of filename \file{.*foo:}. In
Guido van Rossumdf804f81995-03-02 12:38:39 +0000536contrast, the command:
537
Fred Drake19479911998-02-13 06:58:54 +0000538\begin{verbatim}
Fred Drake2cb824c1998-04-09 18:10:35 +0000539print_stats('foo:', .1)
Fred Drake19479911998-02-13 06:58:54 +0000540\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000541
Fred Drakefee6f332004-03-23 21:40:07 +0000542would limit the list to all functions having file names \file{.*foo:},
Guido van Rossumdf804f81995-03-02 12:38:39 +0000543and then proceed to only print the first 10\% of them.
Fred Drake8fe533e1998-03-27 05:27:08 +0000544\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000545
546
Fred Drake20006b22001-07-02 21:22:39 +0000547\begin{methoddesc}[Stats]{print_callers}{\optional{restriction, \moreargs}}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000548This method for the \class{Stats} class prints a list of all functions
Guido van Rossumdf804f81995-03-02 12:38:39 +0000549that called each function in the profiled database. The ordering is
Fred Drake8fa5eb81998-02-27 05:23:37 +0000550identical to that provided by \method{print_stats()}, and the definition
Armin Rigoa871ef22006-02-08 12:53:56 +0000551of the restricting argument is also identical. Each caller is reported on
552its own line. The format differs slightly depending on the profiler that
553produced the stats:
554
555\begin{itemize}
556\item With \module{profile}, a number is shown in parentheses after each
557 caller to show how many times this specific call was made. For
558 convenience, a second non-parenthesized number repeats the cumulative
559 time spent in the function at the right.
560
561\item With \module{cProfile}, each caller is preceeded by three numbers:
562 the number of times this specific call was made, and the total and
563 cumulative times spent in the current function while it was invoked by
564 this specific caller.
565\end{itemize}
Fred Drake8fe533e1998-03-27 05:27:08 +0000566\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000567
Fred Drake20006b22001-07-02 21:22:39 +0000568\begin{methoddesc}[Stats]{print_callees}{\optional{restriction, \moreargs}}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000569This method for the \class{Stats} class prints a list of all function
Guido van Rossumdf804f81995-03-02 12:38:39 +0000570that were called by the indicated function. Aside from this reversal
571of direction of calls (re: called vs was called by), the arguments and
Fred Drake8fa5eb81998-02-27 05:23:37 +0000572ordering are identical to the \method{print_callers()} method.
Fred Drake8fe533e1998-03-27 05:27:08 +0000573\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000574
Guido van Rossumdf804f81995-03-02 12:38:39 +0000575
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000576\section{Limitations \label{profile-limits}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000577
Raymond Hettingerda264122004-12-19 20:31:46 +0000578One limitation has to do with accuracy of timing information.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000579There is a fundamental problem with deterministic profilers involving
580accuracy. The most obvious restriction is that the underlying ``clock''
581is only ticking at a rate (typically) of about .001 seconds. Hence no
Raymond Hettinger999b57c2003-08-25 04:28:05 +0000582measurements will be more accurate than the underlying clock. If
Guido van Rossumdf804f81995-03-02 12:38:39 +0000583enough measurements are taken, then the ``error'' will tend to average
584out. Unfortunately, removing this first error induces a second source
Fred Drakee05c3e02004-03-23 20:30:59 +0000585of error.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000586
587The second problem is that it ``takes a while'' from when an event is
588dispatched until the profiler's call to get the time actually
589\emph{gets} the state of the clock. Similarly, there is a certain lag
590when exiting the profiler event handler from the time that the clock's
591value was obtained (and then squirreled away), until the user's code
592is once again executing. As a result, functions that are called many
593times, or call many functions, will typically accumulate this error.
594The error that accumulates in this fashion is typically less than the
Fred Drake907e76b2001-07-06 20:30:11 +0000595accuracy of the clock (less than one clock tick), but it
Armin Rigoa871ef22006-02-08 12:53:56 +0000596\emph{can} accumulate and become very significant.
597
598The problem is more important with \module{profile} than with the
599lower-overhead \module{cProfile}. For this reason, \module{profile}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000600provides a means of calibrating itself for a given platform so that
Fred Drake907e76b2001-07-06 20:30:11 +0000601this error can be probabilistically (on the average) removed.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000602After the profiler is calibrated, it will be more accurate (in a least
603square sense), but it will sometimes produce negative numbers (when
604call counts are exceptionally low, and the gods of probability work
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000605against you :-). ) Do \emph{not} be alarmed by negative numbers in
Guido van Rossumdf804f81995-03-02 12:38:39 +0000606the profile. They should \emph{only} appear if you have calibrated
607your profiler, and the results are actually better than without
608calibration.
609
610
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000611\section{Calibration \label{profile-calibration}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000612
Armin Rigoa871ef22006-02-08 12:53:56 +0000613The profiler of the \module{profile} module subtracts a constant from each
Guido van Rossumdf804f81995-03-02 12:38:39 +0000614event handling time to compensate for the overhead of calling the time
Tim Peters659a6032001-10-09 20:51:19 +0000615function, and socking away the results. By default, the constant is 0.
616The following procedure can
617be used to obtain a better constant for a given platform (see discussion
Guido van Rossumdf804f81995-03-02 12:38:39 +0000618in section Limitations above).
619
Fred Drake19479911998-02-13 06:58:54 +0000620\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000621import profile
622pr = profile.Profile()
Tim Peters659a6032001-10-09 20:51:19 +0000623for i in range(5):
624 print pr.calibrate(10000)
Fred Drake19479911998-02-13 06:58:54 +0000625\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000626
Tim Peters659a6032001-10-09 20:51:19 +0000627The method executes the number of Python calls given by the argument,
628directly and again under the profiler, measuring the time for both.
629It then computes the hidden overhead per profiler event, and returns
630that as a float. For example, on an 800 MHz Pentium running
631Windows 2000, and using Python's time.clock() as the timer,
632the magical number is about 12.5e-6.
Fred Drake8fa5eb81998-02-27 05:23:37 +0000633
Guido van Rossumdf804f81995-03-02 12:38:39 +0000634The object of this exercise is to get a fairly consistent result.
Tim Peters659a6032001-10-09 20:51:19 +0000635If your computer is \emph{very} fast, or your timer function has poor
636resolution, you might have to pass 100000, or even 1000000, to get
637consistent results.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000638
Tim Peters659a6032001-10-09 20:51:19 +0000639When you have a consistent answer,
640there are three ways you can use it:\footnote{Prior to Python 2.2, it
641 was necessary to edit the profiler source code to embed the bias as
642 a literal number. You still can, but that method is no longer
643 described, because no longer needed.}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000644
Fred Drake19479911998-02-13 06:58:54 +0000645\begin{verbatim}
Tim Peters659a6032001-10-09 20:51:19 +0000646import profile
Guido van Rossume47da0a1997-07-17 16:34:52 +0000647
Tim Peters659a6032001-10-09 20:51:19 +0000648# 1. Apply computed bias to all Profile instances created hereafter.
Tim Peters8cd015c2001-10-09 20:54:23 +0000649profile.Profile.bias = your_computed_bias
Tim Peters659a6032001-10-09 20:51:19 +0000650
651# 2. Apply computed bias to a specific Profile instance.
652pr = profile.Profile()
653pr.bias = your_computed_bias
654
655# 3. Specify computed bias in instance constructor.
656pr = profile.Profile(bias=your_computed_bias)
Fred Drake19479911998-02-13 06:58:54 +0000657\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000658
Tim Peters659a6032001-10-09 20:51:19 +0000659If you have a choice, you are better off choosing a smaller constant, and
660then your results will ``less often'' show up as negative in profile
661statistics.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000662
663
Guido van Rossum86cb0921995-03-20 12:59:56 +0000664\section{Extensions --- Deriving Better Profilers}
665\nodename{Profiler Extensions}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000666
Armin Rigoa871ef22006-02-08 12:53:56 +0000667The \class{Profile} class of both modules, \module{profile} and
668\module{cProfile}, were written so that
Tim Peters0a1fc4e2001-10-07 03:12:08 +0000669derived classes could be developed to extend the profiler. The details
670are not described here, as doing this successfully requires an expert
671understanding of how the \class{Profile} class works internally. Study
Armin Rigoa871ef22006-02-08 12:53:56 +0000672the source code of the module carefully if you want to
Tim Peters0a1fc4e2001-10-07 03:12:08 +0000673pursue this.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000674
Tim Peters0a1fc4e2001-10-07 03:12:08 +0000675If all you want to do is change how current time is determined (for
676example, to force use of wall-clock time or elapsed process time),
677pass the timing function you want to the \class{Profile} class
678constructor:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000679
Fred Drake19479911998-02-13 06:58:54 +0000680\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000681pr = profile.Profile(your_time_func)
Fred Drake19479911998-02-13 06:58:54 +0000682\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000683
Armin Rigoa871ef22006-02-08 12:53:56 +0000684The resulting profiler will then call \function{your_time_func()}.
685
686\begin{description}
687\item[\class{profile.Profile}]
688\function{your_time_func()} should return a single number, or a list of
Tim Peters0a1fc4e2001-10-07 03:12:08 +0000689numbers whose sum is the current time (like what \function{os.times()}
690returns). If the function returns a single time number, or the list of
691returned numbers has length 2, then you will get an especially fast
692version of the dispatch routine.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000693
Tim Peters0a1fc4e2001-10-07 03:12:08 +0000694Be warned that you should calibrate the profiler class for the
Guido van Rossumdf804f81995-03-02 12:38:39 +0000695timer function that you choose. For most machines, a timer that
696returns a lone integer value will provide the best results in terms of
Fred Drake8fa5eb81998-02-27 05:23:37 +0000697low overhead during profiling. (\function{os.times()} is
Tim Peters0a1fc4e2001-10-07 03:12:08 +0000698\emph{pretty} bad, as it returns a tuple of floating point values). If
699you want to substitute a better timer in the cleanest fashion,
700derive a class and hardwire a replacement dispatch method that best
Fred Drake8fa5eb81998-02-27 05:23:37 +0000701handles your timer call, along with the appropriate calibration
Fred Drake62f9d7c2001-06-08 05:04:19 +0000702constant.
Armin Rigoa871ef22006-02-08 12:53:56 +0000703
704\item[\class{cProfile.Profile}]
705\function{your_time_func()} should return a single number. If it returns
706plain integers, you can also invoke the class constructor with a second
707argument specifying the real duration of one unit of time. For example,
708if \function{your_integer_time_func()} returns times measured in thousands
709of seconds, you would constuct the \class{Profile} instance as follows:
710
711\begin{verbatim}
712pr = profile.Profile(your_integer_time_func, 0.001)
713\end{verbatim}
714
715As the \module{cProfile.Profile} class cannot be calibrated, custom
716timer functions should be used with care and should be as fast as
717possible. For the best results with a custom timer, it might be
718necessary to hard-code it in the C source of the internal
719\module{_lsprof} module.
720
721\end{description}