blob: 335c02f88017f86b54dc0a105dc84aed7ed8dede [file] [log] [blame]
Fred Drakeea003fc1999-04-05 21:59:15 +00001\chapter{The Python Profiler \label{profile}}
2
3\sectionauthor{James Roskind}{}
Guido van Rossumdf804f81995-03-02 12:38:39 +00004
Fred Drake4b3f0311996-12-13 22:04:31 +00005Copyright \copyright{} 1994, by InfoSeek Corporation, all rights reserved.
Fred Drake5dabeed1998-04-03 07:02:35 +00006\index{InfoSeek Corporation}
Guido van Rossumdf804f81995-03-02 12:38:39 +00007
Fred Drakeea003fc1999-04-05 21:59:15 +00008Written by James Roskind.\footnote{
9 Updated and converted to \LaTeX\ by Guido van Rossum. The references to
10 the old profiler are left in the text, although it no longer exists.}
Guido van Rossumdf804f81995-03-02 12:38:39 +000011
12Permission to use, copy, modify, and distribute this Python software
13and its associated documentation for any purpose (subject to the
14restriction in the following sentence) without fee is hereby granted,
15provided that the above copyright notice appears in all copies, and
16that both that copyright notice and this permission notice appear in
17supporting documentation, and that the name of InfoSeek not be used in
18advertising or publicity pertaining to distribution of the software
19without specific, written prior permission. This permission is
20explicitly restricted to the copying and modification of the software
21to remain in Python, compiled Python, or other languages (such as C)
22wherein the modified or derived code is exclusively imported into a
23Python module.
24
25INFOSEEK CORPORATION DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
26SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
27FITNESS. IN NO EVENT SHALL INFOSEEK CORPORATION BE LIABLE FOR ANY
28SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
29RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
30CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
31CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
32
33
34The profiler was written after only programming in Python for 3 weeks.
35As a result, it is probably clumsy code, but I don't know for sure yet
36'cause I'm a beginner :-). I did work hard to make the code run fast,
37so that profiling would be a reasonable thing to do. I tried not to
38repeat code fragments, but I'm sure I did some stuff in really awkward
39ways at times. Please send suggestions for improvements to:
Fred Drake8fa5eb81998-02-27 05:23:37 +000040\email{jar@netscape.com}. I won't promise \emph{any} support. ...but
Guido van Rossumdf804f81995-03-02 12:38:39 +000041I'd appreciate the feedback.
42
43
Guido van Rossum470be141995-03-17 16:07:09 +000044\section{Introduction to the profiler}
Guido van Rossum86cb0921995-03-20 12:59:56 +000045\nodename{Profiler Introduction}
Guido van Rossumdf804f81995-03-02 12:38:39 +000046
47A \dfn{profiler} is a program that describes the run time performance
48of a program, providing a variety of statistics. This documentation
49describes the profiler functionality provided in the modules
Fred Drake8fa5eb81998-02-27 05:23:37 +000050\module{profile} and \module{pstats}. This profiler provides
Guido van Rossumdf804f81995-03-02 12:38:39 +000051\dfn{deterministic profiling} of any Python programs. It also
52provides a series of report generation tools to allow users to rapidly
53examine the results of a profile operation.
Fred Drake8fa5eb81998-02-27 05:23:37 +000054\index{deterministic profiling}
55\index{profiling, deterministic}
Guido van Rossumdf804f81995-03-02 12:38:39 +000056
57
58\section{How Is This Profiler Different From The Old Profiler?}
Guido van Rossum86cb0921995-03-20 12:59:56 +000059\nodename{Profiler Changes}
Guido van Rossumdf804f81995-03-02 12:38:39 +000060
Guido van Rossum364e6431997-11-18 15:28:46 +000061(This section is of historical importance only; the old profiler
62discussed here was last seen in Python 1.1.)
63
Guido van Rossumdf804f81995-03-02 12:38:39 +000064The big changes from old profiling module are that you get more
65information, and you pay less CPU time. It's not a trade-off, it's a
66trade-up.
67
68To be specific:
69
70\begin{description}
71
72\item[Bugs removed:]
73Local stack frame is no longer molested, execution time is now charged
74to correct functions.
75
76\item[Accuracy increased:]
77Profiler execution time is no longer charged to user's code,
78calibration for platform is supported, file reads are not done \emph{by}
79profiler \emph{during} profiling (and charged to user's code!).
80
81\item[Speed increased:]
82Overhead CPU cost was reduced by more than a factor of two (perhaps a
83factor of five), lightweight profiler module is all that must be
Fred Drake8fa5eb81998-02-27 05:23:37 +000084loaded, and the report generating module (\module{pstats}) is not needed
Guido van Rossumdf804f81995-03-02 12:38:39 +000085during profiling.
86
87\item[Recursive functions support:]
88Cumulative times in recursive functions are correctly calculated;
89recursive entries are counted.
90
91\item[Large growth in report generating UI:]
92Distinct profiles runs can be added together forming a comprehensive
93report; functions that import statistics take arbitrary lists of
94files; sorting criteria is now based on keywords (instead of 4 integer
95options); reports shows what functions were profiled as well as what
96profile file was referenced; output format has been improved.
97
98\end{description}
99
100
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000101\section{Instant Users Manual \label{profile-instant}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000102
103This section is provided for users that ``don't want to read the
104manual.'' It provides a very brief overview, and allows a user to
105rapidly perform profiling on an existing application.
106
107To profile an application with a main entry point of \samp{foo()}, you
108would add the following to your module:
109
Fred Drake19479911998-02-13 06:58:54 +0000110\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000111import profile
Fred Drake2cb824c1998-04-09 18:10:35 +0000112profile.run('foo()')
Fred Drake19479911998-02-13 06:58:54 +0000113\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000114
Guido van Rossumdf804f81995-03-02 12:38:39 +0000115The above action would cause \samp{foo()} to be run, and a series of
116informative lines (the profile) to be printed. The above approach is
117most useful when working with the interpreter. If you would like to
118save the results of a profile into a file for later examination, you
Fred Drake8fa5eb81998-02-27 05:23:37 +0000119can supply a file name as the second argument to the \function{run()}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000120function:
121
Fred Drake19479911998-02-13 06:58:54 +0000122\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000123import profile
Fred Drake2cb824c1998-04-09 18:10:35 +0000124profile.run('foo()', 'fooprof')
Fred Drake19479911998-02-13 06:58:54 +0000125\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000126
Fred Drake8fa5eb81998-02-27 05:23:37 +0000127The file \file{profile.py} can also be invoked as
Guido van Rossumbac80021997-06-02 17:29:12 +0000128a script to profile another script. For example:
Fred Drake8fa5eb81998-02-27 05:23:37 +0000129
130\begin{verbatim}
Fred Drake5dabeed1998-04-03 07:02:35 +0000131python /usr/local/lib/python1.5/profile.py myscript.py
Fred Drake8fa5eb81998-02-27 05:23:37 +0000132\end{verbatim}
Guido van Rossumbac80021997-06-02 17:29:12 +0000133
Guido van Rossumdf804f81995-03-02 12:38:39 +0000134When you wish to review the profile, you should use the methods in the
Fred Drake8fa5eb81998-02-27 05:23:37 +0000135\module{pstats} module. Typically you would load the statistics data as
Guido van Rossumdf804f81995-03-02 12:38:39 +0000136follows:
137
Fred Drake19479911998-02-13 06:58:54 +0000138\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000139import pstats
140p = pstats.Stats('fooprof')
Fred Drake19479911998-02-13 06:58:54 +0000141\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000142
Fred Drake8fa5eb81998-02-27 05:23:37 +0000143The class \class{Stats} (the above code just created an instance of
Guido van Rossumdf804f81995-03-02 12:38:39 +0000144this class) has a variety of methods for manipulating and printing the
145data that was just read into \samp{p}. When you ran
Fred Drake8fa5eb81998-02-27 05:23:37 +0000146\function{profile.run()} above, what was printed was the result of three
Guido van Rossumdf804f81995-03-02 12:38:39 +0000147method calls:
148
Fred Drake19479911998-02-13 06:58:54 +0000149\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000150p.strip_dirs().sort_stats(-1).print_stats()
Fred Drake19479911998-02-13 06:58:54 +0000151\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000152
Guido van Rossumdf804f81995-03-02 12:38:39 +0000153The first method removed the extraneous path from all the module
154names. The second method sorted all the entries according to the
155standard module/line/name string that is printed (this is to comply
156with the semantics of the old profiler). The third method printed out
157all the statistics. You might try the following sort calls:
158
Fred Drake19479911998-02-13 06:58:54 +0000159\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000160p.sort_stats('name')
161p.print_stats()
Fred Drake19479911998-02-13 06:58:54 +0000162\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000163
Guido van Rossumdf804f81995-03-02 12:38:39 +0000164The first call will actually sort the list by function name, and the
165second call will print out the statistics. The following are some
166interesting calls to experiment with:
167
Fred Drake19479911998-02-13 06:58:54 +0000168\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000169p.sort_stats('cumulative').print_stats(10)
Fred Drake19479911998-02-13 06:58:54 +0000170\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000171
Guido van Rossumdf804f81995-03-02 12:38:39 +0000172This sorts the profile by cumulative time in a function, and then only
173prints the ten most significant lines. If you want to understand what
174algorithms are taking time, the above line is what you would use.
175
176If you were looking to see what functions were looping a lot, and
177taking a lot of time, you would do:
178
Fred Drake19479911998-02-13 06:58:54 +0000179\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000180p.sort_stats('time').print_stats(10)
Fred Drake19479911998-02-13 06:58:54 +0000181\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000182
Guido van Rossumdf804f81995-03-02 12:38:39 +0000183to sort according to time spent within each function, and then print
184the statistics for the top ten functions.
185
186You might also try:
187
Fred Drake19479911998-02-13 06:58:54 +0000188\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000189p.sort_stats('file').print_stats('__init__')
Fred Drake19479911998-02-13 06:58:54 +0000190\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000191
Guido van Rossumdf804f81995-03-02 12:38:39 +0000192This will sort all the statistics by file name, and then print out
193statistics for only the class init methods ('cause they are spelled
Fred Drake8fa5eb81998-02-27 05:23:37 +0000194with \samp{__init__} in them). As one final example, you could try:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000195
Fred Drake19479911998-02-13 06:58:54 +0000196\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000197p.sort_stats('time', 'cum').print_stats(.5, 'init')
Fred Drake19479911998-02-13 06:58:54 +0000198\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000199
Guido van Rossumdf804f81995-03-02 12:38:39 +0000200This line sorts statistics with a primary key of time, and a secondary
201key of cumulative time, and then prints out some of the statistics.
202To be specific, the list is first culled down to 50\% (re: \samp{.5})
203of its original size, then only lines containing \code{init} are
204maintained, and that sub-sub-list is printed.
205
206If you wondered what functions called the above functions, you could
207now (\samp{p} is still sorted according to the last criteria) do:
208
Fred Drake19479911998-02-13 06:58:54 +0000209\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000210p.print_callers(.5, 'init')
Fred Drake19479911998-02-13 06:58:54 +0000211\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000212
Guido van Rossumdf804f81995-03-02 12:38:39 +0000213and you would get a list of callers for each of the listed functions.
214
215If you want more functionality, you're going to have to read the
216manual, or guess what the following functions do:
217
Fred Drake19479911998-02-13 06:58:54 +0000218\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000219p.print_callees()
220p.add('fooprof')
Fred Drake19479911998-02-13 06:58:54 +0000221\end{verbatim}
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000222
Guido van Rossumdf804f81995-03-02 12:38:39 +0000223\section{What Is Deterministic Profiling?}
Guido van Rossum86cb0921995-03-20 12:59:56 +0000224\nodename{Deterministic Profiling}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000225
226\dfn{Deterministic profiling} is meant to reflect the fact that all
227\dfn{function call}, \dfn{function return}, and \dfn{exception} events
228are monitored, and precise timings are made for the intervals between
229these events (during which time the user's code is executing). In
230contrast, \dfn{statistical profiling} (which is not done by this
231module) randomly samples the effective instruction pointer, and
232deduces where time is being spent. The latter technique traditionally
233involves less overhead (as the code does not need to be instrumented),
234but provides only relative indications of where time is being spent.
235
236In Python, since there is an interpreter active during execution, the
237presence of instrumented code is not required to do deterministic
238profiling. Python automatically provides a \dfn{hook} (optional
239callback) for each event. In addition, the interpreted nature of
240Python tends to add so much overhead to execution, that deterministic
241profiling tends to only add small processing overhead in typical
242applications. The result is that deterministic profiling is not that
243expensive, yet provides extensive run time statistics about the
244execution of a Python program.
245
246Call count statistics can be used to identify bugs in code (surprising
247counts), and to identify possible inline-expansion points (high call
248counts). Internal time statistics can be used to identify ``hot
249loops'' that should be carefully optimized. Cumulative time
250statistics should be used to identify high level errors in the
251selection of algorithms. Note that the unusual handling of cumulative
252times in this profiler allows statistics for recursive implementations
253of algorithms to be directly compared to iterative implementations.
254
255
256\section{Reference Manual}
Fred Drakeb91e9341998-07-23 17:59:49 +0000257
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000258\declaremodule{standard}{profile}
259\modulesynopsis{Python profiler}
Fred Drakeb91e9341998-07-23 17:59:49 +0000260
Guido van Rossumdf804f81995-03-02 12:38:39 +0000261
Guido van Rossumdf804f81995-03-02 12:38:39 +0000262
263The primary entry point for the profiler is the global function
Fred Drake8fa5eb81998-02-27 05:23:37 +0000264\function{profile.run()}. It is typically used to create any profile
Guido van Rossumdf804f81995-03-02 12:38:39 +0000265information. The reports are formatted and printed using methods of
Fred Drake8fa5eb81998-02-27 05:23:37 +0000266the class \class{pstats.Stats}. The following is a description of all
Guido van Rossumdf804f81995-03-02 12:38:39 +0000267of these standard entry points and functions. For a more in-depth
268view of some of the code, consider reading the later section on
269Profiler Extensions, which includes discussion of how to derive
270``better'' profilers from the classes presented, or reading the source
271code for these modules.
272
Fred Drake8fe533e1998-03-27 05:27:08 +0000273\begin{funcdesc}{run}{string\optional{, filename\optional{, ...}}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000274
275This function takes a single argument that has can be passed to the
Fred Drake8fa5eb81998-02-27 05:23:37 +0000276\keyword{exec} statement, and an optional file name. In all cases this
277routine attempts to \keyword{exec} its first argument, and gather profiling
Guido van Rossumdf804f81995-03-02 12:38:39 +0000278statistics from the execution. If no file name is present, then this
279function automatically prints a simple profiling report, sorted by the
280standard name string (file/line/function-name) that is presented in
281each line. The following is a typical output from such a call:
282
Fred Drake19479911998-02-13 06:58:54 +0000283\begin{verbatim}
Guido van Rossum96628a91995-04-10 11:34:00 +0000284 main()
285 2706 function calls (2004 primitive calls) in 4.504 CPU seconds
Guido van Rossumdf804f81995-03-02 12:38:39 +0000286
Guido van Rossum96628a91995-04-10 11:34:00 +0000287Ordered by: standard name
Guido van Rossumdf804f81995-03-02 12:38:39 +0000288
Guido van Rossum96628a91995-04-10 11:34:00 +0000289ncalls tottime percall cumtime percall filename:lineno(function)
290 2 0.006 0.003 0.953 0.477 pobject.py:75(save_objects)
291 43/3 0.533 0.012 0.749 0.250 pobject.py:99(evaluate)
292 ...
Fred Drake19479911998-02-13 06:58:54 +0000293\end{verbatim}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000294
295The first line indicates that this profile was generated by the call:\\
296\code{profile.run('main()')}, and hence the exec'ed string is
297\code{'main()'}. The second line indicates that 2706 calls were
298monitored. Of those calls, 2004 were \dfn{primitive}. We define
299\dfn{primitive} to mean that the call was not induced via recursion.
300The next line: \code{Ordered by:\ standard name}, indicates that
301the text string in the far right column was used to sort the output.
302The column headings include:
303
304\begin{description}
305
306\item[ncalls ]
307for the number of calls,
308
309\item[tottime ]
310for the total time spent in the given function (and excluding time
311made in calls to sub-functions),
312
313\item[percall ]
314is the quotient of \code{tottime} divided by \code{ncalls}
315
316\item[cumtime ]
317is the total time spent in this and all subfunctions (i.e., from
318invocation till exit). This figure is accurate \emph{even} for recursive
319functions.
320
321\item[percall ]
322is the quotient of \code{cumtime} divided by primitive calls
323
324\item[filename:lineno(function) ]
325provides the respective data of each function
326
327\end{description}
328
329When there are two numbers in the first column (e.g.: \samp{43/3}),
330then the latter is the number of primitive calls, and the former is
331the actual number of calls. Note that when the function does not
332recurse, these two values are the same, and only the single figure is
333printed.
Guido van Rossum96628a91995-04-10 11:34:00 +0000334
Guido van Rossumdf804f81995-03-02 12:38:39 +0000335\end{funcdesc}
336
Fred Drake8fa5eb81998-02-27 05:23:37 +0000337Analysis of the profiler data is done using this class from the
338\module{pstats} module:
339
Fred Drake8fe533e1998-03-27 05:27:08 +0000340% now switch modules....
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000341% (This \stmodindex use may be hard to change ;-( )
Fred Drake8fe533e1998-03-27 05:27:08 +0000342\stmodindex{pstats}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000343
Fred Drakecce10901998-03-17 06:33:25 +0000344\begin{classdesc}{Stats}{filename\optional{, ...}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000345This class constructor creates an instance of a ``statistics object''
Fred Drake8fa5eb81998-02-27 05:23:37 +0000346from a \var{filename} (or set of filenames). \class{Stats} objects are
Guido van Rossumdf804f81995-03-02 12:38:39 +0000347manipulated by methods, in order to print useful reports.
348
349The file selected by the above constructor must have been created by
Fred Drake8fa5eb81998-02-27 05:23:37 +0000350the corresponding version of \module{profile}. To be specific, there is
351\emph{no} file compatibility guaranteed with future versions of this
Guido van Rossumdf804f81995-03-02 12:38:39 +0000352profiler, and there is no compatibility with files produced by other
353profilers (e.g., the old system profiler).
354
355If several files are provided, all the statistics for identical
356functions will be coalesced, so that an overall view of several
357processes can be considered in a single report. If additional files
Fred Drake8fa5eb81998-02-27 05:23:37 +0000358need to be combined with data in an existing \class{Stats} object, the
359\method{add()} method can be used.
360\end{classdesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000361
362
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000363\subsection{The \class{Stats} Class \label{profile-stats}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000364
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000365\class{Stats} objects have the following methods:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000366
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000367\begin{methoddesc}[Stats]{strip_dirs}{}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000368This method for the \class{Stats} class removes all leading path
369information from file names. It is very useful in reducing the size
370of the printout to fit within (close to) 80 columns. This method
371modifies the object, and the stripped information is lost. After
372performing a strip operation, the object is considered to have its
373entries in a ``random'' order, as it was just after object
374initialization and loading. If \method{strip_dirs()} causes two
375function names to be indistinguishable (i.e., they are on the same
376line of the same filename, and have the same function name), then the
377statistics for these two entries are accumulated into a single entry.
Fred Drake8fe533e1998-03-27 05:27:08 +0000378\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000379
380
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000381\begin{methoddesc}[Stats]{add}{filename\optional{, ...}}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000382This method of the \class{Stats} class accumulates additional
383profiling information into the current profiling object. Its
384arguments should refer to filenames created by the corresponding
385version of \function{profile.run()}. Statistics for identically named
386(re: file, line, name) functions are automatically accumulated into
387single function statistics.
Fred Drake8fe533e1998-03-27 05:27:08 +0000388\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000389
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000390\begin{methoddesc}[Stats]{sort_stats}{key\optional{, ...}}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000391This method modifies the \class{Stats} object by sorting it according
392to the supplied criteria. The argument is typically a string
Fred Drake2cb824c1998-04-09 18:10:35 +0000393identifying the basis of a sort (example: \code{'time'} or
394\code{'name'}).
Guido van Rossumdf804f81995-03-02 12:38:39 +0000395
396When more than one key is provided, then additional keys are used as
397secondary criteria when the there is equality in all keys selected
Fred Drake8fa5eb81998-02-27 05:23:37 +0000398before them. For example, \samp{sort_stats('name', 'file')} will sort
399all the entries according to their function name, and resolve all ties
Guido van Rossumdf804f81995-03-02 12:38:39 +0000400(identical function names) by sorting by file name.
401
402Abbreviations can be used for any key names, as long as the
403abbreviation is unambiguous. The following are the keys currently
404defined:
405
Fred Drakeee601911998-04-11 20:53:03 +0000406\begin{tableii}{l|l}{code}{Valid Arg}{Meaning}
Fred Drake5dabeed1998-04-03 07:02:35 +0000407 \lineii{'calls'}{call count}
408 \lineii{'cumulative'}{cumulative time}
409 \lineii{'file'}{file name}
410 \lineii{'module'}{file name}
411 \lineii{'pcalls'}{primitive call count}
412 \lineii{'line'}{line number}
413 \lineii{'name'}{function name}
414 \lineii{'nfl'}{name/file/line}
415 \lineii{'stdname'}{standard name}
416 \lineii{'time'}{internal time}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000417\end{tableii}
418
419Note that all sorts on statistics are in descending order (placing
420most time consuming items first), where as name, file, and line number
421searches are in ascending order (i.e., alphabetical). The subtle
Fred Drake2cb824c1998-04-09 18:10:35 +0000422distinction between \code{'nfl'} and \code{'stdname'} is that the
Guido van Rossumdf804f81995-03-02 12:38:39 +0000423standard name is a sort of the name as printed, which means that the
424embedded line numbers get compared in an odd way. For example, lines
4253, 20, and 40 would (if the file names were the same) appear in the
Fred Drake2cb824c1998-04-09 18:10:35 +0000426string order 20, 3 and 40. In contrast, \code{'nfl'} does a numeric
427compare of the line numbers. In fact, \code{sort_stats('nfl')} is the
428same as \code{sort_stats('name', 'file', 'line')}.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000429
430For compatibility with the old profiler, the numeric arguments
Fred Drake2cb824c1998-04-09 18:10:35 +0000431\code{-1}, \code{0}, \code{1}, and \code{2} are permitted. They are
432interpreted as \code{'stdname'}, \code{'calls'}, \code{'time'}, and
433\code{'cumulative'} respectively. If this old style format (numeric)
Guido van Rossumdf804f81995-03-02 12:38:39 +0000434is used, only one sort key (the numeric key) will be used, and
435additional arguments will be silently ignored.
Fred Drake8fe533e1998-03-27 05:27:08 +0000436\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000437
438
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000439\begin{methoddesc}[Stats]{reverse_order}{}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000440This method for the \class{Stats} class reverses the ordering of the basic
Guido van Rossumdf804f81995-03-02 12:38:39 +0000441list within the object. This method is provided primarily for
442compatibility with the old profiler. Its utility is questionable
443now that ascending vs descending order is properly selected based on
444the sort key of choice.
Fred Drake8fe533e1998-03-27 05:27:08 +0000445\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000446
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000447\begin{methoddesc}[Stats]{print_stats}{restriction\optional{, ...}}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000448This method for the \class{Stats} class prints out a report as described
449in the \function{profile.run()} definition.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000450
Fred Drake8fa5eb81998-02-27 05:23:37 +0000451The order of the printing is based on the last \method{sort_stats()}
452operation done on the object (subject to caveats in \method{add()} and
453\method{strip_dirs()}.
Guido van Rossumdf804f81995-03-02 12:38:39 +0000454
455The arguments provided (if any) can be used to limit the list down to
456the significant entries. Initially, the list is taken to be the
457complete set of profiled functions. Each restriction is either an
458integer (to select a count of lines), or a decimal fraction between
4590.0 and 1.0 inclusive (to select a percentage of lines), or a regular
Guido van Rossum364e6431997-11-18 15:28:46 +0000460expression (to pattern match the standard name that is printed; as of
461Python 1.5b1, this uses the Perl-style regular expression syntax
Fred Drakeffbe6871999-04-22 21:23:22 +0000462defined by the \refmodule{re} module). If several restrictions are
Guido van Rossum364e6431997-11-18 15:28:46 +0000463provided, then they are applied sequentially. For example:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000464
Fred Drake19479911998-02-13 06:58:54 +0000465\begin{verbatim}
Fred Drake2cb824c1998-04-09 18:10:35 +0000466print_stats(.1, 'foo:')
Fred Drake19479911998-02-13 06:58:54 +0000467\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000468
Guido van Rossumdf804f81995-03-02 12:38:39 +0000469would first limit the printing to first 10\% of list, and then only
470print functions that were part of filename \samp{.*foo:}. In
471contrast, the command:
472
Fred Drake19479911998-02-13 06:58:54 +0000473\begin{verbatim}
Fred Drake2cb824c1998-04-09 18:10:35 +0000474print_stats('foo:', .1)
Fred Drake19479911998-02-13 06:58:54 +0000475\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000476
Guido van Rossumdf804f81995-03-02 12:38:39 +0000477would limit the list to all functions having file names \samp{.*foo:},
478and then proceed to only print the first 10\% of them.
Fred Drake8fe533e1998-03-27 05:27:08 +0000479\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000480
481
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000482\begin{methoddesc}[Stats]{print_callers}{restrictions\optional{, ...}}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000483This method for the \class{Stats} class prints a list of all functions
Guido van Rossumdf804f81995-03-02 12:38:39 +0000484that called each function in the profiled database. The ordering is
Fred Drake8fa5eb81998-02-27 05:23:37 +0000485identical to that provided by \method{print_stats()}, and the definition
Guido van Rossumdf804f81995-03-02 12:38:39 +0000486of the restricting argument is also identical. For convenience, a
487number is shown in parentheses after each caller to show how many
488times this specific call was made. A second non-parenthesized number
489is the cumulative time spent in the function at the right.
Fred Drake8fe533e1998-03-27 05:27:08 +0000490\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000491
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000492\begin{methoddesc}[Stats]{print_callees}{restrictions\optional{, ...}}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000493This method for the \class{Stats} class prints a list of all function
Guido van Rossumdf804f81995-03-02 12:38:39 +0000494that were called by the indicated function. Aside from this reversal
495of direction of calls (re: called vs was called by), the arguments and
Fred Drake8fa5eb81998-02-27 05:23:37 +0000496ordering are identical to the \method{print_callers()} method.
Fred Drake8fe533e1998-03-27 05:27:08 +0000497\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000498
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000499\begin{methoddesc}[Stats]{ignore}{}
Fred Drakeea003fc1999-04-05 21:59:15 +0000500\deprecated{1.5.1}{This is not needed in modern versions of
501Python.\footnote{
502 This was once necessary, when Python would print any unused expression
503 result that was not \code{None}. The method is still defined for
504 backward compatibility.}}
Fred Drake8fe533e1998-03-27 05:27:08 +0000505\end{methoddesc}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000506
507
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000508\section{Limitations \label{profile-limits}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000509
510There are two fundamental limitations on this profiler. The first is
511that it relies on the Python interpreter to dispatch \dfn{call},
Fred Drake8fa5eb81998-02-27 05:23:37 +0000512\dfn{return}, and \dfn{exception} events. Compiled \C{} code does not
Guido van Rossumdf804f81995-03-02 12:38:39 +0000513get interpreted, and hence is ``invisible'' to the profiler. All time
Fred Drake3a18f3b1998-04-02 19:36:25 +0000514spent in \C{} code (including built-in functions) will be charged to the
Fred Drake8fa5eb81998-02-27 05:23:37 +0000515Python function that invoked the \C{} code. If the \C{} code calls out
Guido van Rossumdf804f81995-03-02 12:38:39 +0000516to some native Python code, then those calls will be profiled
517properly.
518
519The second limitation has to do with accuracy of timing information.
520There is a fundamental problem with deterministic profilers involving
521accuracy. The most obvious restriction is that the underlying ``clock''
522is only ticking at a rate (typically) of about .001 seconds. Hence no
523measurements will be more accurate that that underlying clock. If
524enough measurements are taken, then the ``error'' will tend to average
525out. Unfortunately, removing this first error induces a second source
526of error...
527
528The second problem is that it ``takes a while'' from when an event is
529dispatched until the profiler's call to get the time actually
530\emph{gets} the state of the clock. Similarly, there is a certain lag
531when exiting the profiler event handler from the time that the clock's
532value was obtained (and then squirreled away), until the user's code
533is once again executing. As a result, functions that are called many
534times, or call many functions, will typically accumulate this error.
535The error that accumulates in this fashion is typically less than the
536accuracy of the clock (i.e., less than one clock tick), but it
537\emph{can} accumulate and become very significant. This profiler
538provides a means of calibrating itself for a given platform so that
539this error can be probabilistically (i.e., on the average) removed.
540After the profiler is calibrated, it will be more accurate (in a least
541square sense), but it will sometimes produce negative numbers (when
542call counts are exceptionally low, and the gods of probability work
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000543against you :-). ) Do \emph{not} be alarmed by negative numbers in
Guido van Rossumdf804f81995-03-02 12:38:39 +0000544the profile. They should \emph{only} appear if you have calibrated
545your profiler, and the results are actually better than without
546calibration.
547
548
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000549\section{Calibration \label{profile-calibration}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000550
551The profiler class has a hard coded constant that is added to each
552event handling time to compensate for the overhead of calling the time
553function, and socking away the results. The following procedure can
554be used to obtain this constant for a given platform (see discussion
555in section Limitations above).
556
Fred Drake19479911998-02-13 06:58:54 +0000557\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000558import profile
559pr = profile.Profile()
Guido van Rossum685ef4e1998-03-17 14:37:48 +0000560print pr.calibrate(100)
561print pr.calibrate(100)
562print pr.calibrate(100)
Fred Drake19479911998-02-13 06:58:54 +0000563\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000564
565The argument to \method{calibrate()} is the number of times to try to
566do the sample calls to get the CPU times. If your computer is
567\emph{very} fast, you might have to do:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000568
Fred Drake19479911998-02-13 06:58:54 +0000569\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000570pr.calibrate(1000)
Fred Drake19479911998-02-13 06:58:54 +0000571\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000572
Guido van Rossumdf804f81995-03-02 12:38:39 +0000573or even:
574
Fred Drake19479911998-02-13 06:58:54 +0000575\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000576pr.calibrate(10000)
Fred Drake19479911998-02-13 06:58:54 +0000577\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000578
Guido van Rossumdf804f81995-03-02 12:38:39 +0000579The object of this exercise is to get a fairly consistent result.
580When you have a consistent answer, you are ready to use that number in
581the source code. For a Sun Sparcstation 1000 running Solaris 2.3, the
582magical number is about .00053. If you have a choice, you are better
583off with a smaller constant, and your results will ``less often'' show
584up as negative in profile statistics.
585
586The following shows how the trace_dispatch() method in the Profile
587class should be modified to install the calibration constant on a Sun
588Sparcstation 1000:
589
Fred Drake19479911998-02-13 06:58:54 +0000590\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000591def trace_dispatch(self, frame, event, arg):
592 t = self.timer()
593 t = t[0] + t[1] - self.t - .00053 # Calibration constant
594
595 if self.dispatch[event](frame,t):
Guido van Rossumdf804f81995-03-02 12:38:39 +0000596 t = self.timer()
Guido van Rossume47da0a1997-07-17 16:34:52 +0000597 self.t = t[0] + t[1]
598 else:
599 r = self.timer()
600 self.t = r[0] + r[1] - t # put back unrecorded delta
601 return
Fred Drake19479911998-02-13 06:58:54 +0000602\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000603
Guido van Rossumdf804f81995-03-02 12:38:39 +0000604Note that if there is no calibration constant, then the line
605containing the callibration constant should simply say:
606
Fred Drake19479911998-02-13 06:58:54 +0000607\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000608t = t[0] + t[1] - self.t # no calibration constant
Fred Drake19479911998-02-13 06:58:54 +0000609\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000610
Guido van Rossumdf804f81995-03-02 12:38:39 +0000611You can also achieve the same results using a derived class (and the
612profiler will actually run equally fast!!), but the above method is
613the simplest to use. I could have made the profiler ``self
614calibrating'', but it would have made the initialization of the
615profiler class slower, and would have required some \emph{very} fancy
616coding, or else the use of a variable where the constant \samp{.00053}
617was placed in the code shown. This is a \strong{VERY} critical
618performance section, and there is no reason to use a variable lookup
619at this point, when a constant can be used.
620
621
Guido van Rossum86cb0921995-03-20 12:59:56 +0000622\section{Extensions --- Deriving Better Profilers}
623\nodename{Profiler Extensions}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000624
Fred Drake8fa5eb81998-02-27 05:23:37 +0000625The \class{Profile} class of module \module{profile} was written so that
Guido van Rossumdf804f81995-03-02 12:38:39 +0000626derived classes could be developed to extend the profiler. Rather
627than describing all the details of such an effort, I'll just present
628the following two examples of derived classes that can be used to do
629profiling. If the reader is an avid Python programmer, then it should
630be possible to use these as a model and create similar (and perchance
631better) profile classes.
632
633If all you want to do is change how the timer is called, or which
634timer function is used, then the basic class has an option for that in
635the constructor for the class. Consider passing the name of a
636function to call into the constructor:
637
Fred Drake19479911998-02-13 06:58:54 +0000638\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000639pr = profile.Profile(your_time_func)
Fred Drake19479911998-02-13 06:58:54 +0000640\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000641
Guido van Rossumdf804f81995-03-02 12:38:39 +0000642The resulting profiler will call \code{your_time_func()} instead of
Fred Drake8fa5eb81998-02-27 05:23:37 +0000643\function{os.times()}. The function should return either a single number
644or a list of numbers (like what \function{os.times()} returns). If the
Guido van Rossumdf804f81995-03-02 12:38:39 +0000645function returns a single time number, or the list of returned numbers
646has length 2, then you will get an especially fast version of the
647dispatch routine.
648
649Be warned that you \emph{should} calibrate the profiler class for the
650timer function that you choose. For most machines, a timer that
651returns a lone integer value will provide the best results in terms of
Fred Drake8fa5eb81998-02-27 05:23:37 +0000652low overhead during profiling. (\function{os.times()} is
653\emph{pretty} bad, 'cause it returns a tuple of floating point values,
654so all arithmetic is floating point in the profiler!). If you want to
655substitute a better timer in the cleanest fashion, you should derive a
656class, and simply put in the replacement dispatch method that better
657handles your timer call, along with the appropriate calibration
658constant :-).
Guido van Rossumdf804f81995-03-02 12:38:39 +0000659
660
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000661\subsection{OldProfile Class \label{profile-old}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000662
663The following derived profiler simulates the old style profiler,
664providing errant results on recursive functions. The reason for the
665usefulness of this profiler is that it runs faster (i.e., less
666overhead) than the old profiler. It still creates all the caller
667stats, and is quite useful when there is \emph{no} recursion in the
668user's code. It is also a lot more accurate than the old profiler, as
669it does not charge all its overhead time to the user's code.
670
Fred Drake19479911998-02-13 06:58:54 +0000671\begin{verbatim}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000672class OldProfile(Profile):
673
674 def trace_dispatch_exception(self, frame, t):
675 rt, rtt, rct, rfn, rframe, rcur = self.cur
676 if rcur and not rframe is frame:
677 return self.trace_dispatch_return(rframe, t)
678 return 0
679
680 def trace_dispatch_call(self, frame, t):
681 fn = `frame.f_code`
682
683 self.cur = (t, 0, 0, fn, frame, self.cur)
684 if self.timings.has_key(fn):
685 tt, ct, callers = self.timings[fn]
686 self.timings[fn] = tt, ct, callers
687 else:
688 self.timings[fn] = 0, 0, {}
689 return 1
690
691 def trace_dispatch_return(self, frame, t):
692 rt, rtt, rct, rfn, frame, rcur = self.cur
693 rtt = rtt + t
694 sft = rtt + rct
695
696 pt, ptt, pct, pfn, pframe, pcur = rcur
697 self.cur = pt, ptt+rt, pct+sft, pfn, pframe, pcur
698
699 tt, ct, callers = self.timings[rfn]
700 if callers.has_key(pfn):
701 callers[pfn] = callers[pfn] + 1
702 else:
703 callers[pfn] = 1
704 self.timings[rfn] = tt+rtt, ct + sft, callers
705
706 return 1
707
708
709 def snapshot_stats(self):
710 self.stats = {}
711 for func in self.timings.keys():
712 tt, ct, callers = self.timings[func]
713 nor_func = self.func_normalize(func)
714 nor_callers = {}
715 nc = 0
716 for func_caller in callers.keys():
Fred Drake5dabeed1998-04-03 07:02:35 +0000717 nor_callers[self.func_normalize(func_caller)] = \
718 callers[func_caller]
Guido van Rossumdf804f81995-03-02 12:38:39 +0000719 nc = nc + callers[func_caller]
720 self.stats[nor_func] = nc, nc, tt, ct, nor_callers
Fred Drake19479911998-02-13 06:58:54 +0000721\end{verbatim}
Fred Drake8fa5eb81998-02-27 05:23:37 +0000722
Fred Drakeb9f1f6d1999-04-21 21:43:17 +0000723\subsection{HotProfile Class \label{profile-HotProfile}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000724
725This profiler is the fastest derived profile example. It does not
726calculate caller-callee relationships, and does not calculate
727cumulative time under a function. It only calculates time spent in a
728function, so it runs very quickly (re: very low overhead). In truth,
729the basic profiler is so fast, that is probably not worth the savings
730to give up the data, but this class still provides a nice example.
731
Fred Drake19479911998-02-13 06:58:54 +0000732\begin{verbatim}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000733class HotProfile(Profile):
734
735 def trace_dispatch_exception(self, frame, t):
736 rt, rtt, rfn, rframe, rcur = self.cur
737 if rcur and not rframe is frame:
738 return self.trace_dispatch_return(rframe, t)
739 return 0
740
741 def trace_dispatch_call(self, frame, t):
742 self.cur = (t, 0, frame, self.cur)
743 return 1
744
745 def trace_dispatch_return(self, frame, t):
746 rt, rtt, frame, rcur = self.cur
747
748 rfn = `frame.f_code`
749
750 pt, ptt, pframe, pcur = rcur
751 self.cur = pt, ptt+rt, pframe, pcur
752
753 if self.timings.has_key(rfn):
754 nc, tt = self.timings[rfn]
755 self.timings[rfn] = nc + 1, rt + rtt + tt
756 else:
757 self.timings[rfn] = 1, rt + rtt
758
759 return 1
760
761
762 def snapshot_stats(self):
763 self.stats = {}
764 for func in self.timings.keys():
765 nc, tt = self.timings[func]
766 nor_func = self.func_normalize(func)
767 self.stats[nor_func] = nc, nc, tt, 0, {}
Fred Drake19479911998-02-13 06:58:54 +0000768\end{verbatim}