blob: 20da0b6bd2041797a0400b23fdcb238b1b68e102 [file] [log] [blame]
Guido van Rossumdf804f81995-03-02 12:38:39 +00001\chapter{The Python Profiler}
2\stmodindex{profile}
3\stmodindex{pstats}
4
Fred Drake4b3f0311996-12-13 22:04:31 +00005Copyright \copyright{} 1994, by InfoSeek Corporation, all rights reserved.
Guido van Rossumdf804f81995-03-02 12:38:39 +00006
7Written by James Roskind%
8\footnote{
Guido van Rossum6c4f0031995-03-07 10:14:09 +00009Updated and converted to \LaTeX\ by Guido van Rossum. The references to
Guido van Rossumdf804f81995-03-02 12:38:39 +000010the old profiler are left in the text, although it no longer exists.
11}
12
13Permission to use, copy, modify, and distribute this Python software
14and its associated documentation for any purpose (subject to the
15restriction in the following sentence) without fee is hereby granted,
16provided that the above copyright notice appears in all copies, and
17that both that copyright notice and this permission notice appear in
18supporting documentation, and that the name of InfoSeek not be used in
19advertising or publicity pertaining to distribution of the software
20without specific, written prior permission. This permission is
21explicitly restricted to the copying and modification of the software
22to remain in Python, compiled Python, or other languages (such as C)
23wherein the modified or derived code is exclusively imported into a
24Python module.
25
26INFOSEEK CORPORATION DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
27SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
28FITNESS. IN NO EVENT SHALL INFOSEEK CORPORATION BE LIABLE FOR ANY
29SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
30RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
31CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
32CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
33
34
35The profiler was written after only programming in Python for 3 weeks.
36As a result, it is probably clumsy code, but I don't know for sure yet
37'cause I'm a beginner :-). I did work hard to make the code run fast,
38so that profiling would be a reasonable thing to do. I tried not to
39repeat code fragments, but I'm sure I did some stuff in really awkward
40ways at times. Please send suggestions for improvements to:
Guido van Rossum789742b1996-02-12 23:17:40 +000041\code{jar@netscape.com}. I won't promise \emph{any} support. ...but
Guido van Rossumdf804f81995-03-02 12:38:39 +000042I'd appreciate the feedback.
43
44
Guido van Rossum470be141995-03-17 16:07:09 +000045\section{Introduction to the profiler}
Guido van Rossum86cb0921995-03-20 12:59:56 +000046\nodename{Profiler Introduction}
Guido van Rossumdf804f81995-03-02 12:38:39 +000047
48A \dfn{profiler} is a program that describes the run time performance
49of a program, providing a variety of statistics. This documentation
50describes the profiler functionality provided in the modules
51\code{profile} and \code{pstats.} This profiler provides
52\dfn{deterministic profiling} of any Python programs. It also
53provides a series of report generation tools to allow users to rapidly
54examine the results of a profile operation.
55
56
57\section{How Is This Profiler Different From The Old Profiler?}
Guido van Rossum86cb0921995-03-20 12:59:56 +000058\nodename{Profiler Changes}
Guido van Rossumdf804f81995-03-02 12:38:39 +000059
Guido van Rossum364e6431997-11-18 15:28:46 +000060(This section is of historical importance only; the old profiler
61discussed here was last seen in Python 1.1.)
62
Guido van Rossumdf804f81995-03-02 12:38:39 +000063The big changes from old profiling module are that you get more
64information, and you pay less CPU time. It's not a trade-off, it's a
65trade-up.
66
67To be specific:
68
69\begin{description}
70
71\item[Bugs removed:]
72Local stack frame is no longer molested, execution time is now charged
73to correct functions.
74
75\item[Accuracy increased:]
76Profiler execution time is no longer charged to user's code,
77calibration for platform is supported, file reads are not done \emph{by}
78profiler \emph{during} profiling (and charged to user's code!).
79
80\item[Speed increased:]
81Overhead CPU cost was reduced by more than a factor of two (perhaps a
82factor of five), lightweight profiler module is all that must be
83loaded, and the report generating module (\code{pstats}) is not needed
84during profiling.
85
86\item[Recursive functions support:]
87Cumulative times in recursive functions are correctly calculated;
88recursive entries are counted.
89
90\item[Large growth in report generating UI:]
91Distinct profiles runs can be added together forming a comprehensive
92report; functions that import statistics take arbitrary lists of
93files; sorting criteria is now based on keywords (instead of 4 integer
94options); reports shows what functions were profiled as well as what
95profile file was referenced; output format has been improved.
96
97\end{description}
98
99
100\section{Instant Users Manual}
101
102This section is provided for users that ``don't want to read the
103manual.'' It provides a very brief overview, and allows a user to
104rapidly perform profiling on an existing application.
105
106To profile an application with a main entry point of \samp{foo()}, you
107would add the following to your module:
108
Fred Drake19479911998-02-13 06:58:54 +0000109\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000110import profile
111profile.run("foo()")
Fred Drake19479911998-02-13 06:58:54 +0000112\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000113%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000114The above action would cause \samp{foo()} to be run, and a series of
115informative lines (the profile) to be printed. The above approach is
116most useful when working with the interpreter. If you would like to
117save the results of a profile into a file for later examination, you
118can supply a file name as the second argument to the \code{run()}
119function:
120
Fred Drake19479911998-02-13 06:58:54 +0000121\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000122import profile
123profile.run("foo()", 'fooprof')
Fred Drake19479911998-02-13 06:58:54 +0000124\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000125%
Guido van Rossumbac80021997-06-02 17:29:12 +0000126\code{profile.py} can also be invoked as
127a script to profile another script. For example:
128\code{python /usr/local/lib/python1.4/profile.py myscript.py}
129
Guido van Rossumdf804f81995-03-02 12:38:39 +0000130When you wish to review the profile, you should use the methods in the
131\code{pstats} module. Typically you would load the statistics data as
132follows:
133
Fred Drake19479911998-02-13 06:58:54 +0000134\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000135import pstats
136p = pstats.Stats('fooprof')
Fred Drake19479911998-02-13 06:58:54 +0000137\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000138%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000139The class \code{Stats} (the above code just created an instance of
140this class) has a variety of methods for manipulating and printing the
141data that was just read into \samp{p}. When you ran
142\code{profile.run()} above, what was printed was the result of three
143method calls:
144
Fred Drake19479911998-02-13 06:58:54 +0000145\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000146p.strip_dirs().sort_stats(-1).print_stats()
Fred Drake19479911998-02-13 06:58:54 +0000147\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000148%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000149The first method removed the extraneous path from all the module
150names. The second method sorted all the entries according to the
151standard module/line/name string that is printed (this is to comply
152with the semantics of the old profiler). The third method printed out
153all the statistics. You might try the following sort calls:
154
Fred Drake19479911998-02-13 06:58:54 +0000155\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000156p.sort_stats('name')
157p.print_stats()
Fred Drake19479911998-02-13 06:58:54 +0000158\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000159%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000160The first call will actually sort the list by function name, and the
161second call will print out the statistics. The following are some
162interesting calls to experiment with:
163
Fred Drake19479911998-02-13 06:58:54 +0000164\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000165p.sort_stats('cumulative').print_stats(10)
Fred Drake19479911998-02-13 06:58:54 +0000166\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000167%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000168This sorts the profile by cumulative time in a function, and then only
169prints the ten most significant lines. If you want to understand what
170algorithms are taking time, the above line is what you would use.
171
172If you were looking to see what functions were looping a lot, and
173taking a lot of time, you would do:
174
Fred Drake19479911998-02-13 06:58:54 +0000175\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000176p.sort_stats('time').print_stats(10)
Fred Drake19479911998-02-13 06:58:54 +0000177\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000178%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000179to sort according to time spent within each function, and then print
180the statistics for the top ten functions.
181
182You might also try:
183
Fred Drake19479911998-02-13 06:58:54 +0000184\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000185p.sort_stats('file').print_stats('__init__')
Fred Drake19479911998-02-13 06:58:54 +0000186\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000187%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000188This will sort all the statistics by file name, and then print out
189statistics for only the class init methods ('cause they are spelled
190with \code{__init__} in them). As one final example, you could try:
191
Fred Drake19479911998-02-13 06:58:54 +0000192\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000193p.sort_stats('time', 'cum').print_stats(.5, 'init')
Fred Drake19479911998-02-13 06:58:54 +0000194\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000195%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000196This line sorts statistics with a primary key of time, and a secondary
197key of cumulative time, and then prints out some of the statistics.
198To be specific, the list is first culled down to 50\% (re: \samp{.5})
199of its original size, then only lines containing \code{init} are
200maintained, and that sub-sub-list is printed.
201
202If you wondered what functions called the above functions, you could
203now (\samp{p} is still sorted according to the last criteria) do:
204
Fred Drake19479911998-02-13 06:58:54 +0000205\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000206p.print_callers(.5, 'init')
Fred Drake19479911998-02-13 06:58:54 +0000207\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000208%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000209and you would get a list of callers for each of the listed functions.
210
211If you want more functionality, you're going to have to read the
212manual, or guess what the following functions do:
213
Fred Drake19479911998-02-13 06:58:54 +0000214\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000215p.print_callees()
216p.add('fooprof')
Fred Drake19479911998-02-13 06:58:54 +0000217\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000218%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000219\section{What Is Deterministic Profiling?}
Guido van Rossum86cb0921995-03-20 12:59:56 +0000220\nodename{Deterministic Profiling}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000221
222\dfn{Deterministic profiling} is meant to reflect the fact that all
223\dfn{function call}, \dfn{function return}, and \dfn{exception} events
224are monitored, and precise timings are made for the intervals between
225these events (during which time the user's code is executing). In
226contrast, \dfn{statistical profiling} (which is not done by this
227module) randomly samples the effective instruction pointer, and
228deduces where time is being spent. The latter technique traditionally
229involves less overhead (as the code does not need to be instrumented),
230but provides only relative indications of where time is being spent.
231
232In Python, since there is an interpreter active during execution, the
233presence of instrumented code is not required to do deterministic
234profiling. Python automatically provides a \dfn{hook} (optional
235callback) for each event. In addition, the interpreted nature of
236Python tends to add so much overhead to execution, that deterministic
237profiling tends to only add small processing overhead in typical
238applications. The result is that deterministic profiling is not that
239expensive, yet provides extensive run time statistics about the
240execution of a Python program.
241
242Call count statistics can be used to identify bugs in code (surprising
243counts), and to identify possible inline-expansion points (high call
244counts). Internal time statistics can be used to identify ``hot
245loops'' that should be carefully optimized. Cumulative time
246statistics should be used to identify high level errors in the
247selection of algorithms. Note that the unusual handling of cumulative
248times in this profiler allows statistics for recursive implementations
249of algorithms to be directly compared to iterative implementations.
250
251
252\section{Reference Manual}
253
Fred Drake19479911998-02-13 06:58:54 +0000254\setindexsubitem{(profiler function)}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000255
256The primary entry point for the profiler is the global function
257\code{profile.run()}. It is typically used to create any profile
258information. The reports are formatted and printed using methods of
259the class \code{pstats.Stats}. The following is a description of all
260of these standard entry points and functions. For a more in-depth
261view of some of the code, consider reading the later section on
262Profiler Extensions, which includes discussion of how to derive
263``better'' profilers from the classes presented, or reading the source
264code for these modules.
265
Guido van Rossum470be141995-03-17 16:07:09 +0000266\begin{funcdesc}{profile.run}{string\optional{\, filename\optional{\, ...}}}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000267
268This function takes a single argument that has can be passed to the
269\code{exec} statement, and an optional file name. In all cases this
270routine attempts to \code{exec} its first argument, and gather profiling
271statistics from the execution. If no file name is present, then this
272function automatically prints a simple profiling report, sorted by the
273standard name string (file/line/function-name) that is presented in
274each line. The following is a typical output from such a call:
275
Fred Drake19479911998-02-13 06:58:54 +0000276\begin{verbatim}
Guido van Rossum96628a91995-04-10 11:34:00 +0000277 main()
278 2706 function calls (2004 primitive calls) in 4.504 CPU seconds
Guido van Rossumdf804f81995-03-02 12:38:39 +0000279
Guido van Rossum96628a91995-04-10 11:34:00 +0000280Ordered by: standard name
Guido van Rossumdf804f81995-03-02 12:38:39 +0000281
Guido van Rossum96628a91995-04-10 11:34:00 +0000282ncalls tottime percall cumtime percall filename:lineno(function)
283 2 0.006 0.003 0.953 0.477 pobject.py:75(save_objects)
284 43/3 0.533 0.012 0.749 0.250 pobject.py:99(evaluate)
285 ...
Fred Drake19479911998-02-13 06:58:54 +0000286\end{verbatim}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000287
288The first line indicates that this profile was generated by the call:\\
289\code{profile.run('main()')}, and hence the exec'ed string is
290\code{'main()'}. The second line indicates that 2706 calls were
291monitored. Of those calls, 2004 were \dfn{primitive}. We define
292\dfn{primitive} to mean that the call was not induced via recursion.
293The next line: \code{Ordered by:\ standard name}, indicates that
294the text string in the far right column was used to sort the output.
295The column headings include:
296
297\begin{description}
298
299\item[ncalls ]
300for the number of calls,
301
302\item[tottime ]
303for the total time spent in the given function (and excluding time
304made in calls to sub-functions),
305
306\item[percall ]
307is the quotient of \code{tottime} divided by \code{ncalls}
308
309\item[cumtime ]
310is the total time spent in this and all subfunctions (i.e., from
311invocation till exit). This figure is accurate \emph{even} for recursive
312functions.
313
314\item[percall ]
315is the quotient of \code{cumtime} divided by primitive calls
316
317\item[filename:lineno(function) ]
318provides the respective data of each function
319
320\end{description}
321
322When there are two numbers in the first column (e.g.: \samp{43/3}),
323then the latter is the number of primitive calls, and the former is
324the actual number of calls. Note that when the function does not
325recurse, these two values are the same, and only the single figure is
326printed.
Guido van Rossum96628a91995-04-10 11:34:00 +0000327
Guido van Rossumdf804f81995-03-02 12:38:39 +0000328\end{funcdesc}
329
330\begin{funcdesc}{pstats.Stats}{filename\optional{\, ...}}
331This class constructor creates an instance of a ``statistics object''
332from a \var{filename} (or set of filenames). \code{Stats} objects are
333manipulated by methods, in order to print useful reports.
334
335The file selected by the above constructor must have been created by
336the corresponding version of \code{profile}. To be specific, there is
337\emph{NO} file compatibility guaranteed with future versions of this
338profiler, and there is no compatibility with files produced by other
339profilers (e.g., the old system profiler).
340
341If several files are provided, all the statistics for identical
342functions will be coalesced, so that an overall view of several
343processes can be considered in a single report. If additional files
344need to be combined with data in an existing \code{Stats} object, the
345\code{add()} method can be used.
346\end{funcdesc}
347
348
Guido van Rossum470be141995-03-17 16:07:09 +0000349\subsection{The \sectcode{Stats} Class}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000350
Fred Drake19479911998-02-13 06:58:54 +0000351\setindexsubitem{(Stats method)}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000352
353\begin{funcdesc}{strip_dirs}{}
Guido van Rossum470be141995-03-17 16:07:09 +0000354This method for the \code{Stats} class removes all leading path information
Guido van Rossumdf804f81995-03-02 12:38:39 +0000355from file names. It is very useful in reducing the size of the
356printout to fit within (close to) 80 columns. This method modifies
357the object, and the stripped information is lost. After performing a
358strip operation, the object is considered to have its entries in a
359``random'' order, as it was just after object initialization and
360loading. If \code{strip_dirs()} causes two function names to be
361indistinguishable (i.e., they are on the same line of the same
362filename, and have the same function name), then the statistics for
363these two entries are accumulated into a single entry.
364\end{funcdesc}
365
366
367\begin{funcdesc}{add}{filename\optional{\, ...}}
Guido van Rossum470be141995-03-17 16:07:09 +0000368This method of the \code{Stats} class accumulates additional profiling
Guido van Rossumdf804f81995-03-02 12:38:39 +0000369information into the current profiling object. Its arguments should
370refer to filenames created by the corresponding version of
371\code{profile.run()}. Statistics for identically named (re: file,
372line, name) functions are automatically accumulated into single
373function statistics.
374\end{funcdesc}
375
376\begin{funcdesc}{sort_stats}{key\optional{\, ...}}
Guido van Rossum470be141995-03-17 16:07:09 +0000377This method modifies the \code{Stats} object by sorting it according to the
Guido van Rossumdf804f81995-03-02 12:38:39 +0000378supplied criteria. The argument is typically a string identifying the
379basis of a sort (example: \code{"time"} or \code{"name"}).
380
381When more than one key is provided, then additional keys are used as
382secondary criteria when the there is equality in all keys selected
383before them. For example, sort_stats('name', 'file') will sort all
384the entries according to their function name, and resolve all ties
385(identical function names) by sorting by file name.
386
387Abbreviations can be used for any key names, as long as the
388abbreviation is unambiguous. The following are the keys currently
389defined:
390
391\begin{tableii}{|l|l|}{code}{Valid Arg}{Meaning}
392\lineii{"calls"}{call count}
393\lineii{"cumulative"}{cumulative time}
394\lineii{"file"}{file name}
395\lineii{"module"}{file name}
396\lineii{"pcalls"}{primitive call count}
397\lineii{"line"}{line number}
398\lineii{"name"}{function name}
399\lineii{"nfl"}{name/file/line}
400\lineii{"stdname"}{standard name}
401\lineii{"time"}{internal time}
402\end{tableii}
403
404Note that all sorts on statistics are in descending order (placing
405most time consuming items first), where as name, file, and line number
406searches are in ascending order (i.e., alphabetical). The subtle
407distinction between \code{"nfl"} and \code{"stdname"} is that the
408standard name is a sort of the name as printed, which means that the
409embedded line numbers get compared in an odd way. For example, lines
4103, 20, and 40 would (if the file names were the same) appear in the
411string order 20, 3 and 40. In contrast, \code{"nfl"} does a numeric
412compare of the line numbers. In fact, \code{sort_stats("nfl")} is the
413same as \code{sort_stats("name", "file", "line")}.
414
415For compatibility with the old profiler, the numeric arguments
416\samp{-1}, \samp{0}, \samp{1}, and \samp{2} are permitted. They are
417interpreted as \code{"stdname"}, \code{"calls"}, \code{"time"}, and
418\code{"cumulative"} respectively. If this old style format (numeric)
419is used, only one sort key (the numeric key) will be used, and
420additional arguments will be silently ignored.
421\end{funcdesc}
422
423
424\begin{funcdesc}{reverse_order}{}
Guido van Rossum470be141995-03-17 16:07:09 +0000425This method for the \code{Stats} class reverses the ordering of the basic
Guido van Rossumdf804f81995-03-02 12:38:39 +0000426list within the object. This method is provided primarily for
427compatibility with the old profiler. Its utility is questionable
428now that ascending vs descending order is properly selected based on
429the sort key of choice.
430\end{funcdesc}
431
432\begin{funcdesc}{print_stats}{restriction\optional{\, ...}}
Guido van Rossum470be141995-03-17 16:07:09 +0000433This method for the \code{Stats} class prints out a report as described
Guido van Rossumdf804f81995-03-02 12:38:39 +0000434in the \code{profile.run()} definition.
435
436The order of the printing is based on the last \code{sort_stats()}
437operation done on the object (subject to caveats in \code{add()} and
438\code{strip_dirs())}.
439
440The arguments provided (if any) can be used to limit the list down to
441the significant entries. Initially, the list is taken to be the
442complete set of profiled functions. Each restriction is either an
443integer (to select a count of lines), or a decimal fraction between
4440.0 and 1.0 inclusive (to select a percentage of lines), or a regular
Guido van Rossum364e6431997-11-18 15:28:46 +0000445expression (to pattern match the standard name that is printed; as of
446Python 1.5b1, this uses the Perl-style regular expression syntax
447defined by the \code{re} module). If several restrictions are
448provided, then they are applied sequentially. For example:
Guido van Rossumdf804f81995-03-02 12:38:39 +0000449
Fred Drake19479911998-02-13 06:58:54 +0000450\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000451print_stats(.1, "foo:")
Fred Drake19479911998-02-13 06:58:54 +0000452\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000453%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000454would first limit the printing to first 10\% of list, and then only
455print functions that were part of filename \samp{.*foo:}. In
456contrast, the command:
457
Fred Drake19479911998-02-13 06:58:54 +0000458\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000459print_stats("foo:", .1)
Fred Drake19479911998-02-13 06:58:54 +0000460\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000461%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000462would limit the list to all functions having file names \samp{.*foo:},
463and then proceed to only print the first 10\% of them.
464\end{funcdesc}
465
466
467\begin{funcdesc}{print_callers}{restrictions\optional{\, ...}}
Guido van Rossum470be141995-03-17 16:07:09 +0000468This method for the \code{Stats} class prints a list of all functions
Guido van Rossumdf804f81995-03-02 12:38:39 +0000469that called each function in the profiled database. The ordering is
470identical to that provided by \code{print_stats()}, and the definition
471of the restricting argument is also identical. For convenience, a
472number is shown in parentheses after each caller to show how many
473times this specific call was made. A second non-parenthesized number
474is the cumulative time spent in the function at the right.
475\end{funcdesc}
476
477\begin{funcdesc}{print_callees}{restrictions\optional{\, ...}}
Guido van Rossum470be141995-03-17 16:07:09 +0000478This method for the \code{Stats} class prints a list of all function
Guido van Rossumdf804f81995-03-02 12:38:39 +0000479that were called by the indicated function. Aside from this reversal
480of direction of calls (re: called vs was called by), the arguments and
481ordering are identical to the \code{print_callers()} method.
482\end{funcdesc}
483
484\begin{funcdesc}{ignore}{}
Guido van Rossum470be141995-03-17 16:07:09 +0000485This method of the \code{Stats} class is used to dispose of the value
Guido van Rossumdf804f81995-03-02 12:38:39 +0000486returned by earlier methods. All standard methods in this class
487return the instance that is being processed, so that the commands can
488be strung together. For example:
489
Fred Drake19479911998-02-13 06:58:54 +0000490\begin{verbatim}
Guido van Rossum96628a91995-04-10 11:34:00 +0000491pstats.Stats('foofile').strip_dirs().sort_stats('cum') \
492 .print_stats().ignore()
Fred Drake19479911998-02-13 06:58:54 +0000493\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000494%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000495would perform all the indicated functions, but it would not return
Guido van Rossum470be141995-03-17 16:07:09 +0000496the final reference to the \code{Stats} instance.%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000497\footnote{
498This was once necessary, when Python would print any unused expression
499result that was not \code{None}. The method is still defined for
500backward compatibility.
501}
502\end{funcdesc}
503
504
505\section{Limitations}
506
507There are two fundamental limitations on this profiler. The first is
508that it relies on the Python interpreter to dispatch \dfn{call},
509\dfn{return}, and \dfn{exception} events. Compiled C code does not
510get interpreted, and hence is ``invisible'' to the profiler. All time
511spent in C code (including builtin functions) will be charged to the
Guido van Rossumcca8d2b1995-03-22 15:48:46 +0000512Python function that invoked the C code. If the C code calls out
Guido van Rossumdf804f81995-03-02 12:38:39 +0000513to some native Python code, then those calls will be profiled
514properly.
515
516The second limitation has to do with accuracy of timing information.
517There is a fundamental problem with deterministic profilers involving
518accuracy. The most obvious restriction is that the underlying ``clock''
519is only ticking at a rate (typically) of about .001 seconds. Hence no
520measurements will be more accurate that that underlying clock. If
521enough measurements are taken, then the ``error'' will tend to average
522out. Unfortunately, removing this first error induces a second source
523of error...
524
525The second problem is that it ``takes a while'' from when an event is
526dispatched until the profiler's call to get the time actually
527\emph{gets} the state of the clock. Similarly, there is a certain lag
528when exiting the profiler event handler from the time that the clock's
529value was obtained (and then squirreled away), until the user's code
530is once again executing. As a result, functions that are called many
531times, or call many functions, will typically accumulate this error.
532The error that accumulates in this fashion is typically less than the
533accuracy of the clock (i.e., less than one clock tick), but it
534\emph{can} accumulate and become very significant. This profiler
535provides a means of calibrating itself for a given platform so that
536this error can be probabilistically (i.e., on the average) removed.
537After the profiler is calibrated, it will be more accurate (in a least
538square sense), but it will sometimes produce negative numbers (when
539call counts are exceptionally low, and the gods of probability work
540against you :-). ) Do \emph{NOT} be alarmed by negative numbers in
541the profile. They should \emph{only} appear if you have calibrated
542your profiler, and the results are actually better than without
543calibration.
544
545
546\section{Calibration}
547
548The profiler class has a hard coded constant that is added to each
549event handling time to compensate for the overhead of calling the time
550function, and socking away the results. The following procedure can
551be used to obtain this constant for a given platform (see discussion
552in section Limitations above).
553
Fred Drake19479911998-02-13 06:58:54 +0000554\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000555import profile
556pr = profile.Profile()
557pr.calibrate(100)
558pr.calibrate(100)
559pr.calibrate(100)
Fred Drake19479911998-02-13 06:58:54 +0000560\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000561%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000562The argument to calibrate() is the number of times to try to do the
563sample calls to get the CPU times. If your computer is \emph{very}
564fast, you might have to do:
565
Fred Drake19479911998-02-13 06:58:54 +0000566\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000567pr.calibrate(1000)
Fred Drake19479911998-02-13 06:58:54 +0000568\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000569%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000570or even:
571
Fred Drake19479911998-02-13 06:58:54 +0000572\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000573pr.calibrate(10000)
Fred Drake19479911998-02-13 06:58:54 +0000574\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000575%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000576The object of this exercise is to get a fairly consistent result.
577When you have a consistent answer, you are ready to use that number in
578the source code. For a Sun Sparcstation 1000 running Solaris 2.3, the
579magical number is about .00053. If you have a choice, you are better
580off with a smaller constant, and your results will ``less often'' show
581up as negative in profile statistics.
582
583The following shows how the trace_dispatch() method in the Profile
584class should be modified to install the calibration constant on a Sun
585Sparcstation 1000:
586
Fred Drake19479911998-02-13 06:58:54 +0000587\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000588def trace_dispatch(self, frame, event, arg):
589 t = self.timer()
590 t = t[0] + t[1] - self.t - .00053 # Calibration constant
591
592 if self.dispatch[event](frame,t):
Guido van Rossumdf804f81995-03-02 12:38:39 +0000593 t = self.timer()
Guido van Rossume47da0a1997-07-17 16:34:52 +0000594 self.t = t[0] + t[1]
595 else:
596 r = self.timer()
597 self.t = r[0] + r[1] - t # put back unrecorded delta
598 return
Fred Drake19479911998-02-13 06:58:54 +0000599\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000600%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000601Note that if there is no calibration constant, then the line
602containing the callibration constant should simply say:
603
Fred Drake19479911998-02-13 06:58:54 +0000604\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000605t = t[0] + t[1] - self.t # no calibration constant
Fred Drake19479911998-02-13 06:58:54 +0000606\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000607%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000608You can also achieve the same results using a derived class (and the
609profiler will actually run equally fast!!), but the above method is
610the simplest to use. I could have made the profiler ``self
611calibrating'', but it would have made the initialization of the
612profiler class slower, and would have required some \emph{very} fancy
613coding, or else the use of a variable where the constant \samp{.00053}
614was placed in the code shown. This is a \strong{VERY} critical
615performance section, and there is no reason to use a variable lookup
616at this point, when a constant can be used.
617
618
Guido van Rossum86cb0921995-03-20 12:59:56 +0000619\section{Extensions --- Deriving Better Profilers}
620\nodename{Profiler Extensions}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000621
622The \code{Profile} class of module \code{profile} was written so that
623derived classes could be developed to extend the profiler. Rather
624than describing all the details of such an effort, I'll just present
625the following two examples of derived classes that can be used to do
626profiling. If the reader is an avid Python programmer, then it should
627be possible to use these as a model and create similar (and perchance
628better) profile classes.
629
630If all you want to do is change how the timer is called, or which
631timer function is used, then the basic class has an option for that in
632the constructor for the class. Consider passing the name of a
633function to call into the constructor:
634
Fred Drake19479911998-02-13 06:58:54 +0000635\begin{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000636pr = profile.Profile(your_time_func)
Fred Drake19479911998-02-13 06:58:54 +0000637\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000638%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000639The resulting profiler will call \code{your_time_func()} instead of
640\code{os.times()}. The function should return either a single number
641or a list of numbers (like what \code{os.times()} returns). If the
642function returns a single time number, or the list of returned numbers
643has length 2, then you will get an especially fast version of the
644dispatch routine.
645
646Be warned that you \emph{should} calibrate the profiler class for the
647timer function that you choose. For most machines, a timer that
648returns a lone integer value will provide the best results in terms of
649low overhead during profiling. (os.times is \emph{pretty} bad, 'cause
650it returns a tuple of floating point values, so all arithmetic is
651floating point in the profiler!). If you want to substitute a
652better timer in the cleanest fashion, you should derive a class, and
653simply put in the replacement dispatch method that better handles your
654timer call, along with the appropriate calibration constant :-).
655
656
657\subsection{OldProfile Class}
658
659The following derived profiler simulates the old style profiler,
660providing errant results on recursive functions. The reason for the
661usefulness of this profiler is that it runs faster (i.e., less
662overhead) than the old profiler. It still creates all the caller
663stats, and is quite useful when there is \emph{no} recursion in the
664user's code. It is also a lot more accurate than the old profiler, as
665it does not charge all its overhead time to the user's code.
666
Fred Drake19479911998-02-13 06:58:54 +0000667\begin{verbatim}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000668class OldProfile(Profile):
669
670 def trace_dispatch_exception(self, frame, t):
671 rt, rtt, rct, rfn, rframe, rcur = self.cur
672 if rcur and not rframe is frame:
673 return self.trace_dispatch_return(rframe, t)
674 return 0
675
676 def trace_dispatch_call(self, frame, t):
677 fn = `frame.f_code`
678
679 self.cur = (t, 0, 0, fn, frame, self.cur)
680 if self.timings.has_key(fn):
681 tt, ct, callers = self.timings[fn]
682 self.timings[fn] = tt, ct, callers
683 else:
684 self.timings[fn] = 0, 0, {}
685 return 1
686
687 def trace_dispatch_return(self, frame, t):
688 rt, rtt, rct, rfn, frame, rcur = self.cur
689 rtt = rtt + t
690 sft = rtt + rct
691
692 pt, ptt, pct, pfn, pframe, pcur = rcur
693 self.cur = pt, ptt+rt, pct+sft, pfn, pframe, pcur
694
695 tt, ct, callers = self.timings[rfn]
696 if callers.has_key(pfn):
697 callers[pfn] = callers[pfn] + 1
698 else:
699 callers[pfn] = 1
700 self.timings[rfn] = tt+rtt, ct + sft, callers
701
702 return 1
703
704
705 def snapshot_stats(self):
706 self.stats = {}
707 for func in self.timings.keys():
708 tt, ct, callers = self.timings[func]
709 nor_func = self.func_normalize(func)
710 nor_callers = {}
711 nc = 0
712 for func_caller in callers.keys():
713 nor_callers[self.func_normalize(func_caller)]=\
714 callers[func_caller]
715 nc = nc + callers[func_caller]
716 self.stats[nor_func] = nc, nc, tt, ct, nor_callers
Fred Drake19479911998-02-13 06:58:54 +0000717\end{verbatim}
Guido van Rossume47da0a1997-07-17 16:34:52 +0000718%
Guido van Rossumdf804f81995-03-02 12:38:39 +0000719\subsection{HotProfile Class}
720
721This profiler is the fastest derived profile example. It does not
722calculate caller-callee relationships, and does not calculate
723cumulative time under a function. It only calculates time spent in a
724function, so it runs very quickly (re: very low overhead). In truth,
725the basic profiler is so fast, that is probably not worth the savings
726to give up the data, but this class still provides a nice example.
727
Fred Drake19479911998-02-13 06:58:54 +0000728\begin{verbatim}
Guido van Rossumdf804f81995-03-02 12:38:39 +0000729class HotProfile(Profile):
730
731 def trace_dispatch_exception(self, frame, t):
732 rt, rtt, rfn, rframe, rcur = self.cur
733 if rcur and not rframe is frame:
734 return self.trace_dispatch_return(rframe, t)
735 return 0
736
737 def trace_dispatch_call(self, frame, t):
738 self.cur = (t, 0, frame, self.cur)
739 return 1
740
741 def trace_dispatch_return(self, frame, t):
742 rt, rtt, frame, rcur = self.cur
743
744 rfn = `frame.f_code`
745
746 pt, ptt, pframe, pcur = rcur
747 self.cur = pt, ptt+rt, pframe, pcur
748
749 if self.timings.has_key(rfn):
750 nc, tt = self.timings[rfn]
751 self.timings[rfn] = nc + 1, rt + rtt + tt
752 else:
753 self.timings[rfn] = 1, rt + rtt
754
755 return 1
756
757
758 def snapshot_stats(self):
759 self.stats = {}
760 for func in self.timings.keys():
761 nc, tt = self.timings[func]
762 nor_func = self.func_normalize(func)
763 self.stats[nor_func] = nc, nc, tt, 0, {}
Fred Drake19479911998-02-13 06:58:54 +0000764\end{verbatim}