Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 1 | \chapter{The Python Profilers \label{profile}} |
Fred Drake | ea003fc | 1999-04-05 21:59:15 +0000 | [diff] [blame] | 2 | |
| 3 | \sectionauthor{James Roskind}{} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 4 | |
Fred Drake | 4b3f031 | 1996-12-13 22:04:31 +0000 | [diff] [blame] | 5 | Copyright \copyright{} 1994, by InfoSeek Corporation, all rights reserved. |
Fred Drake | 5dabeed | 1998-04-03 07:02:35 +0000 | [diff] [blame] | 6 | \index{InfoSeek Corporation} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 7 | |
Fred Drake | ea003fc | 1999-04-05 21:59:15 +0000 | [diff] [blame] | 8 | Written by James Roskind.\footnote{ |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 9 | Updated and converted to \LaTeX\ by Guido van Rossum. |
| 10 | Further updated by Armin Rigo to integrate the documentation for the new |
| 11 | \module{cProfile} module of Python 2.5.} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 12 | |
| 13 | Permission to use, copy, modify, and distribute this Python software |
| 14 | and its associated documentation for any purpose (subject to the |
| 15 | restriction in the following sentence) without fee is hereby granted, |
| 16 | provided that the above copyright notice appears in all copies, and |
| 17 | that both that copyright notice and this permission notice appear in |
| 18 | supporting documentation, and that the name of InfoSeek not be used in |
| 19 | advertising or publicity pertaining to distribution of the software |
| 20 | without specific, written prior permission. This permission is |
| 21 | explicitly restricted to the copying and modification of the software |
| 22 | to remain in Python, compiled Python, or other languages (such as C) |
| 23 | wherein the modified or derived code is exclusively imported into a |
| 24 | Python module. |
| 25 | |
| 26 | INFOSEEK CORPORATION DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS |
| 27 | SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND |
| 28 | FITNESS. IN NO EVENT SHALL INFOSEEK CORPORATION BE LIABLE FOR ANY |
| 29 | SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER |
| 30 | RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF |
| 31 | CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN |
| 32 | CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. |
| 33 | |
| 34 | |
| 35 | The profiler was written after only programming in Python for 3 weeks. |
| 36 | As a result, it is probably clumsy code, but I don't know for sure yet |
| 37 | 'cause I'm a beginner :-). I did work hard to make the code run fast, |
| 38 | so that profiling would be a reasonable thing to do. I tried not to |
| 39 | repeat code fragments, but I'm sure I did some stuff in really awkward |
| 40 | ways at times. Please send suggestions for improvements to: |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 41 | \email{jar@netscape.com}. I won't promise \emph{any} support. ...but |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 42 | I'd appreciate the feedback. |
| 43 | |
| 44 | |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 45 | \section{Introduction to the profilers} |
Guido van Rossum | 86cb092 | 1995-03-20 12:59:56 +0000 | [diff] [blame] | 46 | \nodename{Profiler Introduction} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 47 | |
| 48 | A \dfn{profiler} is a program that describes the run time performance |
| 49 | of a program, providing a variety of statistics. This documentation |
| 50 | describes the profiler functionality provided in the modules |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 51 | \module{profile} and \module{pstats}. This profiler provides |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 52 | \dfn{deterministic profiling} of any Python programs. It also |
| 53 | provides a series of report generation tools to allow users to rapidly |
| 54 | examine the results of a profile operation. |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 55 | \index{deterministic profiling} |
| 56 | \index{profiling, deterministic} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 57 | |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 58 | The Python standard library provides three different profilers: |
| 59 | |
| 60 | \begin{enumerate} |
| 61 | \item \module{profile}, a pure Python module, described in the sequel. |
| 62 | Copyright \copyright{} 1994, by InfoSeek Corporation. |
| 63 | \versionchanged[also reports the time spent in calls to built-in |
| 64 | functions and methods]{2.4} |
| 65 | |
| 66 | \item \module{cProfile}, a module written in C, with a reasonable |
| 67 | overhead that makes it suitable for profiling long-running programs. |
| 68 | Based on \module{lsprof}, contributed by Brett Rosen and Ted Czotter. |
| 69 | \versionadded{2.5} |
| 70 | |
| 71 | \item \module{hotshot}, a C module focusing on minimizing the overhead |
| 72 | while profiling, at the expense of long data post-processing times. |
| 73 | \versionchanged[the results should be more meaningful than in the |
| 74 | past: the timing core contained a critical bug]{2.5} |
| 75 | \end{enumerate} |
| 76 | |
| 77 | The \module{profile} and \module{cProfile} modules export the same |
| 78 | interface, so they are mostly interchangeables; \module{cProfile} has a |
| 79 | much lower overhead but is not so far as well-tested and might not be |
| 80 | available on all systems. \module{cProfile} is really a compatibility |
| 81 | layer on top of the internal \module{_lsprof} module. The |
| 82 | \module{hotshot} module is reserved to specialized usages. |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 83 | |
Georg Brandl | 6c1908d | 2005-12-26 23:44:29 +0000 | [diff] [blame] | 84 | %\section{How Is This Profiler Different From The Old Profiler?} |
| 85 | %\nodename{Profiler Changes} |
| 86 | % |
| 87 | %(This section is of historical importance only; the old profiler |
| 88 | %discussed here was last seen in Python 1.1.) |
| 89 | % |
| 90 | %The big changes from old profiling module are that you get more |
| 91 | %information, and you pay less CPU time. It's not a trade-off, it's a |
| 92 | %trade-up. |
| 93 | % |
| 94 | %To be specific: |
| 95 | % |
| 96 | %\begin{description} |
| 97 | % |
| 98 | %\item[Bugs removed:] |
| 99 | %Local stack frame is no longer molested, execution time is now charged |
| 100 | %to correct functions. |
| 101 | % |
| 102 | %\item[Accuracy increased:] |
| 103 | %Profiler execution time is no longer charged to user's code, |
| 104 | %calibration for platform is supported, file reads are not done \emph{by} |
| 105 | %profiler \emph{during} profiling (and charged to user's code!). |
| 106 | % |
| 107 | %\item[Speed increased:] |
| 108 | %Overhead CPU cost was reduced by more than a factor of two (perhaps a |
| 109 | %factor of five), lightweight profiler module is all that must be |
| 110 | %loaded, and the report generating module (\module{pstats}) is not needed |
| 111 | %during profiling. |
| 112 | % |
| 113 | %\item[Recursive functions support:] |
| 114 | %Cumulative times in recursive functions are correctly calculated; |
| 115 | %recursive entries are counted. |
| 116 | % |
| 117 | %\item[Large growth in report generating UI:] |
| 118 | %Distinct profiles runs can be added together forming a comprehensive |
| 119 | %report; functions that import statistics take arbitrary lists of |
| 120 | %files; sorting criteria is now based on keywords (instead of 4 integer |
| 121 | %options); reports shows what functions were profiled as well as what |
| 122 | %profile file was referenced; output format has been improved. |
| 123 | % |
| 124 | %\end{description} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 125 | |
| 126 | |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 127 | \section{Instant Users Manual \label{profile-instant}} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 128 | |
| 129 | This section is provided for users that ``don't want to read the |
| 130 | manual.'' It provides a very brief overview, and allows a user to |
| 131 | rapidly perform profiling on an existing application. |
| 132 | |
Fred Drake | fee6f33 | 2004-03-23 21:40:07 +0000 | [diff] [blame] | 133 | To profile an application with a main entry point of \function{foo()}, |
| 134 | you would add the following to your module: |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 135 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 136 | \begin{verbatim} |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 137 | import cProfile |
| 138 | cProfile.run('foo()') |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 139 | \end{verbatim} |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 140 | |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 141 | (Use \module{profile} instead of \module{cProfile} if the latter is not |
| 142 | available on your system.) |
| 143 | |
Fred Drake | fee6f33 | 2004-03-23 21:40:07 +0000 | [diff] [blame] | 144 | The above action would cause \function{foo()} to be run, and a series of |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 145 | informative lines (the profile) to be printed. The above approach is |
| 146 | most useful when working with the interpreter. If you would like to |
| 147 | save the results of a profile into a file for later examination, you |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 148 | can supply a file name as the second argument to the \function{run()} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 149 | function: |
| 150 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 151 | \begin{verbatim} |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 152 | import cProfile |
| 153 | cProfile.run('foo()', 'fooprof') |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 154 | \end{verbatim} |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 155 | |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 156 | The file \file{cProfile.py} can also be invoked as |
Guido van Rossum | bac8002 | 1997-06-02 17:29:12 +0000 | [diff] [blame] | 157 | a script to profile another script. For example: |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 158 | |
| 159 | \begin{verbatim} |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 160 | python -m cProfile myscript.py |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 161 | \end{verbatim} |
Guido van Rossum | bac8002 | 1997-06-02 17:29:12 +0000 | [diff] [blame] | 162 | |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 163 | \file{cProfile.py} accepts two optional arguments on the command line: |
Nicholas Bastin | 824b1b2 | 2004-03-23 18:44:39 +0000 | [diff] [blame] | 164 | |
| 165 | \begin{verbatim} |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 166 | cProfile.py [-o output_file] [-s sort_order] |
Nicholas Bastin | 824b1b2 | 2004-03-23 18:44:39 +0000 | [diff] [blame] | 167 | \end{verbatim} |
| 168 | |
Fred Drake | fee6f33 | 2004-03-23 21:40:07 +0000 | [diff] [blame] | 169 | \programopt{-s} only applies to standard output (\programopt{-o} is |
| 170 | not supplied). Look in the \class{Stats} documentation for valid sort |
| 171 | values. |
Nicholas Bastin | 824b1b2 | 2004-03-23 18:44:39 +0000 | [diff] [blame] | 172 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 173 | When you wish to review the profile, you should use the methods in the |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 174 | \module{pstats} module. Typically you would load the statistics data as |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 175 | follows: |
| 176 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 177 | \begin{verbatim} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 178 | import pstats |
| 179 | p = pstats.Stats('fooprof') |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 180 | \end{verbatim} |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 181 | |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 182 | The class \class{Stats} (the above code just created an instance of |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 183 | this class) has a variety of methods for manipulating and printing the |
Fred Drake | fee6f33 | 2004-03-23 21:40:07 +0000 | [diff] [blame] | 184 | data that was just read into \code{p}. When you ran |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 185 | \function{cProfile.run()} above, what was printed was the result of three |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 186 | method calls: |
| 187 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 188 | \begin{verbatim} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 189 | p.strip_dirs().sort_stats(-1).print_stats() |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 190 | \end{verbatim} |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 191 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 192 | The first method removed the extraneous path from all the module |
| 193 | names. The second method sorted all the entries according to the |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 194 | standard module/line/name string that is printed. |
| 195 | %(this is to comply with the semantics of the old profiler). |
| 196 | The third method printed out |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 197 | all the statistics. You might try the following sort calls: |
| 198 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 199 | \begin{verbatim} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 200 | p.sort_stats('name') |
| 201 | p.print_stats() |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 202 | \end{verbatim} |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 203 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 204 | The first call will actually sort the list by function name, and the |
| 205 | second call will print out the statistics. The following are some |
| 206 | interesting calls to experiment with: |
| 207 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 208 | \begin{verbatim} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 209 | p.sort_stats('cumulative').print_stats(10) |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 210 | \end{verbatim} |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 211 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 212 | This sorts the profile by cumulative time in a function, and then only |
| 213 | prints the ten most significant lines. If you want to understand what |
| 214 | algorithms are taking time, the above line is what you would use. |
| 215 | |
| 216 | If you were looking to see what functions were looping a lot, and |
| 217 | taking a lot of time, you would do: |
| 218 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 219 | \begin{verbatim} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 220 | p.sort_stats('time').print_stats(10) |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 221 | \end{verbatim} |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 222 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 223 | to sort according to time spent within each function, and then print |
| 224 | the statistics for the top ten functions. |
| 225 | |
| 226 | You might also try: |
| 227 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 228 | \begin{verbatim} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 229 | p.sort_stats('file').print_stats('__init__') |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 230 | \end{verbatim} |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 231 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 232 | This will sort all the statistics by file name, and then print out |
Fred Drake | fee6f33 | 2004-03-23 21:40:07 +0000 | [diff] [blame] | 233 | statistics for only the class init methods (since they are spelled |
| 234 | with \code{__init__} in them). As one final example, you could try: |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 235 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 236 | \begin{verbatim} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 237 | p.sort_stats('time', 'cum').print_stats(.5, 'init') |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 238 | \end{verbatim} |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 239 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 240 | This line sorts statistics with a primary key of time, and a secondary |
| 241 | key of cumulative time, and then prints out some of the statistics. |
| 242 | To be specific, the list is first culled down to 50\% (re: \samp{.5}) |
| 243 | of its original size, then only lines containing \code{init} are |
| 244 | maintained, and that sub-sub-list is printed. |
| 245 | |
| 246 | If you wondered what functions called the above functions, you could |
Fred Drake | fee6f33 | 2004-03-23 21:40:07 +0000 | [diff] [blame] | 247 | now (\code{p} is still sorted according to the last criteria) do: |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 248 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 249 | \begin{verbatim} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 250 | p.print_callers(.5, 'init') |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 251 | \end{verbatim} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 252 | |
Tim Peters | 0a1fc4e | 2001-10-07 03:12:08 +0000 | [diff] [blame] | 253 | and you would get a list of callers for each of the listed functions. |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 254 | |
| 255 | If you want more functionality, you're going to have to read the |
| 256 | manual, or guess what the following functions do: |
| 257 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 258 | \begin{verbatim} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 259 | p.print_callees() |
| 260 | p.add('fooprof') |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 261 | \end{verbatim} |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 262 | |
Eric S. Raymond | 4f3980d | 2001-04-13 00:23:01 +0000 | [diff] [blame] | 263 | Invoked as a script, the \module{pstats} module is a statistics |
| 264 | browser for reading and examining profile dumps. It has a simple |
Fred Drake | a3e56a6 | 2001-04-13 14:34:58 +0000 | [diff] [blame] | 265 | line-oriented interface (implemented using \refmodule{cmd}) and |
Eric S. Raymond | 4f3980d | 2001-04-13 00:23:01 +0000 | [diff] [blame] | 266 | interactive help. |
| 267 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 268 | \section{What Is Deterministic Profiling?} |
Guido van Rossum | 86cb092 | 1995-03-20 12:59:56 +0000 | [diff] [blame] | 269 | \nodename{Deterministic Profiling} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 270 | |
| 271 | \dfn{Deterministic profiling} is meant to reflect the fact that all |
Fred Drake | a3e56a6 | 2001-04-13 14:34:58 +0000 | [diff] [blame] | 272 | \emph{function call}, \emph{function return}, and \emph{exception} events |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 273 | are monitored, and precise timings are made for the intervals between |
| 274 | these events (during which time the user's code is executing). In |
| 275 | contrast, \dfn{statistical profiling} (which is not done by this |
| 276 | module) randomly samples the effective instruction pointer, and |
| 277 | deduces where time is being spent. The latter technique traditionally |
| 278 | involves less overhead (as the code does not need to be instrumented), |
| 279 | but provides only relative indications of where time is being spent. |
| 280 | |
| 281 | In Python, since there is an interpreter active during execution, the |
| 282 | presence of instrumented code is not required to do deterministic |
| 283 | profiling. Python automatically provides a \dfn{hook} (optional |
| 284 | callback) for each event. In addition, the interpreted nature of |
| 285 | Python tends to add so much overhead to execution, that deterministic |
| 286 | profiling tends to only add small processing overhead in typical |
| 287 | applications. The result is that deterministic profiling is not that |
| 288 | expensive, yet provides extensive run time statistics about the |
| 289 | execution of a Python program. |
| 290 | |
| 291 | Call count statistics can be used to identify bugs in code (surprising |
| 292 | counts), and to identify possible inline-expansion points (high call |
| 293 | counts). Internal time statistics can be used to identify ``hot |
| 294 | loops'' that should be carefully optimized. Cumulative time |
| 295 | statistics should be used to identify high level errors in the |
| 296 | selection of algorithms. Note that the unusual handling of cumulative |
| 297 | times in this profiler allows statistics for recursive implementations |
| 298 | of algorithms to be directly compared to iterative implementations. |
| 299 | |
| 300 | |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 301 | \section{Reference Manual -- \module{profile} and \module{cProfile}} |
Fred Drake | b91e934 | 1998-07-23 17:59:49 +0000 | [diff] [blame] | 302 | |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 303 | \declaremodule{standard}{profile} |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 304 | \declaremodule{standard}{cProfile} |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 305 | \modulesynopsis{Python profiler} |
Fred Drake | b91e934 | 1998-07-23 17:59:49 +0000 | [diff] [blame] | 306 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 307 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 308 | |
| 309 | The primary entry point for the profiler is the global function |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 310 | \function{profile.run()} (resp. \function{cProfile.run()}). |
| 311 | It is typically used to create any profile |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 312 | information. The reports are formatted and printed using methods of |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 313 | the class \class{pstats.Stats}. The following is a description of all |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 314 | of these standard entry points and functions. For a more in-depth |
| 315 | view of some of the code, consider reading the later section on |
| 316 | Profiler Extensions, which includes discussion of how to derive |
| 317 | ``better'' profilers from the classes presented, or reading the source |
| 318 | code for these modules. |
| 319 | |
Nicholas Bastin | 1eb4bfc | 2004-03-22 20:12:56 +0000 | [diff] [blame] | 320 | \begin{funcdesc}{run}{command\optional{, filename}} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 321 | |
| 322 | This function takes a single argument that has can be passed to the |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 323 | \keyword{exec} statement, and an optional file name. In all cases this |
| 324 | routine attempts to \keyword{exec} its first argument, and gather profiling |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 325 | statistics from the execution. If no file name is present, then this |
| 326 | function automatically prints a simple profiling report, sorted by the |
| 327 | standard name string (file/line/function-name) that is presented in |
| 328 | each line. The following is a typical output from such a call: |
| 329 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 330 | \begin{verbatim} |
Guido van Rossum | 96628a9 | 1995-04-10 11:34:00 +0000 | [diff] [blame] | 331 | 2706 function calls (2004 primitive calls) in 4.504 CPU seconds |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 332 | |
Guido van Rossum | 96628a9 | 1995-04-10 11:34:00 +0000 | [diff] [blame] | 333 | Ordered by: standard name |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 334 | |
Guido van Rossum | 96628a9 | 1995-04-10 11:34:00 +0000 | [diff] [blame] | 335 | ncalls tottime percall cumtime percall filename:lineno(function) |
| 336 | 2 0.006 0.003 0.953 0.477 pobject.py:75(save_objects) |
| 337 | 43/3 0.533 0.012 0.749 0.250 pobject.py:99(evaluate) |
| 338 | ... |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 339 | \end{verbatim} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 340 | |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 341 | The first line indicates that 2706 calls were |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 342 | monitored. Of those calls, 2004 were \dfn{primitive}. We define |
| 343 | \dfn{primitive} to mean that the call was not induced via recursion. |
| 344 | The next line: \code{Ordered by:\ standard name}, indicates that |
| 345 | the text string in the far right column was used to sort the output. |
| 346 | The column headings include: |
| 347 | |
| 348 | \begin{description} |
| 349 | |
| 350 | \item[ncalls ] |
Tim Peters | 0a1fc4e | 2001-10-07 03:12:08 +0000 | [diff] [blame] | 351 | for the number of calls, |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 352 | |
| 353 | \item[tottime ] |
| 354 | for the total time spent in the given function (and excluding time |
| 355 | made in calls to sub-functions), |
| 356 | |
| 357 | \item[percall ] |
| 358 | is the quotient of \code{tottime} divided by \code{ncalls} |
| 359 | |
| 360 | \item[cumtime ] |
Fred Drake | 907e76b | 2001-07-06 20:30:11 +0000 | [diff] [blame] | 361 | is the total time spent in this and all subfunctions (from invocation |
| 362 | till exit). This figure is accurate \emph{even} for recursive |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 363 | functions. |
| 364 | |
| 365 | \item[percall ] |
| 366 | is the quotient of \code{cumtime} divided by primitive calls |
| 367 | |
| 368 | \item[filename:lineno(function) ] |
| 369 | provides the respective data of each function |
| 370 | |
| 371 | \end{description} |
| 372 | |
Fred Drake | 907e76b | 2001-07-06 20:30:11 +0000 | [diff] [blame] | 373 | When there are two numbers in the first column (for example, |
| 374 | \samp{43/3}), then the latter is the number of primitive calls, and |
| 375 | the former is the actual number of calls. Note that when the function |
| 376 | does not recurse, these two values are the same, and only the single |
| 377 | figure is printed. |
Guido van Rossum | 96628a9 | 1995-04-10 11:34:00 +0000 | [diff] [blame] | 378 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 379 | \end{funcdesc} |
| 380 | |
Nicholas Bastin | 1eb4bfc | 2004-03-22 20:12:56 +0000 | [diff] [blame] | 381 | \begin{funcdesc}{runctx}{command, globals, locals\optional{, filename}} |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 382 | This function is similar to \function{run()}, with added |
Nicholas Bastin | 1eb4bfc | 2004-03-22 20:12:56 +0000 | [diff] [blame] | 383 | arguments to supply the globals and locals dictionaries for the |
| 384 | \var{command} string. |
| 385 | \end{funcdesc} |
| 386 | |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 387 | Analysis of the profiler data is done using this class from the |
| 388 | \module{pstats} module: |
| 389 | |
Fred Drake | 8fe533e | 1998-03-27 05:27:08 +0000 | [diff] [blame] | 390 | % now switch modules.... |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 391 | % (This \stmodindex use may be hard to change ;-( ) |
Fred Drake | 8fe533e | 1998-03-27 05:27:08 +0000 | [diff] [blame] | 392 | \stmodindex{pstats} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 393 | |
Fred Drake | e05c3e0 | 2004-03-23 20:30:59 +0000 | [diff] [blame] | 394 | \begin{classdesc}{Stats}{filename\optional{, \moreargs}} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 395 | This class constructor creates an instance of a ``statistics object'' |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 396 | from a \var{filename} (or set of filenames). \class{Stats} objects are |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 397 | manipulated by methods, in order to print useful reports. |
| 398 | |
| 399 | The file selected by the above constructor must have been created by |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 400 | the corresponding version of \module{profile} or \module{cProfile}. |
| 401 | To be specific, there is |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 402 | \emph{no} file compatibility guaranteed with future versions of this |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 403 | profiler, and there is no compatibility with files produced by other |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 404 | profilers. |
| 405 | %(such as the old system profiler). |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 406 | |
| 407 | If several files are provided, all the statistics for identical |
| 408 | functions will be coalesced, so that an overall view of several |
| 409 | processes can be considered in a single report. If additional files |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 410 | need to be combined with data in an existing \class{Stats} object, the |
| 411 | \method{add()} method can be used. |
| 412 | \end{classdesc} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 413 | |
| 414 | |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 415 | \subsection{The \class{Stats} Class \label{profile-stats}} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 416 | |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 417 | \class{Stats} objects have the following methods: |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 418 | |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 419 | \begin{methoddesc}[Stats]{strip_dirs}{} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 420 | This method for the \class{Stats} class removes all leading path |
| 421 | information from file names. It is very useful in reducing the size |
| 422 | of the printout to fit within (close to) 80 columns. This method |
| 423 | modifies the object, and the stripped information is lost. After |
| 424 | performing a strip operation, the object is considered to have its |
| 425 | entries in a ``random'' order, as it was just after object |
| 426 | initialization and loading. If \method{strip_dirs()} causes two |
Fred Drake | 907e76b | 2001-07-06 20:30:11 +0000 | [diff] [blame] | 427 | function names to be indistinguishable (they are on the same |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 428 | line of the same filename, and have the same function name), then the |
| 429 | statistics for these two entries are accumulated into a single entry. |
Fred Drake | 8fe533e | 1998-03-27 05:27:08 +0000 | [diff] [blame] | 430 | \end{methoddesc} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 431 | |
| 432 | |
Fred Drake | e05c3e0 | 2004-03-23 20:30:59 +0000 | [diff] [blame] | 433 | \begin{methoddesc}[Stats]{add}{filename\optional{, \moreargs}} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 434 | This method of the \class{Stats} class accumulates additional |
| 435 | profiling information into the current profiling object. Its |
| 436 | arguments should refer to filenames created by the corresponding |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 437 | version of \function{profile.run()} or \function{cProfile.run()}. |
| 438 | Statistics for identically named |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 439 | (re: file, line, name) functions are automatically accumulated into |
| 440 | single function statistics. |
Fred Drake | 8fe533e | 1998-03-27 05:27:08 +0000 | [diff] [blame] | 441 | \end{methoddesc} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 442 | |
Fred Drake | 126d366 | 2003-05-14 14:29:27 +0000 | [diff] [blame] | 443 | \begin{methoddesc}[Stats]{dump_stats}{filename} |
| 444 | Save the data loaded into the \class{Stats} object to a file named |
| 445 | \var{filename}. The file is created if it does not exist, and is |
| 446 | overwritten if it already exists. This is equivalent to the method of |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 447 | the same name on the \class{profile.Profile} and |
| 448 | \class{cProfile.Profile} classes. |
Fred Drake | 126d366 | 2003-05-14 14:29:27 +0000 | [diff] [blame] | 449 | \versionadded{2.3} |
| 450 | \end{methoddesc} |
| 451 | |
Fred Drake | e05c3e0 | 2004-03-23 20:30:59 +0000 | [diff] [blame] | 452 | \begin{methoddesc}[Stats]{sort_stats}{key\optional{, \moreargs}} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 453 | This method modifies the \class{Stats} object by sorting it according |
| 454 | to the supplied criteria. The argument is typically a string |
Fred Drake | 2cb824c | 1998-04-09 18:10:35 +0000 | [diff] [blame] | 455 | identifying the basis of a sort (example: \code{'time'} or |
| 456 | \code{'name'}). |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 457 | |
| 458 | When more than one key is provided, then additional keys are used as |
Walter Dörwald | f0dfc7a | 2003-10-20 14:01:56 +0000 | [diff] [blame] | 459 | secondary criteria when there is equality in all keys selected |
Fred Drake | fee6f33 | 2004-03-23 21:40:07 +0000 | [diff] [blame] | 460 | before them. For example, \code{sort_stats('name', 'file')} will sort |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 461 | all the entries according to their function name, and resolve all ties |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 462 | (identical function names) by sorting by file name. |
| 463 | |
| 464 | Abbreviations can be used for any key names, as long as the |
| 465 | abbreviation is unambiguous. The following are the keys currently |
Tim Peters | 0a1fc4e | 2001-10-07 03:12:08 +0000 | [diff] [blame] | 466 | defined: |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 467 | |
Fred Drake | ee60191 | 1998-04-11 20:53:03 +0000 | [diff] [blame] | 468 | \begin{tableii}{l|l}{code}{Valid Arg}{Meaning} |
Fred Drake | 5dabeed | 1998-04-03 07:02:35 +0000 | [diff] [blame] | 469 | \lineii{'calls'}{call count} |
| 470 | \lineii{'cumulative'}{cumulative time} |
| 471 | \lineii{'file'}{file name} |
| 472 | \lineii{'module'}{file name} |
| 473 | \lineii{'pcalls'}{primitive call count} |
| 474 | \lineii{'line'}{line number} |
| 475 | \lineii{'name'}{function name} |
| 476 | \lineii{'nfl'}{name/file/line} |
| 477 | \lineii{'stdname'}{standard name} |
| 478 | \lineii{'time'}{internal time} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 479 | \end{tableii} |
| 480 | |
| 481 | Note that all sorts on statistics are in descending order (placing |
| 482 | most time consuming items first), where as name, file, and line number |
Fred Drake | 907e76b | 2001-07-06 20:30:11 +0000 | [diff] [blame] | 483 | searches are in ascending order (alphabetical). The subtle |
Fred Drake | 2cb824c | 1998-04-09 18:10:35 +0000 | [diff] [blame] | 484 | distinction between \code{'nfl'} and \code{'stdname'} is that the |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 485 | standard name is a sort of the name as printed, which means that the |
| 486 | embedded line numbers get compared in an odd way. For example, lines |
| 487 | 3, 20, and 40 would (if the file names were the same) appear in the |
Fred Drake | 2cb824c | 1998-04-09 18:10:35 +0000 | [diff] [blame] | 488 | string order 20, 3 and 40. In contrast, \code{'nfl'} does a numeric |
| 489 | compare of the line numbers. In fact, \code{sort_stats('nfl')} is the |
| 490 | same as \code{sort_stats('name', 'file', 'line')}. |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 491 | |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 492 | %For compatibility with the old profiler, |
| 493 | For backward-compatibility reasons, the numeric arguments |
Fred Drake | 2cb824c | 1998-04-09 18:10:35 +0000 | [diff] [blame] | 494 | \code{-1}, \code{0}, \code{1}, and \code{2} are permitted. They are |
| 495 | interpreted as \code{'stdname'}, \code{'calls'}, \code{'time'}, and |
| 496 | \code{'cumulative'} respectively. If this old style format (numeric) |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 497 | is used, only one sort key (the numeric key) will be used, and |
| 498 | additional arguments will be silently ignored. |
Fred Drake | 8fe533e | 1998-03-27 05:27:08 +0000 | [diff] [blame] | 499 | \end{methoddesc} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 500 | |
| 501 | |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 502 | \begin{methoddesc}[Stats]{reverse_order}{} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 503 | This method for the \class{Stats} class reverses the ordering of the basic |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 504 | list within the object. %This method is provided primarily for |
| 505 | %compatibility with the old profiler. |
| 506 | Note that by default ascending vs descending order is properly selected |
| 507 | based on the sort key of choice. |
Fred Drake | 8fe533e | 1998-03-27 05:27:08 +0000 | [diff] [blame] | 508 | \end{methoddesc} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 509 | |
Fred Drake | 20006b2 | 2001-07-02 21:22:39 +0000 | [diff] [blame] | 510 | \begin{methoddesc}[Stats]{print_stats}{\optional{restriction, \moreargs}} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 511 | This method for the \class{Stats} class prints out a report as described |
| 512 | in the \function{profile.run()} definition. |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 513 | |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 514 | The order of the printing is based on the last \method{sort_stats()} |
| 515 | operation done on the object (subject to caveats in \method{add()} and |
Raymond Hettinger | 0e53d23 | 2003-07-14 18:24:26 +0000 | [diff] [blame] | 516 | \method{strip_dirs()}). |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 517 | |
| 518 | The arguments provided (if any) can be used to limit the list down to |
| 519 | the significant entries. Initially, the list is taken to be the |
| 520 | complete set of profiled functions. Each restriction is either an |
| 521 | integer (to select a count of lines), or a decimal fraction between |
| 522 | 0.0 and 1.0 inclusive (to select a percentage of lines), or a regular |
Guido van Rossum | 364e643 | 1997-11-18 15:28:46 +0000 | [diff] [blame] | 523 | expression (to pattern match the standard name that is printed; as of |
| 524 | Python 1.5b1, this uses the Perl-style regular expression syntax |
Fred Drake | ffbe687 | 1999-04-22 21:23:22 +0000 | [diff] [blame] | 525 | defined by the \refmodule{re} module). If several restrictions are |
Guido van Rossum | 364e643 | 1997-11-18 15:28:46 +0000 | [diff] [blame] | 526 | provided, then they are applied sequentially. For example: |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 527 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 528 | \begin{verbatim} |
Fred Drake | 2cb824c | 1998-04-09 18:10:35 +0000 | [diff] [blame] | 529 | print_stats(.1, 'foo:') |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 530 | \end{verbatim} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 531 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 532 | would first limit the printing to first 10\% of list, and then only |
Fred Drake | fee6f33 | 2004-03-23 21:40:07 +0000 | [diff] [blame] | 533 | print functions that were part of filename \file{.*foo:}. In |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 534 | contrast, the command: |
| 535 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 536 | \begin{verbatim} |
Fred Drake | 2cb824c | 1998-04-09 18:10:35 +0000 | [diff] [blame] | 537 | print_stats('foo:', .1) |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 538 | \end{verbatim} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 539 | |
Fred Drake | fee6f33 | 2004-03-23 21:40:07 +0000 | [diff] [blame] | 540 | would limit the list to all functions having file names \file{.*foo:}, |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 541 | and then proceed to only print the first 10\% of them. |
Fred Drake | 8fe533e | 1998-03-27 05:27:08 +0000 | [diff] [blame] | 542 | \end{methoddesc} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 543 | |
| 544 | |
Fred Drake | 20006b2 | 2001-07-02 21:22:39 +0000 | [diff] [blame] | 545 | \begin{methoddesc}[Stats]{print_callers}{\optional{restriction, \moreargs}} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 546 | This method for the \class{Stats} class prints a list of all functions |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 547 | that called each function in the profiled database. The ordering is |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 548 | identical to that provided by \method{print_stats()}, and the definition |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 549 | of the restricting argument is also identical. Each caller is reported on |
| 550 | its own line. The format differs slightly depending on the profiler that |
| 551 | produced the stats: |
| 552 | |
| 553 | \begin{itemize} |
| 554 | \item With \module{profile}, a number is shown in parentheses after each |
| 555 | caller to show how many times this specific call was made. For |
| 556 | convenience, a second non-parenthesized number repeats the cumulative |
| 557 | time spent in the function at the right. |
| 558 | |
| 559 | \item With \module{cProfile}, each caller is preceeded by three numbers: |
| 560 | the number of times this specific call was made, and the total and |
| 561 | cumulative times spent in the current function while it was invoked by |
| 562 | this specific caller. |
| 563 | \end{itemize} |
Fred Drake | 8fe533e | 1998-03-27 05:27:08 +0000 | [diff] [blame] | 564 | \end{methoddesc} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 565 | |
Fred Drake | 20006b2 | 2001-07-02 21:22:39 +0000 | [diff] [blame] | 566 | \begin{methoddesc}[Stats]{print_callees}{\optional{restriction, \moreargs}} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 567 | This method for the \class{Stats} class prints a list of all function |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 568 | that were called by the indicated function. Aside from this reversal |
| 569 | of direction of calls (re: called vs was called by), the arguments and |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 570 | ordering are identical to the \method{print_callers()} method. |
Fred Drake | 8fe533e | 1998-03-27 05:27:08 +0000 | [diff] [blame] | 571 | \end{methoddesc} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 572 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 573 | |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 574 | \section{Limitations \label{profile-limits}} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 575 | |
Raymond Hettinger | da26412 | 2004-12-19 20:31:46 +0000 | [diff] [blame] | 576 | One limitation has to do with accuracy of timing information. |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 577 | There is a fundamental problem with deterministic profilers involving |
| 578 | accuracy. The most obvious restriction is that the underlying ``clock'' |
| 579 | is only ticking at a rate (typically) of about .001 seconds. Hence no |
Raymond Hettinger | 999b57c | 2003-08-25 04:28:05 +0000 | [diff] [blame] | 580 | measurements will be more accurate than the underlying clock. If |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 581 | enough measurements are taken, then the ``error'' will tend to average |
| 582 | out. Unfortunately, removing this first error induces a second source |
Fred Drake | e05c3e0 | 2004-03-23 20:30:59 +0000 | [diff] [blame] | 583 | of error. |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 584 | |
| 585 | The second problem is that it ``takes a while'' from when an event is |
| 586 | dispatched until the profiler's call to get the time actually |
| 587 | \emph{gets} the state of the clock. Similarly, there is a certain lag |
| 588 | when exiting the profiler event handler from the time that the clock's |
| 589 | value was obtained (and then squirreled away), until the user's code |
| 590 | is once again executing. As a result, functions that are called many |
| 591 | times, or call many functions, will typically accumulate this error. |
| 592 | The error that accumulates in this fashion is typically less than the |
Fred Drake | 907e76b | 2001-07-06 20:30:11 +0000 | [diff] [blame] | 593 | accuracy of the clock (less than one clock tick), but it |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 594 | \emph{can} accumulate and become very significant. |
| 595 | |
| 596 | The problem is more important with \module{profile} than with the |
| 597 | lower-overhead \module{cProfile}. For this reason, \module{profile} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 598 | provides a means of calibrating itself for a given platform so that |
Fred Drake | 907e76b | 2001-07-06 20:30:11 +0000 | [diff] [blame] | 599 | this error can be probabilistically (on the average) removed. |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 600 | After the profiler is calibrated, it will be more accurate (in a least |
| 601 | square sense), but it will sometimes produce negative numbers (when |
| 602 | call counts are exceptionally low, and the gods of probability work |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 603 | against you :-). ) Do \emph{not} be alarmed by negative numbers in |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 604 | the profile. They should \emph{only} appear if you have calibrated |
| 605 | your profiler, and the results are actually better than without |
| 606 | calibration. |
| 607 | |
| 608 | |
Fred Drake | b9f1f6d | 1999-04-21 21:43:17 +0000 | [diff] [blame] | 609 | \section{Calibration \label{profile-calibration}} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 610 | |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 611 | The profiler of the \module{profile} module subtracts a constant from each |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 612 | event handling time to compensate for the overhead of calling the time |
Tim Peters | 659a603 | 2001-10-09 20:51:19 +0000 | [diff] [blame] | 613 | function, and socking away the results. By default, the constant is 0. |
| 614 | The following procedure can |
| 615 | be used to obtain a better constant for a given platform (see discussion |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 616 | in section Limitations above). |
| 617 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 618 | \begin{verbatim} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 619 | import profile |
| 620 | pr = profile.Profile() |
Tim Peters | 659a603 | 2001-10-09 20:51:19 +0000 | [diff] [blame] | 621 | for i in range(5): |
| 622 | print pr.calibrate(10000) |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 623 | \end{verbatim} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 624 | |
Tim Peters | 659a603 | 2001-10-09 20:51:19 +0000 | [diff] [blame] | 625 | The method executes the number of Python calls given by the argument, |
| 626 | directly and again under the profiler, measuring the time for both. |
| 627 | It then computes the hidden overhead per profiler event, and returns |
| 628 | that as a float. For example, on an 800 MHz Pentium running |
| 629 | Windows 2000, and using Python's time.clock() as the timer, |
| 630 | the magical number is about 12.5e-6. |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 631 | |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 632 | The object of this exercise is to get a fairly consistent result. |
Tim Peters | 659a603 | 2001-10-09 20:51:19 +0000 | [diff] [blame] | 633 | If your computer is \emph{very} fast, or your timer function has poor |
| 634 | resolution, you might have to pass 100000, or even 1000000, to get |
| 635 | consistent results. |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 636 | |
Tim Peters | 659a603 | 2001-10-09 20:51:19 +0000 | [diff] [blame] | 637 | When you have a consistent answer, |
| 638 | there are three ways you can use it:\footnote{Prior to Python 2.2, it |
| 639 | was necessary to edit the profiler source code to embed the bias as |
| 640 | a literal number. You still can, but that method is no longer |
| 641 | described, because no longer needed.} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 642 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 643 | \begin{verbatim} |
Tim Peters | 659a603 | 2001-10-09 20:51:19 +0000 | [diff] [blame] | 644 | import profile |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 645 | |
Tim Peters | 659a603 | 2001-10-09 20:51:19 +0000 | [diff] [blame] | 646 | # 1. Apply computed bias to all Profile instances created hereafter. |
Tim Peters | 8cd015c | 2001-10-09 20:54:23 +0000 | [diff] [blame] | 647 | profile.Profile.bias = your_computed_bias |
Tim Peters | 659a603 | 2001-10-09 20:51:19 +0000 | [diff] [blame] | 648 | |
| 649 | # 2. Apply computed bias to a specific Profile instance. |
| 650 | pr = profile.Profile() |
| 651 | pr.bias = your_computed_bias |
| 652 | |
| 653 | # 3. Specify computed bias in instance constructor. |
| 654 | pr = profile.Profile(bias=your_computed_bias) |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 655 | \end{verbatim} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 656 | |
Tim Peters | 659a603 | 2001-10-09 20:51:19 +0000 | [diff] [blame] | 657 | If you have a choice, you are better off choosing a smaller constant, and |
| 658 | then your results will ``less often'' show up as negative in profile |
| 659 | statistics. |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 660 | |
| 661 | |
Guido van Rossum | 86cb092 | 1995-03-20 12:59:56 +0000 | [diff] [blame] | 662 | \section{Extensions --- Deriving Better Profilers} |
| 663 | \nodename{Profiler Extensions} |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 664 | |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 665 | The \class{Profile} class of both modules, \module{profile} and |
| 666 | \module{cProfile}, were written so that |
Tim Peters | 0a1fc4e | 2001-10-07 03:12:08 +0000 | [diff] [blame] | 667 | derived classes could be developed to extend the profiler. The details |
| 668 | are not described here, as doing this successfully requires an expert |
| 669 | understanding of how the \class{Profile} class works internally. Study |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 670 | the source code of the module carefully if you want to |
Tim Peters | 0a1fc4e | 2001-10-07 03:12:08 +0000 | [diff] [blame] | 671 | pursue this. |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 672 | |
Tim Peters | 0a1fc4e | 2001-10-07 03:12:08 +0000 | [diff] [blame] | 673 | If all you want to do is change how current time is determined (for |
| 674 | example, to force use of wall-clock time or elapsed process time), |
| 675 | pass the timing function you want to the \class{Profile} class |
| 676 | constructor: |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 677 | |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 678 | \begin{verbatim} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 679 | pr = profile.Profile(your_time_func) |
Fred Drake | 1947991 | 1998-02-13 06:58:54 +0000 | [diff] [blame] | 680 | \end{verbatim} |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 681 | |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 682 | The resulting profiler will then call \function{your_time_func()}. |
| 683 | |
| 684 | \begin{description} |
| 685 | \item[\class{profile.Profile}] |
| 686 | \function{your_time_func()} should return a single number, or a list of |
Tim Peters | 0a1fc4e | 2001-10-07 03:12:08 +0000 | [diff] [blame] | 687 | numbers whose sum is the current time (like what \function{os.times()} |
| 688 | returns). If the function returns a single time number, or the list of |
| 689 | returned numbers has length 2, then you will get an especially fast |
| 690 | version of the dispatch routine. |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 691 | |
Tim Peters | 0a1fc4e | 2001-10-07 03:12:08 +0000 | [diff] [blame] | 692 | Be warned that you should calibrate the profiler class for the |
Guido van Rossum | df804f8 | 1995-03-02 12:38:39 +0000 | [diff] [blame] | 693 | timer function that you choose. For most machines, a timer that |
| 694 | returns a lone integer value will provide the best results in terms of |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 695 | low overhead during profiling. (\function{os.times()} is |
Tim Peters | 0a1fc4e | 2001-10-07 03:12:08 +0000 | [diff] [blame] | 696 | \emph{pretty} bad, as it returns a tuple of floating point values). If |
| 697 | you want to substitute a better timer in the cleanest fashion, |
| 698 | derive a class and hardwire a replacement dispatch method that best |
Fred Drake | 8fa5eb8 | 1998-02-27 05:23:37 +0000 | [diff] [blame] | 699 | handles your timer call, along with the appropriate calibration |
Fred Drake | 62f9d7c | 2001-06-08 05:04:19 +0000 | [diff] [blame] | 700 | constant. |
Armin Rigo | a871ef2 | 2006-02-08 12:53:56 +0000 | [diff] [blame] | 701 | |
| 702 | \item[\class{cProfile.Profile}] |
| 703 | \function{your_time_func()} should return a single number. If it returns |
| 704 | plain integers, you can also invoke the class constructor with a second |
| 705 | argument specifying the real duration of one unit of time. For example, |
| 706 | if \function{your_integer_time_func()} returns times measured in thousands |
| 707 | of seconds, you would constuct the \class{Profile} instance as follows: |
| 708 | |
| 709 | \begin{verbatim} |
| 710 | pr = profile.Profile(your_integer_time_func, 0.001) |
| 711 | \end{verbatim} |
| 712 | |
| 713 | As the \module{cProfile.Profile} class cannot be calibrated, custom |
| 714 | timer functions should be used with care and should be as fast as |
| 715 | possible. For the best results with a custom timer, it might be |
| 716 | necessary to hard-code it in the C source of the internal |
| 717 | \module{_lsprof} module. |
| 718 | |
| 719 | \end{description} |