blob: e49d4ca4512c9991ec8543fdbce19e55d7102a5e [file] [log] [blame]
Victor Stinnered3b0bc2013-11-23 12:27:24 +01001:mod:`tracemalloc` --- Trace memory allocations
2===============================================
3
4.. module:: tracemalloc
5 :synopsis: Trace memory allocations.
6
R David Murray6d5cb072013-12-20 14:48:50 -05007.. versionadded:: 3.4
8
Victor Stinnered3b0bc2013-11-23 12:27:24 +01009The tracemalloc module is a debug tool to trace memory blocks allocated by
10Python. It provides the following information:
11
12* Traceback where an object was allocated
13* Statistics on allocated memory blocks per filename and per line number:
14 total size, number and average size of allocated memory blocks
15* Compute the differences between two snapshots to detect memory leaks
16
17To trace most memory blocks allocated by Python, the module should be started
18as early as possible by setting the :envvar:`PYTHONTRACEMALLOC` environment
19variable to ``1``, or by using :option:`-X` ``tracemalloc`` command line
20option. The :func:`tracemalloc.start` function can be called at runtime to
21start tracing Python memory allocations.
22
23By default, a trace of an allocated memory block only stores the most recent
24frame (1 frame). To store 25 frames at startup: set the
25:envvar:`PYTHONTRACEMALLOC` environment variable to ``25``, or use the
Victor Stinner3728d6c2013-11-23 12:37:20 +010026:option:`-X` ``tracemalloc=25`` command line option.
Victor Stinnered3b0bc2013-11-23 12:27:24 +010027
Victor Stinnered3b0bc2013-11-23 12:27:24 +010028
29Examples
30========
31
32Display the top 10
33------------------
34
35Display the 10 files allocating the most memory::
36
37 import tracemalloc
38
39 tracemalloc.start()
40
41 # ... run your application ...
42
43 snapshot = tracemalloc.take_snapshot()
44 top_stats = snapshot.statistics('lineno')
45
46 print("[ Top 10 ]")
47 for stat in top_stats[:10]:
48 print(stat)
49
50
51Example of output of the Python test suite::
52
53 [ Top 10 ]
54 <frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B
55 <frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 B
56 /usr/lib/python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B
57 /usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B
58 /usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B
59 /usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B
60 <frozen importlib._bootstrap>:1446: size=70.4 KiB, count=911, average=79 B
61 <frozen importlib._bootstrap>:1454: size=52.0 KiB, count=25, average=2131 B
62 <string>:5: size=49.7 KiB, count=148, average=344 B
63 /usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB
64
65We can see that Python loaded ``4.8 MiB`` data (bytecode and constants) from
66modules and that the :mod:`collections` module allocated ``244 KiB`` to build
67:class:`~collections.namedtuple` types.
68
69See :meth:`Snapshot.statistics` for more options.
70
71
72Compute differences
73-------------------
74
75Take two snapshots and display the differences::
76
77 import tracemalloc
78 tracemalloc.start()
79 # ... start your application ...
80
81 snapshot1 = tracemalloc.take_snapshot()
82 # ... call the function leaking memory ...
83 snapshot2 = tracemalloc.take_snapshot()
84
85 top_stats = snapshot2.compare_to(snapshot1, 'lineno')
86
87 print("[ Top 10 differences ]")
88 for stat in top_stats[:10]:
89 print(stat)
90
91Example of output before/after running some tests of the Python test suite::
92
93 [ Top 10 differences ]
94 <frozen importlib._bootstrap>:716: size=8173 KiB (+4428 KiB), count=71332 (+39369), average=117 B
95 /usr/lib/python3.4/linecache.py:127: size=940 KiB (+940 KiB), count=8106 (+8106), average=119 B
96 /usr/lib/python3.4/unittest/case.py:571: size=298 KiB (+298 KiB), count=589 (+589), average=519 B
97 <frozen importlib._bootstrap>:284: size=1005 KiB (+166 KiB), count=7423 (+1526), average=139 B
98 /usr/lib/python3.4/mimetypes.py:217: size=112 KiB (+112 KiB), count=1334 (+1334), average=86 B
99 /usr/lib/python3.4/http/server.py:848: size=96.0 KiB (+96.0 KiB), count=1 (+1), average=96.0 KiB
100 /usr/lib/python3.4/inspect.py:1465: size=83.5 KiB (+83.5 KiB), count=109 (+109), average=784 B
101 /usr/lib/python3.4/unittest/mock.py:491: size=77.7 KiB (+77.7 KiB), count=143 (+143), average=557 B
102 /usr/lib/python3.4/urllib/parse.py:476: size=71.8 KiB (+71.8 KiB), count=969 (+969), average=76 B
103 /usr/lib/python3.4/contextlib.py:38: size=67.2 KiB (+67.2 KiB), count=126 (+126), average=546 B
104
Victor Stinner00773df2013-11-26 00:40:10 +0100105We can see that Python has loaded ``8.2 MiB`` of module data (bytecode and
106constants), and that this is ``4.4 MiB`` more than had been loaded before the
107tests, when the previous snapshot was taken. Similarly, the :mod:`linecache`
108module has cached ``940 KiB`` of Python source code to format tracebacks, all
109of it since the previous snapshot.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100110
111If the system has little free memory, snapshots can be written on disk using
112the :meth:`Snapshot.dump` method to analyze the snapshot offline. Then use the
113:meth:`Snapshot.load` method reload the snapshot.
114
115
116Get the traceback of a memory block
117-----------------------------------
118
119Code to display the traceback of the biggest memory block::
120
121 import linecache
122 import tracemalloc
123
Victor Stinner3728d6c2013-11-23 12:37:20 +0100124 # Store 25 frames
125 tracemalloc.start(25)
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100126
127 # ... run your application ...
128
129 snapshot = tracemalloc.take_snapshot()
130 top_stats = snapshot.statistics('traceback')
131
132 # pick the biggest memory block
133 stat = top_stats[0]
134 print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024))
135 for frame in stat.traceback:
136 print(' File "%s", line %s' % (frame.filename, frame.lineno))
137 line = linecache.getline(frame.filename, frame.lineno)
138 line = line.strip()
139 if line:
140 print(' ' + line)
141
142Example of output of the Python test suite (traceback limited to 25 frames)::
143
144 903 memory blocks: 870.1 KiB
145 File "<frozen importlib._bootstrap>", line 716
146 File "<frozen importlib._bootstrap>", line 1036
147 File "<frozen importlib._bootstrap>", line 934
148 File "<frozen importlib._bootstrap>", line 1068
149 File "<frozen importlib._bootstrap>", line 619
150 File "<frozen importlib._bootstrap>", line 1581
151 File "<frozen importlib._bootstrap>", line 1614
152 File "/usr/lib/python3.4/doctest.py", line 101
153 import pdb
154 File "<frozen importlib._bootstrap>", line 284
155 File "<frozen importlib._bootstrap>", line 938
156 File "<frozen importlib._bootstrap>", line 1068
157 File "<frozen importlib._bootstrap>", line 619
158 File "<frozen importlib._bootstrap>", line 1581
159 File "<frozen importlib._bootstrap>", line 1614
160 File "/usr/lib/python3.4/test/support/__init__.py", line 1728
161 import doctest
162 File "/usr/lib/python3.4/test/test_pickletools.py", line 21
163 support.run_doctest(pickletools)
164 File "/usr/lib/python3.4/test/regrtest.py", line 1276
165 test_runner()
166 File "/usr/lib/python3.4/test/regrtest.py", line 976
167 display_failure=not verbose)
168 File "/usr/lib/python3.4/test/regrtest.py", line 761
169 match_tests=ns.match_tests)
170 File "/usr/lib/python3.4/test/regrtest.py", line 1563
171 main()
172 File "/usr/lib/python3.4/test/__main__.py", line 3
173 regrtest.main_in_temp_cwd()
174 File "/usr/lib/python3.4/runpy.py", line 73
175 exec(code, run_globals)
176 File "/usr/lib/python3.4/runpy.py", line 160
177 "__main__", fname, loader, pkg_name)
178
Victor Stinner00773df2013-11-26 00:40:10 +0100179We can see that the most memory was allocated in the :mod:`importlib` module to
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100180load data (bytecode and constants) from modules: ``870 KiB``. The traceback is
Victor Stinner00773df2013-11-26 00:40:10 +0100181where the :mod:`importlib` loaded data most recently: on the ``import pdb``
182line of the :mod:`doctest` module. The traceback may change if a new module is
183loaded.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100184
185
186Pretty top
187----------
188
189Code to display the 10 lines allocating the most memory with a pretty output,
190ignoring ``<frozen importlib._bootstrap>`` and ``<unknown>`` files::
191
192 import os
193 import tracemalloc
194
195 def display_top(snapshot, group_by='lineno', limit=10):
196 snapshot = snapshot.filter_traces((
197 tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
198 tracemalloc.Filter(False, "<unknown>"),
199 ))
200 top_stats = snapshot.statistics(group_by)
201
202 print("Top %s lines" % limit)
203 for index, stat in enumerate(top_stats[:limit], 1):
204 frame = stat.traceback[0]
205 # replace "/path/to/module/file.py" with "module/file.py"
206 filename = os.sep.join(frame.filename.split(os.sep)[-2:])
207 print("#%s: %s:%s: %.1f KiB"
208 % (index, filename, frame.lineno,
209 stat.size / 1024))
210
211 other = top_stats[limit:]
212 if other:
213 size = sum(stat.size for stat in other)
214 print("%s other: %.1f KiB" % (len(other), size / 1024))
215 total = sum(stat.size for stat in top_stats)
216 print("Total allocated size: %.1f KiB" % (total / 1024))
217
218 tracemalloc.start()
219
220 # ... run your application ...
221
222 snapshot = tracemalloc.take_snapshot()
223 display_top(snapshot, 10)
224
225Example of output of the Python test suite::
226
227 2013-11-08 14:16:58.149320: Top 10 lines
228 #1: collections/__init__.py:368: 291.9 KiB
229 #2: Lib/doctest.py:1291: 200.2 KiB
230 #3: unittest/case.py:571: 160.3 KiB
231 #4: Lib/abc.py:133: 99.8 KiB
232 #5: urllib/parse.py:476: 71.8 KiB
233 #6: <string>:5: 62.7 KiB
234 #7: Lib/base64.py:140: 59.8 KiB
235 #8: Lib/_weakrefset.py:37: 51.8 KiB
236 #9: collections/__init__.py:362: 50.6 KiB
237 #10: test/test_site.py:56: 48.0 KiB
238 7496 other: 4161.9 KiB
239 Total allocated size: 5258.8 KiB
240
241See :meth:`Snapshot.statistics` for more options.
242
243
244API
245===
246
247Functions
248---------
249
250.. function:: clear_traces()
251
252 Clear traces of memory blocks allocated by Python.
253
254 See also :func:`stop`.
255
256
257.. function:: get_object_traceback(obj)
258
259 Get the traceback where the Python object *obj* was allocated.
260 Return a :class:`Traceback` instance, or ``None`` if the :mod:`tracemalloc`
261 module is not tracing memory allocations or did not trace the allocation of
262 the object.
263
264 See also :func:`gc.get_referrers` and :func:`sys.getsizeof` functions.
265
266
267.. function:: get_traceback_limit()
268
269 Get the maximum number of frames stored in the traceback of a trace.
270
Victor Stinner3728d6c2013-11-23 12:37:20 +0100271 The :mod:`tracemalloc` module must be tracing memory allocations to
272 get the limit, otherwise an exception is raised.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100273
Victor Stinner3728d6c2013-11-23 12:37:20 +0100274 The limit is set by the :func:`start` function.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100275
276
277.. function:: get_traced_memory()
278
Victor Stinner3c0481d2013-11-27 21:39:49 +0100279 Get the current size and peak size of memory blocks traced by the
280 :mod:`tracemalloc` module as a tuple: ``(current: int, peak: int)``.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100281
282
283.. function:: get_tracemalloc_memory()
284
285 Get the memory usage in bytes of the :mod:`tracemalloc` module used to store
286 traces of memory blocks.
287 Return an :class:`int`.
288
289
290.. function:: is_tracing()
291
292 ``True`` if the :mod:`tracemalloc` module is tracing Python memory
293 allocations, ``False`` otherwise.
294
295 See also :func:`start` and :func:`stop` functions.
296
297
Victor Stinner3728d6c2013-11-23 12:37:20 +0100298.. function:: start(nframe: int=1)
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100299
Victor Stinner3728d6c2013-11-23 12:37:20 +0100300 Start tracing Python memory allocations: install hooks on Python memory
301 allocators. Collected tracebacks of traces will be limited to *nframe*
302 frames. By default, a trace of a memory block only stores the most recent
303 frame: the limit is ``1``. *nframe* must be greater or equal to ``1``.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100304
305 Storing more than ``1`` frame is only useful to compute statistics grouped
306 by ``'traceback'`` or to compute cumulative statistics: see the
307 :meth:`Snapshot.compare_to` and :meth:`Snapshot.statistics` methods.
308
309 Storing more frames increases the memory and CPU overhead of the
310 :mod:`tracemalloc` module. Use the :func:`get_tracemalloc_memory` function
311 to measure how much memory is used by the :mod:`tracemalloc` module.
312
313 The :envvar:`PYTHONTRACEMALLOC` environment variable
314 (``PYTHONTRACEMALLOC=NFRAME``) and the :option:`-X` ``tracemalloc=NFRAME``
Victor Stinner3728d6c2013-11-23 12:37:20 +0100315 command line option can be used to start tracing at startup.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100316
Victor Stinner3728d6c2013-11-23 12:37:20 +0100317 See also :func:`stop`, :func:`is_tracing` and :func:`get_traceback_limit`
318 functions.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100319
320
321.. function:: stop()
322
323 Stop tracing Python memory allocations: uninstall hooks on Python memory
Victor Stinner00773df2013-11-26 00:40:10 +0100324 allocators. Also clears all previously collected traces of memory blocks
325 allocated by Python.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100326
327 Call :func:`take_snapshot` function to take a snapshot of traces before
328 clearing them.
329
Victor Stinner00773df2013-11-26 00:40:10 +0100330 See also :func:`start`, :func:`is_tracing` and :func:`clear_traces`
331 functions.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100332
333
334.. function:: take_snapshot()
335
336 Take a snapshot of traces of memory blocks allocated by Python. Return a new
337 :class:`Snapshot` instance.
338
339 The snapshot does not include memory blocks allocated before the
340 :mod:`tracemalloc` module started to trace memory allocations.
341
342 Tracebacks of traces are limited to :func:`get_traceback_limit` frames. Use
Victor Stinner3728d6c2013-11-23 12:37:20 +0100343 the *nframe* parameter of the :func:`start` function to store more frames.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100344
345 The :mod:`tracemalloc` module must be tracing memory allocations to take a
346 snapshot, see the the :func:`start` function.
347
348 See also the :func:`get_object_traceback` function.
349
350
351Filter
352------
353
354.. class:: Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False)
355
356 Filter on traces of memory blocks.
357
358 See the :func:`fnmatch.fnmatch` function for the syntax of
359 *filename_pattern*. The ``'.pyc'`` and ``'.pyo'`` file extensions are
360 replaced with ``'.py'``.
361
362 Examples:
363
364 * ``Filter(True, subprocess.__file__)`` only includes traces of the
365 :mod:`subprocess` module
366 * ``Filter(False, tracemalloc.__file__)`` excludes traces of the
367 :mod:`tracemalloc` module
368 * ``Filter(False, "<unknown>")`` excludes empty tracebacks
369
370 .. attribute:: inclusive
371
372 If *inclusive* is ``True`` (include), only trace memory blocks allocated
373 in a file with a name matching :attr:`filename_pattern` at line number
374 :attr:`lineno`.
375
376 If *inclusive* is ``False`` (exclude), ignore memory blocks allocated in
377 a file with a name matching :attr:`filename_pattern` at line number
378 :attr:`lineno`.
379
380 .. attribute:: lineno
381
382 Line number (``int``) of the filter. If *lineno* is ``None``, the filter
383 matches any line number.
384
385 .. attribute:: filename_pattern
386
387 Filename pattern of the filter (``str``).
388
389 .. attribute:: all_frames
390
391 If *all_frames* is ``True``, all frames of the traceback are checked. If
392 *all_frames* is ``False``, only the most recent frame is checked.
393
Victor Stinner5362abf2013-11-27 23:39:55 +0100394 This attribute has no effect if the traceback limit is ``1``. See the
395 :func:`get_traceback_limit` function and :attr:`Snapshot.traceback_limit`
396 attribute.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100397
398
399Frame
400-----
401
402.. class:: Frame
403
404 Frame of a traceback.
405
406 The :class:`Traceback` class is a sequence of :class:`Frame` instances.
407
408 .. attribute:: filename
409
410 Filename (``str``).
411
412 .. attribute:: lineno
413
414 Line number (``int``).
415
416
417Snapshot
418--------
419
420.. class:: Snapshot
421
422 Snapshot of traces of memory blocks allocated by Python.
423
424 The :func:`take_snapshot` function creates a snapshot instance.
425
426 .. method:: compare_to(old_snapshot: Snapshot, group_by: str, cumulative: bool=False)
427
428 Compute the differences with an old snapshot. Get statistics as a sorted
429 list of :class:`StatisticDiff` instances grouped by *group_by*.
430
431 See the :meth:`statistics` method for *group_by* and *cumulative*
432 parameters.
433
434 The result is sorted from the biggest to the smallest by: absolute value
435 of :attr:`StatisticDiff.size_diff`, :attr:`StatisticDiff.size`, absolute
436 value of :attr:`StatisticDiff.count_diff`, :attr:`Statistic.count` and
437 then by :attr:`StatisticDiff.traceback`.
438
439
440 .. method:: dump(filename)
441
442 Write the snapshot into a file.
443
444 Use :meth:`load` to reload the snapshot.
445
446
447 .. method:: filter_traces(filters)
448
449 Create a new :class:`Snapshot` instance with a filtered :attr:`traces`
450 sequence, *filters* is a list of :class:`Filter` instances. If *filters*
451 is an empty list, return a new :class:`Snapshot` instance with a copy of
452 the traces.
453
454 All inclusive filters are applied at once, a trace is ignored if no
455 inclusive filters match it. A trace is ignored if at least one exclusive
456 filter matchs it.
457
458
459 .. classmethod:: load(filename)
460
461 Load a snapshot from a file.
462
463 See also :meth:`dump`.
464
465
466 .. method:: statistics(group_by: str, cumulative: bool=False)
467
468 Get statistics as a sorted list of :class:`Statistic` instances grouped
469 by *group_by*:
470
471 ===================== ========================
472 group_by description
473 ===================== ========================
474 ``'filename'`` filename
475 ``'lineno'`` filename and line number
476 ``'traceback'`` traceback
477 ===================== ========================
478
479 If *cumulative* is ``True``, cumulate size and count of memory blocks of
480 all frames of the traceback of a trace, not only the most recent frame.
481 The cumulative mode can only be used with *group_by* equals to
Victor Stinner8e3708d2013-11-26 00:45:47 +0100482 ``'filename'`` and ``'lineno'``.
Victor Stinnered3b0bc2013-11-23 12:27:24 +0100483
484 The result is sorted from the biggest to the smallest by:
485 :attr:`Statistic.size`, :attr:`Statistic.count` and then by
486 :attr:`Statistic.traceback`.
487
488
489 .. attribute:: traceback_limit
490
491 Maximum number of frames stored in the traceback of :attr:`traces`:
492 result of the :func:`get_traceback_limit` when the snapshot was taken.
493
494 .. attribute:: traces
495
496 Traces of all memory blocks allocated by Python: sequence of
497 :class:`Trace` instances.
498
499 The sequence has an undefined order. Use the :meth:`Snapshot.statistics`
500 method to get a sorted list of statistics.
501
502
503Statistic
504---------
505
506.. class:: Statistic
507
508 Statistic on memory allocations.
509
510 :func:`Snapshot.statistics` returns a list of :class:`Statistic` instances.
511
512 See also the :class:`StatisticDiff` class.
513
514 .. attribute:: count
515
516 Number of memory blocks (``int``).
517
518 .. attribute:: size
519
520 Total size of memory blocks in bytes (``int``).
521
522 .. attribute:: traceback
523
524 Traceback where the memory block was allocated, :class:`Traceback`
525 instance.
526
527
528StatisticDiff
529-------------
530
531.. class:: StatisticDiff
532
533 Statistic difference on memory allocations between an old and a new
534 :class:`Snapshot` instance.
535
536 :func:`Snapshot.compare_to` returns a list of :class:`StatisticDiff`
537 instances. See also the :class:`Statistic` class.
538
539 .. attribute:: count
540
541 Number of memory blocks in the new snapshot (``int``): ``0`` if
542 the memory blocks have been released in the new snapshot.
543
544 .. attribute:: count_diff
545
546 Difference of number of memory blocks between the old and the new
547 snapshots (``int``): ``0`` if the memory blocks have been allocated in
548 the new snapshot.
549
550 .. attribute:: size
551
552 Total size of memory blocks in bytes in the new snapshot (``int``):
553 ``0`` if the memory blocks have been released in the new snapshot.
554
555 .. attribute:: size_diff
556
557 Difference of total size of memory blocks in bytes between the old and
558 the new snapshots (``int``): ``0`` if the memory blocks have been
559 allocated in the new snapshot.
560
561 .. attribute:: traceback
562
563 Traceback where the memory blocks were allocated, :class:`Traceback`
564 instance.
565
566
567Trace
568-----
569
570.. class:: Trace
571
572 Trace of a memory block.
573
574 The :attr:`Snapshot.traces` attribute is a sequence of :class:`Trace`
575 instances.
576
577 .. attribute:: size
578
579 Size of the memory block in bytes (``int``).
580
581 .. attribute:: traceback
582
583 Traceback where the memory block was allocated, :class:`Traceback`
584 instance.
585
586
587Traceback
588---------
589
590.. class:: Traceback
591
592 Sequence of :class:`Frame` instances sorted from the most recent frame to
593 the oldest frame.
594
595 A traceback contains at least ``1`` frame. If the ``tracemalloc`` module
596 failed to get a frame, the filename ``"<unknown>"`` at line number ``0`` is
597 used.
598
599 When a snapshot is taken, tracebacks of traces are limited to
600 :func:`get_traceback_limit` frames. See the :func:`take_snapshot` function.
601
602 The :attr:`Trace.traceback` attribute is an instance of :class:`Traceback`
603 instance.
604
605