tracemalloc —- 跟踪内存分配

3.4 新版功能.

源代码： Lib/tracemalloc.py

---

tracemalloc 模块是一个用于对 python 已申请的内存块进行debug的工具。它能提供以下信息:

• 回溯对象分配内存的位置

• 按文件、按行统计python的内存块分配情况: 内存块总大小、数量以及块平均大小。

• 对比两个内存快照的差异，以便排查内存泄漏

To trace most memory blocks allocated by Python, the module should be started as early as possible by setting the PYTHONTRACEMALLOC environment variable to 1, or by using -X tracemalloc command line option. The tracemalloc.start() function can be called at runtime to start tracing Python memory allocations.

By default, a trace of an allocated memory block only stores the most recent frame (1 frame). To store 25 frames at startup: set the PYTHONTRACEMALLOC environment variable to 25, or use the -X tracemalloc=25 command line option.

例子

显示前10项

显示内存分配最多的10个文件：

 
 
 
 

  
  
  
  
import tracemalloc
  
  
  
  
  
  
  
  
tracemalloc.start()
  
  
  
  
  
  
  
  
# ... run your application ...
  
  
  
  
  
  
  
  
snapshot = tracemalloc.take_snapshot()
  
  
  
  
top_stats = snapshot.statistics('lineno')
  
  
  
  
  
  
  
  
print("[ Top 10 ]")
  
  
  
  
for stat in top_stats[:10]:
  
  
  
  
 print(stat)
 
 
 
 

Python测试套件的输出示例：

 
 
 
 

  
  
  
  
[ Top 10 ]
  
  
  
  
:716: size=4855 KiB, count=39328, average=126 B
  
  
  
  
:284: size=521 KiB, count=3199, average=167 B
  
  
  
  
/usr/lib/Python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B
  
  
  
  
/usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B
  
  
  
  
/usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B
  
  
  
  
/usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B
  
  
  
  
:1446: size=70.4 KiB, count=911, average=79 B
  
  
  
  
:1454: size=52.0 KiB, count=25, average=2131 B
  
  
  
  
:5: size=49.7 KiB, count=148, average=344 B
  
  
  
  
/usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB
 
 
 
 

We can see that Python loaded 4855 KiB data (bytecode and constants) from modules and that the collections module allocated 244 KiB to build namedtuple types.

更多选项，请参见 Snapshot.statistics()

计算差异

获取两个快照并显示差异：

 
 
 
 

  
  
  
  
import tracemalloc
  
  
  
  
tracemalloc.start()
  
  
  
  
# ... start your application ...
  
  
  
  
  
  
  
  
snapshot1 = tracemalloc.take_snapshot()
  
  
  
  
# ... call the function leaking memory ...
  
  
  
  
snapshot2 = tracemalloc.take_snapshot()
  
  
  
  
  
  
  
  
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
  
  
  
  
  
  
  
  
print("[ Top 10 differences ]")
  
  
  
  
for stat in top_stats[:10]:
  
  
  
  
 print(stat)
 
 
 
 

Example of output before/after running some tests of the Python test suite:

 
 
 
 

  
  
  
  
[ Top 10 differences ]
  
  
  
  
:716: size=8173 KiB (+4428 KiB), count=71332 (+39369), average=117 B
  
  
  
  
/usr/lib/python3.4/linecache.py:127: size=940 KiB (+940 KiB), count=8106 (+8106), average=119 B
  
  
  
  
/usr/lib/python3.4/unittest/case.py:571: size=298 KiB (+298 KiB), count=589 (+589), average=519 B
  
  
  
  
:284: size=1005 KiB (+166 KiB), count=7423 (+1526), average=139 B
  
  
  
  
/usr/lib/python3.4/mimetypes.py:217: size=112 KiB (+112 KiB), count=1334 (+1334), average=86 B
  
  
  
  
/usr/lib/python3.4/http/server.py:848: size=96.0 KiB (+96.0 KiB), count=1 (+1), average=96.0 KiB
  
  
  
  
/usr/lib/python3.4/inspect.py:1465: size=83.5 KiB (+83.5 KiB), count=109 (+109), average=784 B
  
  
  
  
/usr/lib/python3.4/unittest/mock.py:491: size=77.7 KiB (+77.7 KiB), count=143 (+143), average=557 B
  
  
  
  
/usr/lib/python3.4/urllib/parse.py:476: size=71.8 KiB (+71.8 KiB), count=969 (+969), average=76 B
  
  
  
  
/usr/lib/python3.4/contextlib.py:38: size=67.2 KiB (+67.2 KiB), count=126 (+126), average=546 B
 
 
 
 

We can see that Python has loaded 8173 KiB of module data (bytecode and constants), and that this is 4428 KiB more than had been loaded before the tests, when the previous snapshot was taken. Similarly, the linecache module has cached 940 KiB of Python source code to format tracebacks, all of it since the previous snapshot.

If the system has little free memory, snapshots can be written on disk using the Snapshot.dump() method to analyze the snapshot offline. Then use the Snapshot.load() method reload the snapshot.

获取一个内存块的溯源

一段找出程序中最大内存块溯源的代码:

 
 
 
 

  
  
  
  
import tracemalloc
  
  
  
  
  
  
  
  
# Store 25 frames
  
  
  
  
tracemalloc.start(25)
  
  
  
  
  
  
  
  
# ... run your application ...
  
  
  
  
  
  
  
  
snapshot = tracemalloc.take_snapshot()
  
  
  
  
top_stats = snapshot.statistics('traceback')
  
  
  
  
  
  
  
  
# pick the biggest memory block
  
  
  
  
stat = top_stats[0]
  
  
  
  
print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024))
  
  
  
  
for line in stat.traceback.format():
  
  
  
  
 print(line)
 
 
 
 

一段Python单元测试的输出案例(限制最大25层堆栈)

 
 
 
 

  
  
  
  
903 memory blocks: 870.1 KiB
  
  
  
  
 File "", line 716
  
  
  
  
 File "", line 1036
  
  
  
  
 File "", line 934
  
  
  
  
 File "", line 1068
  
  
  
  
 File "", line 619
  
  
  
  
 File "", line 1581
  
  
  
  
 File "", line 1614
  
  
  
  
 File "/usr/lib/python3.4/doctest.py", line 101
  
  
  
  
 import pdb
  
  
  
  
 File "", line 284
  
  
  
  
 File "", line 938
  
  
  
  
 File "", line 1068
  
  
  
  
 File "", line 619
  
  
  
  
 File "", line 1581
  
  
  
  
 File "", line 1614
  
  
  
  
 File "/usr/lib/python3.4/test/support/__init__.py", line 1728
  
  
  
  
 import doctest
  
  
  
  
 File "/usr/lib/python3.4/test/test_pickletools.py", line 21
  
  
  
  
 support.run_doctest(pickletools)
  
  
  
  
 File "/usr/lib/python3.4/test/regrtest.py", line 1276
  
  
  
  
 test_runner()
  
  
  
  
 File "/usr/lib/python3.4/test/regrtest.py", line 976
  
  
  
  
 display_failure=not verbose)
  
  
  
  
 File "/usr/lib/python3.4/test/regrtest.py", line 761
  
  
  
  
 match_tests=ns.match_tests)
  
  
  
  
 File "/usr/lib/python3.4/test/regrtest.py", line 1563
  
  
  
  
 main()
  
  
  
  
 File "/usr/lib/python3.4/test/__main__.py", line 3
  
  
  
  
 regrtest.main_in_temp_cwd()
  
  
  
  
 File "/usr/lib/python3.4/runpy.py", line 73
  
  
  
  
 exec(code, run_globals)
  
  
  
  
 File "/usr/lib/python3.4/runpy.py", line 160
  
  
  
  
 "__main__", fname, loader, pkg_name)
 
 
 
 

We can see that the most memory was allocated in the importlib module to load data (bytecode and constants) from modules: 870.1 KiB. The traceback is where the importlib loaded data most recently: on the import pdb line of the doctest module. The traceback may change if a new module is loaded.

Pretty top

Code to display the 10 lines allocating the most memory with a pretty output, ignoring and files:

 
 
 
 

  
  
  
  
import linecache
  
  
  
  
import os
  
  
  
  
import tracemalloc
  
  
  
  
  
  
  
  
def display_top(snapshot, key_type='lineno', limit=10):
  
  
  
  
 snapshot = snapshot.filter_traces((
  
  
  
  
 tracemalloc.Filter(False, ""),
  
  
  
  
 tracemalloc.Filter(False, ""),
  
  
  
  
 ))
  
  
  
  
 top_stats = snapshot.statistics(key_type)
  
  
  
  
  
  
  
  
 print("Top %s lines" % limit)
  
  
  
  
 for index, stat in enumerate(top_stats[:limit], 1):
  
  
  
  
 frame = stat.traceback[0]
  
  
  
  
 print("#%s: %s:%s: %.1f KiB"
  
  
  
  
 % (index, frame.filename, frame.lineno, stat.size / 1024))
  
  
  
  
 line = linecache.getline(frame.filename, frame.lineno).strip()
  
  
  
  
 if line:
  
  
  
  
 print(' %s' % line)
  
  
  
  
  
  
  
  
 other = top_stats[limit:]
  
  
  
  
 if other:
  
  
  
  
 size = sum(stat.size for stat in other)
  
  
  
  
 print("%s other: %.1f KiB" % (len(other), size / 1024))
  
  
  
  
 total = sum(stat.size for stat in top_stats)
  
  
  
  
 print("Total allocated size: %.1f KiB" % (total / 1024))
  
  
  
  
  
  
  
  
tracemalloc.start()
  
  
  
  
  
  
  
  
# ... run your application ...
  
  
  
  
  
  
  
  
snapshot = tracemalloc.take_snapshot()
  
  
  
  
display_top(snapshot)
 
 
 
 

Python测试套件的输出示例：

 
 
 
 

  
  
  
  
Top 10 lines
  
  
  
  
#1: Lib/base64.py:414: 419.8 KiB
  
  
  
  
 _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars]
  
  
  
  
#2: Lib/base64.py:306: 419.8 KiB
  
  
  
  
 _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars]
  
  
  
  
#3: collections/__init__.py:368: 293.6 KiB
  
  
  
  
 exec(class_definition, namespace)
  
  
  
  
#4: Lib/abc.py:133: 115.2 KiB
  
  
  
  
 cls = super().__new__(mcls, name, bases, namespace)
  
  
  
  
#5: unittest/case.py:574: 103.1 KiB
  
  
  
  
 testMethod()
  
  
  
  
#6: Lib/linecache.py:127: 95.4 KiB
  
  
  
  
 lines = fp.readlines()
  
  
  
  
#7: urllib/parse.py:476: 71.8 KiB
  
  
  
  
 for a in _hexdig for b in _hexdig}
  
  
  
  
#8: :5: 62.0 KiB
  
  
  
  
#9: Lib/_weakrefset.py:37: 60.0 KiB
  
  
  
  
 self.data = set()
  
  
  
  
#10: Lib/base64.py:142: 59.8 KiB
  
  
  
  
 _b32tab2 = [a + b for a in _b32tab for b in _b32tab]
  
  
  
  
6220 other: 3602.8 KiB
  
  
  
  
Total allocated size: 5303.1 KiB
 
 
 
 

更多选项，请参见 Snapshot.statistics()

Record the current and peak size of all traced memory blocks

The following code computes two sums like 0 + 1 + 2 + ... inefficiently, by creating a list of those numbers. This list consumes a lot of memory temporarily. We can use get_traced_memory() and reset_peak() to observe the small memory usage after the sum is computed as well as the peak memory usage during the computations:

 
 
 
 

  
  
  
  
import tracemalloc
  
  
  
  
  
  
  
  
tracemalloc.start()
  
  
  
  
  
  
  
  
# Example code: compute a sum with a large temporary list
  
  
  
  
large_sum = sum(list(range(100000)))
  
  
  
  
  
  
  
  
first_size, first_peak = tracemalloc.get_traced_memory()
  
  
  
  
  
  
  
  
tracemalloc.reset_peak()
  
  
  
  
  
  
  
  
# Example code: compute a sum with a small temporary list
  
  
  
  
small_sum = sum(list(range(1000)))
  
  
  
  
  
  
  
  
second_size, second_peak = tracemalloc.get_traced_memory()
  
  
  
  
  
  
  
  
print(f"{first_size=}, {first_peak=}")
  
  
  
  
print(f"{second_size=}, {second_peak=}")
 
 
 
 

输出:

 
 
 
 

  
  
  
  
first_size=664, first_peak=3592984
  
  
  
  
second_size=804, second_peak=29704
 
 
 
 

Using reset_peak() ensured we could accurately record the peak during the computation of small_sum, even though it is much smaller than the overall peak size of memory blocks since the start() call. Without the call to reset_peak(), second_peak would still be the peak from the computation large_sum (that is, equal to first_peak). In this case, both peaks are much higher than the final memory usage, and which suggests we could optimise (by removing the unnecessary call to list, and writing sum(range(...))).

API

函数

tracemalloc.clear_traces()

清空 Python 所分配的内存块的追踪数据。

另见 stop().

tracemalloc.get_object_traceback(obj)

Get the traceback where the Python object obj was allocated. Return a Traceback instance, or None if the tracemalloc module is not tracing memory allocations or did not trace the allocation of the object.

See also gc.get_referrers() and sys.getsizeof() functions.

tracemalloc.get_traceback_limit()

Get the maximum number of frames stored in the traceback of a trace.

The tracemalloc module must be tracing memory allocations to get the limit, otherwise an exception is raised.

The limit is set by the start() function.

tracemalloc.get_traced_memory()

Get the current size and peak size of memory blocks traced by the tracemalloc module as a tuple: (current: int, peak: int).

tracemalloc.reset_peak()

Set the peak size of memory blocks traced by the tracemalloc module to the current size.

Do nothing if the tracemalloc module is not tracing memory allocations.

This function only modifies the recorded peak size, and does not modify or clear any traces, unlike clear_traces(). Snapshots taken with take_snapshot() before a call to reset_peak() can be meaningfully compared to snapshots taken after the call.

See also get_traced_memory().

3.9 新版功能.

tracemalloc.get_tracemalloc_memory()

Get the memory usage in bytes of the tracemalloc module used to store traces of memory blocks. Return an int.

tracemalloc.is_tracing()

True if the tracemalloc module is tracing Python memory allocations, False otherwise.

See also start() and stop() functions.

tracemalloc.start(nframe: int = 1)

Start tracing Python memory allocations: install hooks on Python memory allocators. Collected tracebacks of traces will be limited to nframe frames. By default, a trace of a memory block only stores the most recent frame: the limit is 1. nframe must be greater or equal to 1.

You can still read the original number of total frames that composed the traceback by looking at the Traceback.total_nframe attribute.

Storing more than 1 frame is only useful to compute statistics grouped by 'traceback' or to compute cumulative statistics: see the Snapshot.compare_to() and Snapshot.statistics() methods.

Storing more frames increases the memory and CPU overhead of the tracemalloc module. Use the get_tracemalloc_memory() function to measure how much memory is used by the tracemalloc module.

The PYTHONTRACEMALLOC environment variable (PYTHONTRACEMALLOC=NFRAME) and the -X tracemalloc=NFRAME command line option can be used to start tracing at startup.

See also stop(), is_tracing() and get_traceback_limit() functions.

tracemalloc.stop()

Stop tracing Python memory allocations: uninstall hooks on Python memory allocators. Also clears all previously collected traces of memory blocks allocated by Python.

Call take_snapshot() function to take a snapshot of traces before clearing them.

See also start(), is_tracing() and clear_traces() functions.

tracemalloc.take_snapshot()

Take a snapshot of traces of memory blocks allocated by Python. Return a new Snapshot instance.

The snapshot does not include memory blocks allocated before the tracemalloc module started to trace memory allocations.

Tracebacks of traces are limited to get_traceback_limit() frames. Use the nframe parameter of the start() function to store more frames.

The tracemalloc module must be tracing memory allocations to take a snapshot, see the start() function.

See also the get_object_traceback() function.

域过滤器

class tracemalloc.DomainFilter(inclusive: bool, domain: int)

Filter traces of memory blocks by their address space (domain).

3.6 新版功能.

• inclusive

  If inclusive is True (include), match memory blocks allocated in the address space domain.

  If inclusive is False (exclude), match memory blocks not allocated in the address space domain.

• domain

  Address space of a memory block (int). Read-only property.

过滤器

class tracemalloc.Filter(inclusive: bool, filename_pattern: str, lineno: int = None, all_frames: bool = False, domain: int = None)

对内存块的跟踪进行筛选。

See the fnmatch.fnmatch() function for the syntax of filename_pattern. The '.pyc' file extension is replaced with '.py'.

示例：

• Filter(True, subprocess.__file__) 只包括 subprocess 模块的追踪数据

• Filter(False, tracemalloc.__file__) 排除了 tracemalloc 模块的追踪数据

• Filter(False, "") 排除了空的回溯信息

在 3.5 版更改: '.pyo' 文件扩展名不会再被替换为 '.py'。

在 3.6 版更改: 增加了 domain 属性。

• domain

  Address space of a memory block (int or None).

  tracemalloc uses the domain 0 to trace memory allocations made by Python. C extensions can use other domains to trace other resources.

• inclusive

  If inclusive is True (include), only match memory blocks allocated in a file with a name matching filename_pattern at line number lineno.

  If inclusive is False (exclude), ignore memory blocks allocated in a file with a name matching filename_pattern at line number lineno.

• lineno

  Line number (int) of the filter. If lineno is None, the filter matches any line number.

• filename_pattern

  Filename pattern of the filter (str). Read-only property.

• all_frames

  If all_frames is True, all frames of the traceback are checked. If all_frames is False, only the most recent frame is checked.

  This attribute has no effect if the traceback limit is 1. See the get_traceback_limit() function and Snapshot.traceback_limit attribute.

Frame

class tracemalloc.Frame

Frame of a traceback.

The Traceback class is a sequence of Frame instances.

• filename

  文件名（字符串）

• lineno

  行号（整形）

快照

class tracemalloc.Snapshot

Snapshot of traces of memory blocks allocated by Python.

The take_snapshot() function creates a snapshot instance.

• compare_to(old_snapshot: Snapshot, key_type: str, cumulative: bool = False)

  Compute the differences with an old snapshot. Get statistics as a sorted list of StatisticDiff instances grouped by key_type.

  See the Snapshot.statistics() method for key_type and cumulative parameters.

  The result is sorted from the biggest to the smallest by: absolute value of StatisticDiff.size_diff, StatisticDiff.size, absolute value of StatisticDiff.count_diff, Statistic.count and then by StatisticDiff.traceback.

• dump(filename)

  将快照写入文件

  使用 load() 重载快照。

• filter_traces(filters)

  Create a new Snapshot instance with a filtered traces sequence, filters is a list of DomainFilter and Filter instances. If filters is an empty list, return a new Snapshot instance with a copy of the traces.

  All inclusive filters are applied at once, a trace is ignored if no inclusive filters match it. A trace is ignored if at least one exclusive filter matches it.

  在 3.6 版更改: DomainFilter instances are now also accepted in filters.

• classmethod load(filename)

  从文件载入快照。

  另见 dump().

• statistics(key_type: str, cumulative: bool = False)

  获取 Statistic 信息列表，按 key_type 分组排序：

  key_type	description
  ‘filename’	文件名
  ‘lineno’	文件名和行号
  ‘traceback’	回溯

  If cumulative is True, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The cumulative mode can only be used with key_type equals to 'filename' and 'lineno'.

  The result is sorted from the biggest to the smallest by: Statistic.size, Statistic.count and then by Statistic.traceback.

• traceback_limit

  Maximum number of frames stored in the traceback of traces: result of the get_traceback_limit() when the snapshot was taken.

• traces

  Traces of all memory blocks allocated by Python: sequence of Trace instances.

  The sequence has an undefined order. Use the Snapshot.statistics() method to get a sorted list of statistics.

统计

class tracemalloc.Statistic

统计内存分配

Snapshot.statistics() 返回 Statistic 实例的列表。.

参见 StatisticDiff 类。

• count

  内存块数（整形）。

• size

  Total size of memory blocks in bytes (int).

• traceback

  Traceback where the memory block was allocated, Traceback instance.

StatisticDiff

class tracemalloc.StatisticDiff

Statistic difference on memory allocations between an old and a new Snapshot instance.

Snapshot.compare_to() returns a list of StatisticDiff instances. See also the Statistic class.

• count

  Number of memory blocks in the new snapshot (int): 0 if the memory blocks have been released in the new snapshot.

• count_diff

  Difference of number of memory blocks between the old and the new snapshots (int): 0 if the memory blocks have been allocated in the new snapshot.

• size

  Total size of memory blocks in bytes in the new snapshot (int): 0 if the memory blocks have been released in the new snapshot.

• size_diff

  Difference of total size of memory blocks in bytes between the old and the new snapshots (int): 0 if the memory blocks have been allocated in the new snapshot.

• traceback

  Traceback where the memory blocks were allocated, Traceback instance.

跟踪

class tracemalloc.Trace

Trace of a memory block.

The Snapshot.traces attribute is a sequence of Trace instances.

在 3.6 版更改: 增加了 domain 属性。

• domain

  Address space of a memory block (int). Read-only property.

  tracemalloc uses the domain 0 to trace memory allocations made by Python. C extensions can use other domains to trace other resources.

• size

  Size of the memory block in bytes (int).

• traceback

  Traceback where the memory block was allocated, Traceback instance.

回溯

class tracemalloc.Traceback

Sequence of Frame instances sorted from the oldest frame to the most recent frame.

A traceback contains at least 1 frame. If the tracemalloc module failed to get a frame, the filename "" at line number 0 is used.

When a snapshot is taken, tracebacks of traces are limited to get_traceback_limit() frames. See the take_snapshot() function. The original number of frames of the traceback is stored in the Traceback.total_nframe attribute. That allows to know if a traceback has been truncated by the traceback limit.

The Trace.traceback attribute is an instance of Traceback instance.

在 3.7 版更改: Frames are now sorted from the oldest to the most recent, instead of most recent to oldest.

• total_nframe

  Total number of frames that composed the traceback before truncation. This attribute can be set to None if the information is not available.

在 3.9 版更改: The Traceback.total_nframe attribute was added.

• format(limit=None, most_recent_first=False)

  Format the traceback as a list of lines. Use the linecache module to retrieve lines from the source code. If limit is set, format the limit most recent frames if limit is positive. Otherwise, format the abs(limit) oldest frames. If most_recent_first is True, the order of the formatted frames is reversed, returning the most recent frame first instead of last.

  Similar to the traceback.format_tb() function, except that format() does not include newlines.

  示例:

     
     
     
     
  
      
      
      
      
  print("Traceback (most recent call first):")
      
      
      
      
  for line in traceback:
      
      
      
      
   print(line)
     
     
     
     

  输出:

     
     
     
     
  
      
      
      
      
  Traceback (most recent call first):
      
      
      
      
   File "test.py", line 9
      
      
      
      
   obj = Object()
      
      
      
      
   File "test.py", line 12
      
      
      
      
   tb = tracemalloc.get_object_traceback(f())
     
     
     
     

本文名称：创新互联Python教程：tracemalloc—-跟踪内存分配
文章URL：http://cdbrznjsb.com/article/dpgdipo.html