Python memory leak detector

Posted On By admin

Tags: Python, memory leak, leek, object, garbage collection, garbage collector, __del__, inspect, gc, DEBUG_LEAK

I have a nasty memory leak in my Python script which I can’t find. For some random projects my script gobbles up 2gig of memory then quietly dies.

Here is my first attempt at cobbling together a leak detector:

import gc
import inspect

def dump_garbage():
# force collection
print “\nCollecting GARBAGE:”
gc.collect()
# prove they have been collected
print “\nCollecting GARBAGE:”
gc.collect()

print “\nGARBAGE OBJECTS:”
for x in gc.garbage:
s = str(x)
if len(s) > 80: s = “%s…” % s[:80]

print “::”, s
print ”        type:”, type(x)
print ”   referrers:”, len(gc.get_referrers(x))
try:
print ”    is class:”, inspect.isclass(type(x))
print ”      module:”, inspect.getmodule(x)

lines, line_num = inspect.getsourcelines(type(x))
print ”    line num:”, line_num
for l in lines:
print ”        line:”, l.rstrip(“\n”)
except:
pass

print

class tmp(object):
def __init__(self):
a = 0

if __name__==”__main__”:
import gc
gc.enable()
gc.set_debug(gc.DEBUG_LEAK)

# make a leak
l = [tmp()]
l.append(l)
del l

dump_garbage()

When run it outputs:

Collecting GARBAGE:
gc: collectable <tmp 00BE1730>
gc: collectable <list 00BED788>

Collecting GARBAGE:

GARBAGE OBJECTS:
:: <__main__.tmp object at 0x00BE1730>
type: <class ‘__main__.tmp’>
referrers: 4
is class: True
module: <module ‘__main__’ from ‘C:\XXXXXX.py’>
line num: 33
line: class tmp(object):
line:     def __init__(self):
line:         a = 0

:: [<__main__.tmp object at 0x00BE1730>, […]]
type: <type ‘list’>
referrers: 4
is class: True
module: None

As you can see it tries to figure out where the object is defined in your code. This should give you some clues as to where the leak is happening. If you are still having problems try adding some debug comments to the classes as you create them, print the debug info in the dump_garbage() function.

This is based on this python memory leak detector at active state. I am learning all this as I go. I guess outputing this as a .csv would help. If you have any comments or improvements drop me a line in the comments below and I’ll update the code.

These are some links and stuff to help you get started hunting down your memory leak:

  • Zope.org has a class called TrackRefs (google for it) but to use it you need a debug version of Python.
  • Guppy PE, also called heapy. See this post for an example of how to use guppy (the example is near the end of the post)
  • PySizer
  • Hunting memory leaks in Python, and a followup post.

Update: