memory usage (leakage?) in ipython interactive mode

Community matplotlib-users
1

Is there a summary somewhere of the current state of knowledge about memory leaks when using the pylab interface interactively? Doing plot(rand(1000000)) or matshow(rand(1000,1000)) for example eats a big chunk of memory (tried with TkAgg and WxAgg in Windows (mpl v0.98.5.2) and Linux (mpl v0.98.3)), most of which is not returned when the window is closed. The same goes if you create an array, plot it, and explicitly del it after closing the window. I've seen lots of posts over the years about memory leaks, but there's nothing in the FAQ about this. I found old posts about similar things, but nothing that had a clear resolution.

thanks,
Gary

2

There are at least three possible causes of what you're seeing here:

1) ipython stores references to all results in the console. (ipython maintains a history of results so they can easily be accessed later). I don't recall the details, but it may be possible to turn this feature off or limit the number of objects stored.

2) matplotlib stores references to all figures until they are explicitly closed with pyplot.close(fignum)

3) Python uses pools of memory, and is often imposes a significant delay returning memory to the operating system. It is actually very hard to determine from the outside whether something is leaking or just pooling without compiling a special build of Python with memory pooling turned off.

In general, interactive use is somewhat at odds with creating many large plots in a single session, since all of the nice interactive features (history etc.) do not know automagically when the user is done with certain objects.

I am not aware of any memory leaks in current versions of matplotlib with *noninteractive* use, other than small leaks caused by bugs in older versions of some of the GUI toolkits (notably gtk+). If you find a script that produces a leak reproducibly, please share so we can track down the cause.

Gary Ruben wrote:

Doing plot(rand(1000000)) or matshow(rand(1000,1000)) for example eats a big chunk of memory (tried with TkAgg and WxAgg in Windows (mpl v0.98.5.2) and Linux (mpl v0.98.3)), most of which is not returned when the window is closed. The same goes if you create an array, plot it, and explicitly del it after closing the window.

Can you elaborate on these steps? It's possible that the del has little effect, since del only deletes a single reference to the object, not all references which may be keeping it alive (such as the figure, which matplotlib itself keeps a reference to). In general, you need to explicitly call pyplot.close(fignum) to delete a figure.

Cheers,
Mike

···

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

3

Hi Michael,

Thanks for your explanation. It turns out that it is a combination of
(1) and (3). I hadn't thought about (1) and I hadn't done enough
playing to see the python interpreter releasing blocks of memory. As
you suggested, the "solution" is to limit the iPython cache by using
the iPython -cs option.

thanks for your help,
Gary

Michael Droettboom wrote:

···

There are at least three possible causes of what you're seeing here:

1) ipython stores references to all results in the console. (ipython maintains a history of results so they can easily be accessed later). I don't recall the details, but it may be possible to turn this feature off or limit the number of objects stored.

2) matplotlib stores references to all figures until they are explicitly closed with pyplot.close(fignum)

3) Python uses pools of memory, and is often imposes a significant delay returning memory to the operating system. It is actually very hard to determine from the outside whether something is leaking or just pooling without compiling a special build of Python with memory pooling turned off.

In general, interactive use is somewhat at odds with creating many large plots in a single session, since all of the nice interactive features (history etc.) do not know automagically when the user is done with certain objects.

I am not aware of any memory leaks in current versions of matplotlib with *noninteractive* use, other than small leaks caused by bugs in older versions of some of the GUI toolkits (notably gtk+). If you find a script that produces a leak reproducibly, please share so we can track down the cause.

Gary Ruben wrote:

Doing plot(rand(1000000)) or matshow(rand(1000,1000)) for example eats a big chunk of memory (tried with TkAgg and WxAgg in Windows (mpl v0.98.5.2) and Linux (mpl v0.98.3)), most of which is not returned when the window is closed. The same goes if you create an array, plot it, and explicitly del it after closing the window.

Can you elaborate on these steps? It's possible that the del has little effect, since del only deletes a single reference to the object, not all references which may be keeping it alive (such as the figure, which matplotlib itself keeps a reference to). In general, you need to explicitly call pyplot.close(fignum) to delete a figure.

Cheers,
Mike