Memory leak in basemap or matplotlib

Community matplotlib-users
1

Hi matplotlib users,

I'm using matplotlib for a long running process. Unfortunately the memory
usage continue to grow as the process runs. I have appended a simple example
which illustrates this at the end of this mail. Unfortunately I haven't
figured out how to use the information obtainable from gc for anything useful
in this regards.

Kind regards,
Jesper

My system is:

uname -a

Linux sea 2.6.15-28-686 #1 SMP PREEMPT Thu Feb 1 16:14:07 UTC 2007 i686
GNU/Linux

python

Python 2.4.4 (#1, Nov 16 2006, 13:39:46)
[GCC 3.3.3 (Debian)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import matplotlib
print matplotlib.__version__

0.87.6

from matplotlib.toolkits import basemap
print basemap.__version__

0.9.4

Test code:

import os, gc
import PyNGL.Nio as Nio
from matplotlib.toolkits import basemap
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
import pylab

def report_memory(i):
  pid = os.getpid()
  a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
  print i, ' ', a2[1],
  return int(a2[1].split()[1])

def plot():
  #gc.set_debug(gc.DEBUG_LEAK)
  lon = pylab.linspace(-4.08300018311, 30.25, 207)
  lat = pylab.linspace(48.5499992371, 65.8499984741, 174)
  xo, yo = pylab.meshgrid(lon, lat)
  bmap = basemap.Basemap(-4, 48, 30, 66)
  xlon, ylat = bmap(xo,yo)

  fig = pylab.Figure()
  canvas = FigureCanvas(fig)

  i = 0
  while True:
    report_memory(i)

    fig.clear()
    cs = bmap.contourf(xlon, ylat, xo)
    del cs

    i += 1

if __name__ == '__main__': plot()

2

Jesper Larsen wrote:

Hi matplotlib users,

I'm using matplotlib for a long running process. Unfortunately the memory usage continue to grow as the process runs. I have appended a simple example which illustrates this at the end of this mail. Unfortunately I haven't figured out how to use the information obtainable from gc for anything useful in this regards.

Kind regards,
Jesper

My system is:

uname -a
    

Linux sea 2.6.15-28-686 #1 SMP PREEMPT Thu Feb 1 16:14:07 UTC 2007 i686 GNU/Linux

python
    

Python 2.4.4 (#1, Nov 16 2006, 13:39:46)
[GCC 3.3.3 (Debian)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
  

import matplotlib
print matplotlib.__version__
        

0.87.6
  

from matplotlib.toolkits import basemap
print basemap.__version__
        

0.9.4

Test code:

import os, gc
import PyNGL.Nio as Nio
from matplotlib.toolkits import basemap
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
import pylab

def report_memory(i):
  pid = os.getpid()
  a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
  print i, ' ', a2[1],
  return int(a2[1].split()[1])

def plot():
  #gc.set_debug(gc.DEBUG_LEAK)
  lon = pylab.linspace(-4.08300018311, 30.25, 207)
  lat = pylab.linspace(48.5499992371, 65.8499984741, 174)
  xo, yo = pylab.meshgrid(lon, lat)
  bmap = basemap.Basemap(-4, 48, 30, 66)
  xlon, ylat = bmap(xo,yo)

  fig = pylab.Figure()
  canvas = FigureCanvas(fig)

  i = 0
  while True:
    report_memory(i)

    fig.clear()
    cs = bmap.contourf(xlon, ylat, xo)
    del cs

    i += 1

if __name__ == '__main__': plot()
  

Jesper: Looks to me like the memory leak is not in Basemap, but somehow the fig.clear() is not working properly. Replacing the fig.clear() with pylab.gca() gets rid of the memory leak. I will investigate further ...

-Jeff

···

--
Jeffrey S. Whitaker Phone : (303)497-6313
NOAA/OAR/CDC R/PSD1 FAX : (303)497-6449
325 Broadway Boulder, CO, USA 80305-3328

3

Jesper Larsen wrote:

Hi matplotlib users,

I'm using matplotlib for a long running process. Unfortunately the memory usage continue to grow as the process runs. I have appended a simple example which illustrates this at the end of this mail. Unfortunately I haven't figured out how to use the information obtainable from gc for anything useful in this regards.

Kind regards,
Jesper

My system is:

uname -a
    

Linux sea 2.6.15-28-686 #1 SMP PREEMPT Thu Feb 1 16:14:07 UTC 2007 i686 GNU/Linux

python
    

Python 2.4.4 (#1, Nov 16 2006, 13:39:46)
[GCC 3.3.3 (Debian)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
  

import matplotlib
print matplotlib.__version__
        

0.87.6
  

from matplotlib.toolkits import basemap
print basemap.__version__
        

0.9.4

Test code:

import os, gc
import PyNGL.Nio as Nio
from matplotlib.toolkits import basemap
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
import pylab

def report_memory(i):
  pid = os.getpid()
  a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
  print i, ' ', a2[1],
  return int(a2[1].split()[1])

def plot():
  #gc.set_debug(gc.DEBUG_LEAK)
  lon = pylab.linspace(-4.08300018311, 30.25, 207)
  lat = pylab.linspace(48.5499992371, 65.8499984741, 174)
  xo, yo = pylab.meshgrid(lon, lat)
  bmap = basemap.Basemap(-4, 48, 30, 66)
  xlon, ylat = bmap(xo,yo)

  fig = pylab.Figure()
  canvas = FigureCanvas(fig)

  i = 0
  while True:
    report_memory(i)

    fig.clear()
    cs = bmap.contourf(xlon, ylat, xo)
    del cs

    i += 1

if __name__ == '__main__': plot()

Jesper: For now I recommend adding 'pylab.cla()' after the fig.clear() as a workaround. I don't really understand why that is necessary - perhaps John or Eric can clarify whether this is indeed a bug.

-Jeff

···

--
Jeffrey S. Whitaker Phone : (303)497-6313
Meteorologist FAX : (303)497-6449
NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@...259...
325 Broadway Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web : Jeffrey S. Whitaker: NOAA Physical Sciences Laboratory

4

Jeff Whitaker wrote:

Jesper Larsen wrote:

Hi matplotlib users,

I'm using matplotlib for a long running process. Unfortunately the memory usage continue to grow as the process runs. I have appended a simple example which illustrates this at the end of this mail. Unfortunately I haven't figured out how to use the information obtainable from gc for anything useful in this regards.

[...]

  

Jesper: Looks to me like the memory leak is not in Basemap, but somehow the fig.clear() is not working properly. Replacing the fig.clear() with pylab.gca() gets rid of the memory leak. I will investigate further ...

-Jeff

Jeff, this might be related to the leak illustrated in the attached script, based on an earlier report. (I don't recall who reported it.)

Eric

mem_minimal.py (731 Bytes)

5

Eric Firing wrote:

Jeff Whitaker wrote:

Jesper Larsen wrote:

Hi matplotlib users,

I'm using matplotlib for a long running process. Unfortunately the memory usage continue to grow as the process runs. I have appended a simple example which illustrates this at the end of this mail. Unfortunately I haven't figured out how to use the information obtainable from gc for anything useful in this regards.

[...]

  

Jesper: Looks to me like the memory leak is not in Basemap, but somehow the fig.clear() is not working properly. Replacing the fig.clear() with pylab.gca() gets rid of the memory leak. I will investigate further ...

-Jeff

Jeff, this might be related to the leak illustrated in the attached script, based on an earlier report. (I don't recall who reported it.)

Eric

Eric: Don't think so, since Jesper's script leaks even when the backend is set to 'Agg' (while the one you sent doesn't).

-Jeff

···

--
Jeffrey S. Whitaker Phone : (303)497-6313
Meteorologist FAX : (303)497-6449
NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@...259...
325 Broadway Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web : Jeffrey S. Whitaker: NOAA Physical Sciences Laboratory

6

I suggest adding a gc.collect after the figure.clear() and see if
this fixes things.

JDH

···

On 3/26/07, Jeff Whitaker <jswhit@...146...> wrote:

Jesper: For now I recommend adding 'pylab.cla()' after the fig.clear()
as a workaround. I don't really understand why that is necessary -
perhaps John or Eric can clarify whether this is indeed a bug.

7

John Hunter wrote:

···

On 3/26/07, Jeff Whitaker <jswhit@...146...> wrote:

Jesper: For now I recommend adding 'pylab.cla()' after the fig.clear()
as a workaround. I don't really understand why that is necessary -
perhaps John or Eric can clarify whether this is indeed a bug.

I suggest adding a gc.collect after the figure.clear() and see if
this fixes things.

JDH

John: Nope, that has no effect. What does cla clean up that clf doesn't?

-Jeff

--
Jeffrey S. Whitaker Phone : (303)497-6313
Meteorologist FAX : (303)497-6449
NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@...259...
325 Broadway Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web : Jeffrey S. Whitaker: NOAA Physical Sciences Laboratory

8

Well, it flushes all the lines, removes the ticks, that kind of thing.
But fig.clear sets

self.axes =

so if there are no references to the axes anywhere, gc.collect()
should force a clean up and a purging of all the axes artists. Ie, it
should matter if Axes.cla does self.lines= if the figure clear
removes all references to the axes. I'm wondering if someone
somewhere in mpl is keeping a reference to the axes....

···

On 3/26/07, Jeff Whitaker <jswhit@...146...> wrote:

John Hunter wrote:

John: Nope, that has no effect. What does cla clean up that clf doesn't?

9

John Hunter wrote:

John Hunter wrote:

John: Nope, that has no effect. What does cla clean up that clf doesn't?

Well, it flushes all the lines, removes the ticks, that kind of thing.
But fig.clear sets

self.axes =

so if there are no references to the axes anywhere, gc.collect()
should force a clean up and a purging of all the axes artists. Ie, it
should matter if Axes.cla does self.lines= if the figure clear
removes all references to the axes. I'm wondering if someone
somewhere in mpl is keeping a reference to the axes....

It would have to be something like a growing list of axes, wouldn't it? Otherwise the old axes would be replaced by the new one on the next run through the loop, and it would not be a leak.

(The other leak, with interactive backends and nothing but pylab.figure(); pylab.close() in the loop is also keeping me baffled. It is not the same as the above, but it is the same sort of thing.)

Are there any python debugging tools that can show at the python level where memory is being used?

Eric

···

On 3/26/07, Jeff Whitaker <jswhit@...146...> wrote: