Hi matplotlib developers and users,
I have had some problems with a memory leak in a long running
matplotlib based web application that I have developed
(www.worldwildweather.com). I believe the problem is due to a memory
leak in the Agg backend but I am not sure. Below is a script which for
me results in a consistently increasing amount of memory usage. I am
using mpl version 0.98.6svn. The problem does not occur when the
savefig command is commented out. And it does not occur when "cs =
ax.contourf(z)" and "ax.cla()" are moved outside the loop (before and
after respectively).
Best regards,
Jesper
import os, gc
import numpy as npy
import matplotlib as mpl
from matplotlib.figure import Figure
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
def report_memory():
"""Report memory."""
gc.collect()
pid = os.getpid()
a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
return int(a2[1].split()[1])
fig = Figure(dpi=100)
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])
FigureCanvas(fig)
n = 1000
z = npy.zeros((n,n))
for i in range(2000):
cs = ax.contourf(z)
fig.savefig('test.png')
ax.cla()
print report_memory(), i
I have not pasted in all of the output but just the first and last 25 lines:
53356 0
53360 1
53360 2
53360 3
53360 4
53360 5
53360 6
53360 7
53360 8
53360 9
53360 10
53360 11
53360 12
53360 13
53360 14
53360 15
53360 16
53360 17
53356 18
53360 19
53360 20
53360 21
53360 22
53360 23
53356 24
...
57552 1975
57552 1976
57552 1977
57552 1978
57552 1979
57552 1980
57552 1981
57552 1982
57552 1983
57552 1984
57552 1985
57552 1986
57552 1987
57552 1988
57552 1989
57552 1990
57552 1991
57552 1992
57552 1993
57552 1994
57552 1995
57552 1996
57552 1997
57552 1998
57552 1999
I am not able to reproduce this leak here with 0.98.6svn from today on RHEL4. What platform are you on?
(See attached massif profile -- the memory usage is flat...)
Mike
Jesper Larsen wrote:
massif.26713.ps (79 KB)
···
Hi matplotlib developers and users,
I have had some problems with a memory leak in a long running
matplotlib based web application that I have developed
(www.worldwildweather.com). I believe the problem is due to a memory
leak in the Agg backend but I am not sure. Below is a script which for
me results in a consistently increasing amount of memory usage. I am
using mpl version 0.98.6svn. The problem does not occur when the
savefig command is commented out. And it does not occur when "cs =
ax.contourf(z)" and "ax.cla()" are moved outside the loop (before and
after respectively).
Best regards,
Jesper
import os, gc
import numpy as npy
import matplotlib as mpl
from matplotlib.figure import Figure
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
def report_memory():
"""Report memory."""
gc.collect()
pid = os.getpid()
a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
return int(a2[1].split()[1])
fig = Figure(dpi=100)
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])
FigureCanvas(fig)
n = 1000
z = npy.zeros((n,n))
for i in range(2000):
cs = ax.contourf(z)
fig.savefig('test.png')
ax.cla()
print report_memory(), i
I have not pasted in all of the output but just the first and last 25 lines:
53356 0
53360 1
53360 2
53360 3
53360 4
53360 5
53360 6
53360 7
53360 8
53360 9
53360 10
53360 11
53360 12
53360 13
53360 14
53360 15
53360 16
53360 17
53356 18
53360 19
53360 20
53360 21
53360 22
53360 23
53356 24
...
57552 1975
57552 1976
57552 1977
57552 1978
57552 1979
57552 1980
57552 1981
57552 1982
57552 1983
57552 1984
57552 1985
57552 1986
57552 1987
57552 1988
57552 1989
57552 1990
57552 1991
57552 1992
57552 1993
57552 1994
57552 1995
57552 1996
57552 1997
57552 1998
57552 1999
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA
Michael Droettboom wrote:
I am not able to reproduce this leak here with 0.98.6svn from today on RHEL4. What platform are you on?
(See attached massif profile -- the memory usage is flat...)
Mike
Well, in my setup I see similar behaviour as in the initial question. Ubuntu 8.10 32bit, 2-days old svn verion of matplotlib.
Andres
···
Jesper Larsen wrote:
Hi matplotlib developers and users,
I have had some problems with a memory leak in a long running
matplotlib based web application that I have developed
(www.worldwildweather.com). I believe the problem is due to a memory
leak in the Agg backend but I am not sure. Below is a script which for
me results in a consistently increasing amount of memory usage. I am
using mpl version 0.98.6svn. The problem does not occur when the
savefig command is commented out. And it does not occur when "cs =
ax.contourf(z)" and "ax.cla()" are moved outside the loop (before and
after respectively).
Best regards,
Jesper
import os, gc
import numpy as npy
import matplotlib as mpl
from matplotlib.figure import Figure
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
def report_memory():
"""Report memory."""
gc.collect()
pid = os.getpid()
a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
return int(a2[1].split()[1])
fig = Figure(dpi=100)
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])
FigureCanvas(fig)
n = 1000
z = npy.zeros((n,n))
for i in range(2000):
cs = ax.contourf(z)
fig.savefig('test.png')
ax.cla()
print report_memory(), i
I have not pasted in all of the output but just the first and last 25 lines:
53356 0
53360 1
53360 2
53360 3
53360 4
53360 5
53360 6
53360 7
53360 8
53360 9
53360 10
53360 11
53360 12
53360 13
53360 14
53360 15
53360 16
53360 17
53356 18
53360 19
53360 20
53360 21
53360 22
53360 23
53356 24
...
57552 1975
57552 1976
57552 1977
57552 1978
57552 1979
57552 1980
57552 1981
57552 1982
57552 1983
57552 1984
57552 1985
57552 1986
57552 1987
57552 1988
57552 1989
57552 1990
57552 1991
57552 1992
57552 1993
57552 1994
57552 1995
57552 1996
57552 1997
57552 1998
57552 1999
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
------------------------------------------------------------------------
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
Are you able to run it inside of valgrind's massif tool? Calling out to ps can be a bit spurious (particularly with a memory-pooling Python build) especially for a leak this small.
Mike
Andres Luhamaa wrote:
···
Michael Droettboom wrote:
I am not able to reproduce this leak here with 0.98.6svn from today on RHEL4. What platform are you on?
(See attached massif profile -- the memory usage is flat...)
Mike
Well, in my setup I see similar behaviour as in the initial question. Ubuntu 8.10 32bit, 2-days old svn verion of matplotlib.
Andres
Jesper Larsen wrote:
Hi matplotlib developers and users,
I have had some problems with a memory leak in a long running
matplotlib based web application that I have developed
(www.worldwildweather.com). I believe the problem is due to a memory
leak in the Agg backend but I am not sure. Below is a script which for
me results in a consistently increasing amount of memory usage. I am
using mpl version 0.98.6svn. The problem does not occur when the
savefig command is commented out. And it does not occur when "cs =
ax.contourf(z)" and "ax.cla()" are moved outside the loop (before and
after respectively).
Best regards,
Jesper
import os, gc
import numpy as npy
import matplotlib as mpl
from matplotlib.figure import Figure
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
def report_memory():
"""Report memory."""
gc.collect()
pid = os.getpid()
a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
return int(a2[1].split()[1])
fig = Figure(dpi=100)
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])
FigureCanvas(fig)
n = 1000
z = npy.zeros((n,n))
for i in range(2000):
cs = ax.contourf(z)
fig.savefig('test.png')
ax.cla()
print report_memory(), i
I have not pasted in all of the output but just the first and last 25 lines:
53356 0
53360 1
53360 2
53360 3
53360 4
53360 5
53360 6
53360 7
53360 8
53360 9
53360 10
53360 11
53360 12
53360 13
53360 14
53360 15
53360 16
53360 17
53356 18
53360 19
53360 20
53360 21
53360 22
53360 23
53356 24
...
57552 1975
57552 1976
57552 1977
57552 1978
57552 1979
57552 1980
57552 1981
57552 1982
57552 1983
57552 1984
57552 1985
57552 1986
57552 1987
57552 1988
57552 1989
57552 1990
57552 1991
57552 1992
57552 1993
57552 1994
57552 1995
57552 1996
57552 1997
57552 1998
57552 1999
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
------------------------------------------------------------------------
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
------------------------------------------------------------------------------
Stay on top of everything new and different, both inside and around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. Use priority code J9JMT32. http://p.sf.net/sfu/p
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA
Well, I have never used it before but here is the output from valgrind masstif, can you figure out something from this? The mem usage from python was still increasing.
Andres
Michael Droettboom wrote:
massif.out.25331.gz (11.5 KB)
···
Are you able to run it inside of valgrind's massif tool? Calling out to ps can be a bit spurious (particularly with a memory-pooling Python build) especially for a leak this small.
Mike
Andres Luhamaa wrote:
Michael Droettboom wrote:
I am not able to reproduce this leak here with 0.98.6svn from today on RHEL4. What platform are you on?
(See attached massif profile -- the memory usage is flat...)
Mike
Well, in my setup I see similar behaviour as in the initial question. Ubuntu 8.10 32bit, 2-days old svn verion of matplotlib.
Andres
Jesper Larsen wrote:
Hi matplotlib developers and users,
I have had some problems with a memory leak in a long running
matplotlib based web application that I have developed
(www.worldwildweather.com). I believe the problem is due to a memory
leak in the Agg backend but I am not sure. Below is a script which for
me results in a consistently increasing amount of memory usage. I am
using mpl version 0.98.6svn. The problem does not occur when the
savefig command is commented out. And it does not occur when "cs =
ax.contourf(z)" and "ax.cla()" are moved outside the loop (before and
after respectively).
Best regards,
Jesper
import os, gc
import numpy as npy
import matplotlib as mpl
from matplotlib.figure import Figure
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
def report_memory():
"""Report memory."""
gc.collect()
pid = os.getpid()
a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
return int(a2[1].split()[1])
fig = Figure(dpi=100)
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])
FigureCanvas(fig)
n = 1000
z = npy.zeros((n,n))
for i in range(2000):
cs = ax.contourf(z)
fig.savefig('test.png')
ax.cla()
print report_memory(), i
I have not pasted in all of the output but just the first and last 25 lines:
53356 0
53360 1
53360 2
53360 3
53360 4
53360 5
53360 6
53360 7
53360 8
53360 9
53360 10
53360 11
53360 12
53360 13
53360 14
53360 15
53360 16
53360 17
53356 18
53360 19
53360 20
53360 21
53360 22
53360 23
53356 24
...
57552 1975
57552 1976
57552 1977
57552 1978
57552 1979
57552 1980
57552 1981
57552 1982
57552 1983
57552 1984
57552 1985
57552 1986
57552 1987
57552 1988
57552 1989
57552 1990
57552 1991
57552 1992
57552 1993
57552 1994
57552 1995
57552 1996
57552 1997
57552 1998
57552 1999
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
------------------------------------------------------------------------
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
------------------------------------------------------------------------------
Stay on top of everything new and different, both inside and around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. Use priority code J9JMT32. http://p.sf.net/sfu/p
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
Hi Michael and others,
Sorry for the late answer. I am on Ubuntu 8.10. Unfortunately I have
not had time to look any more into the issue (which might very well be
an error on my part) but I will return when I have more info (I have
made a temporary fix).
I tried using valgrind-massif to reproduce your plot but unfortunately
newer versions of valgrind-massif do not support ps plots.
Best regards,
Jesper
2009/4/15 Michael Droettboom <mdroe@...86...>:
···
I am not able to reproduce this leak here with 0.98.6svn from today on
RHEL4. What platform are you on?
(See attached massif profile -- the memory usage is flat...)
Mike
Jesper Larsen wrote:
Hi matplotlib developers and users,
I have had some problems with a memory leak in a long running
matplotlib based web application that I have developed
(www.worldwildweather.com). I believe the problem is due to a memory
leak in the Agg backend but I am not sure. Below is a script which for
me results in a consistently increasing amount of memory usage. I am
using mpl version 0.98.6svn. The problem does not occur when the
savefig command is commented out. And it does not occur when "cs =
ax.contourf(z)" and "ax.cla()" are moved outside the loop (before and
after respectively).
Best regards,
Jesper
import os, gc
import numpy as npy
import matplotlib as mpl
from matplotlib.figure import Figure
from matplotlib.backends.backend_agg import FigureCanvasAgg as
FigureCanvas
def report_memory():
"""Report memory."""
gc.collect()
pid = os.getpid()
a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
return int(a2[1].split()[1])
fig = Figure(dpi=100)
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])
FigureCanvas(fig)
n = 1000
z = npy.zeros((n,n))
for i in range(2000):
cs = ax.contourf(z)
fig.savefig('test.png')
ax.cla()
print report_memory(), i
I have not pasted in all of the output but just the first and last 25
lines:
53356 0
53360 1
53360 2
53360 3
53360 4
53360 5
53360 6
53360 7
53360 8
53360 9
53360 10
53360 11
53360 12
53360 13
53360 14
53360 15
53360 16
53360 17
53356 18
53360 19
53360 20
53360 21
53360 22
53360 23
53356 24
...
57552 1975
57552 1976
57552 1977
57552 1978
57552 1979
57552 1980
57552 1981
57552 1982
57552 1983
57552 1984
57552 1985
57552 1986
57552 1987
57552 1988
57552 1989
57552 1990
57552 1991
57552 1992
57552 1993
57552 1994
57552 1995
57552 1996
57552 1997
57552 1998
57552 1999
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA
I forgot to mention that it seems like the issue is related to the
Figure object (and not any foreign language extensions?). When a
fig.clf() is added before the call to report_memory the memory usage
is constant. Since I am reusing the Figure instance this is a problem
for me. But maybe I just need to figure out what to delete from the
Figure instance - or should it be unchanged for the case I presented?
Best regards,
Jesper
2009/4/15 Michael Droettboom <mdroe@...86...>:
···
I am not able to reproduce this leak here with 0.98.6svn from today on
RHEL4. What platform are you on?
(See attached massif profile -- the memory usage is flat...)
Mike
Jesper Larsen wrote:
Hi matplotlib developers and users,
I have had some problems with a memory leak in a long running
matplotlib based web application that I have developed
(www.worldwildweather.com). I believe the problem is due to a memory
leak in the Agg backend but I am not sure. Below is a script which for
me results in a consistently increasing amount of memory usage. I am
using mpl version 0.98.6svn. The problem does not occur when the
savefig command is commented out. And it does not occur when "cs =
ax.contourf(z)" and "ax.cla()" are moved outside the loop (before and
after respectively).
Best regards,
Jesper
import os, gc
import numpy as npy
import matplotlib as mpl
from matplotlib.figure import Figure
from matplotlib.backends.backend_agg import FigureCanvasAgg as
FigureCanvas
def report_memory():
"""Report memory."""
gc.collect()
pid = os.getpid()
a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
return int(a2[1].split()[1])
fig = Figure(dpi=100)
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])
FigureCanvas(fig)
n = 1000
z = npy.zeros((n,n))
for i in range(2000):
cs = ax.contourf(z)
fig.savefig('test.png')
ax.cla()
print report_memory(), i
I have not pasted in all of the output but just the first and last 25
lines:
53356 0
53360 1
53360 2
53360 3
53360 4
53360 5
53360 6
53360 7
53360 8
53360 9
53360 10
53360 11
53360 12
53360 13
53360 14
53360 15
53360 16
53360 17
53356 18
53360 19
53360 20
53360 21
53360 22
53360 23
53356 24
...
57552 1975
57552 1976
57552 1977
57552 1978
57552 1979
57552 1980
57552 1981
57552 1982
57552 1983
57552 1984
57552 1985
57552 1986
57552 1987
57552 1988
57552 1989
57552 1990
57552 1991
57552 1992
57552 1993
57552 1994
57552 1995
57552 1996
57552 1997
57552 1998
57552 1999
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA