I'm writing a program that processes ~ 25,000 jobs and each iteration
draws a histogram and writes out some of the output. I let it run all
night and when I came back, python was filling up all my memory
(2Gigs) and was thrashing on and off of swap. I narrowed the problem
down to my calling of the hist() function, and was able to reproduce
it in the following code I copied from the mpl website.
Am I not properly closing the figure somehow?
Has this issue already been addressed?
I recently installed version 0.90.1 of matplotlib, although I don't
see any easy way to verify that version number from within python.
-Luke Robison
Code:
···
-------------------
import os,time,sys
from pylab import *
def report_memory(i):
pid = os.getpid()
a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
print i, ' ', a2[1],
return int(a2[1].split()[1])
# take a memory snapshot on indStart and compare it with indEnd
indStart, indEnd = 100, 150
for i in range(indStart,indEnd):
ind = arange(100)
xx = rand(len(ind))
figure(1)
hist(xx)
close(1)
# wait a few cycles for memory usage to stabilize
if i==indStart: start = val
if i>indStart:
end = val
print 'Average memory consumed per loop: %1.4fk bytes' % \
((end-start)/float(indEnd-indStart))
-----------------
Output:
python memtest.py
Average memory consumed per loop: 0.0000k bytes
102 39808 21991
Average memory consumed per loop: 0.0000k bytes
103 39828 21991
Average memory consumed per loop: 0.0000k bytes
104 39852 22024
Average memory consumed per loop: 0.6600k bytes
105 39876 22024
Average memory consumed per loop: 0.6600k bytes
106 39908 22024
Average memory consumed per loop: 0.6600k bytes
107 39932 22024
Average memory consumed per loop: 0.6600k bytes
108 39960 22024
Average memory consumed per loop: 0.6600k bytes
109 39980 22057
Average memory consumed per loop: 1.3200k bytes
110 40008 22057
Average memory consumed per loop: 1.3200k bytes
111 40032 22057
Average memory consumed per loop: 1.3200k bytes
112 40056 22057
Average memory consumed per loop: 1.3200k bytes
113 40084 22057
Average memory consumed per loop: 1.3200k bytes
114 40104 22090
Average memory consumed per loop: 1.9800k bytes
115 40132 22090
Average memory consumed per loop: 1.9800k bytes
116 40156 22090
Average memory consumed per loop: 1.9800k bytes
117 40180 22090
Average memory consumed per loop: 1.9800k bytes
118 40208 22090
Average memory consumed per loop: 1.9800k bytes
119 40232 22123
Average memory consumed per loop: 2.6400k bytes
120 40256 22123
Average memory consumed per loop: 2.6400k bytes
121 40280 22123
Average memory consumed per loop: 2.6400k bytes
122 40304 22123
Average memory consumed per loop: 2.6400k bytes
123 40328 22123
Average memory consumed per loop: 2.6400k bytes
124 40356 22123
Average memory consumed per loop: 2.6400k bytes
125 40380 22156
Average memory consumed per loop: 3.3000k bytes
126 40404 22156
Average memory consumed per loop: 3.3000k bytes
127 40428 22156
Average memory consumed per loop: 3.3000k bytes
128 40452 22156
Average memory consumed per loop: 3.3000k bytes
129 40476 22156
Average memory consumed per loop: 3.3000k bytes
130 40500 22189
Average memory consumed per loop: 3.9600k bytes
131 40528 22189
Average memory consumed per loop: 3.9600k bytes
132 40548 22189
Average memory consumed per loop: 3.9600k bytes
133 40576 22189
Average memory consumed per loop: 3.9600k bytes
134 40596 22189
Average memory consumed per loop: 3.9600k bytes
135 40624 22222
Average memory consumed per loop: 4.6200k bytes
136 40652 22222
Average memory consumed per loop: 4.6200k bytes
137 40676 22222
Average memory consumed per loop: 4.6200k bytes
138 40700 22222
Average memory consumed per loop: 4.6200k bytes
139 40724 22222
Average memory consumed per loop: 4.6200k bytes
140 40744 22222
Average memory consumed per loop: 4.6200k bytes
141 40768 22255
Average memory consumed per loop: 5.2800k bytes
142 40800 22255
Average memory consumed per loop: 5.2800k bytes
143 40824 22255
Average memory consumed per loop: 5.2800k bytes
144 40848 22255
Average memory consumed per loop: 5.2800k bytes
145 40872 22255
Average memory consumed per loop: 5.2800k bytes
146 40896 22288
Average memory consumed per loop: 5.9400k bytes
147 40916 22288
Average memory consumed per loop: 5.9400k bytes
148 40940 22288
Average memory consumed per loop: 5.9400k bytes
149 40972 22288
Average memory consumed per loop: 5.9400k bytes
as you can see, the memory consumption is increasing each loop, and
furthermore, and an increasing rate 
There have been a number of memory leaks resolved since the 0.90.1 release. However, there are still known memory leaks in all of the GUI backends, some of which are unfortunately just beyond easy reach of matplotlib. If this is an automated process and you only care about the file output, you could try using the "Agg", "Pdf", "Ps" or "Svg" backends, e.g.:
import matplotlib
matplotlib.use("Agg")
I tried your script with both 0.90.1 and the latest svn, and I could reproduce your leak with the TkAgg backend, but not with the Agg backend.
If you need a GUI, you may want to try using the latest svn version if you can. The leaks still exist there, but they are much smaller.
BTW -- you can get the version of matplotlib with:
>>> import matplotlib
>>> matplotlib.__version__
'0.90.1'
Cheers,
Mike
Luke Robison wrote:
···
I'm writing a program that processes ~ 25,000 jobs and each iteration
draws a histogram and writes out some of the output. I let it run all
night and when I came back, python was filling up all my memory
(2Gigs) and was thrashing on and off of swap. I narrowed the problem
down to my calling of the hist() function, and was able to reproduce
it in the following code I copied from the mpl website.
Am I not properly closing the figure somehow?
Has this issue already been addressed?
I recently installed version 0.90.1 of matplotlib, although I don't
see any easy way to verify that version number from within python.
-Luke Robison
Code:
-------------------
import os,time,sys
from pylab import *
def report_memory(i):
pid = os.getpid()
a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
print i, ' ', a2[1],
return int(a2[1].split()[1])
# take a memory snapshot on indStart and compare it with indEnd
indStart, indEnd = 100, 150
for i in range(indStart,indEnd):
ind = arange(100)
xx = rand(len(ind))
figure(1)
hist(xx)
close(1)
# wait a few cycles for memory usage to stabilize
if i==indStart: start = val
if i>indStart:
end = val
print 'Average memory consumed per loop: %1.4fk bytes' % \
((end-start)/float(indEnd-indStart))
-----------------
Output:
python memtest.py
Average memory consumed per loop: 0.0000k bytes
102 39808 21991
Average memory consumed per loop: 0.0000k bytes
103 39828 21991
Average memory consumed per loop: 0.0000k bytes
104 39852 22024
Average memory consumed per loop: 0.6600k bytes
105 39876 22024
Average memory consumed per loop: 0.6600k bytes
106 39908 22024
Average memory consumed per loop: 0.6600k bytes
107 39932 22024
Average memory consumed per loop: 0.6600k bytes
108 39960 22024
Average memory consumed per loop: 0.6600k bytes
109 39980 22057
Average memory consumed per loop: 1.3200k bytes
110 40008 22057
Average memory consumed per loop: 1.3200k bytes
111 40032 22057
Average memory consumed per loop: 1.3200k bytes
112 40056 22057
Average memory consumed per loop: 1.3200k bytes
113 40084 22057
Average memory consumed per loop: 1.3200k bytes
114 40104 22090
Average memory consumed per loop: 1.9800k bytes
115 40132 22090
Average memory consumed per loop: 1.9800k bytes
116 40156 22090
Average memory consumed per loop: 1.9800k bytes
117 40180 22090
Average memory consumed per loop: 1.9800k bytes
118 40208 22090
Average memory consumed per loop: 1.9800k bytes
119 40232 22123
Average memory consumed per loop: 2.6400k bytes
120 40256 22123
Average memory consumed per loop: 2.6400k bytes
121 40280 22123
Average memory consumed per loop: 2.6400k bytes
122 40304 22123
Average memory consumed per loop: 2.6400k bytes
123 40328 22123
Average memory consumed per loop: 2.6400k bytes
124 40356 22123
Average memory consumed per loop: 2.6400k bytes
125 40380 22156
Average memory consumed per loop: 3.3000k bytes
126 40404 22156
Average memory consumed per loop: 3.3000k bytes
127 40428 22156
Average memory consumed per loop: 3.3000k bytes
128 40452 22156
Average memory consumed per loop: 3.3000k bytes
129 40476 22156
Average memory consumed per loop: 3.3000k bytes
130 40500 22189
Average memory consumed per loop: 3.9600k bytes
131 40528 22189
Average memory consumed per loop: 3.9600k bytes
132 40548 22189
Average memory consumed per loop: 3.9600k bytes
133 40576 22189
Average memory consumed per loop: 3.9600k bytes
134 40596 22189
Average memory consumed per loop: 3.9600k bytes
135 40624 22222
Average memory consumed per loop: 4.6200k bytes
136 40652 22222
Average memory consumed per loop: 4.6200k bytes
137 40676 22222
Average memory consumed per loop: 4.6200k bytes
138 40700 22222
Average memory consumed per loop: 4.6200k bytes
139 40724 22222
Average memory consumed per loop: 4.6200k bytes
140 40744 22222
Average memory consumed per loop: 4.6200k bytes
141 40768 22255
Average memory consumed per loop: 5.2800k bytes
142 40800 22255
Average memory consumed per loop: 5.2800k bytes
143 40824 22255
Average memory consumed per loop: 5.2800k bytes
144 40848 22255
Average memory consumed per loop: 5.2800k bytes
145 40872 22255
Average memory consumed per loop: 5.2800k bytes
146 40896 22288
Average memory consumed per loop: 5.9400k bytes
147 40916 22288
Average memory consumed per loop: 5.9400k bytes
148 40940 22288
Average memory consumed per loop: 5.9400k bytes
149 40972 22288
Average memory consumed per loop: 5.9400k bytes
as you can see, the memory consumption is increasing each loop, and
furthermore, and an increasing rate 
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
I *am* able to see the same leak in 0.90.1 using Agg, Pdg, Ps, and
Svg with the same memory amount being leaked each time, so its
probably in matplotlib code, although I do not have the SVN sources to
check it there to see if it has been fixed there. I was originally
using the wxAgg backend, but during the batch job really all i needed
was the actual histogram numbers, so I looked through the code and
found matplotlib.mlab.hist which is all I really need, and doesn't
leak ;-).
Thanks for the help,
Luke Robison
···
On 8/9/07, Michael Droettboom <mdroe@...86...> wrote:
There have been a number of memory leaks resolved since the 0.90.1
release. However, there are still known memory leaks in all of the GUI
backends, some of which are unfortunately just beyond easy reach of
matplotlib. If this is an automated process and you only care about the
file output, you could try using the "Agg", "Pdf", "Ps" or "Svg"
backends, e.g.:
import matplotlib
matplotlib.use("Agg")
I tried your script with both 0.90.1 and the latest svn, and I could
reproduce your leak with the TkAgg backend, but not with the Agg backend.
If you need a GUI, you may want to try using the latest svn version if
you can. The leaks still exist there, but they are much smaller.
BTW -- you can get the version of matplotlib with:
>>> import matplotlib
>>> matplotlib.__version__
'0.90.1'
Cheers,
Mike
Luke Robison wrote:
> I'm writing a program that processes ~ 25,000 jobs and each iteration
> draws a histogram and writes out some of the output. I let it run all
> night and when I came back, python was filling up all my memory
> (2Gigs) and was thrashing on and off of swap. I narrowed the problem
> down to my calling of the hist() function, and was able to reproduce
> it in the following code I copied from the mpl website.
>
> Am I not properly closing the figure somehow?
> Has this issue already been addressed?
>
> I recently installed version 0.90.1 of matplotlib, although I don't
> see any easy way to verify that version number from within python.
>
> -Luke Robison
>
>
> Code:
> -------------------
> import os,time,sys
> from pylab import *
>
> def report_memory(i):
> pid = os.getpid()
> a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
> print i, ' ', a2[1],
> return int(a2[1].split()[1])
>
> # take a memory snapshot on indStart and compare it with indEnd
> indStart, indEnd = 100, 150
> for i in range(indStart,indEnd):
> ind = arange(100)
> xx = rand(len(ind))
>
> figure(1)
> hist(xx)
> close(1)
>
> # wait a few cycles for memory usage to stabilize
> if i==indStart: start = val
> if i>indStart:
> end = val
> print 'Average memory consumed per loop: %1.4fk bytes' % \
> ((end-start)/float(indEnd-indStart))
>
> -----------------
>
> Output:
>
>
> python memtest.py
>
> Average memory consumed per loop: 0.0000k bytes
> 102 39808 21991
> Average memory consumed per loop: 0.0000k bytes
> 103 39828 21991
> Average memory consumed per loop: 0.0000k bytes
> 104 39852 22024
> Average memory consumed per loop: 0.6600k bytes
> 105 39876 22024
> Average memory consumed per loop: 0.6600k bytes
> 106 39908 22024
> Average memory consumed per loop: 0.6600k bytes
> 107 39932 22024
> Average memory consumed per loop: 0.6600k bytes
> 108 39960 22024
> Average memory consumed per loop: 0.6600k bytes
> 109 39980 22057
> Average memory consumed per loop: 1.3200k bytes
> 110 40008 22057
> Average memory consumed per loop: 1.3200k bytes
> 111 40032 22057
> Average memory consumed per loop: 1.3200k bytes
> 112 40056 22057
> Average memory consumed per loop: 1.3200k bytes
> 113 40084 22057
> Average memory consumed per loop: 1.3200k bytes
> 114 40104 22090
> Average memory consumed per loop: 1.9800k bytes
> 115 40132 22090
> Average memory consumed per loop: 1.9800k bytes
> 116 40156 22090
> Average memory consumed per loop: 1.9800k bytes
> 117 40180 22090
> Average memory consumed per loop: 1.9800k bytes
> 118 40208 22090
> Average memory consumed per loop: 1.9800k bytes
> 119 40232 22123
> Average memory consumed per loop: 2.6400k bytes
> 120 40256 22123
> Average memory consumed per loop: 2.6400k bytes
> 121 40280 22123
> Average memory consumed per loop: 2.6400k bytes
> 122 40304 22123
> Average memory consumed per loop: 2.6400k bytes
> 123 40328 22123
> Average memory consumed per loop: 2.6400k bytes
> 124 40356 22123
> Average memory consumed per loop: 2.6400k bytes
> 125 40380 22156
> Average memory consumed per loop: 3.3000k bytes
> 126 40404 22156
> Average memory consumed per loop: 3.3000k bytes
> 127 40428 22156
> Average memory consumed per loop: 3.3000k bytes
> 128 40452 22156
> Average memory consumed per loop: 3.3000k bytes
> 129 40476 22156
> Average memory consumed per loop: 3.3000k bytes
> 130 40500 22189
> Average memory consumed per loop: 3.9600k bytes
> 131 40528 22189
> Average memory consumed per loop: 3.9600k bytes
> 132 40548 22189
> Average memory consumed per loop: 3.9600k bytes
> 133 40576 22189
> Average memory consumed per loop: 3.9600k bytes
> 134 40596 22189
> Average memory consumed per loop: 3.9600k bytes
> 135 40624 22222
> Average memory consumed per loop: 4.6200k bytes
> 136 40652 22222
> Average memory consumed per loop: 4.6200k bytes
> 137 40676 22222
> Average memory consumed per loop: 4.6200k bytes
> 138 40700 22222
> Average memory consumed per loop: 4.6200k bytes
> 139 40724 22222
> Average memory consumed per loop: 4.6200k bytes
> 140 40744 22222
> Average memory consumed per loop: 4.6200k bytes
> 141 40768 22255
> Average memory consumed per loop: 5.2800k bytes
> 142 40800 22255
> Average memory consumed per loop: 5.2800k bytes
> 143 40824 22255
> Average memory consumed per loop: 5.2800k bytes
> 144 40848 22255
> Average memory consumed per loop: 5.2800k bytes
> 145 40872 22255
> Average memory consumed per loop: 5.2800k bytes
> 146 40896 22288
> Average memory consumed per loop: 5.9400k bytes
> 147 40916 22288
> Average memory consumed per loop: 5.9400k bytes
> 148 40940 22288
> Average memory consumed per loop: 5.9400k bytes
> 149 40972 22288
> Average memory consumed per loop: 5.9400k bytes
>
>
> as you can see, the memory consumption is increasing each loop, and
> furthermore, and an increasing rate
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Matplotlib-users mailing list
> Matplotlib-users@lists.sourceforge.net
> matplotlib-users List Signup and Options
Luke,
Just to be sure: how are you selecting the backend?
If you use the "matplotlib.use('Agg')" method, this must appear *before* importing pylab for the first time; it is something of a "gotcha" that if it appears after pylab has been imported it doesn't give you any clue that it is ineffective.
Eric
Luke Robison wrote:
···
I *am* able to see the same leak in 0.90.1 using Agg, Pdg, Ps, and
Svg with the same memory amount being leaked each time, so its
probably in matplotlib code, although I do not have the SVN sources to
check it there to see if it has been fixed there. I was originally
using the wxAgg backend, but during the batch job really all i needed
was the actual histogram numbers, so I looked through the code and
found matplotlib.mlab.hist which is all I really need, and doesn't
leak ;-).
Thanks for the help,
Luke Robison
On 8/9/07, Michael Droettboom <mdroe@...86...> wrote:
There have been a number of memory leaks resolved since the 0.90.1
release. However, there are still known memory leaks in all of the GUI
backends, some of which are unfortunately just beyond easy reach of
matplotlib. If this is an automated process and you only care about the
file output, you could try using the "Agg", "Pdf", "Ps" or "Svg"
backends, e.g.:
import matplotlib
matplotlib.use("Agg")
I tried your script with both 0.90.1 and the latest svn, and I could
reproduce your leak with the TkAgg backend, but not with the Agg backend.
If you need a GUI, you may want to try using the latest svn version if
you can. The leaks still exist there, but they are much smaller.
BTW -- you can get the version of matplotlib with:
>>> import matplotlib
>>> matplotlib.__version__
'0.90.1'
Cheers,
Mike
Luke Robison wrote:
I'm writing a program that processes ~ 25,000 jobs and each iteration
draws a histogram and writes out some of the output. I let it run all
night and when I came back, python was filling up all my memory
(2Gigs) and was thrashing on and off of swap. I narrowed the problem
down to my calling of the hist() function, and was able to reproduce
it in the following code I copied from the mpl website.
Am I not properly closing the figure somehow?
Has this issue already been addressed?
I recently installed version 0.90.1 of matplotlib, although I don't
see any easy way to verify that version number from within python.
-Luke Robison
Code:
-------------------
import os,time,sys
from pylab import *
def report_memory(i):
pid = os.getpid()
a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
print i, ' ', a2[1],
return int(a2[1].split()[1])
# take a memory snapshot on indStart and compare it with indEnd
indStart, indEnd = 100, 150
for i in range(indStart,indEnd):
ind = arange(100)
xx = rand(len(ind))
figure(1)
hist(xx)
close(1)
# wait a few cycles for memory usage to stabilize
if i==indStart: start = val
if i>indStart:
end = val
print 'Average memory consumed per loop: %1.4fk bytes' % \
((end-start)/float(indEnd-indStart))
-----------------
Output:
python memtest.py
Average memory consumed per loop: 0.0000k bytes
102 39808 21991
Average memory consumed per loop: 0.0000k bytes
103 39828 21991
Average memory consumed per loop: 0.0000k bytes
104 39852 22024
Average memory consumed per loop: 0.6600k bytes
105 39876 22024
Average memory consumed per loop: 0.6600k bytes
106 39908 22024
Average memory consumed per loop: 0.6600k bytes
107 39932 22024
Average memory consumed per loop: 0.6600k bytes
108 39960 22024
Average memory consumed per loop: 0.6600k bytes
109 39980 22057
Average memory consumed per loop: 1.3200k bytes
110 40008 22057
Average memory consumed per loop: 1.3200k bytes
111 40032 22057
Average memory consumed per loop: 1.3200k bytes
112 40056 22057
Average memory consumed per loop: 1.3200k bytes
113 40084 22057
Average memory consumed per loop: 1.3200k bytes
114 40104 22090
Average memory consumed per loop: 1.9800k bytes
115 40132 22090
Average memory consumed per loop: 1.9800k bytes
116 40156 22090
Average memory consumed per loop: 1.9800k bytes
117 40180 22090
Average memory consumed per loop: 1.9800k bytes
118 40208 22090
Average memory consumed per loop: 1.9800k bytes
119 40232 22123
Average memory consumed per loop: 2.6400k bytes
120 40256 22123
Average memory consumed per loop: 2.6400k bytes
121 40280 22123
Average memory consumed per loop: 2.6400k bytes
122 40304 22123
Average memory consumed per loop: 2.6400k bytes
123 40328 22123
Average memory consumed per loop: 2.6400k bytes
124 40356 22123
Average memory consumed per loop: 2.6400k bytes
125 40380 22156
Average memory consumed per loop: 3.3000k bytes
126 40404 22156
Average memory consumed per loop: 3.3000k bytes
127 40428 22156
Average memory consumed per loop: 3.3000k bytes
128 40452 22156
Average memory consumed per loop: 3.3000k bytes
129 40476 22156
Average memory consumed per loop: 3.3000k bytes
130 40500 22189
Average memory consumed per loop: 3.9600k bytes
131 40528 22189
Average memory consumed per loop: 3.9600k bytes
132 40548 22189
Average memory consumed per loop: 3.9600k bytes
133 40576 22189
Average memory consumed per loop: 3.9600k bytes
134 40596 22189
Average memory consumed per loop: 3.9600k bytes
135 40624 22222
Average memory consumed per loop: 4.6200k bytes
136 40652 22222
Average memory consumed per loop: 4.6200k bytes
137 40676 22222
Average memory consumed per loop: 4.6200k bytes
138 40700 22222
Average memory consumed per loop: 4.6200k bytes
139 40724 22222
Average memory consumed per loop: 4.6200k bytes
140 40744 22222
Average memory consumed per loop: 4.6200k bytes
141 40768 22255
Average memory consumed per loop: 5.2800k bytes
142 40800 22255
Average memory consumed per loop: 5.2800k bytes
143 40824 22255
Average memory consumed per loop: 5.2800k bytes
144 40848 22255
Average memory consumed per loop: 5.2800k bytes
145 40872 22255
Average memory consumed per loop: 5.2800k bytes
146 40896 22288
Average memory consumed per loop: 5.9400k bytes
147 40916 22288
Average memory consumed per loop: 5.9400k bytes
148 40940 22288
Average memory consumed per loop: 5.9400k bytes
149 40972 22288
Average memory consumed per loop: 5.9400k bytes
as you can see, the memory consumption is increasing each loop, and
furthermore, and an increasing rate
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
hm, i was setting it after i imported pylab. You were right. I just
now tried calling matplotlib.use("Agg") just before import pylab, and
that did give me better results.
wxAgg did start leaking memory very quickly, and ps,svg, and pdf were
all holding constant - until i hit around loop number 440, at which
point they all started growing at about the same rate as the wxAgg was
-Luke
···
On 8/9/07, Eric Firing <efiring@...202...> wrote:
Luke,
Just to be sure: how are you selecting the backend?
If you use the "matplotlib.use('Agg')" method, this must appear *before*
importing pylab for the first time; it is something of a "gotcha" that
if it appears after pylab has been imported it doesn't give you any clue
that it is ineffective.
Eric
Luke Robison wrote:
> I *am* able to see the same leak in 0.90.1 using Agg, Pdg, Ps, and
> Svg with the same memory amount being leaked each time, so its
> probably in matplotlib code, although I do not have the SVN sources to
> check it there to see if it has been fixed there. I was originally
> using the wxAgg backend, but during the batch job really all i needed
> was the actual histogram numbers, so I looked through the code and
> found matplotlib.mlab.hist which is all I really need, and doesn't
> leak ;-).
>
> Thanks for the help,
> Luke Robison
>
>
> On 8/9/07, Michael Droettboom <mdroe@...86...> wrote:
>> There have been a number of memory leaks resolved since the 0.90.1
>> release. However, there are still known memory leaks in all of the GUI
>> backends, some of which are unfortunately just beyond easy reach of
>> matplotlib. If this is an automated process and you only care about the
>> file output, you could try using the "Agg", "Pdf", "Ps" or "Svg"
>> backends, e.g.:
>>
>> import matplotlib
>> matplotlib.use("Agg")
>>
>> I tried your script with both 0.90.1 and the latest svn, and I could
>> reproduce your leak with the TkAgg backend, but not with the Agg backend.
>>
>> If you need a GUI, you may want to try using the latest svn version if
>> you can. The leaks still exist there, but they are much smaller.
>>
>> BTW -- you can get the version of matplotlib with:
>>
>> >>> import matplotlib
>> >>> matplotlib.__version__
>> '0.90.1'
>>
>> Cheers,
>> Mike
>>
>>
>> Luke Robison wrote:
>>> I'm writing a program that processes ~ 25,000 jobs and each iteration
>>> draws a histogram and writes out some of the output. I let it run all
>>> night and when I came back, python was filling up all my memory
>>> (2Gigs) and was thrashing on and off of swap. I narrowed the problem
>>> down to my calling of the hist() function, and was able to reproduce
>>> it in the following code I copied from the mpl website.
>>>
>>> Am I not properly closing the figure somehow?
>>> Has this issue already been addressed?
>>>
>>> I recently installed version 0.90.1 of matplotlib, although I don't
>>> see any easy way to verify that version number from within python.
>>>
>>> -Luke Robison
>>>
>>>
>>> Code:
>>> -------------------
>>> import os,time,sys
>>> from pylab import *
>>>
>>> def report_memory(i):
>>> pid = os.getpid()
>>> a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
>>> print i, ' ', a2[1],
>>> return int(a2[1].split()[1])
>>>
>>> # take a memory snapshot on indStart and compare it with indEnd
>>> indStart, indEnd = 100, 150
>>> for i in range(indStart,indEnd):
>>> ind = arange(100)
>>> xx = rand(len(ind))
>>>
>>> figure(1)
>>> hist(xx)
>>> close(1)
>>>
>>> # wait a few cycles for memory usage to stabilize
>>> if i==indStart: start = val
>>> if i>indStart:
>>> end = val
>>> print 'Average memory consumed per loop: %1.4fk bytes' % \
>>> ((end-start)/float(indEnd-indStart))
>>>
>>> -----------------
>>>
>>> Output:
>>>
>>>
>>> python memtest.py
>>>
>>> Average memory consumed per loop: 0.0000k bytes
>>> 102 39808 21991
>>> Average memory consumed per loop: 0.0000k bytes
>>> 103 39828 21991
>>> Average memory consumed per loop: 0.0000k bytes
>>> 104 39852 22024
>>> Average memory consumed per loop: 0.6600k bytes
>>> 105 39876 22024
>>> Average memory consumed per loop: 0.6600k bytes
>>> 106 39908 22024
>>> Average memory consumed per loop: 0.6600k bytes
>>> 107 39932 22024
>>> Average memory consumed per loop: 0.6600k bytes
>>> 108 39960 22024
>>> Average memory consumed per loop: 0.6600k bytes
>>> 109 39980 22057
>>> Average memory consumed per loop: 1.3200k bytes
>>> 110 40008 22057
>>> Average memory consumed per loop: 1.3200k bytes
>>> 111 40032 22057
>>> Average memory consumed per loop: 1.3200k bytes
>>> 112 40056 22057
>>> Average memory consumed per loop: 1.3200k bytes
>>> 113 40084 22057
>>> Average memory consumed per loop: 1.3200k bytes
>>> 114 40104 22090
>>> Average memory consumed per loop: 1.9800k bytes
>>> 115 40132 22090
>>> Average memory consumed per loop: 1.9800k bytes
>>> 116 40156 22090
>>> Average memory consumed per loop: 1.9800k bytes
>>> 117 40180 22090
>>> Average memory consumed per loop: 1.9800k bytes
>>> 118 40208 22090
>>> Average memory consumed per loop: 1.9800k bytes
>>> 119 40232 22123
>>> Average memory consumed per loop: 2.6400k bytes
>>> 120 40256 22123
>>> Average memory consumed per loop: 2.6400k bytes
>>> 121 40280 22123
>>> Average memory consumed per loop: 2.6400k bytes
>>> 122 40304 22123
>>> Average memory consumed per loop: 2.6400k bytes
>>> 123 40328 22123
>>> Average memory consumed per loop: 2.6400k bytes
>>> 124 40356 22123
>>> Average memory consumed per loop: 2.6400k bytes
>>> 125 40380 22156
>>> Average memory consumed per loop: 3.3000k bytes
>>> 126 40404 22156
>>> Average memory consumed per loop: 3.3000k bytes
>>> 127 40428 22156
>>> Average memory consumed per loop: 3.3000k bytes
>>> 128 40452 22156
>>> Average memory consumed per loop: 3.3000k bytes
>>> 129 40476 22156
>>> Average memory consumed per loop: 3.3000k bytes
>>> 130 40500 22189
>>> Average memory consumed per loop: 3.9600k bytes
>>> 131 40528 22189
>>> Average memory consumed per loop: 3.9600k bytes
>>> 132 40548 22189
>>> Average memory consumed per loop: 3.9600k bytes
>>> 133 40576 22189
>>> Average memory consumed per loop: 3.9600k bytes
>>> 134 40596 22189
>>> Average memory consumed per loop: 3.9600k bytes
>>> 135 40624 22222
>>> Average memory consumed per loop: 4.6200k bytes
>>> 136 40652 22222
>>> Average memory consumed per loop: 4.6200k bytes
>>> 137 40676 22222
>>> Average memory consumed per loop: 4.6200k bytes
>>> 138 40700 22222
>>> Average memory consumed per loop: 4.6200k bytes
>>> 139 40724 22222
>>> Average memory consumed per loop: 4.6200k bytes
>>> 140 40744 22222
>>> Average memory consumed per loop: 4.6200k bytes
>>> 141 40768 22255
>>> Average memory consumed per loop: 5.2800k bytes
>>> 142 40800 22255
>>> Average memory consumed per loop: 5.2800k bytes
>>> 143 40824 22255
>>> Average memory consumed per loop: 5.2800k bytes
>>> 144 40848 22255
>>> Average memory consumed per loop: 5.2800k bytes
>>> 145 40872 22255
>>> Average memory consumed per loop: 5.2800k bytes
>>> 146 40896 22288
>>> Average memory consumed per loop: 5.9400k bytes
>>> 147 40916 22288
>>> Average memory consumed per loop: 5.9400k bytes
>>> 148 40940 22288
>>> Average memory consumed per loop: 5.9400k bytes
>>> 149 40972 22288
>>> Average memory consumed per loop: 5.9400k bytes
>>>
>>>
>>> as you can see, the memory consumption is increasing each loop, and
>>> furthermore, and an increasing rate
>>>
>>> -------------------------------------------------------------------------
>>> This SF.net email is sponsored by: Splunk Inc.
>>> Still grepping through log files to find problems? Stop.
>>> Now Search log events and configuration files using AJAX and a browser.
>>> Download your FREE copy of Splunk now >> http://get.splunk.com/
>>> _______________________________________________
>>> Matplotlib-users mailing list
>>> Matplotlib-users@lists.sourceforge.net
>>> matplotlib-users List Signup and Options
>>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Matplotlib-users mailing list
> Matplotlib-users@lists.sourceforge.net
> matplotlib-users List Signup and Options