Dear all,
I posted a message two days ago mentionning memory problems while saving a large
number of images, which I guess was not so clear... I've worked a bit on it and
I think the questions seem clearer to me now. I have paid particular attention
to the threads in the user mailing list dealing with memory problems.
I use a function that generates arrays of typically 500x1380 elements.
For each column, I transform the data and image them using imshow. I needn't
display them, all I do is save them. I therefore wrote a function
generateImshow which I show below.
This function allows to produce about 300 out of 500 images and then crashes. I
therefore output the memory usage at each step. I run this from a pyWin shell
on windows XP using a 2.8GHz processor with 512 Kb memory. The result is shown
below. The conclusions are the following :
- each cycle leads to an increase of about 6Mb in memory usage, which explains
why it crashes after some cycles
- if I comment the line pylab.savefig(), then I only get an overall increase of
about 0.6 Mb, which is a lot, but I can live with this
- obviously, clf() does a good job in freeing memory, but calling the garbage
collector gc.collector does not help, nor does clearing the cached text as
suggested in one of the mails
The problem clearly seems to originate from the savefig operation but I can't
figure out what's going on... anyone has a clue or an idea of how to overcome
the problem ?
Cheers,
Aurélien
···
----
def generateImshow(yarray,
xscansize,yscansize,
xscanstep,yscanstep,
outputfilename,
initimagesize = 10):
'''Use this function to generate matplotlib images without displaying
them'''
#
val = float(getMemoryUsage())/1000000
print 'memory used for image - step1 ','%.2f' % val,' Mb\n'
##normalize file containing intensity correction factors
#xlist, ylist = fileHandling(filename).readChi()
##fill in missing data if any
#ylist.extend([0]*(scanx*scany-len(ylist)))
#reshape according to scan
yarray = na.reshape(yarray,(yscansize,-1))
val = float(getMemoryUsage())/1000000
print 'memory used for image - step2 ','%.2f' % val,' Mb'
#build image via matplotlib
ximagesize = xscansize*xscanstep
yimagesize = yscansize*yscanstep
xyimageratio = float(ximagesize)/yimagesize
#print xyimageratio
if xyimageratio > 1: ximagesize,yimagesize =
initimagesize,initimagesize*xyimageratio
else: ximagesize,yimagesize = initimagesize*xyimageratio,initimagesize
val = float(getMemoryUsage())/1000000
print 'memory used for image - step3 ','%.2f' % val,' Mb'
fig1 = pylab.figure(figsize=(ximagesize,yimagesize),dpi=100)
val = float(getMemoryUsage())/1000000
print 'memory used for image - step4 ','%.2f' % val,' Mb'
fig1.clear()
val = float(getMemoryUsage())/1000000
print 'memory used for image - step5 ','%.2f' % val,' Mb'
#pylab.title('blahblah')
im1 = pylab.imshow(yarray,
origin='lower',
#interpolation='nearest', #i.e. pixel
interpolation='bicubic', #i.e. smooth
#vmin=minvalue,
#vmax=maxvalue,
cmap = pylab.cm.bone,
)
val = float(getMemoryUsage())/1000000
print 'memory used for image - step6 ','%.2f' % val,' Mb'
colbar1 = pylab.colorbar()
val = float(getMemoryUsage())/1000000
print 'memory used for image - step7 ','%.2f' % val,' Mb'
#pylab.bone()
pylab.axis('off')
val = float(getMemoryUsage())/1000000
print 'memory used for image - step8 ','%.2f' % val,' Mb'
#save figure
#outputfilename = filename[:-4]+'.png'
pylab.savefig(outputfilename)
val = float(getMemoryUsage())/1000000
print 'memory used for image - step9 ','%.2f' % val,' Mb'
mpl.text.Text.cached = {}
val = float(getMemoryUsage())/1000000
print 'memory used for image - step10 ','%.2f' % val,' Mb'
pylab.cla()
val = float(getMemoryUsage())/1000000
print 'memory used for image - step11 ','%.2f' % val,' Mb'
del im1,colbar1 #doesn't bring anything
val = float(getMemoryUsage())/1000000
print 'memory used for image - step12 ','%.2f' % val,' Mb'
pylab.close(fig1)
val = float(getMemoryUsage())/1000000
print 'memory used for image - step13 ','%.2f' % val,' Mb'
#pylab.close('all')
gc.collect()
val = float(getMemoryUsage())/1000000
print 'memory used for image - step14 ','%.2f' % val,' Mb'
=======================
Result for three steps :
memory used for image - step1 296.57 Mb
memory used for image - step2 296.57 Mb
memory used for image - step3 296.57 Mb
memory used for image - step4 310.75 Mb
memory used for image - step5 310.75 Mb
memory used for image - step6 308.61 Mb
memory used for image - step7 308.71 Mb
memory used for image - step8 308.71 Mb
memory used for image - step9 316.77 Mb
memory used for image - step10 316.77 Mb
memory used for image - step11 316.77 Mb
memory used for image - step12 316.77 Mb
memory used for image - step13 302.92 Mb
memory used for image - step14 302.92 Mb
total time = 1.64100003242
done
memory used 302.92 Mb
building file ...
memory used for image - step1 302.92 Mb
memory used for image - step2 302.92 Mb
memory used for image - step3 302.92 Mb
memory used for image - step4 317.10 Mb
memory used for image - step5 317.10 Mb
memory used for image - step6 314.70 Mb
memory used for image - step7 314.80 Mb
memory used for image - step8 314.80 Mb
memory used for image - step9 322.87 Mb
memory used for image - step10 322.87 Mb
memory used for image - step11 322.87 Mb
memory used for image - step12 322.87 Mb
memory used for image - step13 309.02 Mb
memory used for image - step14 309.02 Mb
total time = 1.65599989891
done
memory used 309.02 Mb
building file ...
memory used for image - step1 309.02 Mb
memory used for image - step2 309.02 Mb
memory used for image - step3 309.02 Mb
memory used for image - step4 323.20 Mb
memory used for image - step5 323.20 Mb
memory used for image - step6 320.79 Mb
memory used for image - step7 320.90 Mb
memory used for image - step8 320.90 Mb
memory used for image - step9 328.97 Mb
memory used for image - step10 328.99 Mb
memory used for image - step11 328.98 Mb
memory used for image - step12 328.98 Mb
memory used for image - step13 315.14 Mb
memory used for image - step14 315.14 Mb
total time = 1.875
done
--
Just Auré