Memory problem using pylab.savefig() [bis]

Dear all,

I posted a message two days ago mentionning memory problems while saving a large
number of images, which I guess was not so clear... I've worked a bit on it and
I think the questions seem clearer to me now. I have paid particular attention
to the threads in the user mailing list dealing with memory problems.

I use a function that generates arrays of typically 500x1380 elements.
For each column, I transform the data and image them using imshow. I needn't
display them, all I do is save them. I therefore wrote a function
generateImshow which I show below.

This function allows to produce about 300 out of 500 images and then crashes. I
therefore output the memory usage at each step. I run this from a pyWin shell
on windows XP using a 2.8GHz processor with 512 Kb memory. The result is shown
below. The conclusions are the following :

- each cycle leads to an increase of about 6Mb in memory usage, which explains
why it crashes after some cycles

- if I comment the line pylab.savefig(), then I only get an overall increase of
about 0.6 Mb, which is a lot, but I can live with this

- obviously, clf() does a good job in freeing memory, but calling the garbage
collector gc.collector does not help, nor does clearing the cached text as
suggested in one of the mails

The problem clearly seems to originate from the savefig operation but I can't
figure out what's going on... anyone has a clue or an idea of how to overcome
the problem ?

Cheers,

Aurélien

···

----
def generateImshow(yarray,
                   xscansize,yscansize,
                   xscanstep,yscanstep,
                   outputfilename,
                   initimagesize = 10):
    '''Use this function to generate matplotlib images without displaying
them'''

    #
    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step1 ','%.2f' % val,' Mb\n'

    ##normalize file containing intensity correction factors
    #xlist, ylist = fileHandling(filename).readChi()
    ##fill in missing data if any
    #ylist.extend([0]*(scanx*scany-len(ylist)))
    #reshape according to scan
    yarray = na.reshape(yarray,(yscansize,-1))

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step2 ','%.2f' % val,' Mb'

    #build image via matplotlib
    ximagesize = xscansize*xscanstep
    yimagesize = yscansize*yscanstep
    xyimageratio = float(ximagesize)/yimagesize
    #print xyimageratio
    if xyimageratio > 1: ximagesize,yimagesize =
initimagesize,initimagesize*xyimageratio
    else: ximagesize,yimagesize = initimagesize*xyimageratio,initimagesize

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step3 ','%.2f' % val,' Mb'

    fig1 = pylab.figure(figsize=(ximagesize,yimagesize),dpi=100)

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step4 ','%.2f' % val,' Mb'

    fig1.clear()

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step5 ','%.2f' % val,' Mb'

    #pylab.title('blahblah')
    im1 = pylab.imshow(yarray,
                       origin='lower',
                       #interpolation='nearest', #i.e. pixel
                       interpolation='bicubic', #i.e. smooth
                       #vmin=minvalue,
                       #vmax=maxvalue,
                       cmap = pylab.cm.bone,
                       )

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step6 ','%.2f' % val,' Mb'

    colbar1 = pylab.colorbar()

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step7 ','%.2f' % val,' Mb'

    #pylab.bone()
    pylab.axis('off')

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step8 ','%.2f' % val,' Mb'

    #save figure
    #outputfilename = filename[:-4]+'.png'
    pylab.savefig(outputfilename)

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step9 ','%.2f' % val,' Mb'

    mpl.text.Text.cached = {}

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step10 ','%.2f' % val,' Mb'

    pylab.cla()

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step11 ','%.2f' % val,' Mb'

    del im1,colbar1 #doesn't bring anything

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step12 ','%.2f' % val,' Mb'

    pylab.close(fig1)

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step13 ','%.2f' % val,' Mb'

    #pylab.close('all')
    gc.collect()

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step14 ','%.2f' % val,' Mb'

=======================
Result for three steps :

memory used for image - step1 296.57 Mb
memory used for image - step2 296.57 Mb
memory used for image - step3 296.57 Mb
memory used for image - step4 310.75 Mb
memory used for image - step5 310.75 Mb
memory used for image - step6 308.61 Mb
memory used for image - step7 308.71 Mb
memory used for image - step8 308.71 Mb
memory used for image - step9 316.77 Mb
memory used for image - step10 316.77 Mb
memory used for image - step11 316.77 Mb
memory used for image - step12 316.77 Mb
memory used for image - step13 302.92 Mb
memory used for image - step14 302.92 Mb
total time = 1.64100003242
done
memory used 302.92 Mb
building file ...

memory used for image - step1 302.92 Mb
memory used for image - step2 302.92 Mb
memory used for image - step3 302.92 Mb
memory used for image - step4 317.10 Mb
memory used for image - step5 317.10 Mb
memory used for image - step6 314.70 Mb
memory used for image - step7 314.80 Mb
memory used for image - step8 314.80 Mb
memory used for image - step9 322.87 Mb
memory used for image - step10 322.87 Mb
memory used for image - step11 322.87 Mb
memory used for image - step12 322.87 Mb
memory used for image - step13 309.02 Mb
memory used for image - step14 309.02 Mb
total time = 1.65599989891
done
memory used 309.02 Mb
building file ...

memory used for image - step1 309.02 Mb
memory used for image - step2 309.02 Mb
memory used for image - step3 309.02 Mb
memory used for image - step4 323.20 Mb
memory used for image - step5 323.20 Mb
memory used for image - step6 320.79 Mb
memory used for image - step7 320.90 Mb
memory used for image - step8 320.90 Mb
memory used for image - step9 328.97 Mb
memory used for image - step10 328.99 Mb
memory used for image - step11 328.98 Mb
memory used for image - step12 328.98 Mb
memory used for image - step13 315.14 Mb
memory used for image - step14 315.14 Mb
total time = 1.875
done

--
Just Auré

Just a quick suggestion. Since you are using pylab and you don't
really show a need to recreate a figure everytime, just use "fig1 =
pylab.gcf()" and don't delete it at the end. Reusing current memory
is always going to help prevent leaks.

···

On 2/10/06, aurelien.gourrier@...185... <aurelien.gourrier@...185...> wrote:

Dear all,

I posted a message two days ago mentionning memory problems while saving a large
number of images, which I guess was not so clear... I've worked a bit on it and
I think the questions seem clearer to me now. I have paid particular attention
to the threads in the user mailing list dealing with memory problems.

I use a function that generates arrays of typically 500x1380 elements.
For each column, I transform the data and image them using imshow. I needn't
display them, all I do is save them. I therefore wrote a function
generateImshow which I show below.

This function allows to produce about 300 out of 500 images and then crashes. I
therefore output the memory usage at each step. I run this from a pyWin shell
on windows XP using a 2.8GHz processor with 512 Kb memory. The result is shown
below. The conclusions are the following :

- each cycle leads to an increase of about 6Mb in memory usage, which explains
why it crashes after some cycles

- if I comment the line pylab.savefig(), then I only get an overall increase of
about 0.6 Mb, which is a lot, but I can live with this

- obviously, clf() does a good job in freeing memory, but calling the garbage
collector gc.collector does not help, nor does clearing the cached text as
suggested in one of the mails

The problem clearly seems to originate from the savefig operation but I can't
figure out what's going on... anyone has a clue or an idea of how to overcome
the problem ?

Cheers,

Aurélien

----
def generateImshow(yarray,
                   xscansize,yscansize,
                   xscanstep,yscanstep,
                   outputfilename,
                   initimagesize = 10):
    '''Use this function to generate matplotlib images without displaying
them'''

    #
    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step1 ','%.2f' % val,' Mb\n'

    ##normalize file containing intensity correction factors
    #xlist, ylist = fileHandling(filename).readChi()
    ##fill in missing data if any
    #ylist.extend([0]*(scanx*scany-len(ylist)))
    #reshape according to scan
    yarray = na.reshape(yarray,(yscansize,-1))

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step2 ','%.2f' % val,' Mb'

    #build image via matplotlib
    ximagesize = xscansize*xscanstep
    yimagesize = yscansize*yscanstep
    xyimageratio = float(ximagesize)/yimagesize
    #print xyimageratio
    if xyimageratio > 1: ximagesize,yimagesize =
initimagesize,initimagesize*xyimageratio
    else: ximagesize,yimagesize = initimagesize*xyimageratio,initimagesize

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step3 ','%.2f' % val,' Mb'

    fig1 = pylab.figure(figsize=(ximagesize,yimagesize),dpi=100)

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step4 ','%.2f' % val,' Mb'

    fig1.clear()

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step5 ','%.2f' % val,' Mb'

    #pylab.title('blahblah')
    im1 = pylab.imshow(yarray,
                       origin='lower',
                       #interpolation='nearest', #i.e. pixel
                       interpolation='bicubic', #i.e. smooth
                       #vmin=minvalue,
                       #vmax=maxvalue,
                       cmap = pylab.cm.bone,
                       )

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step6 ','%.2f' % val,' Mb'

    colbar1 = pylab.colorbar()

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step7 ','%.2f' % val,' Mb'

    #pylab.bone()
    pylab.axis('off')

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step8 ','%.2f' % val,' Mb'

    #save figure
    #outputfilename = filename[:-4]+'.png'
    pylab.savefig(outputfilename)

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step9 ','%.2f' % val,' Mb'

    mpl.text.Text.cached = {}

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step10 ','%.2f' % val,' Mb'

    pylab.cla()

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step11 ','%.2f' % val,' Mb'

    del im1,colbar1 #doesn't bring anything

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step12 ','%.2f' % val,' Mb'

    pylab.close(fig1)

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step13 ','%.2f' % val,' Mb'

    #pylab.close('all')
    gc.collect()

    val = float(getMemoryUsage())/1000000
    print 'memory used for image - step14 ','%.2f' % val,' Mb'

=======================
Result for three steps :

memory used for image - step1 296.57 Mb
memory used for image - step2 296.57 Mb
memory used for image - step3 296.57 Mb
memory used for image - step4 310.75 Mb
memory used for image - step5 310.75 Mb
memory used for image - step6 308.61 Mb
memory used for image - step7 308.71 Mb
memory used for image - step8 308.71 Mb
memory used for image - step9 316.77 Mb
memory used for image - step10 316.77 Mb
memory used for image - step11 316.77 Mb
memory used for image - step12 316.77 Mb
memory used for image - step13 302.92 Mb
memory used for image - step14 302.92 Mb
total time = 1.64100003242
done
memory used 302.92 Mb
building file ...

memory used for image - step1 302.92 Mb
memory used for image - step2 302.92 Mb
memory used for image - step3 302.92 Mb
memory used for image - step4 317.10 Mb
memory used for image - step5 317.10 Mb
memory used for image - step6 314.70 Mb
memory used for image - step7 314.80 Mb
memory used for image - step8 314.80 Mb
memory used for image - step9 322.87 Mb
memory used for image - step10 322.87 Mb
memory used for image - step11 322.87 Mb
memory used for image - step12 322.87 Mb
memory used for image - step13 309.02 Mb
memory used for image - step14 309.02 Mb
total time = 1.65599989891
done
memory used 309.02 Mb
building file ...

memory used for image - step1 309.02 Mb
memory used for image - step2 309.02 Mb
memory used for image - step3 309.02 Mb
memory used for image - step4 323.20 Mb
memory used for image - step5 323.20 Mb
memory used for image - step6 320.79 Mb
memory used for image - step7 320.90 Mb
memory used for image - step8 320.90 Mb
memory used for image - step9 328.97 Mb
memory used for image - step10 328.99 Mb
memory used for image - step11 328.98 Mb
memory used for image - step12 328.98 Mb
memory used for image - step13 315.14 Mb
memory used for image - step14 315.14 Mb
total time = 1.875
done

--
Just Auré

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmdlnk&kid3432&bid#0486&dat1642
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users