Memory leak

All, I followed up the 'memory leak' discussion in the

    > sourceforge list and I know the Matplotlib-FAQ entry about
    > this subject. I've also seen John Hunter's post about the
    > need of matching figure/close pairs. Anyway, I still feel
    > that there are problems in this subject, which can be
    > exposed by the following script (for Windows, but can
    > easily be adapted to Unix).

    > As can be seen by the results (also given below), there is
    > a steady increase in memory usage which is not recovered!

If I recall correctly, there is a known leak in tkagg when you create
multiple canvases, and this is in Tk and not matplotlib proper. Todd
may have something to add here.

JDH

"clovis" == clovis <clovis@...339...> writes:
       
   > All, I followed up the 'memory leak' discussion in the
   > sourceforge list and I know the Matplotlib-FAQ entry about
   > this subject. I've also seen John Hunter's post about the
   > need of matching figure/close pairs. Anyway, I still feel
   > that there are problems in this subject, which can be
   > exposed by the following script (for Windows, but can
   > easily be adapted to Unix).

   > As can be seen by the results (also given below), there is
   > a steady increase in memory usage which is not recovered!

If I recall correctly, there is a known leak in tkagg when you create
multiple canvases, and this is in Tk and not matplotlib proper. Todd
may have something to add here.

JDH

Yesterday you mentioned that that 'memory leak' would probably be caused by Tkagg
and not by matplotlib. You also mentioned that it would be good to hear Todd Miller
about this subject (I agree).

Following your idea I tested the memory usage under different backends.
The results are given below:

···

##############
#TKAgg results
##############
Date/time of test = Wed May 17 10:08:39 2006
OS version = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 bit (Intel)]
OS platform = win32
Matplotlib version = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend = TkAgg
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False SAVEFIG=True
Memory usage before/after figure[ 0] = 15740 21564 5824
Memory usage before/after figure[ 1] = 21564 25944 4380
Memory usage before/after figure[ 2] = 25944 30316 4372
Memory usage before/after figure[ 3] = 30316 34668 4352
Memory usage before/after figure[ 4] = 34668 39020 4352
Memory usage before/after figure[ 5] = 39020 43376 4356
Memory usage before/after figure[ 6] = 43376 47740 4364
Memory usage before/after figure[ 7] = 47740 52096 4356
Memory usage before/after figure[ 8] = 52096 56472 4376
Memory usage before/after figure[ 9] = 56472 60836 4364

############
#Agg results
############
Date/time of test = Wed May 17 09:24:26 2006
OS version = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 bit (Intel)]
OS platform = win32
Matplotlib version = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend = Agg
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False SAVEFIG=True
Memory usage before/after figure[ 0] = 14160 17008 2848
Memory usage before/after figure[ 1] = 17008 17504 496
Memory usage before/after figure[ 2] = 17504 17604 100
Memory usage before/after figure[ 3] = 17604 17604 0
Memory usage before/after figure[ 4] = 17604 17588 -16
Memory usage before/after figure[ 5] = 17588 17608 20
Memory usage before/after figure[ 6] = 17608 17600 -8
Memory usage before/after figure[ 7] = 17600 17596 -4
Memory usage before/after figure[ 8] = 17596 17584 -12
Memory usage before/after figure[ 9] = 17584 17604 20

##############
#Cairo results
##############
Date/time of test = Wed May 17 10:29:09 2006
OS version = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 bit (Intel)]
OS platform = win32
Matplotlib version = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend = Cairo
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False SAVEFIG=True
Memory usage before/after figure[ 0] = 14024 15436 1412
Memory usage before/after figure[ 1] = 15436 15944 508
Memory usage before/after figure[ 2] = 15944 16252 308
Memory usage before/after figure[ 3] = 16252 16460 208
Memory usage before/after figure[ 4] = 16460 16468 8
Memory usage before/after figure[ 5] = 16468 18448 1980
Memory usage before/after figure[ 6] = 18448 18464 16
Memory usage before/after figure[ 7] = 18464 16744 -1720
Memory usage before/after figure[ 8] = 16744 18144 1400
Memory usage before/after figure[ 9] = 18144 18488 344

#################
#GTKCairo results
#################
Date/time of test = Wed May 17 10:28:45 2006
OS version = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 bit (Intel)]
OS platform = win32
Matplotlib version = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend = GTKCairo
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False SAVEFIG=True
Memory usage before/after figure[ 0] = 20888 26384 5496
Memory usage before/after figure[ 1] = 26384 26972 588
Memory usage before/after figure[ 2] = 26972 28224 1252
Memory usage before/after figure[ 3] = 28224 27848 -376
Memory usage before/after figure[ 4] = 27848 27440 -408
Memory usage before/after figure[ 5] = 27440 29444 2004
Memory usage before/after figure[ 6] = 29444 29668 224
Memory usage before/after figure[ 7] = 29668 28840 -828
Memory usage before/after figure[ 8] = 28840 28804 -36
Memory usage before/after figure[ 9] = 28804 29884 1080

############
#GTK results
############
Date/time of test = Wed May 17 09:20:02 2006
OS version = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 bit (Intel)]
OS platform = win32
Matplotlib version = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend = GTK
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False SAVEFIG=True
Memory usage before/after figure[ 0] = 22128 28044 5916
Memory usage before/after figure[ 1] = 28048 28624 576
Memory usage before/after figure[ 2] = 28624 28804 180
Memory usage before/after figure[ 3] = 28804 28872 68
Memory usage before/after figure[ 4] = 28872 28948 76
Memory usage before/after figure[ 5] = 28948 29020 72
Memory usage before/after figure[ 6] = 29020 29080 60
Memory usage before/after figure[ 7] = 29080 29144 64
Memory usage before/after figure[ 8] = 29144 29240 96
Memory usage before/after figure[ 9] = 29240 29292 52

###############
#GTKAgg results
###############
Date/time of test = Wed May 17 09:20:24 2006
OS version = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 bit (Intel)]
OS platform = win32
Matplotlib version = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend = GTKAgg
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False SAVEFIG=True
Memory usage before/after figure[ 0] = 23880 31124 7244
Memory usage before/after figure[ 1] = 31124 31708 584
Memory usage before/after figure[ 2] = 31708 31904 196
Memory usage before/after figure[ 3] = 31904 32308 404
Memory usage before/after figure[ 4] = 32308 32388 80
Memory usage before/after figure[ 5] = 32388 32144 -244
Memory usage before/after figure[ 6] = 32144 32560 416
Memory usage before/after figure[ 7] = 32560 32276 -284
Memory usage before/after figure[ 8] = 32276 32380 104
Memory usage before/after figure[ 9] = 32380 32444 64

############
#EMF results
############
Date/time of test = Wed May 17 09:18:19 2006
OS version = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 bit (Intel)]
OS platform = win32
Matplotlib version = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend = EMF
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after figure
Column #3 = (after-before) memory

Configuration SHOWFIG=True SAVEFIG=True
Memory usage before/after figure[ 0] = 12352 13268 916
Memory usage before/after figure[ 1] = 13268 13732 464
Memory usage before/after figure[ 2] = 13732 13896 164
Memory usage before/after figure[ 3] = 13896 13900 4
Memory usage before/after figure[ 4] = 13900 13804 -96
Memory usage before/after figure[ 5] = 13804 13904 100
Memory usage before/after figure[ 6] = 13904 13904 0
Memory usage before/after figure[ 7] = 13904 13912 8
Memory usage before/after figure[ 8] = 13912 13836 -76
Memory usage before/after figure[ 9] = 13836 13920 84

###########
#PS results
###########
Date/time of test = Wed May 17 09:21:15 2006
OS version = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 bit (Intel)]
OS platform = win32
Matplotlib version = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend = PS
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False SAVEFIG=True
Memory usage before/after figure[ 0] = 12912 14032 1120
Memory usage before/after figure[ 1] = 14032 14508 476
Memory usage before/after figure[ 2] = 14508 16344 1836
Memory usage before/after figure[ 3] = 16344 16548 204
Memory usage before/after figure[ 4] = 16548 17032 484
Memory usage before/after figure[ 5] = 17032 16996 -36
Memory usage before/after figure[ 6] = 16996 16648 -348
Memory usage before/after figure[ 7] = 16648 14876 -1772
Memory usage before/after figure[ 8] = 14876 16928 2052
Memory usage before/after figure[ 9] = 16928 16804 -124

############
#SVG results
############
Date/time of test = Wed May 17 09:21:34 2006
OS version = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 bit (Intel)]
OS platform = win32
Matplotlib version = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend = SVG
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False SAVEFIG=True
Memory usage before/after figure[ 0] = 12844 13648 804
Memory usage before/after figure[ 1] = 13648 14128 480
Memory usage before/after figure[ 2] = 14128 14224 96
Memory usage before/after figure[ 3] = 14224 14224 0
Memory usage before/after figure[ 4] = 14224 14224 0
Memory usage before/after figure[ 5] = 14224 14228 4
Memory usage before/after figure[ 6] = 14228 14228 0
Memory usage before/after figure[ 7] = 14228 14228 0
Memory usage before/after figure[ 8] = 14228 14228 0
Memory usage before/after figure[ 9] = 14228 14228 0

As can be seen by the previously show results, it seems that the TKAgg still has some problems.
I'm not worried the the 'DC level' of the backends or with the results concerning figure[0].
The problem I'm pointing is the continuous increase in memory usage in the TKAgg backend.
The script used for these tests is:

import pylab
import os
import time
N = 10 # number of loops to execute
SAVEFIG = True # SAVEFIG execution flag
SHOWFIG = False # SHOWFIG execution flag

report_filename = 'memory_report_%s.txt' % pylab.matplotlib.get_backend()
fid = file(report_filename,'wt')
fid.write('Date/time of test = %s\n' % time.asctime())
fid.write('OS version = %s\n' % os.sys.version)
fid.write('OS platform = %s\n' % os.sys.platform)
fid.write('Matplotlib version = %s\n' % pylab.matplotlib.__version__)
fid.write('Matplotlib revision = %s\n' % pylab.matplotlib.__revision__)
fid.write('Matplotlib backend = %s\n' % pylab.matplotlib.get_backend())
fid.write('Column #0 = figure index\n')
fid.write('Column #1 = memory usage before figure\n')
fid.write('Column #2 = memory usage after figure\n')
fid.write('Column #3 = (after-before) memory\n')
      pylab.ion()
a=pylab.arange(0,10)

def report_memory():
    ### Attention: the path to the pslist utility should be adjusted according to installation!
    if os.sys.platform == 'win32':
        ps_exe_filename = os.path.join(os.getcwd(),'pslist.exe') #Build ps filename
        a = os.popen('%s -m python' % ps_exe_filename).readlines() #Build and execute command
        b = a[8]
        c = b.split()
        return int(c[3])
    else:
        print 'Sorry, you have to adapt the command for your OS!'
        return 0

def figureloop(N):
    for i in range(0,N):
        memory_usage_before = report_memory()
        fid.write('Memory usage before/after figure[%2d] = %8d' % (i, memory_usage_before))
        pylab.figure(i)
        pylab.plot(a,2*a)
        figurename = 'fig%02d' % i
        if SAVEFIG:
            pylab.savefig(figurename)
            pylab.savefig(figurename+'.eps')
        if SHOWFIG:
            pylab.show()
        pylab.close(i)
        time.sleep(1.0) # wait 1.0 second before inspecting memory usage
        if os.path.isfile(figurename): # remove figure ...
            #os.remove(figurename)
            pass
        memory_usage_after = report_memory()
        delta_memory = memory_usage_after - memory_usage_before
        fid.write(' %8d %8d\n' % (memory_usage_after, delta_memory))
        print '%2d %6d %6d %6d' % (i, memory_usage_before, memory_usage_after, delta_memory)
       
print 'Column #0 = figure index'
print 'Column #1 = memory usage before figure'
print 'Column #2 = memory usage after figure'
print 'Column #3 = (after-before) memory'
print 'There is a sleep time of 1s between each figure!'
print 'Close Figure[0] to continue execution!'
print('\nConfiguration SHOWFIG=%s SAVEFIG=%s' % (SHOWFIG, SAVEFIG))
fid.write('\nConfiguration SHOWFIG=%s SAVEFIG=%s\n' % (SHOWFIG, SAVEFIG))
figureloop(N)

This script should be executed with the backend parameters, such as:
    python memory_test.py -dTKAgg
    python memory_test.py -dAgg
    etc
for all the desired backends.

Thanks for your support and congratulations for your great work (matplotlib).
clovis

After waiting for some days for some suggestion, I would like to ask if
there is some suggestion or "starting-point" for solving this issue.
Sincerely yours,
clovis

John Hunter wrote:

···

"clovis" == clovis <clovis@...339...> writes:
           
   > All, I followed up the 'memory leak' discussion in the
   > sourceforge list and I know the Matplotlib-FAQ entry about
   > this subject. I've also seen John Hunter's post about the
   > need of matching figure/close pairs. Anyway, I still feel
   > that there are problems in this subject, which can be
   > exposed by the following script (for Windows, but can
   > easily be adapted to Unix).

   > As can be seen by the results (also given below), there is
   > a steady increase in memory usage which is not recovered!

If I recall correctly, there is a known leak in tkagg when you create
multiple canvases, and this is in Tk and not matplotlib proper. Todd
may have something to add here.

JDH

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users