Memory usage with matplotlib

[Steve_Chaplin]

The question is: "why isn't the memory collected after

    > closing the figure?". The real program I developed builds
    > 5~15 graphic windows and the required memory is very
    > large.

What makes you think the memory is not being collected?
Collecting memory is not necessarily the same as freeing memory, and
freeing memory is not necessarily the same as returning the memory
to the operating system. I think you may be assuming two things
which are incorrect.

1) When an application frees memory it is returned to the operating
system. See

Radar - O’Reilly

where it says
"An additional point is that malloc does not normally return the freed
memory to the operating system; it remains owned by the process until
it terminates."

2) When the garbage collector collects an unused object it frees the
unused memory.
I don't know the specific details of the Python garbage collector, but
in general I believe garbage collectors effectively wrap calls to
malloc() and free(), and manage larger blocks or pools of
memory. When an object is collected the memory is marked as unused but
is not necessarily freed, it can be reused for the next object
allocation - thereby saving one free() and one or more malloc() calls.

Steve

Send instant messages to your online friends http://au.messenger.yahoo.com

[Clovis_Goldemberg]

Steve Chaplin:

I read the link about memory management you indicated. It is very good and you are right
in pointing the differences between "collecting" and "freeing" memory. Anyway, let me try
to expose the problem once more (sorry for the length).

1. The original problem exposed can be resumed in the following table:

Test #1
Mem Usage Action
3152K Python Command Line Opened
15700K from pylab import *
15700K a = arange(0, 10)
19820K figure(1)
19988K plot(a, a)
22288K show()
20100K Memory usage after closing graphic window

The real program I developed builds 5~15 graphic windows and the required memory is very large.

2. John Hunter suggested me to use a small python application named "memleak_hawaii3.py" he developed
in order to make tests under Linux. I downloaded the "memleak_hawaii3.py" script and adapted it to run
under Windows XP. The "ps" utility was subsituted by an equivalent Windows program named "pslist.exe". I made a simple test in order to verify that the memory usage produced the same result shown by the
Windows XP task manager.

3. The "memleak_hawaii4.py" script opens a set of 40 graphs (without closing them).
   After all these graphs are built, they are all closed.

#!/usr/bin/env python

import os, sys, time
import matplotlib
matplotlib.use('Agg')
from pylab import *

GRAPHNUMBER = 40
CYCLENUMBER = 2

# open a file to store results
fid = open('memory_report1.txt','wt')

def report_memory():
    ### Attention: the path to the pslist utility should be adjusted according to installation!
    a = os.popen('c://tmp//utils//zeta//pslist -m python').readlines()
    b = a[8]
    c = b.split()
    return int(c[3])

for runcycle in range(0, CYCLENUMBER):

    # take a memory snapshot on indStart and compare it with indEnd
    report_string = "\nAllocated memory before loop starts = %6dK\n" % report_memory()
    print report_string
    fid.write(report_string)

    indStart, indEnd = 0, GRAPHNUMBER
    for i in range(indEnd):

        figure(i)
        clf()

        subplot(221)
        t1 = arange(0.0, 2.0, 0.01)
        y = sin(2*pi*t1)
        plot(t1,y,'-')
        plot(t1, rand(len(t1)), 's', hold=True)

        subplot(222)
        X = rand(50,50)
        imshow(X)

        subplot(223)
        scatter(rand(50), rand(50), s=100*rand(50), c=rand(50))

        subplot(224)
        pcolor(10*rand(50,50))

        savefig('tmp%d' % i, dpi = 75)
        show()

        val = report_memory()
        report_string = "Running cycle %1d, opening graphic %2d, allocated memory = %6dK" % (runcycle, i, val)
        print report_string
        fid.write(report_string+'\n')
               if i==indStart: start = val # wait a few cycles for memory usage to stabilize

    end = val
    print '\nAverage memory consumed per loop: %1.4fk bytes\n' % ((end-start)/float(indEnd-indStart))

    for i in range(indEnd):
        close(i)
        val = report_memory()
        report_string = "Running cycle %1d, closing graphic %2d, allocated memory = %6dK" % (runcycle, i, val)
        print report_string
        fid.write(report_string+'\n')

fid.close()

This routine is repeated
   2 times and the results are given in "memory_report1.txt". (resuts below)

Allocated memory before loop starts = 14060K
Running cycle 0, opening graphic 0, allocated memory = 17916K
Running cycle 0, opening graphic 1, allocated memory = 21540K
Running cycle 0, opening graphic 2, allocated memory = 25128K
Running cycle 0, opening graphic 3, allocated memory = 28736K
Running cycle 0, opening graphic 4, allocated memory = 32324K
Running cycle 0, opening graphic 5, allocated memory = 35932K
Running cycle 0, opening graphic 6, allocated memory = 39520K
Running cycle 0, opening graphic 7, allocated memory = 43136K
Running cycle 0, opening graphic 8, allocated memory = 46748K
Running cycle 0, opening graphic 9, allocated memory = 50356K
Running cycle 0, opening graphic 10, allocated memory = 53952K
Running cycle 0, opening graphic 11, allocated memory = 57560K
Running cycle 0, opening graphic 12, allocated memory = 61184K
Running cycle 0, opening graphic 13, allocated memory = 64896K
Running cycle 0, opening graphic 14, allocated memory = 68508K
Running cycle 0, opening graphic 15, allocated memory = 72092K
Running cycle 0, opening graphic 16, allocated memory = 75692K
Running cycle 0, opening graphic 17, allocated memory = 79312K
Running cycle 0, opening graphic 18, allocated memory = 82944K
Running cycle 0, opening graphic 19, allocated memory = 86544K
Running cycle 0, opening graphic 20, allocated memory = 90176K
Running cycle 0, opening graphic 21, allocated memory = 93752K
Running cycle 0, opening graphic 22, allocated memory = 97336K
Running cycle 0, opening graphic 23, allocated memory = 100956K
Running cycle 0, opening graphic 24, allocated memory = 104536K
Running cycle 0, opening graphic 25, allocated memory = 108176K
Running cycle 0, opening graphic 26, allocated memory = 111768K
Running cycle 0, opening graphic 27, allocated memory = 115400K
Running cycle 0, opening graphic 28, allocated memory = 118992K
Running cycle 0, opening graphic 29, allocated memory = 122628K
Running cycle 0, opening graphic 30, allocated memory = 126220K
Running cycle 0, opening graphic 31, allocated memory = 129808K
Running cycle 0, opening graphic 32, allocated memory = 133440K
Running cycle 0, opening graphic 33, allocated memory = 137008K
Running cycle 0, opening graphic 34, allocated memory = 140612K
Running cycle 0, opening graphic 35, allocated memory = 144220K
Running cycle 0, opening graphic 36, allocated memory = 147844K
Running cycle 0, opening graphic 37, allocated memory = 151456K
Running cycle 0, opening graphic 38, allocated memory = 155080K
Running cycle 0, opening graphic 39, allocated memory = 158672K
Running cycle 0, closing graphic 0, allocated memory = 158672K
Running cycle 0, closing graphic 1, allocated memory = 157448K
Running cycle 0, closing graphic 2, allocated memory = 155296K
Running cycle 0, closing graphic 3, allocated memory = 153052K
Running cycle 0, closing graphic 4, allocated memory = 150624K
Running cycle 0, closing graphic 5, allocated memory = 148440K
Running cycle 0, closing graphic 6, allocated memory = 146032K
Running cycle 0, closing graphic 7, allocated memory = 143692K
Running cycle 0, closing graphic 8, allocated memory = 141600K
Running cycle 0, closing graphic 9, allocated memory = 139156K
Running cycle 0, closing graphic 10, allocated memory = 136884K
Running cycle 0, closing graphic 11, allocated memory = 134476K
Running cycle 0, closing graphic 12, allocated memory = 132336K
Running cycle 0, closing graphic 13, allocated memory = 130072K
Running cycle 0, closing graphic 14, allocated memory = 127736K
Running cycle 0, closing graphic 15, allocated memory = 125352K
Running cycle 0, closing graphic 16, allocated memory = 123004K
Running cycle 0, closing graphic 17, allocated memory = 120916K
Running cycle 0, closing graphic 18, allocated memory = 118428K
Running cycle 0, closing graphic 19, allocated memory = 115980K
Running cycle 0, closing graphic 20, allocated memory = 113544K
Running cycle 0, closing graphic 21, allocated memory = 111368K
Running cycle 0, closing graphic 22, allocated memory = 108984K
Running cycle 0, closing graphic 23, allocated memory = 106684K
Running cycle 0, closing graphic 24, allocated memory = 104504K
Running cycle 0, closing graphic 25, allocated memory = 102092K
Running cycle 0, closing graphic 26, allocated memory = 99788K
Running cycle 0, closing graphic 27, allocated memory = 97404K
Running cycle 0, closing graphic 28, allocated memory = 95220K
Running cycle 0, closing graphic 29, allocated memory = 92716K
Running cycle 0, closing graphic 30, allocated memory = 90496K
Running cycle 0, closing graphic 31, allocated memory = 88048K
Running cycle 0, closing graphic 32, allocated memory = 85792K
Running cycle 0, closing graphic 33, allocated memory = 83576K
Running cycle 0, closing graphic 34, allocated memory = 81044K
Running cycle 0, closing graphic 35, allocated memory = 78752K
Running cycle 0, closing graphic 36, allocated memory = 76440K
Running cycle 0, closing graphic 37, allocated memory = 74156K
Running cycle 0, closing graphic 38, allocated memory = 71804K
Running cycle 0, closing graphic 39, allocated memory = 69336K

Allocated memory before loop starts = 69336K
Running cycle 1, opening graphic 0, allocated memory = 70988K
Running cycle 1, opening graphic 1, allocated memory = 72296K
Running cycle 1, opening graphic 2, allocated memory = 73368K
Running cycle 1, opening graphic 3, allocated memory = 75164K
Running cycle 1, opening graphic 4, allocated memory = 77404K
Running cycle 1, opening graphic 5, allocated memory = 78128K
Running cycle 1, opening graphic 6, allocated memory = 79668K
Running cycle 1, opening graphic 7, allocated memory = 82072K
Running cycle 1, opening graphic 8, allocated memory = 84292K
Running cycle 1, opening graphic 9, allocated memory = 86620K
Running cycle 1, opening graphic 10, allocated memory = 88864K
Running cycle 1, opening graphic 11, allocated memory = 91168K
Running cycle 1, opening graphic 12, allocated memory = 93560K
Running cycle 1, opening graphic 13, allocated memory = 95944K
Running cycle 1, opening graphic 14, allocated memory = 98356K
Running cycle 1, opening graphic 15, allocated memory = 100780K
Running cycle 1, opening graphic 16, allocated memory = 103248K
Running cycle 1, opening graphic 17, allocated memory = 105576K
Running cycle 1, opening graphic 18, allocated memory = 108080K
Running cycle 1, opening graphic 19, allocated memory = 110400K
Running cycle 1, opening graphic 20, allocated memory = 112808K
Running cycle 1, opening graphic 21, allocated memory = 115212K
Running cycle 1, opening graphic 22, allocated memory = 117628K
Running cycle 1, opening graphic 23, allocated memory = 120032K
Running cycle 1, opening graphic 24, allocated memory = 122576K
Running cycle 1, opening graphic 25, allocated memory = 124860K
Running cycle 1, opening graphic 26, allocated memory = 127280K
Running cycle 1, opening graphic 27, allocated memory = 129704K
Running cycle 1, opening graphic 28, allocated memory = 132200K
Running cycle 1, opening graphic 29, allocated memory = 134536K
Running cycle 1, opening graphic 30, allocated memory = 136960K
Running cycle 1, opening graphic 31, allocated memory = 139428K
Running cycle 1, opening graphic 32, allocated memory = 141748K
Running cycle 1, opening graphic 33, allocated memory = 144148K
Running cycle 1, opening graphic 34, allocated memory = 146648K
Running cycle 1, opening graphic 35, allocated memory = 149004K
Running cycle 1, opening graphic 36, allocated memory = 151416K
Running cycle 1, opening graphic 37, allocated memory = 153844K
Running cycle 1, opening graphic 38, allocated memory = 156256K
Running cycle 1, opening graphic 39, allocated memory = 158736K
Running cycle 1, closing graphic 0, allocated memory = 158736K
Running cycle 1, closing graphic 1, allocated memory = 157412K
Running cycle 1, closing graphic 2, allocated memory = 156100K
Running cycle 1, closing graphic 3, allocated memory = 154684K
Running cycle 1, closing graphic 4, allocated memory = 152848K
Running cycle 1, closing graphic 5, allocated memory = 150648K
Running cycle 1, closing graphic 6, allocated memory = 148652K
Running cycle 1, closing graphic 7, allocated memory = 146392K
Running cycle 1, closing graphic 8, allocated memory = 144176K
Running cycle 1, closing graphic 9, allocated memory = 141972K
Running cycle 1, closing graphic 10, allocated memory = 139768K
Running cycle 1, closing graphic 11, allocated memory = 137472K
Running cycle 1, closing graphic 12, allocated memory = 135220K
Running cycle 1, closing graphic 13, allocated memory = 132548K
Running cycle 1, closing graphic 14, allocated memory = 130092K
Running cycle 1, closing graphic 15, allocated memory = 127720K
Running cycle 1, closing graphic 16, allocated memory = 125372K
Running cycle 1, closing graphic 17, allocated memory = 123028K
Running cycle 1, closing graphic 18, allocated memory = 120584K
Running cycle 1, closing graphic 19, allocated memory = 118296K
Running cycle 1, closing graphic 20, allocated memory = 115652K
Running cycle 1, closing graphic 21, allocated memory = 113472K
Running cycle 1, closing graphic 22, allocated memory = 111044K
Running cycle 1, closing graphic 23, allocated memory = 108508K
Running cycle 1, closing graphic 24, allocated memory = 106016K
Running cycle 1, closing graphic 25, allocated memory = 103724K
Running cycle 1, closing graphic 26, allocated memory = 101368K
Running cycle 1, closing graphic 27, allocated memory = 98792K
Running cycle 1, closing graphic 28, allocated memory = 96276K
Running cycle 1, closing graphic 29, allocated memory = 93960K
Running cycle 1, closing graphic 30, allocated memory = 91508K
Running cycle 1, closing graphic 31, allocated memory = 89160K
Running cycle 1, closing graphic 32, allocated memory = 86576K
Running cycle 1, closing graphic 33, allocated memory = 84296K
Running cycle 1, closing graphic 34, allocated memory = 81900K
Running cycle 1, closing graphic 35, allocated memory = 79532K
Running cycle 1, closing graphic 36, allocated memory = 77160K
Running cycle 1, closing graphic 37, allocated memory = 74712K
Running cycle 1, closing graphic 38, allocated memory = 72368K
Running cycle 1, closing graphic 39, allocated memory = 69652K

Since the peak memory can reach ~150M, which may require swapping (which slows the application).
Another point is the starting point ~14M and the end point ~70M.

4. The "memleak_hawaii5.py" script opens a set of 40 graphs. They are closed after
   creation and a memory snapshot is taken before and after creation. This routine
   runs 2 times and the results are given in "memory_report2.txt".
   The results in this case show that "destroying" the figure after its creation returns almost all the memory to the
OS and this cycling will not required "swapping".

5. A am a Python and Matplotlib novice and (unfortunately) I don't deal well
   with the internals of these applications so probably some of my questions may seem misplaced.

Sincerely yours,
Clovis Goldemberg
University of Sao Paulo

[Bill_Dandreta]

Clovis Goldemberg wrote:

The real program I developed builds 5~15 graphic windows and the required memory is very large.

The simplest solution to your problem is to add more ram to your system.

A cheap 256MB stick of ram sells for around $20, half that if you buy used.

Bill