basemap memory leak?

Community matplotlib-users
1

Hi all,

I'm trying to generate a plot that contains 3 subplots:
2 line plots and a basemap plot showing location. I'm
generating about 200 such plots however my script
doesn't get to complete because it encounters a MemoryError.

I found the test script at http://matplotlib.sourceforge.net/faq.html#LEAKS (which
BTW generates a plot with 4 subplots) and it succeeded
beautifully with no memory issues. However I changed the
last subplot to be a basemap plot and I was able to see
the memory leak. Below is the modified leak test script
I used and the abbreviated output from it. I would truly
appreciate any help on this.

Thanks,

Gerald

···

################################################
#memory leak test
################################################
import os, sys, time
import matplotlib
matplotlib.use('Agg')
from pylab import *
from matplotlib.toolkits.basemap import Basemap

def report_memory(i):
     pid = os.getpid()
     a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
     print i, ' ', a2[1],
     return int(a2[1].split()[1])

# take a memory snapshot on indStart and compare it with indEnd
indStart, indEnd = 30, 150
for i in range(indEnd):
     ind = arange(100)
     xx = rand(len(ind))

     figure(1)
     subplot(221)
     plot(ind, xx)

     subplot(222)
     X = rand(50,50)

     imshow(X)
     subplot(223)
     scatter(rand(50), rand(50))
     subplot(224)
     #pcolor(10*rand(50,50))
     m = Basemap(0, -85, 360, 85)
     m.plot((50,), (20,), 'ro', markersize=12, label='gps')
     m.drawcoastlines()

     savefig('tmp%d' % i, dpi = 75)
     close(1)

     val = report_memory(i)
    # wait a few cycles for memory usage to stabilize
     if i==indStart: start = val

end = val
print 'Average memory consumed per loop: %1.4fk bytes ' % \
       ((end-start)/float(indEnd-indStart))

################################################
#abbreviated results
################################################
0 42724 12739
1 65068 18359
2 83036 22924
3 100980 27463
4 118928 32027
5 136872 36577
6 154828 41129
7 172784 45681
8 190736 50156
9 208692 54711
10 226644 59272
11 244592 63824
12 262548 68375
13 280496 72926
14 298440 77478
15 316392 81964
.
143 1771824 663458
144 1788924 668010
145 1791316 672561
146 1771104 677112
147 1775644 681599
148 1787288 686166
149 1791604 690718
Average memory consumed per loop: 4505.0833k bytes

2

Gerald John M. Manipon wrote:

Hi all,

I'm trying to generate a plot that contains 3 subplots:
2 line plots and a basemap plot showing location. I'm
generating about 200 such plots however my script
doesn't get to complete because it encounters a MemoryError.

I found the test script at http://matplotlib.sourceforge.net/faq.html#LEAKS (which
BTW generates a plot with 4 subplots) and it succeeded
beautifully with no memory issues. However I changed the
last subplot to be a basemap plot and I was able to see
the memory leak. Below is the modified leak test script
I used and the abbreviated output from it. I would truly
appreciate any help on this.

Thanks,

Gerald

################################################
#memory leak test
################################################
import os, sys, time
import matplotlib
matplotlib.use('Agg')
from pylab import *
from matplotlib.toolkits.basemap import Basemap

def report_memory(i):
     pid = os.getpid()
     a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
     print i, ' ', a2[1],
     return int(a2[1].split()[1])

# take a memory snapshot on indStart and compare it with indEnd
indStart, indEnd = 30, 150
for i in range(indEnd):
     ind = arange(100)
     xx = rand(len(ind))

     figure(1)
     subplot(221)
     plot(ind, xx)

     subplot(222)
     X = rand(50,50)

     imshow(X)
     subplot(223)
     scatter(rand(50), rand(50))
     subplot(224)
     #pcolor(10*rand(50,50))
     m = Basemap(0, -85, 360, 85)
     m.plot((50,), (20,), 'ro', markersize=12, label='gps')
     m.drawcoastlines()

     savefig('tmp%d' % i, dpi = 75)
     close(1)

     val = report_memory(i)
    # wait a few cycles for memory usage to stabilize
     if i==indStart: start = val

end = val
print 'Average memory consumed per loop: %1.4fk bytes ' % \
       ((end-start)/float(indEnd-indStart))

################################################
#abbreviated results
################################################
0 42724 12739
1 65068 18359
2 83036 22924
3 100980 27463
4 118928 32027
5 136872 36577
6 154828 41129
7 172784 45681
8 190736 50156
9 208692 54711
10 226644 59272
11 244592 63824
12 262548 68375
13 280496 72926
14 298440 77478
15 316392 81964
.
143 1771824 663458
144 1788924 668010
145 1791316 672561
146 1771104 677112
147 1775644 681599
148 1787288 686166
149 1791604 690718
Average memory consumed per loop: 4505.0833k bytes
  
Gerald: No leak here

[jsw@...1240...:/Users/jsw/python] python memleak.py
0 27856 89672
1 32672 92620
2 33200 92620
3 33200 92620
4 33168 92612
5 33196 92620
6 33196 92620
7 33164 92616
8 33200 92620
9 33184 92616
10 33196 92620
11 33184 92616
12 33176 92616
13 33204 92620

MacOS 10.4.7, matplotlib 0.87.4, basemap latest svn, numpy 1.0b2.

However, if I use Numeric 24.2

[jsw@...1240...:/Users/jsw/python] python memleak.py --Numeric
0 44472 102496
1 67140 123932
2 85376 141308
3 103200 159728
4 120992 177104
5 138800 194496

it leaks! No idea why - but I doubt it has much to do with basemap.

-Jeff

···

--
Jeffrey S. Whitaker Phone : (303)497-6313
NOAA/OAR/CDC R/PSD1 FAX : (303)497-6449
325 Broadway Boulder, CO, USA 80305-3328

3

Gerald John M. Manipon wrote:

Hi all,

I'm trying to generate a plot that contains 3 subplots:
2 line plots and a basemap plot showing location. I'm
generating about 200 such plots however my script
doesn't get to complete because it encounters a MemoryError.

I found the test script at http://matplotlib.sourceforge.net/faq.html#LEAKS (which
BTW generates a plot with 4 subplots) and it succeeded
beautifully with no memory issues. However I changed the
last subplot to be a basemap plot and I was able to see
the memory leak. Below is the modified leak test script
I used and the abbreviated output from it. I would truly
appreciate any help on this.

Thanks,

Gerald

################################################
#memory leak test
################################################
import os, sys, time
import matplotlib
matplotlib.use('Agg')
from pylab import *
from matplotlib.toolkits.basemap import Basemap

def report_memory(i):
     pid = os.getpid()
     a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
     print i, ' ', a2[1],
     return int(a2[1].split()[1])

# take a memory snapshot on indStart and compare it with indEnd
indStart, indEnd = 30, 150
for i in range(indEnd):
     ind = arange(100)
     xx = rand(len(ind))

     figure(1)
     subplot(221)
     plot(ind, xx)

     subplot(222)
     X = rand(50,50)

     imshow(X)
     subplot(223)
     scatter(rand(50), rand(50))
     subplot(224)
     #pcolor(10*rand(50,50))
     m = Basemap(0, -85, 360, 85)
     m.plot((50,), (20,), 'ro', markersize=12, label='gps')
     m.drawcoastlines()

     savefig('tmp%d' % i, dpi = 75)
     close(1)

     val = report_memory(i)
    # wait a few cycles for memory usage to stabilize
     if i==indStart: start = val

end = val
print 'Average memory consumed per loop: %1.4fk bytes ' % \
       ((end-start)/float(indEnd-indStart))

################################################
#abbreviated results
################################################
0 42724 12739
1 65068 18359
2 83036 22924
3 100980 27463
4 118928 32027
5 136872 36577
6 154828 41129
7 172784 45681
8 190736 50156
9 208692 54711
10 226644 59272
11 244592 63824
12 262548 68375
13 280496 72926
14 298440 77478
15 316392 81964
.
143 1771824 663458
144 1788924 668010
145 1791316 672561
146 1771104 677112
147 1775644 681599
148 1787288 686166
149 1791604 690718
Average memory consumed per loop: 4505.0833k bytes
  
Gerald: One way to workaround the memory leak with Numeric is to move the Basemap instance creation

m = Basemap(0, -85, 360, 85)

outside the loop. There's no need to recreate it each time.

As a bonus, the script runs many times faster too.

-Jeff

···

--
Jeffrey S. Whitaker Phone : (303)497-6313
NOAA/OAR/CDC R/PSD1 FAX : (303)497-6449
325 Broadway Boulder, CO, USA 80305-3328

4

You're right, Jeff. When I use numpy in the test script,
I don't see the leak. Alternatively, moving the basemap
object creation outside of the loop (even with Numeric 24.2)
makes it faster and doesn't produce the leak.

Thanks for showing me the way.

Gerald

Jeff Whitaker wrote:

···

Gerald John M. Manipon wrote:

Hi all,

I'm trying to generate a plot that contains 3 subplots:
2 line plots and a basemap plot showing location. I'm
generating about 200 such plots however my script
doesn't get to complete because it encounters a MemoryError.

I found the test script at http://matplotlib.sourceforge.net/faq.html#LEAKS (which
BTW generates a plot with 4 subplots) and it succeeded
beautifully with no memory issues. However I changed the
last subplot to be a basemap plot and I was able to see
the memory leak. Below is the modified leak test script
I used and the abbreviated output from it. I would truly
appreciate any help on this.

Thanks,

Gerald

################################################
#memory leak test
################################################
import os, sys, time
import matplotlib
matplotlib.use('Agg')
from pylab import *
from matplotlib.toolkits.basemap import Basemap

def report_memory(i):
     pid = os.getpid()
     a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
     print i, ' ', a2[1],
     return int(a2[1].split()[1])

# take a memory snapshot on indStart and compare it with indEnd
indStart, indEnd = 30, 150
for i in range(indEnd):
     ind = arange(100)
     xx = rand(len(ind))

     figure(1)
     subplot(221)
     plot(ind, xx)

     subplot(222)
     X = rand(50,50)

     imshow(X)
     subplot(223)
     scatter(rand(50), rand(50))
     subplot(224)
     #pcolor(10*rand(50,50))
     m = Basemap(0, -85, 360, 85)
     m.plot((50,), (20,), 'ro', markersize=12, label='gps')
     m.drawcoastlines()

     savefig('tmp%d' % i, dpi = 75)
     close(1)

     val = report_memory(i)
    # wait a few cycles for memory usage to stabilize
     if i==indStart: start = val

end = val
print 'Average memory consumed per loop: %1.4fk bytes ' % \
       ((end-start)/float(indEnd-indStart))

################################################
#abbreviated results
################################################
0 42724 12739
1 65068 18359
2 83036 22924
3 100980 27463
4 118928 32027
5 136872 36577
6 154828 41129
7 172784 45681
8 190736 50156
9 208692 54711
10 226644 59272
11 244592 63824
12 262548 68375
13 280496 72926
14 298440 77478
15 316392 81964
.
143 1771824 663458
144 1788924 668010
145 1791316 672561
146 1771104 677112
147 1775644 681599
148 1787288 686166
149 1791604 690718
Average memory consumed per loop: 4505.0833k bytes
  
Gerald: One way to workaround the memory leak with Numeric is to move the Basemap instance creation

m = Basemap(0, -85, 360, 85)

outside the loop. There's no need to recreate it each time.

As a bonus, the script runs many times faster too.

-Jeff