Memory leaks in Basemap

Hi all,

I am using matplotlib-0.87.7, Basemap-0.9.3, Numpy-1.1 and agg backend.
I am facing memory leak problems after running following script -

import os, sys, time
import cStringIO
import Image
import matplotlib
matplotlib.use(‘Agg’)

import matplotlib.pylab as p
import matplotlib.numerix as nx
from matplotlib.toolkits.basemap import Basemap

def get_image_bytes(width_fig, height_fig, str_img):
“”"Returns the bytes representing the image generated

The createImage method should be called before calling this method.

:Returns:
    Byte data.
"""
# Get the actual image data
f = cStringIO.StringIO()
dims = (width_fig, height_fig)
im = Image.fromstring("RGB", dims, str_img)

im.save(f, “JPEG”)
return f.getvalue()

def report_memory(i):
pid = os.getpid()
a2 = os.popen(‘ps -p %d -o rss,vsz,%%mem’ % pid).readlines()
print i, ’ ', a2[1],
return int(a2[1].split()[1])

take a memory snapshot on indStart and compare it with indEnd

indStart, indEnd = 30, 150
print ’ rss vsz %mem’
for i in range(indEnd):
p.figure(figsize=(float(640)/80.0,float(480)/80.0),
facecolor=‘w’, edgecolor=‘w’)
p.axes([0.0, 0.0, 1.0, 1.0])
map1 = Basemap(projection=‘cyl’, lat_0=50, lon_0=-100,

area_thresh=1000.)
# draw coastlines, country boundaries, fill continents.
map1.drawcoastlines(linewidth=.5)
map1.drawcountries()
map1.fillcontinents(color="#CEFFCE")
# draw the edge of the map projection region (the projection limb)
map1.drawmapboundary()
# compute the native map projection coordinates for cities.
lats, lons = ([18.728], [20.890])
x,y = map1(lons,lats)
# plot filled circles at the locations of the cities.
map1.plot(x,y,‘bo’)
canvas = p.get_current_fig_manager().canvas
agg = canvas.switch_backends
(matplotlib.backends.backend_agg.FigureCanvasAgg)
agg.draw() # Takes ~.9
seconds
# get the image data
str_img = agg.tostring_rgb()
width_fig, height_fig = map(int, agg.figure.bbox.get_bounds()[2:])
p.cla()
p.close()
bytes = get_image_bytes(width_fig, height_fig, str_img)

val = report_memory(i)

wait a few cycles for memory usage to stabilize

if i==indStart: start = val

end = val
print ‘Average memory consumed per loop: %1.4fk bytes’ %
((end-start)/float(indEnd-indStart))

Here is the O/P

$python memory_leak_map.py
rss vsz %mem
0 47272 50632 9.7
1 74412 77700 15.3
2 93960 97380 19.3
3 113308 116776 23.3
4
132824 136416 27.3
5 152352 155828 31.3
6 171860 175216 35.3
7 191372 194868 39.3
8 210872 214248 43.3
9 230336 233916 47.3
10 249732 253284 51.3
11 269252 272692 55.3
12 288680 292336 59.3
13 308108 311724 63.2
14 305160 331112 62.6
15 301096 350764 61.8
16 304884 370160 62.6
17 298276 389804 61.2
18 305876 409184 62.8
19 298316 428596 61.2
20 307856 448224 63.2
21 308004 467640 63.2
22 308844 487016 63.4
23 306260 506656 62.9
24 300612 526052 61.7
Traceback (most recent call last):
File “memory_leak_map.py”, line 41, in ?

area_thresh=1000.)
File “/usr/local/lib/python2.4/site-packages/matplotlib/toolkits/basemap/basemap.py”, line 815, in init
self.riversegs = segments+segments2+segments3
MemoryError
Killed: 9

Can someone please help me out solving this problem?

Thanks
-Aalok

···

Find out what India is talking about on - Yahoo! Answers India

Send FREE SMS to your friend’s mobile from Yahoo! Messenger Version 8. Get it NOW

Aalok kapoor wrote:

Hi all,

I am using matplotlib-0.87.7, Basemap-0.9.3, Numpy-1.1 and agg backend.
I am facing memory leak problems after running following script -

import os, sys, time
import cStringIO
import Image
import matplotlib
matplotlib.use('Agg')

import matplotlib.pylab as p
import matplotlib.numerix as nx
from matplotlib.toolkits.basemap import Basemap

def get_image_bytes(width_fig, height_fig, str_img):
    """Returns the bytes representing the image generated

    The createImage method should be called before calling this method.

    :Returns:
        Byte data.
    """
    # Get the actual image data
    f = cStringIO.StringIO()
    dims = (width_fig, height_fig)
    im = Image.fromstring("RGB", dims, str_img)
    im.save(f, "JPEG")
    return f.getvalue()

def report_memory(i):
    pid = os.getpid()
    a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
    print i, ' ', a2[1],
    return int(a2[1].split()[1])

# take a memory snapshot on indStart and compare it with indEnd
indStart, indEnd = 30, 150
print ' rss vsz %mem'
for i in range(indEnd):
    p.figure(figsize=(float(640)/80.0,float(480)/80.0),
             facecolor='w', edgecolor='w')
    p.axes([0.0, 0.0, 1.0, 1.0])
    map1 = Basemap(projection='cyl', lat_0=50, lon_0=-100,
                    area_thresh=1000.)
    # draw coastlines, country boundaries, fill continents.
    map1.drawcoastlines(linewidth=.5)
    map1.drawcountries()
    map1.fillcontinents(color="#CEFFCE")
    # draw the edge of the map projection region (the projection limb)
    map1.drawmapboundary()
    # compute the native map projection coordinates for cities.
    lats, lons = ([18.728], [20.890])
    x,y = map1(lons,lats)
    # plot filled circles at the locations of the cities.
    map1.plot(x,y,'bo')
    canvas = p.get_current_fig_manager().canvas
    agg = canvas.switch_backends \
          (matplotlib.backends.backend_agg.FigureCanvasAgg)
    agg.draw() # Takes ~.9 seconds
    # get the image data
    str_img = agg.tostring_rgb()
    width_fig, height_fig = map(int, agg.figure.bbox.get_bounds()[2:])
    p.cla()
    p.close()
    bytes = get_image_bytes(width_fig, height_fig, str_img)

    val = report_memory(i)
   # wait a few cycles for memory usage to stabilize
    if i==indStart: start = val

end = val
print 'Average memory consumed per loop: %1.4fk bytes' % \
      ((end-start)/float(indEnd-indStart))

Here is the O/P

$python memory_leak_map.py
     rss vsz %mem
0 47272 50632 9.7
1 74412 77700 15.3
2 93960 97380 19.3
3 113308 116776 23.3
4 132824 136416 27.3
5 152352 155828 31.3
6 171860 175216 35.3
7 191372 194868 39.3
8 210872 214248 43.3
9 230336 233916 47.3
10 249732 253284 51.3
11 269252 272692 55.3
12 288680 292336 59.3
13 308108 311724 63.2
14 305160 331112 62.6
15 301096 350764 61.8
16 304884 370160 62.6
17 298276 389804 61.2
18 305876 409184 62.8
19 298316 428596 61.2
20 307856 448224 63.2
21 308004 467640 63.2
22 308844 487016 63.4
23 306260 506656 62.9
24 300612 526052 61.7
Traceback (most recent call last):
  File "memory_leak_map.py", line 41, in ?
    area_thresh=1000.)
  File "/usr/local/lib/python2.4/site-packages/matplotlib/toolkits/basemap/basemap.py", line 815, in __init__
    self.riversegs = segments+segments2+segments3
MemoryError
Killed: 9

Can someone please help me out solving this problem?

Thanks
-Aalok

Aaolk: All your plots use the same map projection, yet you are initializing a new Basemap instance every time through the loop. Try moving the line

    map1 = Basemap(projection='cyl', lat_0=50, lon_0=-100, area_thresh=1000.)

outside the loop.

-Jeff

···

--
Jeffrey S. Whitaker Phone : (303)497-6313
Meteorologist FAX : (303)497-6449
NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@...236...
325 Broadway Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web : Jeffrey S. Whitaker: NOAA Physical Sciences Laboratory

Thanks for reply,
This was just an example for mem leak in my application. Actually i want maps with different data to be loaded each time and the map objects taken once changees after ploting so i have to take different object each time. After around 20 maps generation it gives memory error to me.
thanks,
-Aalok

···

Jeff Whitaker <jswhit@…196…> wrote:

Aalok kapoor wrote:

Hi all,

I am using matplotlib-0.87.7, Basemap-0.9.3, Numpy-1.1 and agg backend.
I am facing memory leak problems after running following script -

import os, sys, time
import cStringIO
import Image
import matplotlib
matplotlib.use(‘Agg’)

import matplotlib.pylab as p
import matplotlib.numerix as nx
from
matplotlib.toolkits.basemap import Basemap

def get_image_bytes(width_fig, height_fig, str_img):
“”"Returns the bytes representing the image generated

The createImage method should be called before calling this method.

:Returns:
Byte data.
“”"

Get the actual image data

f = cStringIO.StringIO()
dims = (width_fig, height_fig)
im = Image.fromstring(“RGB”, dims, str_img)
im.save(f, “JPEG”)
return f.getvalue()

def report_memory(i):
pid = os.getpid()
a2 = os.popen(‘ps -p %d -o rss,vsz,%%mem’ % pid).readlines()
print i, ’ ', a2[1],
return int(a2[1].split()[1])

take a memory snapshot on indStart and compare it with indEnd

indStart, indEnd = 30, 150
print ’ rss vsz %mem’
for i in
range(indEnd):
p.figure(figsize=(float(640)/80.0,float(480)/80.0),
facecolor=‘w’, edgecolor=‘w’)
p.axes([0.0, 0.0, 1.0, 1.0])
map1 = Basemap(projection=‘cyl’, lat_0=50, lon_0=-100,
area_thresh=1000.)

draw coastlines, country boundaries, fill continents.

map1.drawcoastlines(linewidth=.5)
map1.drawcountries()
map1.fillcontinents(color=“#CEFFCE”)

draw the edge of the map projection region (the projection limb)

map1.drawmapboundary()

compute the native map projection coordinates for cities.

lats, lons = ([18.728], [20.890])
x,y = map1(lons,lats)

plot filled circles at the locations of the cities.

map1.plot(x,y,‘bo’)
canvas = p.get_current_fig_manager().canvas
agg = canvas.switch_backends
(matplotlib.backends.backend_agg.FigureCanvasAgg)
agg.draw() # Takes ~.9 seconds

get the image data

str_img = agg.tostring_rgb()
width_fig, height_fig = map(int, agg.figure.bbox.get_bounds()[2:])
p.cla()
p.close()
bytes = get_image_bytes(width_fig, height_fig, str_img)

val = report_memory(i)

wait a few cycles for memory usage to stabilize

if i==indStart: start = val

end = val
print ‘Average memory consumed per loop: %1.4fk bytes’ %
((end-start)/float(indEnd-indStart))

Here is the O/P

$python memory_leak_map.py
rss vsz %mem
0 47272 50632 9.7
1 74412 77700 15.3
2 93960 97380 19.3
3 113308 116776 23.3
4 132824 136416 27.3
5 152352 155828 31.3
6 171860 175216 35.3
7 191372 194868 39.3
8 210872 214248 43.3
9 230336 233916 47.3
10 249732 253284 51.3
11 269252 272692 55.3
12 288680 292336 59.3
13 308108 311724 63.2
14 305160 331112 62.6
15 301096 350764 61.8
16 304884 370160 62.6
17 298276 389804 61.2
18 305876 409184 62.8
19 298316 428596 61.2
20 307856 448224 63.2
21 308004 467640 63.2
22 308844 487016 63.4
23 306260 506656 62.9
24 300612 526052 61.7
Traceback (most recent call last):
File “memory_leak_map.py”, line 41, in ?
area_thresh=1000.)
File
“/usr/local/lib/python2.4/site-packages/matplotlib/toolkits/basemap/basemap.py”,
line 815, in init
self.riversegs = segments+segments2+segments3
MemoryError
Killed:
9

Can someone please help me out solving this problem?

Thanks
-Aalok

Aaolk: All your plots use the same map projection, yet you are
initializing a new Basemap instance every time through the loop. Try
moving the line

map1 = Basemap(projection='cyl', lat_0=50, lon_0=-100,

area_thresh=1000.)

outside the loop.

-Jeff


Jeffrey S. Whitaker Phone : (303)497-6313
Meteorologist FAX : (303)497-6449
NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@…236…
325 Broadway Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web : Jeffrey S. Whitaker: NOAA Physical Sciences Laboratory


Find out what India is talking about on - Yahoo! Answers India

Send FREE SMS to your friend’s mobile from Yahoo! Messenger Version 8. Get it NOW

Aalok kapoor wrote:

Thanks for reply,

This was just an example for mem leak in my application. Actually i want maps with different data to be loaded each time and the map objects taken once changees after ploting so i have to take different object each time. After around 20 maps generation it gives memory error to me.

thanks,
-Aalok

    Aalok kapoor wrote:
    > Hi all,
    >
    > I am using matplotlib-0.87.7, Basemap-0.9.3, Numpy-1.1 and agg
    backend.
    > I am facing memory leak problems after running following script -
    >
    > import os, sys, time
    > import cStringIO
    > import Image
    > import matplotlib
    > matplotlib.use('Agg')
    >
    > import matplotlib.pylab as p
    > import matplotlib.numerix as nx
    > from matplotlib.toolkits.basemap import Basemap
    >
    > def get_image_bytes(width_fig, height_fig, str_img):
    > """Returns the bytes representing the image generated
    >
    > The createImage method should be called before calling this method.
    >
    > :Returns:
    > Byte data.
    > """
    > # Get the actual image data
    > f = cStringIO.StringIO()
    > dims = (width_fig, height_fig)
    > im = Image.fromstring("RGB", dims, str_img)
    > im.save(f, "JPEG")
    > return f.getvalue()
    >
    > def report_memory(i):
    > pid = os.getpid()
    > a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
    > print i, ' ', a2[1],
    > return int(a2[1].split()[1])
    >
    > # take a memory snapshot on indStart and compare it with indEnd
    > indStart, indEnd = 30, 150
    > print ' rss vsz %mem'
    > for i in range(indEnd):
    > p.figure(figsize=(float(640)/80.0,float(480)/80.0),
    > facecolor='w', edgecolor='w')
    > p.axes([0.0, 0.0, 1.0, 1.0])
    > map1 = Basemap(projection='cyl', lat_0=50, lon_0=-100,
    > area_thresh=1000.)
    > # draw coastlines, country boundaries, fill continents.
    > map1.drawcoastlines(linewidth=.5)
    > map1.drawcountries()
    > map1.fillcontinents(color="#CEFFCE")
    > # draw the edge of the map projection region (the projection limb)
    > map1.drawmapboundary()
    > # compute the native map projection coordinates for cities.
    > lats, lons = ([18.728], [20.890])
    > x,y = map1(lons,lats)
    > # plot filled circles at the locations of the cities.
    > map1.plot(x,y,'bo')
    > canvas = p.get_current_fig_manager().canvas
    > agg = canvas.switch_backends \
    > (matplotlib.backends.backend_agg.FigureCanvasAgg)
    > agg.draw() # Takes ~.9 seconds
    > # get the image data
    > str_img = agg.tostring_rgb()
    > width_fig, height_fig = map(int, agg.figure.bbox.get_bounds()[2:])
    > p.cla()
    > p.close()
    > bytes = get_image_bytes(width_fig, height_fig, str_img)
    >
    > val = report_memory(i)
    > # wait a few cycles for memory usage to stabilize
    > if i==indStart: start = val
    >
    > end = val
    > print 'Average memory consumed per loop: %1.4fk bytes' % \
    > ((end-start)/float(indEnd-indStart))
    >
    > Here is the O/P
    >
    > $python memory_leak_map.py
    > rss vsz %mem
    > 0 47272 50632 9.7
    > 1 74412 77700 15.3
    > 2 93960 97380 19.3
    > 3 113308 116776 23.3
    > 4 132824 136416 27.3
    > 5 152352 155828 31.3
    > 6 171860 175216 35.3
    > 7 191372 194868 39.3
    > 8 210872 214248 43.3
    > 9 230336 233916 47.3
    > 10 249732 253284 51.3
    > 11 269252 272692 55.3
    > 12 288680 292336 59.3
    > 13 308108 311724 63.2
    > 14 305160 331112 62.6
    > 15 301096 350764 61.8
    > 16 304884 370160 62.6
    > 17 298276 389804 61.2
    > 18 305876 409184 62.8
    > 19 298316 428596 61.2
    > 20 307856 448224 63.2
    > 21 308004 467640 63.2
    > 22 308844 487016 63.4
    > 23 306260 506656 62.9
    > 24 300612 526052 61.7
    > Traceback (most recent call last):
    > File "memory_leak_map.py", line 41, in ?
    > area_thresh=1000.)
    > File
    >
    "/usr/local/lib/python2.4/site-packages/matplotlib/toolkits/basemap/basemap.py",

    > line 815, in __init__
    > self.riversegs = segments+segments2+segments3
    > MemoryError
    > Killed: 9
    >
    > Can someone please help me out solving this problem?
    >
    > Thanks
    > -Aalok
    >
    Aaolk: All your plots use the same map projection, yet you are
    initializing a new Basemap instance every time through the loop. Try
    moving the line

    map1 = Basemap(projection='cyl', lat_0=50, lon_0=-100,
    area_thresh=1000.)

    outside the loop.

    -Jeff

Aaolk: Here's what I get with matplotlib 0.87.7, basemap 0.9.4, numpy 1.0 on MacOS 10.4.9 with python 2.5 (python 2.4 gives the same result):

[mac28:~/matplotlib/toolkits/basemap] jsw% python memleak.py
rss vsz %mem
0 28248 153200 -1.3
1 34028 157316 -1.6
2 34908 158084 -1.7
3 35688 158852 -1.7
4 36200 159364 -1.7
5 36712 159876 -1.8
6 36456 159620 -1.7
7 36712 159876 -1.8
8 36712 159876 -1.8
9 36712 159876 -1.8
10 36712 159876 -1.8
11 36712 159876 -1.8
12 36456 159620 -1.7
13 36712 159876 -1.8
14 36712 159876 -1.8
15 36712 159876 -1.8
16 36712 159876 -1.8
17 36712 159876 -1.8
18 36712 159876 -1.8
19 36712 159876 -1.8
20 36712 159876 -1.8
21 36712 159876 -1.8

The %mem flag doesn't appear to work right on MacOS X, but otherwise it looks fine - no memory leak. I see you are using a pre-release version of numpy - I wonder if that could be the problem?

-Jeff

···

*/Jeff Whitaker <jswhit@...196...>/* wrote:

--
Jeffrey S. Whitaker Phone : (303)497-6313
NOAA/OAR/CDC R/PSD1 FAX : (303)497-6449
325 Broadway Boulder, CO, USA 80305-3328

Hi,
``Thanks for this direction. I am using Numpy-1.1. Do I not supposed to use this version? I will try to use older version and see what happens. I hope this will solve the problem.
thanks
-Aalok

···

``Jeff Whitaker <jswhit@…196…> wrote:

Aalok kapoor wrote:

Thanks for reply,

This was just an example for mem leak in my application. Actually i
want maps with different data to be loaded each time and the map
objects taken once changees after ploting so i have to take different
object each time. After around 20 maps generation it gives memory
error to me.

thanks,
-Aalok

/Jeff Whitaker / wrote:

Aalok kapoor wrote:

Hi all,

I am using matplotlib-0.87.7, Basemap-0.9.3, Numpy-1.1 and agg
backend.
I am facing memory leak problems after running following script -

import os, sys, time
import cStringIO
import Image
import matplotlib
matplotlib.use(‘Agg’)

import matplotlib.pylab as p
import matplotlib.numerix as nx
from matplotlib.toolkits.basemap import Basemap

def get_image_bytes(width_fig, height_fig, str_img):
“”"Returns the bytes representing the image generated

The createImage method should be called before calling this method.

:Returns:
Byte data.
“”"

Get the actual image data

f =
cStringIO.StringIO()
dims = (width_fig, height_fig)
im = Image.fromstring(“RGB”, dims, str_img)
im.save(f, “JPEG”)
return f.getvalue()

def report_memory(i):
pid = os.getpid()
a2 = os.popen(‘ps -p %d -o rss,vsz,%%mem’ % pid).readlines()
print i, ’ ', a2[1],
return int(a2[1].split()[1])

take a memory snapshot on indStart and compare it with indEnd

indStart, indEnd = 30, 150
print ’ rss vsz %mem’
for i in range(indEnd):
p.figure(figsize=(float(640)/80.0,float(480)/80.0),
facecolor=‘w’, edgecolor=‘w’)
p.axes([0.0, 0.0, 1.0, 1.0])
map1 = Basemap(projection=‘cyl’, lat_0=50, lon_0=-100,

area_thresh=1000.)

draw coastlines, country boundaries, fill continents.

map1.drawcoastlines(linewidth=.5)
map1.drawcountries()
map1.fillcontinents(color=“#CEFFCE”)

draw the edge of the map projection region (the projection limb)

map1.drawmapboundary()

compute the native map projection coordinates for cities.

lats, lons = ([18.728], [20.890])
x,y = map1(lons,lats)

plot filled circles at the locations of the cities.

map1.plot(x,y,‘bo’)
canvas = p.get_current_fig_manager().canvas
agg = canvas.switch_backends
(matplotlib.backends.backend_agg.FigureCanvasAgg)
agg.draw() # Takes ~.9 seconds

get the image data

str_img = agg.tostring_rgb()
width_fig, height_fig =
map(int, agg.figure.bbox.get_bounds()[2:])
p.cla()
p.close()
bytes = get_image_bytes(width_fig, height_fig, str_img)

val = report_memory(i)

wait a few cycles for memory usage to stabilize

if i==indStart: start = val

end = val
print ‘Average memory consumed per loop: %1.4fk bytes’ %
((end-start)/float(indEnd-indStart))

Here is the O/P

$python memory_leak_map.py
rss vsz %mem
0 47272 50632 9.7
1 74412 77700 15.3
2 93960 97380 19.3
3 113308 116776 23.3
4 132824 136416 27.3
5 152352 155828 31.3
6 171860 175216 35.3
7 191372 194868 39.3
8 210872 214248 43.3
9 230336 233916 47.3
10 249732 253284 51.3
11 269252 272692 55.3
12 288680 292336 59.3
13 308108 311724 63.2
14 305160 331112 62.6
15 301096 350764 61.8
16 304884 370160 62.6
17 298276 389804 61.2
18 305876 409184 62.8
19 298316 428596 61.2
20 307856 448224 63.2
21 308004 467640 63.2
22 308844 487016 63.4
23 306260 506656 62.9
24 300612 526052 61.7
Traceback (most recent call last):
File “memory_leak_map.py”, line 41, in ?
area_thresh=1000.)
File

“/usr/local/lib/python2.4/site-packages/matplotlib/toolkits/basemap/basemap.py”,

line 815, in init
self.riversegs = segments+segments2+segments3
MemoryError
Killed: 9

Can someone please help me out solving this problem?

Thanks
-Aalok

Aaolk: All your plots use the same map projection, yet you are
initializing a new Basemap instance every time through the loop. Try
moving the line

map1 = Basemap(projection=‘cyl’, lat_0=50, lon_0=-100,
area_thresh=1000.)

outside the loop.

-Jeff

Aaolk: Here’s what I get with matplotlib 0.87.7, basemap 0.9.4, numpy
1.0 on MacOS 10.4.9 with python 2.5 (python 2.4 gives the same result):

[mac28:~/matplotlib/toolkits/basemap] jsw% python memleak.py
rss vsz %mem
0 28248 153200 -1.3
1 34028 157316 -1.6
2 34908 158084 -1.7
3 35688 158852 -1.7
4 36200 159364 -1.7
5 36712 159876 -1.8
6 36456 159620 -1.7
7 36712 159876 -1.8
8 36712 159876 -1.8
9 36712 159876 -1.8
10 36712 159876 -1.8
11 36712 159876 -1.8
12 36456 159620 -1.7
13 36712 159876 -1.8
14 36712 159876 -1.8
15 36712 159876 -1.8
16 36712 159876 -1.8
17 36712 159876 -1.8
18 36712 159876 -1.8
19 36712 159876 -1.8
20 36712 159876 -1.8
21 36712 159876 -1.8

The %mem flag doesn’t appear to work right on MacOS X, but otherwise it
looks fine - no memory leak. I see you are using a pre-release version
of numpy - I wonder if that could be the problem?

-Jeff


Jeffrey S. Whitaker Phone : (303)497-6313
NOAA/OAR/CDC R/PSD1 FAX : (303)497-6449
325 Broadway Boulder, CO,
USA 80305-3328


Find out what India is talking about on - Yahoo! Answers India

Send FREE SMS to your friend’s mobile from Yahoo! Messenger Version 8. Get it NOW

After using numpy-1.0 the results are more bad. After 15 maps it is reaching to 71% memory usage. I am working on freeBSD box.
python memory_leak_map.py
rss vsz %mem
0 47940 50636 9.8
1 75080 77700 15.4
2 94636 97380 19.4
3 114156 116776 23.4
4 133672 136416 27.4
5 153204 155832 31.4
6 172708 175216 35.5
7 192224 194872 39.5
8 211720 214248 43.5
9 231244 233916 47.5
10 250732 253280 51.5
11 270252 272688 55.5
12 289756 292332 59.5
13 309272 311728 63.5
14 328764 331108 67.5
15 347804 350760 71.4
thanks,
-Aalok
*** Aalok kapoor

···

<aalok_kapoor1984@…477…>*** wrote:

Hi,
Thanks for this direction. I am using Numpy-1.1. Do I not supposed to use this version? I will try to use older version and see what happens. I hope this will solve the problem. thanks -Aalok Jeff Whitaker <jswhit@…196…> wrote:

Aalok kapoor wrote:

Thanks for reply,

This was just an example for mem leak in my application. Actually i
want maps with different data to be loaded each time and the map
objects taken once changees after ploting so i have to take different
object each time. After around 20 maps generation it gives memory
error to me.

thanks,

-Aalok

/Jeff Whitaker / wrote:

Aalok kapoor wrote:

Hi all,

I am using matplotlib-0.87.7, Basemap-0.9.3, Numpy-1.1 and agg
backend.
I am facing memory leak problems after running following script -

import os, sys, time
import cStringIO
import Image
import matplotlib
matplotlib.use(‘Agg’)

import matplotlib.pylab as p
import matplotlib.numerix as nx
from matplotlib.toolkits.basemap import Basemap

def get_image_bytes(width_fig, height_fig, str_img):
“”"Returns the bytes representing the image generated

The createImage method should be called before calling
this method.

:Returns:
Byte data.
“”"

Get the actual image data

f = cStringIO.StringIO()
dims = (width_fig, height_fig)
im = Image.fromstring(“RGB”, dims, str_img)
im.save(f, “JPEG”)
return f.getvalue()

def report_memory(i):
pid = os.getpid()
a2 = os.popen(‘ps -p %d -o rss,vsz,%%mem’ % pid).readlines()
print i, ’ ', a2[1],
return int(a2[1].split()[1])

take a memory snapshot on indStart and compare it with indEnd

indStart, indEnd = 30, 150
print ’ rss vsz %mem’
for i in range(indEnd):
p.figure(figsize=(float(640)/80.0,float(480)/80.0),
facecolor=‘w’,
edgecolor=‘w’)
p.axes([0.0, 0.0, 1.0, 1.0])
map1 = Basemap(projection=‘cyl’, lat_0=50, lon_0=-100,
area_thresh=1000.)

draw coastlines, country boundaries, fill continents.

map1.drawcoastlines(linewidth=.5)
map1.drawcountries()
map1.fillcontinents(color=“#CEFFCE”)

draw the edge of the map projection region (the projection limb)

map1.drawmapboundary()

compute the native map projection coordinates for cities.

lats, lons = ([18.728], [20.890])
x,y = map1(lons,lats)

plot filled circles at the locations of the cities.

map1.plot(x,y,‘bo’)
canvas = p.get_current_fig_manager().canvas
agg = canvas.switch_backends
(matplotlib.backends.backend_agg.FigureCanvasAgg)

agg.draw() # Takes ~.9 seconds

get the image data

str_img = agg.tostring_rgb()
width_fig, height_fig = map(int, agg.figure.bbox.get_bounds()[2:])
p.cla()
p.close()
bytes = get_image_bytes(width_fig, height_fig, str_img)

val = report_memory(i)

wait a few cycles for memory usage to stabilize

if i==indStart: start = val

end = val
print ‘Average memory consumed per loop: %1.4fk bytes’ %
((end-start)/float(indEnd-indStart))

Here is the O/P

$python memory_leak_map.py
rss vsz %mem
0 47272 50632 9.7
1 74412 77700 15.3
2 93960 97380
19.3
3 113308 116776 23.3
4 132824 136416 27.3
5 152352 155828 31.3
6 171860 175216 35.3
7 191372 194868 39.3
8 210872 214248 43.3
9 230336 233916 47.3
10 249732 253284 51.3
11 269252 272692 55.3
12 288680 292336 59.3
13 308108 311724 63.2
14 305160 331112 62.6
15 301096 350764 61.8
16 304884 370160 62.6
17 298276 389804 61.2
18 305876 409184 62.8
19 298316 428596 61.2
20 307856 448224 63.2
21 308004 467640 63.2
22 308844 487016 63.4
23 306260 506656 62.9
24 300612 526052 61.7
Traceback (most recent call last):
File “memory_leak_map.py”, line 41, in ?

area_thresh=1000.)

File

“/usr/local/lib/python2.4/site-packages/matplotlib/toolkits/basemap/basemap.py”,

line 815, in init
self.riversegs = segments+segments2+segments3
MemoryError
Killed: 9

Can someone please help me out solving this problem?

Thanks
-Aalok

Aaolk: All your plots use the same map projection, yet you are
initializing a new Basemap instance every time through the loop. Try
moving the line

map1 = Basemap(projection=‘cyl’, lat_0=50, lon_0=-100,
area_thresh=1000.)

outside the loop.

-Jeff

Aaolk: Here’s what I get with matplotlib 0.87.7, basemap 0.9.4, numpy
1.0 on MacOS 10.4.9
with python 2.5 (python 2.4 gives the same result):

[mac28:~/matplotlib/toolkits/basemap] jsw% python memleak.py
rss vsz %mem
0 28248 153200 -1.3
1 34028 157316 -1.6
2 34908 158084 -1.7
3 35688 158852 -1.7
4 36200 159364 -1.7
5 36712 159876 -1.8
6 36456 159620 -1.7
7 36712 159876 -1.8
8 36712 159876 -1.8
9 36712 159876 -1.8
10 36712 159876 -1.8
11 36712 159876 -1.8
12 36456 159620 -1.7
13 36712 159876 -1.8
14 36712 159876 -1.8
15 36712 159876 -1.8
16 36712 159876 -1.8
17 36712 159876 -1.8
18 36712 159876 -1.8
19 36712 159876 -1.8
20 36712 159876 -1.8
21 36712 159876 -1.8

The %mem flag doesn’t appear to work right on MacOS X, but otherwise it
looks fine - no memory leak. I see you are using a pre-release version
of numpy - I wonder if that could be the
problem?

-Jeff


Jeffrey S. Whitaker Phone : (303)497-6313
NOAA/OAR/CDC R/PSD1 FAX : (303)497-6449
325 Broadway Boulder, CO, USA 80305-3328


Find out what India is talking about on - Yahoo! Answers India
Send FREE SMS to your friend’s mobile from Yahoo! Messenger Version 8. Get it NOW-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net’s Techsay panel and you’ll get the chance to share your
opinions on IT & business topics through brief surveys - and earn
cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
matplotlib-devel List Signup and Options


Find out what India is talking about on - Yahoo! Answers India

Send FREE SMS to your friend’s mobile from Yahoo! Messenger Version 8. Get it NOW

Jeff and Aalok,

I am in central MO and it doesn’t bother me but, for some unknown reason I’m recieving correspondense between you two. You might want to start a new email instead of replying to each other. Have a nice day. Curt Bridges

···

Aalok kapoor <aalok_kapoor1984@…477…> wrote:

After using numpy-1.0 the results are more bad. After 15 maps it is reaching to 71% memory usage. I am working on freeBSD box.
python memory_leak_map.py
rss vsz %mem
0 47940 50636 9.8
1 75080 77700 15.4
2 94636 97380 19.4
3 114156 116776 23.4
4 133672 136416 27.4
5 153204 155832 31.4
6 172708 175216
35.5
7 192224 194872 39.5
8 211720 214248 43.5
9 231244 233916 47.5
10 250732 253280 51.5
11 270252 272688 55.5
12 289756 292332 59.5
13 309272 311728 63.5
14 328764 331108 67.5
15 347804 350760 71.4
thanks,
-Aalok
Aalok kapoor <aalok_kapoor1984@…477…> wrote:

Hi,
Thanks for this direction. I am using Numpy-1.1. Do I not supposed to use this version? I will try to use older version and see what happens. I hope this will solve the problem. thanks -Aalok Jeff Whitaker <jswhit@…196…> wrote:

Aalok kapoor wrote:

Thanks for reply,

This was just an example for mem leak in my application. Actually i
want maps with different data to be loaded each time and the map
objects taken once changees after ploting so i have to take different
object each time. After around 20 maps generation it gives memory
error to me.

thanks,
-Aalok

/Jeff Whitaker / wrote:

Aalok kapoor wrote:

Hi all,

I am using matplotlib-0.87.7, Basemap-0.9.3, Numpy-1.1 and agg
backend.
I am facing memory leak problems after running following script -

import os, sys, time
import cStringIO
import Image
import matplotlib
matplotlib.use(‘Agg’)

import matplotlib.pylab as
p
import matplotlib.numerix as nx
from matplotlib.toolkits.basemap import Basemap

def get_image_bytes(width_fig, height_fig, str_img):
“”"Returns the bytes representing the image generated

The createImage method should be called before calling this method.

:Returns:
Byte data.
“”"

Get the actual image data

f = cStringIO.StringIO()
dims = (width_fig, height_fig)
im = Image.fromstring(“RGB”, dims, str_img)
im.save(f, “JPEG”)
return f.getvalue()

def report_memory(i):
pid = os.getpid()
a2 = os.popen(‘ps -p %d -o rss,vsz,%%mem’ % pid).readlines()
print i, ’ ', a2[1],
return int(a2[1].split()[1])

take a memory snapshot on indStart

and compare it with indEnd

indStart, indEnd = 30, 150
print ’ rss vsz %mem’
for i in range(indEnd):
p.figure(figsize=(float(640)/80.0,float(480)/80.0),
facecolor=‘w’, edgecolor=‘w’)
p.axes([0.0, 0.0, 1.0, 1.0])
map1 = Basemap(projection=‘cyl’, lat_0=50, lon_0=-100,
area_thresh=1000.)

draw coastlines, country boundaries, fill continents.

map1.drawcoastlines(linewidth=.5)
map1.drawcountries()
map1.fillcontinents(color=“#CEFFCE”)

draw the edge of the map projection region (the projection limb)

map1.drawmapboundary()

compute the native map projection coordinates for cities.

lats, lons = ([18.728], [20.890])
x,y = map1(lons,lats)

plot filled circles at the locations of the cities.

map1.plot(x,y,‘bo’)
canvas =
p.get_current_fig_manager().canvas
agg = canvas.switch_backends
(matplotlib.backends.backend_agg.FigureCanvasAgg)
agg.draw() # Takes ~.9 seconds

get the image data

str_img = agg.tostring_rgb()
width_fig, height_fig = map(int, agg.figure.bbox.get_bounds()[2:])
p.cla()
p.close()
bytes = get_image_bytes(width_fig, height_fig, str_img)

val = report_memory(i)

wait a few cycles for memory usage to stabilize

if i==indStart: start = val

end = val
print ‘Average memory consumed per loop: %1.4fk bytes’ %
((end-start)/float(indEnd-indStart))

Here is the O/P

$python memory_leak_map.py
rss vsz %mem
0 47272 50632 9.7
1
74412 77700 15.3
2 93960 97380 19.3
3 113308 116776 23.3
4 132824 136416 27.3
5 152352 155828 31.3
6 171860 175216 35.3
7 191372 194868 39.3
8 210872 214248 43.3
9 230336 233916 47.3
10 249732 253284 51.3
11 269252 272692 55.3
12 288680 292336 59.3
13 308108 311724 63.2
14 305160 331112 62.6
15 301096 350764 61.8
16 304884 370160 62.6
17 298276 389804 61.2
18 305876 409184 62.8
19 298316 428596 61.2
20 307856 448224 63.2
21 308004 467640 63.2
22 308844 487016 63.4
23 306260 506656 62.9
24 300612 526052 61.7
Traceback (most recent call last):
File “memory_leak_map.py”, line 41, in ?
area_thresh=1000.)
File

“/usr/local/lib/python2.4/site-packages/matplotlib/toolkits/basemap/basemap.py”,

line 815, in init
self.riversegs = segments+segments2+segments3
MemoryError
Killed: 9

Can someone please help me out solving this problem?

Thanks
-Aalok

Aaolk: All your plots use the same map projection, yet you are
initializing a new Basemap instance every time through the loop. Try
moving the line

map1 = Basemap(projection=‘cyl’, lat_0=50, lon_0=-100,
area_thresh=1000.)

outside the loop.

-Jeff

Aaolk: Here’s what I get with matplotlib 0.87.7, basemap 0.9.4, numpy
1.0 on MacOS 10.4.9 with python 2.5 (python 2.4 gives the same result):

[mac28:~/matplotlib/toolkits/basemap] jsw% python memleak.py
rss vsz %mem
0 28248
153200 -1.3
1 34028 157316 -1.6
2 34908 158084 -1.7
3 35688 158852 -1.7
4 36200 159364 -1.7
5 36712 159876 -1.8
6 36456 159620 -1.7
7 36712 159876 -1.8
8 36712 159876 -1.8
9 36712 159876 -1.8
10 36712 159876 -1.8
11 36712 159876 -1.8
12 36456 159620 -1.7
13 36712 159876 -1.8
14 36712 159876 -1.8
15 36712 159876 -1.8
16 36712 159876 -1.8
17 36712 159876 -1.8
18 36712 159876 -1.8
19 36712 159876 -1.8
20 36712 159876 -1.8
21 36712 159876 -1.8

The %mem flag doesn’t appear to work right on MacOS X, but otherwise it
looks fine - no memory leak. I see you are using a pre-release version
of numpy - I wonder if that could be the problem?

-Jeff


Jeffrey S. Whitaker Phone : (303)497-6313
NOAA/OAR/CDC R/PSD1 FAX : (303)497-6449
325 Broadway Boulder, CO, USA 80305-3328


Find out what India is talking
about on - Yahoo! Answers India
Send FREE SMS to your friend’s mobile from Yahoo! Messenger Version 8. Get it NOW-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net’s Techsay panel and you’ll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
matplotlib-devel List Signup and Options


Find out what India is talking about on - Yahoo! Answers India
Send FREE SMS to your friend’s mobile from Yahoo! Messenger Version 8. Get it NOW-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net’s Techsay panel and you’ll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
matplotlib-devel List Signup and Options


Want to start your own business? Learn how on Yahoo! Small Business.

Aalok kapoor wrote:

After using numpy-1.0 the results are more bad. After 15 maps it is reaching to 71% memory usage. I am working on freeBSD box.

python memory_leak_map.py
     rss vsz %mem
0 47940 50636 9.8
1 75080 77700 15.4
2 94636 97380 19.4
3 114156 116776 23.4
4 133672 136416 27.4
5 153204 155832 31.4
6 172708 175216 35.5
7 192224 194872 39.5
8 211720 214248 43.5
9 231244 233916 47.5
10 250732 253280 51.5
11 270252 272688 55.5
12 289756 292332 59.5
13 309272 311728 63.5
14 328764 331108 67.5
15 347804 350760 71.4

thanks,
-Aalok

    Hi,

    Thanks for this direction. I am using Numpy-1.1. Do I not supposed
    to use this version? I will try to use older version and see what
    happens. I hope this will solve the problem.

    thanks
    -Aalok

        Aalok kapoor wrote:
        > Thanks for reply,
        >
        > This was just an example for mem leak in my application.
        Actually i
        > want maps with different data to be loaded each time and the
        map
        > objects taken once changees after ploting so i have to take
        different
        > object each time. After around 20 maps generation it gives
        memory
        > error to me.
        >
        > thanks,
        > -Aalok
        >
        > */Jeff Whitaker /* wrote:
        >
        > Aalok kapoor wrote:
        > > Hi all,
        > >
        > > I am using matplotlib-0.87.7, Basemap-0.9.3, Numpy-1.1 and agg
        > backend.
        > > I am facing memory leak problems after running following
        script -
        > >
        > > import os, sys, time
        > > import cStringIO
        > > import Image
        > > import matplotlib
        > > matplotlib.use('Agg')
        > >
        > > import matplotlib.pylab as p
        > > import matplotlib.numerix as nx
        > > from matplotlib.toolkits.basemap import Basemap
        > >
        > > def get_image_bytes(width_fig, height_fig, str_img):
        > > """Returns the bytes representing the image generated
        > >
        > > The createImage method should be called before calling
        this method.
        > >
        > > :Returns:
        > > Byte data.
        > > """
        > > # Get the actual image data
        > > f = cStringIO.StringIO()
        > > dims = (width_fig, height_fig)
        > > im = Image.fromstring("RGB", dims, str_img)
        > > im.save(f, "JPEG")
        > > return f.getvalue()
        > >
        > > def report_memory(i):
        > > pid = os.getpid()
        > > a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
        > > print i, ' ', a2[1],
        > > return int(a2[1].split()[1])
        > >
        > > # take a memory snapshot on indStart and compare it with
        indEnd
        > > indStart, indEnd = 30, 150
        > > print ' rss vsz %mem'
        > > for i in range(indEnd):
        > > p.figure(figsize=(float(640)/80.0,float(480)/80.0),
        > > facecolor='w', edgecolor='w')
        > > p.axes([0.0, 0.0, 1.0, 1.0])
        > > map1 = Basemap(projection='cyl', lat_0=50, lon_0=-100,
        > > area_thresh=1000.)
        > > # draw coastlines, country boundaries, fill continents.
        > > map1.drawcoastlines(linewidth=.5)
        > > map1.drawcountries()
        > > map1.fillcontinents(color="#CEFFCE")
        > > # draw the edge of the map projection region (the
        projection limb)
        > > map1.drawmapboundary()
        > > # compute the native map projection coordinates for cities.
        > > lats, lons = ([18.728], [20.890])
        > > x,y = map1(lons,lats)
        > > # plot filled circles at the locations of the cities.
        > > map1.plot(x,y,'bo')
        > > canvas = p.get_current_fig_manager().canvas
        > > agg = canvas.switch_backends \
        > > (matplotlib.backends.backend_agg.FigureCanvasAgg)
        > > agg.draw() # Takes ~.9 seconds
        > > # get the image data
        > > str_img = agg.tostring_rgb()
        > > width_fig, height_fig = map(int,
        agg.figure.bbox.get_bounds()[2:])
        > > p.cla()
        > > p.close()
        > > bytes = get_image_bytes(width_fig, height_fig, str_img)
        > >
        > > val = report_memory(i)
        > > # wait a few cycles for memory usage to stabilize
        > > if i==indStart: start = val
        > >
        > > end = val
        > > print 'Average memory consumed per loop: %1.4fk bytes' % \
        > > ((end-start)/float(indEnd-indStart))
        > >
        > > Here is the O/P
        > >
        > > $python memory_leak_map.py
        > > rss vsz %mem
        > > 0 47272 50632 9.7
        > > 1 74412 77700 15.3
        > > 2 93960 97380 19.3
        > > 3 113308 116776 23.3
        > > 4 132824 136416 27.3
        > > 5 152352 155828 31.3
        > > 6 171860 175216 35.3
        > > 7 191372 194868 39.3
        > > 8 210872 214248 43.3
        > > 9 230336 233916 47.3
        > > 10 249732 253284 51.3
        > > 11 269252 272692 55.3
        > > 12 288680 292336 59.3
        > > 13 308108 311724 63.2
        > > 14 305160 331112 62.6
        > > 15 301096 350764 61.8
        > > 16 304884 370160 62.6
        > > 17 298276 389804 61.2
        > > 18 305876 409184 62.8
        > > 19 298316 428596 61.2
        > > 20 307856 448224 63.2
        > > 21 308004 467640 63.2
        > > 22 308844 487016 63.4
        > > 23 306260 506656 62.9
        > > 24 300612 526052 61.7
        > > Traceback (most recent call last):
        > > File "memory_leak_map.py", line 41, in ?
        > > area_thresh=1000.)
        > > File
        > >
        >
        "/usr/local/lib/python2.4/site-packages/matplotlib/toolkits/basemap/basemap.py",
        >
        > > line 815, in __init__
        > > self.riversegs = segments+segments2+segments3
        > > MemoryError
        > > Killed: 9
        > >
        > > Can someone please help me out solving this problem?
        > >
        > > Thanks
        > > -Aalok
        > >
        > Aaolk: All your plots use the same map projection, yet you are
        > initializing a new Basemap instance every time through the
        loop. Try
        > moving the line
        >
        > map1 = Basemap(projection='cyl', lat_0=50, lon_0=-100,
        > area_thresh=1000.)
        >
        > outside the loop.
        >
        > -Jeff
        >

        Aaolk: Here's what I get with matplotlib 0.87.7, basemap
        0.9.4, numpy
        1.0 on MacOS 10.4.9 with python 2.5 (python 2.4 gives the same
        result):

        [mac28:~/matplotlib/toolkits/basemap] jsw% python memleak.py
        rss vsz %mem
        0 28248 153200 -1.3
        1 34028 157316 -1.6
        2 34908 158084 -1.7
        3 35688 158852 -1.7
        4 36200 159364 -1.7
        5 36712 159876 -1.8
        6 36456 159620 -1.7
        7 36712 159876 -1.8
        8 36712 159876 -1.8
        9 36712 159876 -1.8
        10 36712 159876 -1.8
        11 36712 159876 -1.8
        12 36456 159620 -1.7
        13 36712 159876 -1.8
        14 36712 159876 -1.8
        15 36712 159876 -1.8
        16 36712 159876 -1.8
        17 36712 159876 -1.8
        18 36712 159876 -1.8
        19 36712 159876 -1.8
        20 36712 159876 -1.8
        21 36712 159876 -1.8

        The %mem flag doesn't appear to work right on MacOS X, but
        otherwise it
        looks fine - no memory leak. I see you are using a pre-release
        version
        of numpy - I wonder if that could be the problem?

        -Jeff

        -- Jeffrey S. Whitaker Phone : (303)497-6313
        NOAA/OAR/CDC R/PSD1 FAX : (303)497-6449
        325 Broadway Boulder, CO, USA 80305-3328

Aaolk: Since I can't reproduce your problem, I'm afraid I can't give you any concrete advice. It could be Basemap, matplotlib, numpy, PIL or even python itself. To narrow down the possibilities, you can try moving trimming your loop line by line and see if you can make it go away. The python garbage collector interface (http://www.python.org/doc/2.4/lib/module-gc.html) might be of some help also.

-Jeff

···

*/Aalok kapoor <aalok_kapoor1984@...477...>/* wrote:
    */Jeff Whitaker <jswhit@...196...>/* wrote:

--
Jeffrey S. Whitaker Phone : (303)497-6313
Meteorologist FAX : (303)497-6449
NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@...236...
325 Broadway Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web : Jeffrey S. Whitaker: NOAA Physical Sciences Laboratory