Memory issues with imshow

Well, I am trying to create an overlay, *one* picture showing all 34
images. So I am only trying to create a single figure.

I just attached an example so you can get an idea (it was downsampled
for mailing, the original picture has ca. 5500 x 6500 pixels). In the
end, I just want to save the image to the disk, so I am using 'Agg' as
the backend - hope this also saves me some memory.

And about "old" images: I am always starting a completely new
python-process for each stitching (one at a time).

Cheers, Gerd

Perry Greenfield wrote:

···

Are you clearing the figure after each image display? The figure retains references to the image if you don't do a clf() and thus you will eventually run out of memory, even if you delete the images (they don't go away while matplotlib is using them).

Perry

On Dec 15, 2009, at 10:32 AM, Wellenreuther, Gerd wrote:

Dear all,

I am trying to write a script to be used with our microscope, stitching
images of various magnifications together to yield a big picture of a
sample. The preprocessing involves operations like rotating the picture
etc., and finally those pictures are being plotted using imshow.

Unfortunately, I am running into memory problems, e.g.:

C:\Python26\lib\site-packages\PIL\Image.py:1264: DeprecationWarning: integer argument expected, got float
im = self.im.stretch(size, resample)
Traceback (most recent call last):
File "F:\Procs\Find_dendrites.py", line 1093, in <module>
   file_type="PNG",do_stitching=do_stitching,do_dendrite_finding=do_dendrite_finding,down_sizing_factor=48,dpi=600)

File "F:\Procs\Find_dendrites.py", line 1052, in process_images
   scale,aspect_ratio,dpi,left,right,bottom,top)
File "F:\Procs\Find_dendrites.py", line 145, in stitch_images
   pylab.draw()
File "C:\Python26\lib\site-packages\matplotlib\pyplot.py", line 352, in draw
   get_current_fig_manager().canvas.draw()
File "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", line 313, in draw
   self.renderer = self.get_renderer()
File "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", line 324, in get_renderer
   self.renderer = RendererAgg(w, h, self.figure.dpi)
File "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", line 59, in __init__
   self._renderer = _RendererAgg(int(width), int(height), dpi, debug=False)
RuntimeError: Could not allocate memory for image

or

Traceback (most recent call last):
File "F:\Procs\Find_dendrites.py", line 1093, in <module>
   file_type="PNG",do_stitching=do_stitching,do_dendrite_finding=do_dendrite_finding,down_sizing_factor=48,dpi=75)

File "F:\Procs\Find_dendrites.py", line 1052, in process_images
   scale,aspect_ratio,dpi,left,right,bottom,top)
File "F:\Procs\Find_dendrites.py", line 142, in stitch_images
   pylab.imshow(rotated_images[i],aspect='auto')
File "C:\Python26\lib\site-packages\matplotlib\pyplot.py", line 2046, in imshow
   ret = ax.imshow(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs)
File "C:\Python26\lib\site-packages\matplotlib\axes.py", line 6275, in imshow
   im.set_data(X)
File "C:\Python26\lib\site-packages\matplotlib\image.py", line 291, in set_data
   self._A = pil_to_array(A)
File "C:\Python26\lib\site-packages\matplotlib\image.py", line 856, in pil_to_array
   x = toarray(im)
File "C:\Python26\lib\site-packages\matplotlib\image.py", line 831, in toarray
   x = np.fromstring(x_str,np.uint8)
MemoryError

I already implemented some downscaling of the original images (ca. 3200
x 2400 pixels), to roughly match the figures dpi-setting. But this does
not seem to be the only issue. The script does work for dpi of 600 or
150 for 11 individual images, yielding e.g. a 23 MB file with 600 dpi
and 36 Megapixels. But it fails for e.g. 35 images even for 75 dpi.

I was trying to throw away any unneccessary data using del + triggering
the garbage collection, but this did not help beyond a certain point.
Maybe somebody could tell me what kind of limitations there are using
imshow to plot a lot of images together, and how to improve?

Some more info: I am using Windows. Just by judging from the
task-manager, the preprocessing is not the problem. But *plotting* the
images using imshow seems to cause an increase of memory consumption of
the task of 32-33 MB *each* time. Somewhere around a total of 1.3 - 1.5
Gigs the process dies ...

Thanks in advance,

Gerd
--
Dr. Gerd Wellenreuther
beamline scientist P06 "Hard X-Ray Micro/Nano-Probe"
Petra III project
HASYLAB at DESY
Notkestr. 85
22603 Hamburg

Tel.: + 49 40 8998 5701

------------------------------------------------------------------------------

Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

--
Dr. Gerd Wellenreuther
beamline scientist P06 "Hard X-Ray Micro/Nano-Probe"
Petra III project
HASYLAB at DESY
Notkestr. 85
22603 Hamburg

Tel.: + 49 40 8998 5701

Hello,

switching to 64-bit Python and OS might help. I can display 8bit images up to 8459x8459 with imshow on Windows 7 64-bit with 8GB RAM. Python then uses about 5.5 GB RAM according to task manager. A 8460x8460 or larger 8bit images crash Python (definitely a bug). The 32-bit interpreter starts throwing MemoryErrors around 4350x4350 pixels. The exact limits will depend on the application, OS, and free RAM.

Christoph

···

On 12/15/2009 9:15 AM, Wellenreuther, Gerd wrote:

Well, I am trying to create an overlay, *one* picture showing all 34
images. So I am only trying to create a single figure.

I just attached an example so you can get an idea (it was downsampled
for mailing, the original picture has ca. 5500 x 6500 pixels). In the
end, I just want to save the image to the disk, so I am using 'Agg' as
the backend - hope this also saves me some memory.

And about "old" images: I am always starting a completely new
python-process for each stitching (one at a time).

Cheers, Gerd

Perry Greenfield wrote:

Are you clearing the figure after each image display? The figure
retains references to the image if you don't do a clf() and thus you
will eventually run out of memory, even if you delete the images (they
don't go away while matplotlib is using them).

Perry

On Dec 15, 2009, at 10:32 AM, Wellenreuther, Gerd wrote:

Dear all,

I am trying to write a script to be used with our microscope, stitching
images of various magnifications together to yield a big picture of a
sample. The preprocessing involves operations like rotating the picture
etc., and finally those pictures are being plotted using imshow.

Unfortunately, I am running into memory problems, e.g.:

C:\Python26\lib\site-packages\PIL\Image.py:1264: DeprecationWarning:
integer argument expected, got float
im = self.im.stretch(size, resample)
Traceback (most recent call last):
File "F:\Procs\Find_dendrites.py", line 1093, in <module>
file_type="PNG",do_stitching=do_stitching,do_dendrite_finding=do_dendrite_finding,down_sizing_factor=48,dpi=600)

File "F:\Procs\Find_dendrites.py", line 1052, in process_images
scale,aspect_ratio,dpi,left,right,bottom,top)
File "F:\Procs\Find_dendrites.py", line 145, in stitch_images
pylab.draw()
File "C:\Python26\lib\site-packages\matplotlib\pyplot.py", line 352,
in draw
get_current_fig_manager().canvas.draw()
File
"C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py",
line 313, in draw
self.renderer = self.get_renderer()
File
"C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py",
line 324, in get_renderer
self.renderer = RendererAgg(w, h, self.figure.dpi)
File
"C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py",
line 59, in __init__
self._renderer = _RendererAgg(int(width), int(height), dpi,
debug=False)
RuntimeError: Could not allocate memory for image

or

Traceback (most recent call last):
File "F:\Procs\Find_dendrites.py", line 1093, in <module>
file_type="PNG",do_stitching=do_stitching,do_dendrite_finding=do_dendrite_finding,down_sizing_factor=48,dpi=75)

File "F:\Procs\Find_dendrites.py", line 1052, in process_images
scale,aspect_ratio,dpi,left,right,bottom,top)
File "F:\Procs\Find_dendrites.py", line 142, in stitch_images
pylab.imshow(rotated_images[i],aspect='auto')
File "C:\Python26\lib\site-packages\matplotlib\pyplot.py", line
2046, in imshow
ret = ax.imshow(X, cmap, norm, aspect, interpolation, alpha, vmin,
vmax, origin, extent, shape, filternorm, filterrad, imlim, resample,
url, **kwargs)
File "C:\Python26\lib\site-packages\matplotlib\axes.py", line 6275,
in imshow
im.set_data(X)
File "C:\Python26\lib\site-packages\matplotlib\image.py", line 291,
in set_data
self._A = pil_to_array(A)
File "C:\Python26\lib\site-packages\matplotlib\image.py", line 856,
in pil_to_array
x = toarray(im)
File "C:\Python26\lib\site-packages\matplotlib\image.py", line 831,
in toarray
x = np.fromstring(x_str,np.uint8)
MemoryError

I already implemented some downscaling of the original images (ca. 3200
x 2400 pixels), to roughly match the figures dpi-setting. But this does
not seem to be the only issue. The script does work for dpi of 600 or
150 for 11 individual images, yielding e.g. a 23 MB file with 600 dpi
and 36 Megapixels. But it fails for e.g. 35 images even for 75 dpi.

I was trying to throw away any unneccessary data using del + triggering
the garbage collection, but this did not help beyond a certain point.
Maybe somebody could tell me what kind of limitations there are using
imshow to plot a lot of images together, and how to improve?

Some more info: I am using Windows. Just by judging from the
task-manager, the preprocessing is not the problem. But *plotting* the
images using imshow seems to cause an increase of memory consumption of
the task of 32-33 MB *each* time. Somewhere around a total of 1.3 - 1.5
Gigs the process dies ...

Thanks in advance,

Gerd
--
Dr. Gerd Wellenreuther
beamline scientist P06 "Hard X-Ray Micro/Nano-Probe"
Petra III project
HASYLAB at DESY
Notkestr. 85
22603 Hamburg

Tel.: + 49 40 8998 5701

------------------------------------------------------------------------------

Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev

_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options