Plotting large images

I am trying to use imshow to plot some semi-large fits images.
Here is the code:

from math import *
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import cm as cm
import pyfits

cat = "/Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/"
fname = "o4985g0263o.warp.MD09.skycell.092"

hdu = pyfits.open(cat+fname+".fits")
print hdu.info()
img = hdu[1].data.astype("int")

plt.figure(figsize=[12,12])
plt.imshow(img,cmap=cm.cool)
plt.savefig("test.png")

Which gives the result:

Filename: /Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/o4985g0263o.warp.MD09.skycell.092.fits
No. Name Type Cards Dimensions Format
0 PRIMARY PrimaryHDU 6 () int16
1 CompImageHDU 101 (6000, 6000) float32
None
Python(23117,0xa04f2720) malloc: *** mmap(size=1152000000) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
   File "quick_look.py", line 16, in <module>
     plt.savefig("test.png")
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/pyplot.py", line 345, in savefig
     return fig.savefig(*args, **kwargs)
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/figure.py", line 990, in savefig
     self.canvas.print_figure(*args, **kwargs)
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/backend_bases.py", line 1419, in print_figure
     **kwargs)
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/backends/backend_agg.py", line 323, in print_png
     FigureCanvasAgg.draw(self)
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/backends/backend_agg.py", line 279, in draw
     self.figure.draw(self.renderer)
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/figure.py", line 772, in draw
     for a in self.axes: a.draw(renderer)
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/axes.py", line 1545, in draw
     im.draw(renderer)
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/image.py", line 233, in draw
     im = self.make_image(renderer.get_image_magnification())
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/image.py", line 180, in make_image
     x = self.to_rgba(self._A, self._alpha)
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/cm.py", line 79, in to_rgba
     x = self.cmap(x, alpha=alpha, bytes=bytes)
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/colors.py", line 501, in __call__
     rgba = np.empty(shape=xa.shape+(4,), dtype=lut.dtype)
MemoryError

I found the earlier thread of http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg11216.html
but that didn't seem to produce any fixes or good explanations.

I am on a Mac Pro Intel machine running 10.5.7 and
[Heimdall:tgrav ~/Work/myCode/Python/pyPS1SC]> python
ActivePython 2.5.4.3 (ActiveState Software Inc.) based on
Python 2.5.4 (r254:67916, Jan 20 2009, 14:11:42)
[GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.3.0rc2'
>>> import pyfits
>>> pyfits.__version__
'2.1.1dev462'
>>> import matplotlib
>>> matplotlib.__version__
'0.98.5.2'
>>>

Cheers
Tommy

Because the image is so large, and matplotlib carries out various operations on the image (scaling, resampling, etc), it uses a lot of memory. This is one area where a specialized display program will be more efficient. If you need to use matplotlib, decide whether you really only want to display a subsection, or only need a lower resolution version (e.g., boxcar smooth and subsample) before displaying. I've found that image sizes well over 1kx1k can take some time to display, and those that are much larger can cause you to run out of memory. At least, that's what I think is happening.

Perry

···

On Jun 30, 2009, at 7:20 PM, Tommy Grav wrote:

I am trying to use imshow to plot some semi-large fits images.
Here is the code:

from math import *
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import cm as cm
import pyfits

cat = "/Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/"
fname = "o4985g0263o.warp.MD09.skycell.092"

hdu = pyfits.open(cat+fname+".fits")
print hdu.info()
img = hdu[1].data.astype("int")

plt.figure(figsize=[12,12])
plt.imshow(img,cmap=cm.cool)
plt.savefig("test.png")

Which gives the result:

Filename: /Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/
o4985g0263o.warp.MD09.skycell.092.fits
No. Name Type Cards Dimensions Format
0 PRIMARY PrimaryHDU 6 () int16
1 CompImageHDU 101 (6000, 6000) float32
None
Python(23117,0xa04f2720) malloc: *** mmap(size=1152000000) failed
(error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
  File "quick_look.py", line 16, in <module>
    plt.savefig("test.png")
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/pyplot.py", line 345, in savefig
    return fig.savefig(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/figure.py", line 990, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/backend_bases.py", line 1419, in
print_figure
    **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/backends/backend_agg.py", line 323,
in print_png
    FigureCanvasAgg.draw(self)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/backends/backend_agg.py", line 279,
in draw
    self.figure.draw(self.renderer)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/figure.py", line 772, in draw
    for a in self.axes: a.draw(renderer)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/axes.py", line 1545, in draw
    im.draw(renderer)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/image.py", line 233, in draw
    im = self.make_image(renderer.get_image_magnification())
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/image.py", line 180, in make_image
    x = self.to_rgba(self._A, self._alpha)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/cm.py", line 79, in to_rgba
    x = self.cmap(x, alpha=alpha, bytes=bytes)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/colors.py", line 501, in __call__
    rgba = np.empty(shape=xa.shape+(4,), dtype=lut.dtype)
MemoryError

I found the earlier thread of http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg11216.html
but that didn't seem to produce any fixes or good explanations.

I am on a Mac Pro Intel machine running 10.5.7 and
[Heimdall:tgrav ~/Work/myCode/Python/pyPS1SC]> python
ActivePython 2.5.4.3 (ActiveState Software Inc.) based on
Python 2.5.4 (r254:67916, Jan 20 2009, 14:11:42)
[GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import numpy
numpy.__version__

'1.3.0rc2'

import pyfits
pyfits.__version__

'2.1.1dev462'

import matplotlib
matplotlib.__version__

'0.98.5.2'

Cheers
Tommy

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users

That is what I was assuming, but it still seems a little odd that matplotlib generates
that large of a memory footprint. Loading the fits file into the program using pyfits,
with the code only uses 19MB of real memory and 600MB of virtual memory (strangly
adding the line img = hdu[1].data, increases this to 208MB/800MB).

Displaying images of various sizes I get these
numbers from Activity Monitor

Size Real Mem Virtual
3k x 3k 0.68GB 1.57GB
4k x 4k 0.92GB 1.80GB
5k x 5k 1.20GB 2.10GB
5.5k x 5.5k 1.38GB 2.28GB

And the limit seems to be somewhere just above 5.5k by 5.5k (darn :frowning: )

Cheers
   Tommy

···

On Jun 30, 2009, at 7:27 PM, Perry Greenfield wrote:

Because the image is so large, and matplotlib carries out various operations on the image (scaling, resampling, etc), it uses a lot of memory. This is one area where a specialized display program will be more efficient. If you need to use matplotlib, decide whether you really only want to display a subsection, or only need a lower resolution version (e.g., boxcar smooth and subsample) before displaying. I've found that image sizes well over 1kx1k can take some time to display, and those that are much larger can cause you to run out of memory. At least, that's what I think is happening.

Perry

On Jun 30, 2009, at 7:20 PM, Tommy Grav wrote:

I am trying to use imshow to plot some semi-large fits images.
Here is the code:

from math import *
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import cm as cm
import pyfits

cat = "/Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/"
fname = "o4985g0263o.warp.MD09.skycell.092"

hdu = pyfits.open(cat+fname+".fits")
print hdu.info()
img = hdu[1].data.astype("int")

plt.figure(figsize=[12,12])
plt.imshow(img,cmap=cm.cool)
plt.savefig("test.png")

Which gives the result:

Filename: /Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/
o4985g0263o.warp.MD09.skycell.092.fits
No. Name Type Cards Dimensions Format
0 PRIMARY PrimaryHDU 6 () int16
1 CompImageHDU 101 (6000, 6000) float32
None
Python(23117,0xa04f2720) malloc: *** mmap(size=1152000000) failed
(error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "quick_look.py", line 16, in <module>
   plt.savefig("test.png")
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/pyplot.py", line 345, in savefig
   return fig.savefig(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/figure.py", line 990, in savefig
   self.canvas.print_figure(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/backend_bases.py", line 1419, in
print_figure
   **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/backends/backend_agg.py", line 323,
in print_png
   FigureCanvasAgg.draw(self)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/backends/backend_agg.py", line 279,
in draw
   self.figure.draw(self.renderer)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/figure.py", line 772, in draw
   for a in self.axes: a.draw(renderer)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/axes.py", line 1545, in draw
   im.draw(renderer)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/image.py", line 233, in draw
   im = self.make_image(renderer.get_image_magnification())
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/image.py", line 180, in make_image
   x = self.to_rgba(self._A, self._alpha)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/cm.py", line 79, in to_rgba
   x = self.cmap(x, alpha=alpha, bytes=bytes)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/site-packages/matplotlib/colors.py", line 501, in __call__
   rgba = np.empty(shape=xa.shape+(4,), dtype=lut.dtype)
MemoryError

I found the earlier thread of http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg11216.html
but that didn't seem to produce any fixes or good explanations.

I am on a Mac Pro Intel machine running 10.5.7 and
[Heimdall:tgrav ~/Work/myCode/Python/pyPS1SC]> python
ActivePython 2.5.4.3 (ActiveState Software Inc.) based on
Python 2.5.4 (r254:67916, Jan 20 2009, 14:11:42)
[GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import numpy
numpy.__version__

'1.3.0rc2'

import pyfits
pyfits.__version__

'2.1.1dev462'

import matplotlib
matplotlib.__version__

'0.98.5.2'

Cheers
Tommy

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users

That is what I was assuming, but it still seems a little odd that
matplotlib generates
that large of a memory footprint. Loading the fits file into the
program using pyfits,
with the code only uses 19MB of real memory and 600MB of virtual
memory (strangly
adding the line img = hdu[1].data, increases this to 208MB/800MB).

The reason for this is that pyfits doesn't actually load the data until you 'touch' the data attribute (to minimize memory, particularly if you just are interested in the header information).

As for the memory footprint of matplotlib, in order to be able to resize and handle interactive updates, it has to retain references to the original image, perhaps as well to intermediate products (and these references won't be memory collected until you clear the figure (e.g., clf()). It's one of the prices for flexibility and generality. It probably would take a lot of complexity to optimize it for large images (but John is better suited to answer this conclusively).

Perry

···

On Jun 30, 2009, at 7:54 PM, Tommy Grav wrote:

Displaying images of various sizes I get these
numbers from Activity Monitor

Size Real Mem Virtual
3k x 3k 0.68GB 1.57GB
4k x 4k 0.92GB 1.80GB
5k x 5k 1.20GB 2.10GB
5.5k x 5.5k 1.38GB 2.28GB

And the limit seems to be somewhere just above 5.5k by 5.5k (darn :frowning: )

Cheers
Tommy