MemoryError with NetCDF

1. an ipython session is invoked with qtconsole --pylab
2. I load a large NetCDF grid (Grid file format: nf (# 18) GMT netCDF format (float) (COARDS-compliant) [DEFAULT]), approx. 1.15 GB
3. I then try to plot with imshow the data

added below are the lines leading up to the error and the error itself.

This is running on OS X 10.7.4 with a recently installed EPD 7.3.

{code}
from scipy.io import netcdf_file as netcdf
data = netcdf('srtm_43_44_05_06_07_08.grd','r').variables['z'][::-1]

fig, ax = subplots()

data.shape
Out[5]: (24004, 12002)

im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])), interpolation='none')

···

---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
<ipython-input-6-f92e4c4c63b5> in <module>()
----> 1 im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])), interpolation='none')

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs)
   6743 im.set_clim(vmin, vmax)
   6744 else:
-> 6745 im.autoscale_None()
   6746 im.set_url(url)
   6747

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/cm.py in autoscale_None(self)
    281 if self._A is None:
    282 raise TypeError('You must first set_array for mappable')
--> 283 self.norm.autoscale_None(self._A)
    284 self.changed()
    285

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py in autoscale_None(self, A)
    889 ' autoscale only None-valued vmin or vmax'
    890 if self.vmin is None:
--> 891 self.vmin = ma.min(A)
    892 if self.vmax is None:
    893 self.vmax = ma.max(A)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in min(obj, axis, out, fill_value)
   5873 def min(obj, axis=None, out=None, fill_value=None):
   5874 try:
-> 5875 return obj.min(axis=axis, fill_value=fill_value, out=out)
   5876 except (AttributeError, TypeError):
   5877 # If obj doesn't have a max method,

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in min(self, axis, out, fill_value)
   5054 # No explicit output
   5055 if out is None:
-> 5056 result = self.filled(fill_value).min(axis=axis, out=out).view(type(self))
   5057 if result.ndim:
   5058 # Set the mask

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in filled(self, fill_value)
   3388 return self._data
   3389 else:
-> 3390 result = self._data.copy()
   3391 try:
   3392 np.putmask(result, m, fill_value)

MemoryError:
{/code}

This is more a NumPy issue than anything else. We need to know the min and the max of the array in order to automatically scale the colormap for display. Therefore, we query the array object for its min/max. Because we support masked arrays, the array is first cast as a masked array, and then these queries are done.

It appears that in numpy’s masked array module, it calculates the array’s min by making a copy of itself first. I would have figured that it would have done its task differently. In the meantime, I suspect you can work around this problem by explicitly setting the vmin/vmax keyword arguments to imshow if you know them. Therefore, there should be no need to determine the array’s min/max in this inefficient manner.

Ben Root

···

On Fri, Sep 7, 2012 at 8:44 AM, Shahar Shani-Kadmiel <kadmiel@…4191…> wrote:

  1. an ipython session is invoked with qtconsole --pylab

  2. I load a large NetCDF grid (Grid file format: nf (# 18) GMT netCDF format (float) (COARDS-compliant) [DEFAULT]), approx. 1.15 GB

  3. I then try to plot with imshow the data

added below are the lines leading up to the error and the error itself.

This is running on OS X 10.7.4 with a recently installed EPD 7.3.

{code}

from scipy.io import netcdf_file as netcdf

data = netcdf(‘srtm_43_44_05_06_07_08.grd’,‘r’).variables[‘z’][::-1]

fig, ax = subplots()

data.shape

Out[5]: (24004, 12002)

im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])), interpolation=‘none’)


MemoryError Traceback (most recent call last)

in ()

----> 1 im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])), interpolation=‘none’)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs)

6743 im.set_clim(vmin, vmax)

6744 else:

-> 6745 im.autoscale_None()

6746 im.set_url(url)

6747

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/cm.py in autoscale_None(self)

281         if self._A is None:

282             raise TypeError('You must first set_array for mappable')

–> 283 self.norm.autoscale_None(self._A)

284         self.changed()

285

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py in autoscale_None(self, A)

889         ' autoscale only None-valued vmin or vmax'

890         if self.vmin is None:

–> 891 self.vmin = ma.min(A)

892         if self.vmax is None:

893             self.vmax = ma.max(A)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in min(obj, axis, out, fill_value)

5873 def min(obj, axis=None, out=None, fill_value=None):

5874 try:

-> 5875 return obj.min(axis=axis, fill_value=fill_value, out=out)

5876 except (AttributeError, TypeError):

5877 # If obj doesn’t have a max method,

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in min(self, axis, out, fill_value)

5054 # No explicit output

5055 if out is None:

-> 5056 result = self.filled(fill_value).min(axis=axis, out=out).view(type(self))

5057 if result.ndim:

5058 # Set the mask

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in filled(self, fill_value)

3388 return self._data

3389 else:

-> 3390 result = self._data.copy()

3391 try:

3392 np.putmask(result, m, fill_value)

MemoryError:

{/code}

Hi Ben,

I tried adding vmin & vmax the the imshow call but I still get a MemoryError.

The grid file is 1.15 GB and I have ~4.5 out of 8 GB of memory available when I launch ipython, 3.5 when I execute imshow and 2 when I execute plt.draw().

im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])), interpolation=None, vmin=-400., vmax=3000.)

plt.draw()

···

On Sep 7, 2012, at 4:25 PM, Benjamin Root wrote:

On Fri, Sep 7, 2012 at 8:44 AM, Shahar Shani-Kadmiel <kadmiel@…4191…> wrote:

  1. an ipython session is invoked with qtconsole --pylab

  2. I load a large NetCDF grid (Grid file format: nf (# 18) GMT netCDF format (float) (COARDS-compliant) [DEFAULT]), approx. 1.15 GB

  3. I then try to plot with imshow the data

added below are the lines leading up to the error and the error itself.

This is running on OS X 10.7.4 with a recently installed EPD 7.3.

{code}

from scipy.io import netcdf_file as netcdf

data = netcdf(‘srtm_43_44_05_06_07_08.grd’,‘r’).variables[‘z’][::-1]

fig, ax = subplots()

data.shape

Out[5]: (24004, 12002)

im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])), interpolation=‘none’)


MemoryError Traceback (most recent call last)

in ()

----> 1 im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])), interpolation=‘none’)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs)

6743 im.set_clim(vmin, vmax)

6744 else:

-> 6745 im.autoscale_None()

6746 im.set_url(url)

6747

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/cm.py in autoscale_None(self)

281         if self._A is None:

282             raise TypeError('You must first set_array for mappable')

–> 283 self.norm.autoscale_None(self._A)

284         self.changed()

285

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py in autoscale_None(self, A)

889         ' autoscale only None-valued vmin or vmax'

890         if self.vmin is None:

–> 891 self.vmin = ma.min(A)

892         if self.vmax is None:

893             self.vmax = ma.max(A)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in min(obj, axis, out, fill_value)

5873 def min(obj, axis=None, out=None, fill_value=None):

5874 try:

-> 5875 return obj.min(axis=axis, fill_value=fill_value, out=out)

5876 except (AttributeError, TypeError):

5877 # If obj doesn’t have a max method,

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in min(self, axis, out, fill_value)

5054 # No explicit output

5055 if out is None:

-> 5056 result = self.filled(fill_value).min(axis=axis, out=out).view(type(self))

5057 if result.ndim:

5058 # Set the mask

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in filled(self, fill_value)

3388 return self._data

3389 else:

-> 3390 result = self._data.copy()

3391 try:

3392 np.putmask(result, m, fill_value)

MemoryError:

{/code}

This is more a NumPy issue than anything else. We need to know the min and the max of the array in order to automatically scale the colormap for display. Therefore, we query the array object for its min/max. Because we support masked arrays, the array is first cast as a masked array, and then these queries are done.

It appears that in numpy’s masked array module, it calculates the array’s min by making a copy of itself first. I would have figured that it would have done its task differently. In the meantime, I suspect you can work around this problem by explicitly setting the vmin/vmax keyword arguments to imshow if you know them. Therefore, there should be no need to determine the array’s min/max in this inefficient manner.

Ben Root


MemoryError Traceback (most recent call last)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)

 53     def draw_wrapper(artist, renderer, *args, **kwargs):

 54         before(artist, renderer)

—> 55 draw(artist, renderer, *args, **kwargs)

 56         after(artist, renderer)

 57 

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/figure.py in draw(self, renderer)

882         dsu.sort(key=itemgetter(0))

883         for zorder, func, args in dsu:

–> 884 func(*args)

885 

886         renderer.close_group('figure')

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)

 53     def draw_wrapper(artist, renderer, *args, **kwargs):

 54         before(artist, renderer)

—> 55 draw(artist, renderer, *args, **kwargs)

 56         after(artist, renderer)

 57 

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/axes.py in draw(self, renderer, inframe)

1981

1982 for zorder, a in dsu:

-> 1983 a.draw(renderer)

1984

1985 renderer.close_group(‘axes’)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)

 53     def draw_wrapper(artist, renderer, *args, **kwargs):

 54         before(artist, renderer)

—> 55 draw(artist, renderer, *args, **kwargs)

 56         after(artist, renderer)

 57 

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/image.py in draw(self, renderer, *args, **kwargs)

353                 warnings.warn("Image will not be shown correctly with this backend.")

354 

–> 355 im = self.make_image(renderer.get_image_magnification())

356             if im is None:

357                 return

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/image.py in make_image(self, magnification)

573         im, xmin, ymin, dxintv, dyintv, sx, sy = \

574             self._get_unsampled_image(self._A, [_x1, _x2, _y1, _y2],

–> 575 transformed_viewLim)

576 

577         fc = self.axes.patch.get_facecolor()

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/image.py in _get_unsampled_image(self, A, image_extents, viewlim)

200             else:

201                 if self._rgbacache is None:

–> 202 x = self.to_rgba(self._A, self._alpha, bytes=True)

203                     self._rgbacache = x

204                 else:

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/cm.py in to_rgba(self, x, alpha, bytes)

211             pass

212         x = ma.asarray(x)

–> 213 x = self.norm(x)

214         x = self.cmap(x, alpha=alpha, bytes=bytes)

215         return x

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py in call(self, value, clip)

843             clip = self.clip

844 

–> 845 result, is_scalar = self.process_value(value)

846 

847         self.autoscale_None(result)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py in process_value(value)

829             if result.dtype.kind == 'f':

830                 if isinstance(value, np.ndarray):

–> 831 result = result.copy()

832             elif result.dtype.itemsize > 2:

833                 result = result.astype(np.float)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in call(self, *args, **params)

2449 mask = instance._mask

2450 cls = type(instance)

-> 2451 result = getattr(data, methodname)(*args, **params).view(cls)

2452 result._update_from(instance)

2453 if result.ndim:

MemoryError:

Well, it looks like setting vmin/vmax helped, because your traceback shows that the code made significant progress. The issue here is that the process_value() method doesn’t make a lot of sense. I am not sure what is the rationale behind its logic. Hopefully, someone else can chime in with an explanation of what is going on.

In the meantime, are you using a 32 or 64-bit machine?

Ben Root

···

On Fri, Sep 7, 2012 at 9:49 AM, Shahar Shani-Kadmiel <kadmiel@…4191…> wrote:

On Sep 7, 2012, at 4:25 PM, Benjamin Root wrote:

On Fri, Sep 7, 2012 at 8:44 AM, Shahar Shani-Kadmiel <kadmiel@…4191…> wrote:

  1. an ipython session is invoked with qtconsole --pylab

  2. I load a large NetCDF grid (Grid file format: nf (# 18) GMT netCDF format (float) (COARDS-compliant) [DEFAULT]), approx. 1.15 GB

  3. I then try to plot with imshow the data

added below are the lines leading up to the error and the error itself.

This is running on OS X 10.7.4 with a recently installed EPD 7.3.

{code}

from scipy.io import netcdf_file as netcdf

data = netcdf(‘srtm_43_44_05_06_07_08.grd’,‘r’).variables[‘z’][::-1]

fig, ax = subplots()

data.shape

Out[5]: (24004, 12002)

im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])), interpolation=‘none’)


MemoryError Traceback (most recent call last)

in ()

----> 1 im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])), interpolation=‘none’)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs)

6743 im.set_clim(vmin, vmax)

6744 else:

-> 6745 im.autoscale_None()

6746 im.set_url(url)

6747

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/cm.py in autoscale_None(self)

281         if self._A is None:

282             raise TypeError('You must first set_array for mappable')

–> 283 self.norm.autoscale_None(self._A)

284         self.changed()

285

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py in autoscale_None(self, A)

889         ' autoscale only None-valued vmin or vmax'

890         if self.vmin is None:

–> 891 self.vmin = ma.min(A)

892         if self.vmax is None:

893             self.vmax = ma.max(A)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in min(obj, axis, out, fill_value)

5873 def min(obj, axis=None, out=None, fill_value=None):

5874 try:

-> 5875 return obj.min(axis=axis, fill_value=fill_value, out=out)

5876 except (AttributeError, TypeError):

5877 # If obj doesn’t have a max method,

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in min(self, axis, out, fill_value)

5054 # No explicit output

5055 if out is None:

-> 5056 result = self.filled(fill_value).min(axis=axis, out=out).view(type(self))

5057 if result.ndim:

5058 # Set the mask

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in filled(self, fill_value)

3388 return self._data

3389 else:

-> 3390 result = self._data.copy()

3391 try:

3392 np.putmask(result, m, fill_value)

MemoryError:

{/code}

This is more a NumPy issue than anything else. We need to know the min and the max of the array in order to automatically scale the colormap for display. Therefore, we query the array object for its min/max. Because we support masked arrays, the array is first cast as a masked array, and then these queries are done.

It appears that in numpy’s masked array module, it calculates the array’s min by making a copy of itself first. I would have figured that it would have done its task differently. In the meantime, I suspect you can work around this problem by explicitly setting the vmin/vmax keyword arguments to imshow if you know them. Therefore, there should be no need to determine the array’s min/max in this inefficient manner.

Ben Root

Hi Ben,

I tried adding vmin & vmax the the imshow call but I still get a MemoryError.

The grid file is 1.15 GB and I have ~4.5 out of 8 GB of memory available when I launch ipython, 3.5 when I execute imshow and 2 when I execute plt.draw().

I am on a 64 bit machine but 32 bit distro of Enthought.

···

On Sep 7, 2012, at 5:00 PM, Benjamin Root <ben.root@…1304…> wrote:

On Fri, Sep 7, 2012 at 9:49 AM, Shahar Shani-Kadmiel <kadmiel@…4191…> wrote:

On Sep 7, 2012, at 4:25 PM, Benjamin Root wrote:

On Fri, Sep 7, 2012 at 8:44 AM, Shahar Shani-Kadmiel <kadmiel@…4191…> wrote:

  1. an ipython session is invoked with qtconsole --pylab

  2. I load a large NetCDF grid (Grid file format: nf (# 18) GMT netCDF format (float) (COARDS-compliant) [DEFAULT]), approx. 1.15 GB

  3. I then try to plot with imshow the data

added below are the lines leading up to the error and the error itself.

This is running on OS X 10.7.4 with a recently installed EPD 7.3.

{code}

from scipy.io import netcdf_file as netcdf

data = netcdf(‘srtm_43_44_05_06_07_08.grd’,‘r’).variables[‘z’][::-1]

fig, ax = subplots()

data.shape

Out[5]: (24004, 12002)

im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])), interpolation=‘none’)


MemoryError Traceback (most recent call last)

in ()

----> 1 im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])), interpolation=‘none’)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs)

6743 im.set_clim(vmin, vmax)

6744 else:

-> 6745 im.autoscale_None()

6746 im.set_url(url)

6747

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/cm.py in autoscale_None(self)

281         if self._A is None:

282             raise TypeError('You must first set_array for mappable')

–> 283 self.norm.autoscale_None(self._A)

284         self.changed()

285

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py in autoscale_None(self, A)

889         ' autoscale only None-valued vmin or vmax'

890         if self.vmin is None:

–> 891 self.vmin = ma.min(A)

892         if self.vmax is None:

893             self.vmax = ma.max(A)

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in min(obj, axis, out, fill_value)

5873 def min(obj, axis=None, out=None, fill_value=None):

5874 try:

-> 5875 return obj.min(axis=axis, fill_value=fill_value, out=out)

5876 except (AttributeError, TypeError):

5877 # If obj doesn’t have a max method,

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in min(self, axis, out, fill_value)

5054 # No explicit output

5055 if out is None:

-> 5056 result = self.filled(fill_value).min(axis=axis, out=out).view(type(self))

5057 if result.ndim:

5058 # Set the mask

/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in filled(self, fill_value)

3388 return self._data

3389 else:

-> 3390 result = self._data.copy()

3391 try:

3392 np.putmask(result, m, fill_value)

MemoryError:

{/code}

This is more a NumPy issue than anything else. We need to know the min and the max of the array in order to automatically scale the colormap for display. Therefore, we query the array object for its min/max. Because we support masked arrays, the array is first cast as a masked array, and then these queries are done.

It appears that in numpy’s masked array module, it calculates the array’s min by making a copy of itself first. I would have figured that it would have done its task differently. In the meantime, I suspect you can work around this problem by explicitly setting the vmin/vmax keyword arguments to imshow if you know them. Therefore, there should be no need to determine the array’s min/max in this inefficient manner.

Ben Root

Hi Ben,

I tried adding vmin & vmax the the imshow call but I still get a MemoryError.

The grid file is 1.15 GB and I have ~4.5 out of 8 GB of memory available when I launch ipython, 3.5 when I execute imshow and 2 when I execute plt.draw().

Well, it looks like setting vmin/vmax helped, because your traceback shows that the code made significant progress. The issue here is that the process_value() method doesn’t make a lot of sense. I am not sure what is the rationale behind its logic. Hopefully, someone else can chime in with an explanation of what is going on.

In the meantime, are you using a 32 or 64-bit machine?

Ben Root

        1. an ipython session is invoked with qtconsole --pylab
        2. I load a large NetCDF grid (Grid file format: nf (# 18) GMT
        netCDF format (float) (COARDS-compliant) [DEFAULT]), approx.
        1.15 GB
        3. I then try to plot with imshow the data

        added below are the lines leading up to the error and the
        error itself.

        This is running on OS X 10.7.4 with a recently installed EPD 7.3.

        {code}
        from scipy.io <http://scipy.io/> import netcdf_file as netcdf
        data =
        netcdf('srtm_43_44_05_06_07_08.grd','r').variables['z'][::-1]

        fig, ax = subplots()

        data.shape
        Out[5]: (24004, 12002)

        im = ax.imshow(data,
        aspect=((data.shape[1])/float(data.shape[0])),
        interpolation='none')
        ---------------------------------------------------------------------------
        MemoryError Traceback (most
        recent call last)
        <ipython-input-6-f92e4c4c63b5> in <module>()
        ----> 1 im = ax.imshow(data,
        aspect=((data.shape[1])/float(data.shape[0])),
        interpolation='none')

        /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/axes.py
        in imshow(self, X, cmap, norm, aspect, interpolation, alpha,
        vmin, vmax, origin, extent, shape, filternorm, filterrad,
        imlim, resample, url, **kwargs)
           6743 im.set_clim(vmin, vmax)
           6744 else:
        -> 6745 im.autoscale_None()
           6746 im.set_url(url)
           6747

        /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/cm.py
        in autoscale_None(self)
            281 if self._A is None:
            282 raise TypeError('You must first set_array
        for mappable')
        --> 283 self.norm.autoscale_None(self._A)
            284 self.changed()
            285

        /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py
        in autoscale_None(self, A)
            889 ' autoscale only None-valued vmin or vmax'
            890 if self.vmin is None:
        --> 891 self.vmin = ma.min(A)
            892 if self.vmax is None:
            893 self.vmax = ma.max(A)

        /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
        in min(obj, axis, out, fill_value)
           5873 def min(obj, axis=None, out=None, fill_value=None):
           5874 try:
        -> 5875 return obj.min(axis=axis,
        fill_value=fill_value, out=out)
           5876 except (AttributeError, TypeError):
           5877 # If obj doesn't have a max method,

        /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
        in min(self, axis, out, fill_value)
           5054 # No explicit output
           5055 if out is None:
        -> 5056 result =
        self.filled(fill_value).min(axis=axis, out=out).view(type(self))
           5057 if result.ndim:
           5058 # Set the mask

        /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
        in filled(self, fill_value)
           3388 return self._data
           3389 else:
        -> 3390 result = self._data.copy()
           3391 try:
           3392 np.putmask(result, m, fill_value)

        MemoryError:
        {/code}

    This is more a NumPy issue than anything else. We need to know
    the min and the max of the array in order to automatically scale
    the colormap for display. Therefore, we query the array object
    for its min/max. Because we support masked arrays, the array is
    first cast as a masked array, and then these queries are done.

    It appears that in numpy's masked array module, it calculates the
    array's min by making a copy of itself first. I would have
    figured that it would have done its task differently. In the
    meantime, I suspect you can work around this problem by explicitly
    setting the vmin/vmax keyword arguments to imshow if you know
    them. Therefore, there should be no need to determine the array's
    min/max in this inefficient manner.

    Ben Root

    Hi Ben,
    I tried adding vmin & vmax the the imshow call but I still get a
    MemoryError.
    The grid file is 1.15 GB and I have ~4.5 out of 8 GB of memory
    available when I launch ipython, 3.5 when I execute imshow and 2
    when I execute plt.draw().

mpl simply is not designed for image-type operations on huge arrays, vastly larger than what can be displayed. It is up to the user to down-sample or otherwise reduce the size of the array fed to imshow.

Well, it looks like setting vmin/vmax helped, because your traceback

Did a new traceback get posted in a message that was not sent to the list? I found only the original traceback.

shows that the code made significant progress. The issue here is that
the process_value() method doesn't make a lot of sense. I am not sure
what is the rationale behind its logic. Hopefully, someone else can
chime in with an explanation of what is going on.

Normalization has to handle all sorts of inputs--masked or not, all sorts of numbers, scalar or array--and it is much easier to do this efficiently if all these possibilities are reduced to a very few at the start. Specifically, it needs to supply a copy of the input (so that normalization doesn't change the original) in a floating point masked array, using float32 if possible for space efficiency. It needs to keep track of whether the input was a scalar, so that normalization can return a scalar when given a scalar input.

Eric

···

On 2012/09/07 4:00 AM, Benjamin Root wrote:

On Fri, Sep 7, 2012 at 9:49 AM, Shahar Shani-Kadmiel > <kadmiel@…4191… <mailto:kadmiel@…4191…>> wrote:
    On Sep 7, 2012, at 4:25 PM, Benjamin Root wrote:

    On Fri, Sep 7, 2012 at 8:44 AM, Shahar Shani-Kadmiel >> <kadmiel@…4191… <mailto:kadmiel@…4191…>> wrote:

In the meantime, are you using a 32 or 64-bit machine?

Ben Root

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users

Normalization has to handle all sorts of inputs–masked or not, all
sorts of numbers, scalar or array–and it is much easier to do this
efficiently if all these possibilities are reduced to a very few at the
start. Specifically, it needs to supply a copy of the input (so that
normalization doesn’t change the original) in a floating point masked
array, using float32 if possible for space efficiency. It needs to keep
track of whether the input was a scalar, so that normalization can
return a scalar when given a scalar input.

Eric

Another option as I understand it is to pass in a 1D (greyscale) or 3d (color) array (where the 3rd dimension is RGB and optionally A) of type uint8(?). This array does not need to get normalized, it will be displayed as raw pixel values. I don’t remember if you also have to specifically tell it not to normalize the data.

But the easier answer for your case would probably be imshow(data[::10,::10]) which will take every 10th element in x and y thus reducing the size by a factor of 100 (depending on the size of your data you could use ::2 or ::50, etc)

Ethan

···

On Sep 7, 2012, at 11:04 AM, Eric Firing wrote:

On 2012/09/07 4:00 AM, Benjamin Root wrote:

On Fri, Sep 7, 2012 at 9:49 AM, Shahar Shani-Kadmiel > > <kadmiel@…4191… mailto:kadmiel@...4191...> wrote:
On Sep 7, 2012, at 4:25 PM, Benjamin Root wrote:

On Fri, Sep 7, 2012 at 8:44 AM, Shahar Shani-Kadmiel > > > <kadmiel@…4191… mailto:kadmiel@...4191...> wrote: