performance of imshow and numarray

Greetings,

I searched the mail archive for information on this question, and
didn't find anything. Hopefully this hasn't been addressed elsewhere.

I just spent a few hours tracking down a huge performance hit in
imshow(). I'm loading some image data via numarray's fromfile(),
stacking three imgaes to make an RGB, and showing it with imshow.

My image loading code looks like this:

def read_dib(filename):
    f = open(filename, "rb")
    fromfile(f, "UInt8", shape=(52, 1))
    im = fromfile(f, "UInt16", shape=(512, 512))
    if sys.byteorder == "big":
  im.byteswap()
    return im.astype('Float') / (2**12 - 1)

I load three images, and put them into a NxMx3 float numarray, and
pass that to imshow.

The performance drop is at this line in image.py:
          im = _image.fromarray(x, 0)

in AxesImage.make_image()

I eventually tracked it down to x's type being <class
'numarray.ma.MA.MaskedArray'>, which gets passed off to numarray's
NA_InputArray function, which chokes and calls setArrayFromSequence
(i.e., it's not using the fact that the input data is a numarray, and
is instead iterating over each element). This is taking around 30
seconds to convert a 512x512x3 image on a new Intel iMac.

I sped it up significantly by using this line instead:

                    im = _image.fromarray(numerix.ma.filled(x,0), 0)

Not that that's a good idea (but it works for me for now).

There is no invalid data in my image. I'm not sure why it's ever
being turned into a masked array.

I'm using matplotlib 0.86.2 and numarray-1.5.1, on OS X 10.3.5 on an Intel iMac.

Any advice would be appreciated.

Thanks,
Ray Jones

Ray,

You tripped over a bug in cm.py: when I added masked array support, I put the "x = ma.asarray(x)" too early in the to_rgba() method of ScalarMappable. It is fixed now in CVS. Thanks for finding the problem.

If you want to try the fix in your version, in place of your workaround, here is the revised method:

    def to_rgba(self, x, alpha=1.0):
         # assume normalized rgb, rgba
         if len(x.shape)>2: return x
         x = ma.asarray(x)
         x = self.norm(x)
         x = self.cmap(x, alpha)
         return x

Eric

Thouis (Ray) Jones wrote:

···

Greetings,

I searched the mail archive for information on this question, and
didn't find anything. Hopefully this hasn't been addressed elsewhere.

I just spent a few hours tracking down a huge performance hit in
imshow(). I'm loading some image data via numarray's fromfile(),
stacking three imgaes to make an RGB, and showing it with imshow.

My image loading code looks like this:

def read_dib(filename):
    f = open(filename, "rb")
    fromfile(f, "UInt8", shape=(52, 1))
    im = fromfile(f, "UInt16", shape=(512, 512))
    if sys.byteorder == "big":
  im.byteswap()
    return im.astype('Float') / (2**12 - 1)

I load three images, and put them into a NxMx3 float numarray, and
pass that to imshow.

The performance drop is at this line in image.py:
          im = _image.fromarray(x, 0)

in AxesImage.make_image()

I eventually tracked it down to x's type being <class
'numarray.ma.MA.MaskedArray'>, which gets passed off to numarray's
NA_InputArray function, which chokes and calls setArrayFromSequence
(i.e., it's not using the fact that the input data is a numarray, and
is instead iterating over each element). This is taking around 30
seconds to convert a 512x512x3 image on a new Intel iMac.

I sped it up significantly by using this line instead:

                    im = _image.fromarray(numerix.ma.filled(x,0), 0)

Not that that's a good idea (but it works for me for now).

There is no invalid data in my image. I'm not sure why it's ever
being turned into a masked array.

I'm using matplotlib 0.86.2 and numarray-1.5.1, on OS X 10.3.5 on an Intel iMac.

Any advice would be appreciated.

Thanks,
Ray Jones

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=k&kid3432&bid#0486&dat1642
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users

Eric Firing wrote:

Ray,

You tripped over a bug in cm.py: when I added masked array support, I put the "x = ma.asarray(x)" too early in the to_rgba() method of ScalarMappable. It is fixed now in CVS. Thanks for finding the problem.

If you want to try the fix in your version, in place of your workaround, here is the revised method:

   def to_rgba(self, x, alpha=1.0):
        # assume normalized rgb, rgba
        if len(x.shape)>2: return x
        x = ma.asarray(x)
        x = self.norm(x)
        x = self.cmap(x, alpha)
        return x

Eric

Hi Eric: This breaks some of the basemap demos which do this

CS = m.contour(x,y,hgt,15,linewidths=0.5,colors='k')

to draw solid black contour lines. I now get

File "/sw/lib/python2.4/site-packages/matplotlib/cm.py", line 52, in to_rgba
    if len(x.shape)>2: return x
AttributeError: 'list' object has no attribute 'shape'

It appears that colors now has to be an array?

-Jeff

···

--
Jeffrey S. Whitaker Phone : (303)497-6313
Meteorologist FAX : (303)497-6449
NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@...259...
325 Broadway Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web : http://tinyurl.com/5telg

Jeff,

I really blew it on that one--it looked so simple! OK, I think this version is correct; it will land in CVS in a few minutes.

     def to_rgba(self, x, alpha=1.0):
         '''Return a normalized rgba array corresponding to x.
         If x is already an rgb or rgba array, return it unchanged.
         '''
         if hasattr(x, 'shape') and len(x.shape)>2: return x
         x = ma.asarray(x)
         x = self.norm(x)
         x = self.cmap(x, alpha)
         return x
Eric

Jeff Whitaker wrote:

···

Eric Firing wrote:

Ray,

You tripped over a bug in cm.py: when I added masked array support, I put the "x = ma.asarray(x)" too early in the to_rgba() method of ScalarMappable. It is fixed now in CVS. Thanks for finding the problem.

If you want to try the fix in your version, in place of your workaround, here is the revised method:

   def to_rgba(self, x, alpha=1.0):
        # assume normalized rgb, rgba
        if len(x.shape)>2: return x
        x = ma.asarray(x)
        x = self.norm(x)
        x = self.cmap(x, alpha)
        return x

Eric

Hi Eric: This breaks some of the basemap demos which do this

CS = m.contour(x,y,hgt,15,linewidths=0.5,colors='k')

to draw solid black contour lines. I now get

File "/sw/lib/python2.4/site-packages/matplotlib/cm.py", line 52, in to_rgba
   if len(x.shape)>2: return x
AttributeError: 'list' object has no attribute 'shape'

It appears that colors now has to be an array?

-Jeff

Eric Firing wrote:

Jeff,

I really blew it on that one--it looked so simple! OK, I think this version is correct; it will land in CVS in a few minutes.

    def to_rgba(self, x, alpha=1.0):
        '''Return a normalized rgba array corresponding to x.
        If x is already an rgb or rgba array, return it unchanged.
        '''
        if hasattr(x, 'shape') and len(x.shape)>2: return x
        x = ma.asarray(x)
        x = self.norm(x)
        x = self.cmap(x, alpha)
        return x
Eric

Eric: Thanks for the quick fix - that seems to have done the job.

-Jeff

···

--
Jeffrey S. Whitaker Phone : (303)497-6313
Meteorologist FAX : (303)497-6449
NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@...259...
325 Broadway Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web : http://tinyurl.com/5telg

Jeff Whitaker wrote:

Eric Firing wrote:

Jeff,
I really blew it on that one--it looked so simple! OK, I think this version is correct; it will land in CVS in a few minutes.

    def to_rgba(self, x, alpha=1.0):

>> [[[ code removed ]]]
>>

Eric: Thanks for the quick fix - that seems to have done the job.
-Jeff

I don't know if it's related, but as of the latest version, all of my imshow() commands show a single color, no structure at all.

Andrew,

Unfortunately, the 0.87 release was packed up immediately after I made the first of two changes to cm.py, so it contains the bug referred to below. Most likely that is what is wrecking your imshow plots. Please try substituting the revised function in cm.py, and let us know if that solves the problem.

     def to_rgba(self, x, alpha=1.0):
         '''Return a normalized rgba array corresponding to x.
         If x is already an rgb or rgba array, return it unchanged.
         '''
         if hasattr(x, 'shape') and len(x.shape)>2: return x
         x = ma.asarray(x)
         x = self.norm(x)
         x = self.cmap(x, alpha)
         return x

Eric

Andrew Jaffe wrote:

···

Jeff Whitaker wrote:

Eric Firing wrote:

Jeff,
I really blew it on that one--it looked so simple! OK, I think this version is correct; it will land in CVS in a few minutes.

    def to_rgba(self, x, alpha=1.0):

>> [[[ code removed ]]]
>>

Eric: Thanks for the quick fix - that seems to have done the job.
-Jeff

I don't know if it's related, but as of the latest version, all of my imshow() commands show a single color, no structure at all.

Eric (etc)-

Sorry, no dice... in fact I was already using the CVS version so this was the to_rgba() method that failed originally!

Any other possibilities? Could the problem be just a coincidental error elsewhere?

Andrew

···

On 23 Feb 2006, at 17:31, Eric Firing wrote:

Andrew,

Unfortunately, the 0.87 release was packed up immediately after I made the first of two changes to cm.py, so it contains the bug referred to below. Most likely that is what is wrecking your imshow plots. Please try substituting the revised function in cm.py, and let us know if that solves the problem.

    def to_rgba(self, x, alpha=1.0):
        '''Return a normalized rgba array corresponding to x.
        If x is already an rgb or rgba array, return it unchanged.
        '''
        if hasattr(x, 'shape') and len(x.shape)>2: return x
        x = ma.asarray(x)
        x = self.norm(x)
        x = self.cmap(x, alpha)
        return x

Eric

Andrew Jaffe wrote:

Jeff Whitaker wrote:

Eric Firing wrote:

Jeff,
I really blew it on that one--it looked so simple! OK, I think this version is correct; it will land in CVS in a few minutes.

    def to_rgba(self, x, alpha=1.0):

>> [[[ code removed ]]]
>>

Eric: Thanks for the quick fix - that seems to have done the job.
-Jeff

I don't know if it's related, but as of the latest version, all of my imshow() commands show a single color, no structure at all.

______________________________________________________________________
Andrew Jaffe a.jaffe@...762...
Astrophysics Group +44 207 594-7526
Blackett Laboratory, Room 1013 FAX 7541
Imperial College, Prince Consort Road
London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe

Andrew Jaffe wrote:

Eric (etc)-

Sorry, no dice... in fact I was already using the CVS version so this was the to_rgba() method that failed originally!

Any other possibilities? Could the problem be just a coincidental error elsewhere?

Andrew

Andrew and Eric: I apologize, this latest glitch was all my fault. I had made a modification to colors.py to workaround an apparenty bug in numpy masked arrays, and that modification was wrong. It's now fixed in CVS, revision 1.26

Index: colors.py

···

===================================================================
RCS file: /cvsroot/matplotlib/matplotlib/lib/matplotlib/colors.py,v
retrieving revision 1.25
diff -r1.25 colors.py
659c659
< result = (vmax-vmin)/(val-vmin) <-- this is wrong
---
> result = (val-vmin)/(vmax-vmin) <-- this is right

Originally (in 0.87) there was

result = (1.0/(vmax-vmin))*(val-vmin)

which raised an exception when val was a numpy masked array.

Sorry for the screwup.

-Jeff

--
Jeffrey S. Whitaker Phone : (303)497-6313
Meteorologist FAX : (303)497-6449
NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@...259...
325 Broadway Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web : http://tinyurl.com/5telg

Hi-

Thanks for the help. A comment below...

Andrew Jaffe wrote:

Eric (etc)-

Sorry, no dice... in fact I was already using the CVS version so this was the to_rgba() method that failed originally!

Any other possibilities? Could the problem be just a coincidental error elsewhere?

Andrew

Andrew and Eric: I apologize, this latest glitch was all my fault. I had made a modification to colors.py to workaround an apparenty bug in numpy masked arrays, and that modification was wrong. It's now fixed in CVS, revision 1.26

Index: colors.py

RCS file: /cvsroot/matplotlib/matplotlib/lib/matplotlib/colors.py,v
retrieving revision 1.25
diff -r1.25 colors.py
659c659
< result = (vmax-vmin)/(val-vmin) <-- this is wrong
---
> result = (val-vmin)/(vmax-vmin) <-- this is right

Originally (in 0.87) there was

result = (1.0/(vmax-vmin))*(val-vmin)

which raised an exception when val was a numpy masked array.

However, I expect the original line was like this in case vmax, vmin and val are all integers, in which case without 'truedivision' (or 'from __future__ import division'), result will use integer division which probably isn't desired.

So this may need to be fixed to force floating-point division. Not sure the robust way to do this with possibly-masked arrays, etc.

Andrew

Andrew Jaffe wrote:

Hi-

Thanks for the help. A comment below...

Andrew Jaffe wrote:

Eric (etc)-

Sorry, no dice... in fact I was already using the CVS version so this was the to_rgba() method that failed originally!

Any other possibilities? Could the problem be just a coincidental error elsewhere?

Andrew

Andrew and Eric: I apologize, this latest glitch was all my fault. I had made a modification to colors.py to workaround an apparenty bug in numpy masked arrays, and that modification was wrong. It's now fixed in CVS, revision 1.26

Index: colors.py

RCS file: /cvsroot/matplotlib/matplotlib/lib/matplotlib/colors.py,v
retrieving revision 1.25
diff -r1.25 colors.py
659c659
< result = (vmax-vmin)/(val-vmin) <-- this is wrong
---
> result = (val-vmin)/(vmax-vmin) <-- this is right

Originally (in 0.87) there was

result = (1.0/(vmax-vmin))*(val-vmin)

which raised an exception when val was a numpy masked array.

However, I expect the original line was like this in case vmax, vmin and val are all integers, in which case without 'truedivision' (or 'from __future__ import division'), result will use integer division which probably isn't desired.

So this may need to be fixed to force floating-point division. Not sure the robust way to do this with possibly-masked arrays, etc.

Andrew

Andrew: (val-vmin)/float(vmax-vmin) should be equivalent to the original, since I believe vmin and vmax are always scalars.

BTW: you can trigger the numpy.ma/imshow problem by uncommenting the 'colorbar(im)' line in the image_masked.py example. With the original

result = (1.0/(vmax-vmin))*(val-vmin)

line in colors.py, it will throw an exception with numpy 0.9.5, but

result = (val-vmin)/float(vmax-vmin)

works.

-Jeff