"N" parameter of LinearSegmentedColormap

Community matplotlib-users
1

Hi everybody,

I have a problem with LinearSegmentedColormap.
In the example below (see PS), I make a colormap, and use it to plot an
EllipseCollection. My plot is parameterized by a quantity that I have named
"large_value". For large_value equal to 257, a blue point is obtained at
(x=0.3, y=0.4). But for large_value equal to 258, it becomes black.

This is because of the way LinearSegmentedColormap is working. It has a
parameter N which allows to set the "number of colors":

http://matplotlib.org/api/colors_api.html#matplotlib.colors.LinearSegmentedColormap

It is 256 by default, so if I increase N to a greater value, the point remains
blue for large_value equal to 258.

Now, my real plot (not this dummy example) is such that I need N to be very
large so as to obtain the right colors on my plot, although very few colors
are used at the end.
However, when N is too large, the plot becomes very slow, and a lot of memory
is used; I think because an array is probably built with this size, although
in theory there is no need to construct such a complete array.

Is there an easy workaround, or have I to study and modify the matplotlib code
myself?

Thanks,

TP

PS: Here is the test code:

···

##################
from pylab import *
from matplotlib.colors import LinearSegmentedColormap
from matplotlib.collections import CircleCollection

ioff()
large_value = 257 # blue below this value
#large_value = 258 # black above this value
N = 1e5 # 256 by default

cdict = { 'blue': [(0.0, 0.0, 0.0),
                    (2*1/large_value, 1, 1)
                    , (1.0, 1.0, 1.0)]
                    , 'green': [(0.0, 0.0, 0.0),
                        (2*1/large_value, 0, 0)
                        , (1.0, 1.0, 1.0)]
                    , 'red': [(0.0, 0.0, 0.0),
                            (2*1/large_value, 0, 0),
                            (1.0, 1.0, 1.0)] }

measures= array([[ 0.2, 0.3, 1],
       [ 0.3, 0.4, 2],
       [ 0.5, 0.6, large_value]])

cmap = LinearSegmentedColormap( "cmap foobar"
        , cdict
       # , N= N )
        )

fig = figure()
axes = fig.add_subplot(111)
ec = CircleCollection( [80]
        , offsets = measures[:,:2]
        , transOffset = axes.transData
        )

ec.set_array( measures[:,2] )
ec.set_cmap( cmap )
axes.add_collection( ec )

show()
##################

2

Hi everybody,

I have a problem with LinearSegmentedColormap.
In the example below (see PS), I make a colormap, and use it to plot an
EllipseCollection. My plot is parameterized by a quantity that I have named
"large_value". For large_value equal to 257, a blue point is obtained at
(x=0.3, y=0.4). But for large_value equal to 258, it becomes black.

This is because of the way LinearSegmentedColormap is working. It has a
parameter N which allows to set the "number of colors":

http://matplotlib.org/api/colors_api.html#matplotlib.colors.LinearSegmentedColormap

It is 256 by default, so if I increase N to a greater value, the point remains
blue for large_value equal to 258.

Now, my real plot (not this dummy example) is such that I need N to be very
large so as to obtain the right colors on my plot, although very few colors
are used at the end.
However, when N is too large, the plot becomes very slow, and a lot of memory
is used; I think because an array is probably built with this size, although
in theory there is no need to construct such a complete array.

Is there an easy workaround, or have I to study and modify the matplotlib code
myself?

It is not entirely clear to me what you are trying to do, but it sounds like increasing N is not the right way to do it. Three things might help you find a better way:

1) The colormap is intended to work with a norm that handles the translation from your data numbers to the 0-1.0 range used to select values from the colormap (with exceptions--see below). You can choose a non-default norm, you can write your own, or you can set the parameters (vmin, vmax) of the standard linear norm.

2) By creating a colormap and calling its set_under, set_over, and set_invalid methods, you can control the colors assigned to data values that your norm maps respectively to negative numbers, numbers greater than 1, and masked values. See http://matplotlib.org/examples/pylab_examples/contourf_demo.html for an example of using set_under and set_over. See http://matplotlib.org/examples/pylab_examples/image_masked.html for another example, and for an example of controlling the norm parameters or using an alternative norm.

3) It is also possible to index directly into the colormap if you use a norm that returns an integer data type. An example of such is the BoundaryNorm. http://matplotlib.org/examples/pylab_examples/multicolored_line.html

If all you need is a single assignment of a color to a "large value", then using the set_over method will take care of it.

Eric

···

On 2012/11/19 11:42 AM, TP wrote:

Thanks,

TP

PS: Here is the test code:
##################
from pylab import *
from matplotlib.colors import LinearSegmentedColormap
from matplotlib.collections import CircleCollection

ioff()
large_value = 257 # blue below this value
#large_value = 258 # black above this value
N = 1e5 # 256 by default

cdict = { 'blue': [(0.0, 0.0, 0.0),
                     (2*1/large_value, 1, 1)
                     , (1.0, 1.0, 1.0)]
                     , 'green': [(0.0, 0.0, 0.0),
                         (2*1/large_value, 0, 0)
                         , (1.0, 1.0, 1.0)]
                     , 'red': [(0.0, 0.0, 0.0),
                             (2*1/large_value, 0, 0),
                             (1.0, 1.0, 1.0)] }

measures= array([[ 0.2, 0.3, 1],
        [ 0.3, 0.4, 2],
        [ 0.5, 0.6, large_value]])

cmap = LinearSegmentedColormap( "cmap foobar"
         , cdict
        # , N= N )
         )

fig = figure()
axes = fig.add_subplot(111)
ec = CircleCollection( [80]
         , offsets = measures[:,:2]
         , transOffset = axes.transData
         )

ec.set_array( measures[:,2] )
ec.set_cmap( cmap )
axes.add_collection( ec )

show()
##################

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

3

Thanks for your answer.

My goal is to keep the correct color, i.e. blue, for the point located at
(x=0.3, y=0.4), even if there are very large values of z on the plot. As I
said, increasing N is not satisfying because it leads to large amounts of
memory to be used. But for the time being, this is the only solution I have
found.

I cannot use the set_over method to do that, because the "large value" is not
the only one. Indeed, what I want to do is an imshow plot, with a colorbar
containing three different linear portions:
* one portion for the values of z contained between the minimum and maximum
value of z in some measure points.
* one portion for the values of z below the minimum z in the measure points.
* one portion for the values of z above the maximum z in the measure points.

My problem is that I may have very large values on the plot in the range below
or above the measures z. So I have exactly the problem shown in my dummy
example of the previous post: all my measures have the same color, although
they should not, because I have created a colormap that should handle this
situation (three different linear portions in the map). The only workaround I
have found is to increase the value of N, but in my case it has to be very
large, such that the plot is very slow to display, or even can ask for huge
amounts of memory.

Thus it seems to me that my dummy example given in the previous post covers
exactly the problem encountered in my real-world imshow function.

Is there a memory-efficient workaround in my dummy example (instead of
increasing N)?

Thanks,

TP

···

On Monday, November 19, 2012 13:53:21 Eric Firing wrote:

It is not entirely clear to me what you are trying to do, but it sounds
like increasing N is not the right way to do it. Three things might help
you find a better way:

1) The colormap is intended to work with a norm that handles the
translation from your data numbers to the 0-1.0 range used to select
values from the colormap (with exceptions--see below). You can choose a
non-default norm, you can write your own, or you can set the parameters
(vmin, vmax) of the standard linear norm.

2) By creating a colormap and calling its set_under, set_over, and
set_invalid methods, you can control the colors assigned to data values
that your norm maps respectively to negative numbers, numbers greater
than 1, and masked values. See
http://matplotlib.org/examples/pylab_examples/contourf_demo.html for an
example of using set_under and set_over. See
http://matplotlib.org/examples/pylab_examples/image_masked.html for
another example, and for an example of controlling the norm parameters
or using an alternative norm.

3) It is also possible to index directly into the colormap if you use a
norm that returns an integer data type. An example of such is the
BoundaryNorm.
http://matplotlib.org/examples/pylab_examples/multicolored_line.html

If all you need is a single assignment of a color to a "large value",
then using the set_over method will take care of it.

Eric

4

I have modified LinearSegmentedColormap so as to solve my problem. The
difference is that I do not create an huge array in my test case, but instead I
interpolate linearly in the colormap. This is a quick and dirty code that
does work in my case, but which does not deal with all cases (no management of
transparency, no discontinuity in the colormap, etc.)

···

On Thursday, November 22, 2012 23:51:08 TP wrote:

Thus it seems to me that my dummy example given in the previous post covers
exactly the problem encountered in my real-world imshow function.

Is there a memory-efficient workaround in my dummy example (instead of
increasing N)?

#####################
from __future__ import division
from pylab import *
from matplotlib.colors import LinearSegmentedColormap
from matplotlib.collections import CircleCollection

from scipy.interpolate import interp1d

class ContinuousLinearSegmentedColormap( LinearSegmentedColormap ):

    def __init__(self, name, segmentdata, gamma=1.0):

        LinearSegmentedColormap.__init__( self
                , name, segmentdata, gamma = gamma )

    def _init(self):

        self.N = len( self._segmentdata['red'] )
        self._lut = np.ones((self.N, 5), np.float)
        for i in range( self.N ):
            self._lut[i, 0] = self._segmentdata['red'][i][0]
            # 2 because I do not manage discontinuities in color
            self._lut[i, 1] = self._segmentdata['red'][i][2]
            self._lut[i, 2] = self._segmentdata['green'][i][2]
            self._lut[i, 3] = self._segmentdata['blue'][i][2]

        self._isinit = True

    def __call__(self, X, alpha=None, bytes=False):

        if not self._isinit: self._init()
        mask_bad = None
        if not cbook.iterable(X):
            vtype = 'scalar'
            xa = np.array()
        else:
            vtype = 'array'
            xma = ma.array(X, copy=False)
            mask_bad = xma.mask
            xa = xma.data.copy() # Copy here to avoid side effects.
            del xma

        lut = self._lut.copy()
        rgba = np.empty(shape=xa.shape+(4,), dtype=lut.dtype)

        # We construct interpolation functions.
        fred = interp1d( lut[:,0], lut[:,1])
        fgreen = interp1d( lut[:,0], lut[:,2])
        fblue = interp1d( lut[:,0], lut[:,3])

        rgba[:,3] = 1 # alpha=1 for the time being
        for i in range( xa.shape[0] ):
            rgba[i,0] = fred( xa[i] )
            rgba[i,1] = fgreen( xa[i] )
            rgba[i,2] = fblue( xa[i] )

        if vtype == 'scalar':
            rgba = tuple(rgba[0,:])
        return rgba

ioff()

large_value = 257 # blue above this value
large_value = 258 # black above this value
large_value = 1e8

cdict = { 'blue': [(0.0, 0.0, 0.0)
                    , (2*1/large_value, 1, 1)
                    , (1.0, 1.0, 1.0)]
                    , 'green': [(0.0, 0.0, 0.0)
                        , (2*1/large_value, 0, 0)
                        , (1.0, 1.0, 1.0)]
                    , 'red': [(0.0, 0.0, 0.0)
                            , (2*1/large_value, 0, 0)
                            , (1.0, 1.0, 1.0)] }

measures= array( [[ 0.2, 0.3, 0],
       [ 0.3, 0.4, 2],
       [ 0.5, 0.6, large_value]] )

cmap = ContinuousLinearSegmentedColormap( "cmap foobar"
        , cdict
        )

fig = figure()
axes = fig.add_subplot(111)
ec = CircleCollection( [80]
        , offsets = measures[:,:2]
        , transOffset = axes.transData
        )

ec.set_array( measures[:,2] )
ec.set_cmap( cmap )
axes.add_collection( ec )

show()
#####################

5

Thus it seems to me that my dummy example given in the previous post covers
exactly the problem encountered in my real-world imshow function.

Is there a memory-efficient workaround in my dummy example (instead of
increasing N)?

I have modified LinearSegmentedColormap so as to solve my problem. The
difference is that I do not create an huge array in my test case, but instead I
interpolate linearly in the colormap. This is a quick and dirty code that
does work in my case, but which does not deal with all cases (no management of
transparency, no discontinuity in the colormap, etc.)

I'm glad you found a solution, but my sense is that the problem is that you are trying to make the colormap do the work of the norm. The colormap is just a set of discrete colors, with a linear mapping to the 0-1 scale (apart from the special under, over, and invalid values). The norm is for mapping your data to those colors, however you like, by mapping your data to the 0-1 range (again with possible under, over, and invalid values). Did you consider making a custom norm instead of modifying the colormap?

Eric

···

On 2012/11/26 11:37 AM, TP wrote:

On Thursday, November 22, 2012 23:51:08 TP wrote:

#####################
from __future__ import division
from pylab import *
from matplotlib.colors import LinearSegmentedColormap
from matplotlib.collections import CircleCollection

from scipy.interpolate import interp1d

class ContinuousLinearSegmentedColormap( LinearSegmentedColormap ):

     def __init__(self, name, segmentdata, gamma=1.0):

         LinearSegmentedColormap.__init__( self
                 , name, segmentdata, gamma = gamma )

     def _init(self):

         self.N = len( self._segmentdata['red'] )
         self._lut = np.ones((self.N, 5), np.float)
         for i in range( self.N ):
             self._lut[i, 0] = self._segmentdata['red'][i][0]
             # 2 because I do not manage discontinuities in color
             self._lut[i, 1] = self._segmentdata['red'][i][2]
             self._lut[i, 2] = self._segmentdata['green'][i][2]
             self._lut[i, 3] = self._segmentdata['blue'][i][2]

         self._isinit = True

     def __call__(self, X, alpha=None, bytes=False):

         if not self._isinit: self._init()
         mask_bad = None
         if not cbook.iterable(X):
             vtype = 'scalar'
             xa = np.array()
         else:
             vtype = 'array'
             xma = ma.array(X, copy=False)
             mask_bad = xma.mask
             xa = xma.data.copy() # Copy here to avoid side effects.
             del xma

         lut = self._lut.copy()
         rgba = np.empty(shape=xa.shape+(4,), dtype=lut.dtype)

         # We construct interpolation functions.
         fred = interp1d( lut[:,0], lut[:,1])
         fgreen = interp1d( lut[:,0], lut[:,2])
         fblue = interp1d( lut[:,0], lut[:,3])

         rgba[:,3] = 1 # alpha=1 for the time being
         for i in range( xa.shape[0] ):
             rgba[i,0] = fred( xa[i] )
             rgba[i,1] = fgreen( xa[i] )
             rgba[i,2] = fblue( xa[i] )

         if vtype == 'scalar':
             rgba = tuple(rgba[0,:])
         return rgba

ioff()

large_value = 257 # blue above this value
large_value = 258 # black above this value
large_value = 1e8

cdict = { 'blue': [(0.0, 0.0, 0.0)
                     , (2*1/large_value, 1, 1)
                     , (1.0, 1.0, 1.0)]
                     , 'green': [(0.0, 0.0, 0.0)
                         , (2*1/large_value, 0, 0)
                         , (1.0, 1.0, 1.0)]
                     , 'red': [(0.0, 0.0, 0.0)
                             , (2*1/large_value, 0, 0)
                             , (1.0, 1.0, 1.0)] }

measures= array( [[ 0.2, 0.3, 0],
        [ 0.3, 0.4, 2],
        [ 0.5, 0.6, large_value]] )

cmap = ContinuousLinearSegmentedColormap( "cmap foobar"
         , cdict
         )

fig = figure()
axes = fig.add_subplot(111)
ec = CircleCollection( [80]
         , offsets = measures[:,:2]
         , transOffset = axes.transData
         )

ec.set_array( measures[:,2] )
ec.set_cmap( cmap )
axes.add_collection( ec )

show()
#####################

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

6

Yes, I did.
The problem with the default colormap is that it has not enough colors. I have
found (I may be wrong) that no norm can change this state of affair. If you are
able to find a norm to make my example work, i.e. to obtain the middle point in
blue when large_value is for example 1e10, I am interested.

TP

···

On Monday, November 26, 2012 12:06:40 Eric Firing wrote:

I'm glad you found a solution, but my sense is that the problem is that
you are trying to make the colormap do the work of the norm. The
colormap is just a set of discrete colors, with a linear mapping to the
0-1 scale (apart from the special under, over, and invalid values). The
norm is for mapping your data to those colors, however you like, by
mapping your data to the 0-1 range (again with possible under, over, and
invalid values). Did you consider making a custom norm instead of
modifying the colormap?

7

I'm glad you found a solution, but my sense is that the problem is that
you are trying to make the colormap do the work of the norm. The
colormap is just a set of discrete colors, with a linear mapping to the
0-1 scale (apart from the special under, over, and invalid values). The
norm is for mapping your data to those colors, however you like, by
mapping your data to the 0-1 range (again with possible under, over, and
invalid values). Did you consider making a custom norm instead of
modifying the colormap?

Yes, I did.
The problem with the default colormap is that it has not enough colors. I have
found (I may be wrong) that no norm can change this state of affair. If you are
able to find a norm to make my example work, i.e. to obtain the middle point in
blue when large_value is for example 1e10, I am interested.

But how many colors can you actually distinguish on the screen, or in a plot? My impression is that the problem is not lack of colors, but rather mapping to the color you want. There is no reason that having a value in your *data* of 1e10 has to affect how numbers in your data over a "normal" range are mapped.

You are trying to illustrate the problem with an example using 3 colors, so how can the number of colors in the colormap be the fundamental limitation?

Eric

···

On 2012/11/26 12:18 PM, TP wrote:

On Monday, November 26, 2012 12:06:40 Eric Firing wrote:

TP

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

8

Ok, I understand.
I think that my linear interpolation code has to somewhat be written in a norm
instead.
At some time, I have looked at examples on Matplotlib website, and at the code
of pyshared/matplotlib/colors.py, but without having the "flash" to write the
norm. The next time I will try to write a norm instead (I will put the code
here of course).

TP

···

On Monday, November 26, 2012 14:10:31 Eric Firing wrote:

But how many colors can you actually distinguish on the screen, or in a
plot? My impression is that the problem is not lack of colors, but
rather mapping to the color you want. There is no reason that having a
value in your *data* of 1e10 has to affect how numbers in your data over
a "normal" range are mapped.

You are trying to illustrate the problem with an example using 3 colors,
so how can the number of colors in the colormap be the fundamental
limitation?