matplotlib-grayscale

Community matplotlib-users
1

Hi,

I appreciate all the effort and, lastly, the large functionality
offered by matplotlib. But I found no way around formulating things a
bit provocative on
http://friedrichromstedt.github.com/matplotlib-grayscale/index.html.

The project is about matplotlib-grayscale, a matplotlib patch that
aims at grayscale toggle-switch behaviour as well as colour filtering
/ conversion.

The project has a proof of concept (see under the URL above), but I
disregard this since it has major design flaws, although *it works*.
But as most things in matplotlib, *which work*, I would rather prefer
a well thought-through solution. No offense, it's just the way it
goes with matplotlib. I have some examples in the roadmap.

I'm telling upfront that I don't want to get engaged with mpl, I just
want to get this my project getting an ex-project. So I want to do it
quickly and properly, let it go to the main repository, and forget
about it except bug-fixes, the usual maintenance.

Time frame, is, including good testing, say 3 monthes. I want to get
it started quickly and just want to let you all let know about this
project. I'll do it irrespective of if there's clubbing, it's up to
the maintainers to accept or reject it in the end.

I'm asking for ideas about the general design, which do not go too far
with mpl, but use the existing framework to its best. I've put some
of my ideas on the web page noted above.

I did not dive into it again yet after the few monthes of letting it
put aside. I just dug out my old commit messages and have put them
together on the web page together with the superficial things I
remembered. But I think the detail knowledge can be recovered easily.

Only constructive feedback please.

Especially a documenter/tester with a fresh eye on the things I know
now would be great. Same applies for testing. The testing done for
the pre-github proof-of-concept by Benjamin Root must be noted here,
it made flaws and side-tracks apparent I either didn't know about or
did overlook. Documenting and testing of the technical things is up
to me, only the interface documentation and testing boredom.

Last time I kept the project secret, I'm not gonna do commit that mistake again.

Without matplotlib, lastly, also this patch would simply not exist,
neither in my mind nor in reality.

cu,
Friedrich

P.S.: I keep the right of dropping the project at any time. I'll do
my diploma next monthes, so time might be short. But I think it might
serve its purpose.

···

--
I want to be honest and open to you from the beginning who I am and
what I'm doing. So here's a small collection of informative links:

http://github.com/friedrichromstedt/ (github code sharing);
http://friedrichr.blogspot.com/2011/10/home-dashboard.html
(All-Purpose Blog); http://picasaweb.google.com/friedrichromstedt/
(photos which are public); http://vimeo.com/friedrichr/ (videos which
are public). There's a legacy web page
http://www.friedrichromstedt.org/ which is to be replaced. There's a
legacy YouTube channel http://www.youtube.com/user/friedrichromstedt/
which is unmaintained. And there's a news feed on Twitter:
@f_romstedt.

2

Friedrich (or others),

I hope you will consider the following to be constructive.

I understand one use case for a gray switch: sometimes one needs to make gray versions of plots for formal publication, but one wants to have nicer color versions for other purposes.

What is the use case for grayscale support for parts of the Figure? It seems to me that this requires a much more complex and extensive set of changes than a switch for the whole Figure. Is it really worth it?

You indicate that some caching will need to be eliminated. Is this correct? If your branch causes a significant loss of speed in displaying and interacting with large data sets, then I will oppose merging it. Speed matters for a significant fraction of mpl usage.

Improvements in mpl architecture that maintain or increase speed, and that make it more comprehensible, maintainable, and capable, are certainly welcome. Color handling has grown organically, often bedeviled by that pesky alpha, and I have no doubt that it could be cleaned up and improved.

Eric

···

On 10/19/2011 05:15 PM, Friedrich Romstedt wrote:

Hi,

I appreciate all the effort and, lastly, the large functionality
offered by matplotlib. But I found no way around formulating things a
bit provocative on
http://friedrichromstedt.github.com/matplotlib-grayscale/index.html.

The project is about matplotlib-grayscale, a matplotlib patch that
aims at grayscale toggle-switch behaviour as well as colour filtering
/ conversion.

The project has a proof of concept (see under the URL above), but I
disregard this since it has major design flaws, although *it works*.
But as most things in matplotlib, *which work*, I would rather prefer
a well thought-through solution. No offense, it's just the way it
goes with matplotlib. I have some examples in the roadmap.

I'm telling upfront that I don't want to get engaged with mpl, I just
want to get this my project getting an ex-project. So I want to do it
quickly and properly, let it go to the main repository, and forget
about it except bug-fixes, the usual maintenance.

Time frame, is, including good testing, say 3 monthes. I want to get
it started quickly and just want to let you all let know about this
project. I'll do it irrespective of if there's clubbing, it's up to
the maintainers to accept or reject it in the end.

I'm asking for ideas about the general design, which do not go too far
with mpl, but use the existing framework to its best. I've put some
of my ideas on the web page noted above.

I did not dive into it again yet after the few monthes of letting it
put aside. I just dug out my old commit messages and have put them
together on the web page together with the superficial things I
remembered. But I think the detail knowledge can be recovered easily.

Only constructive feedback please.

Especially a documenter/tester with a fresh eye on the things I know
now would be great. Same applies for testing. The testing done for
the pre-github proof-of-concept by Benjamin Root must be noted here,
it made flaws and side-tracks apparent I either didn't know about or
did overlook. Documenting and testing of the technical things is up
to me, only the interface documentation and testing boredom.

Last time I kept the project secret, I'm not gonna do commit that mistake again.

Without matplotlib, lastly, also this patch would simply not exist,
neither in my mind nor in reality.

cu,
Friedrich

P.S.: I keep the right of dropping the project at any time. I'll do
my diploma next monthes, so time might be short. But I think it might
serve its purpose.

3

2011/10/20 Eric Firing <efiring@...202...>:

Friedrich (or others),

I hope you will consider the following to be constructive.

I agree :-/ It was very very late in the night [5:30 a.m.], so things
happen, but I think that's no kind of excuse for my rude tone in that
email. I'm sorry.

I understand one use case for a gray switch: sometimes one needs to make
gray versions of plots for formal publication, but one wants to have
nicer color versions for other purposes.

Exactly, but I believe when we have the functionality the use cases
will just start to emerge - like it always is.

For instance, but this is just guessing, someone came up with the will
to write out his figure in CMYK. Others might want to turn black into
white and vice versa, keeping other colours invariant.

What is the use case for grayscale support for parts of the Figure? It
seems to me that this requires a much more complex and extensive set of
changes than a switch for the whole Figure. Is it really worth it?

I think yes.

It is a bit of effort, yes.

You indicate that some caching will need to be eliminated. Is this
correct? If your branch causes a significant loss of speed in
displaying and interacting with large data sets, then I will oppose
merging it. Speed matters for a significant fraction of mpl usage.

Correct. I think it is measurable, but only of O(2). I believe this
kind of fluctuation can be called noise :slight_smile:

The tests cannot esily resolve this because they do everything once.

And with a bit of work I might be able to keep the caching.

Improvements in mpl architecture that maintain or increase speed, and
that make it more comprehensible, maintainable, and capable, are
certainly welcome. Color handling has grown organically, often
bedeviled by that pesky alpha, and I have no doubt that it could be
cleaned up and improved.

I can capably only speak of colors.py and some conversion functions
somewhere else. I've noticed a rather small-grained approach. I
tried to unify things.

But this you already said.

cu,
Friedrich

4

Friedrich,

Great to hear from you again!

I have been doing some thinking about this problem and how it could relate to issues I have encountered in my work on mplot3d. The key problem I encounter there is that shading of the colors needs to be performed based on perspective (rotation of the scene and such).

In many ways, this is a similar problem to wanting b&w versions of graphs.

The problem I have had in addressing this is that there are two primary ways to specify color in mpl. First, you can specify it directly with strings or rgb(a) tuples. Second, you can specify it through a ScalarMappable (eg, colormap). This makes for two very different code paths to account for, and it can often be difficult to detect which path is needed.

So, what if we unify everything under ScalarMappable? The use case of directly specifying the color could be envisioned as the color spec being the data applied to an “identity colormap”.

Then, we extend ScalarMappable such that a chain of transforms could be performed on the rgba array. I envision this to be very similar to how the projections framework operates. We can then have a set of stock transforms that users could select at will (such as color to b&w).

We can use the lessons and successes of the projections framework to guide our design here.

Thoughts?
Ben Root

···

On Thursday, October 20, 2011, Friedrich Romstedt <friedrichromstedt@…287…> wrote:

2011/10/20 Eric Firing <efiring@…202…>:

Friedrich (or others),

I hope you will consider the following to be constructive.

I agree :-/ It was very very late in the night [5:30 a.m.], so things
happen, but I think that’s no kind of excuse for my rude tone in that

email. I’m sorry.

I understand one use case for a gray switch: sometimes one needs to make
gray versions of plots for formal publication, but one wants to have
nicer color versions for other purposes.

Exactly, but I believe when we have the functionality the use cases
will just start to emerge - like it always is.

For instance, but this is just guessing, someone came up with the will

to write out his figure in CMYK. Others might want to turn black into
white and vice versa, keeping other colours invariant.

What is the use case for grayscale support for parts of the Figure? It

seems to me that this requires a much more complex and extensive set of
changes than a switch for the whole Figure. Is it really worth it?

I think yes.

It is a bit of effort, yes.

You indicate that some caching will need to be eliminated. Is this
correct? If your branch causes a significant loss of speed in
displaying and interacting with large data sets, then I will oppose

merging it. Speed matters for a significant fraction of mpl usage.

Correct. I think it is measurable, but only of O(2). I believe this
kind of fluctuation can be called noise :slight_smile:

The tests cannot esily resolve this because they do everything once.

And with a bit of work I might be able to keep the caching.

Improvements in mpl architecture that maintain or increase speed, and

that make it more comprehensible, maintainable, and capable, are
certainly welcome. Color handling has grown organically, often
bedeviled by that pesky alpha, and I have no doubt that it could be

cleaned up and improved.

I can capably only speak of colors.py and some conversion functions
somewhere else. I’ve noticed a rather small-grained approach. I
tried to unify things.

But this you already said.

cu,
Friedrich

5

2011/10/20 Benjamin Root <ben.root@...1304...>:

I have been doing some thinking about this problem and how it could relate
to issues I have encountered in my work on mplot3d. The key problem I
encounter there is that shading of the colors needs to be performed based on
perspective (rotation of the scene and such).

In many ways, this is a similar problem to wanting b&w versions of graphs.

The problem I have had in addressing this is that there are two primary ways
to specify color in mpl. First, you can specify it directly with strings or
rgb(a) tuples. Second, you can specify it through a ScalarMappable (eg,
colormap). This makes for two very different code paths to account for, and
it can often be difficult to detect which path is needed.

You are facing the OO design of Mpl. I believe they cannot be solved
in mpl, only remedied partially. This is the reason why I don't want
to get entangled with mpl.

So, what if we unify everything under ScalarMappable? The use case of
directly specifying the color could be envisioned as the color spec being
the data applied to an "identity colormap".

No, this would hinder further differentiation. You just gave the
example: Shading. "Uniting", i.e. seeing everything as a
ScalarMappable, would make it impossible without further "unification"
to take two parameters into account: Height + Viewing angle. But the
problem exists, as I said it stems from the Oo approach.

Then, we extend ScalarMappable such that a chain of transforms could be
performed on the rgba array. I envision this to be very similar to how the
projections framework operates. We can then have a set of stock transforms
that users could select at will (such as color to b&w).

Exactly. But this is a functional approach. Isn't it?

We can use the lessons and successes of the projections framework to guide
our design here.

I strongly believe, although I don't know the projections framework,
that is has functional ideas strongly expressed in the way it works.
Is that true?

I just had a walk and tapped in my thoughts into my notebook device
while walking, this is some way of meditation. So here they come (The
postscripta are now-thoughts):

1. There are two different ways of specifying a colour: Either a
thing is simply red or you need something that spits out a colour when
the specification is given. Uniting both under the
spit-out-with-specification constraint hinders further
differentiation. P.S.: In fact, it would be better if the "red" spits
out a colour when called with *nothing*. But hey, this is functional.

2. In fact I believe the Oo approach does not work well for frame
generation (and plotting is a frame generation). This is why I coded
moviemaker using functional paradigm. It just didn't work out the Oo
way - just because you needed a new class for each type of plotting
directive; you have to pass everything through. The functional
approach solved my headaches - Classes are too rigid for plotting.

3. I don't want to "pollute" my mind with the Oo ideas of mpl; rather
I would like to learn to avoid the mistakes made here.

4. Your idea is a functional one.

5. In functional programming, each object (i.e. each function) has
one entry point, in Python this is implemented by the __call__()
method.

···

-

I can give further details but it doesn't matter here, since we're
talking to give mpl grayscale and filter properties, right. I will
try to dig out my old private ideas I sent you some monthes ago - I
believe I already found the solution for our matter here. (i.e.
avoiding rcParams and stuff.)

I believe the one and only solution would, if thought thru completely,
unveil that we need functional approach to get better results. In
matplotlib, each type of structure is an object of a class, which is
not callable mostly, and this means, you have to formulate each
alteration explicitly. It's not easy in this frame of thinking to get
the flexibility we in fact want - this might be the reason why mpl is
so monolithic with all its different kind of Artist objects ... with
the counterparts in the Renderer objects and so on.

It's hard to keep a common style when differentiating things in
plotting using the Oo approach.

This is just what we see in fact. We feel that things are somehow
united and should work better together and be more exchangable, but
they aren't, because it has never been formulated in terms of code, of
a common base class Function, as I would do it (and did it in
moviemaker). Addendum: It has been attempted with the Artist class,
with the .render() method. I cannot judge how well it worked out, but
to my understanding, it didn't foster plug-in architecture much. ADD:
I think this is because .render() happens on high-level, so it's not
attractive for data-munging objects. The functional approach is
attractive there.

This can be understood as some advertisement for plotlayers, but
believe me, I have coded not a single line of it so far. There exists
moviemaker3, though, and that one works fine. What's missing are
things like polygons and fast drawing routines for scaled pixmaps.
I'll not go into details here.

For me it's hard to stay focussed on mpl here.

I will try to dig out that emails.

Friedrich

6

I want to do some corrections and alterations to my last mail, because
it troubles me.

2011/10/21 Friedrich Romstedt <friedrichromstedt@...287...>:

I believe the one and only solution would, if thought thru completely,
unveil that we need functional approach to get better results. In
matplotlib, each type of structure is an object of a class, which is
not callable mostly, and this means, you have to formulate each
alteration explicitly. It's not easy in this frame of thinking to get
the flexibility we in fact want - this might be the reason why mpl is
so monolithic with all its different kind of Artist objects ... with
the counterparts in the Renderer objects and so on.

I in fact think the "one and only" solution is a very dangerous deceit.

There might be some advantages of a functional approach, but there are
also severe drawbacks, for instance, that you get rubbish tracebacks
if something goes wrong.

This I just wanted to say to prevent piping errors.

Friedrich

7

2011/10/21 Friedrich Romstedt <friedrichromstedt@...287...>:

I will try to dig out that emails.

Did that, the email I meant dates back to 10 November 2010! Here's the snippet:

<snippet>

(Ben Root):

I am curious, could this approach you have done be generalized to any sort
of color transformation? Admittedly, a gray mode is probably the most
common request, but what if someone wants a different transformation? Maybe
even apply a filter that would produce sepia colors, or high-contrast
colors, or a different grayscale transform? Heck, I could imagine a use
where a user might want to do a color substitution?

Oh Yes, this is *ealily* possible. The new framework in the
ColorConverter class just uses a filter function, if we want to see it
like that, already. It's just the apply_rc_gray_setting() function or
sth like that. If you want to, you can try to add the functionality.
But we should discuss beforehand how to design it. There are several
possibilites. In fact, I like this filter function quite a lot!

1) Hardcoding other filters in ColorConverter (what a decent name!),
and switching them on or off
2) Inserting filters as functions on runtime into the ColorConverter
instance, some hooking mechanism
3) Providing a dedicated Filter class, that can be fed to
set_filter() instead of set_gray(). This I like best.

I will, when i find some time, first implement the propagation of gray
settings by allowing objects in set_gray(). Might be a good time to
rename it to set_filter() right away, or maybe not do it, if
set_gray() goes in, and expects a bool, it might break compat when
changing the argument spec later. So later set_gray() would just call
set_filter() or add_filter() or whatever.

</snippet>

So the filter idea was Ben's idea. I still like idea (3) for the
design best. I will check if it is possible to inject the filter code
into the renderer framework, since all colour settings arguments
somewhen flow into a call into the renderer. Pro: No rcParams
addition necessary, no modification of the peculiar colors.py
ColorConverter framework necessary, just leaving those things
untouched and hence no worries and headaches about them. No disabling
of higher-level caching and at the same time negligible small impact
if there is no filter active. Con: I don't know yet if it works. I
vaguely remember there were some problems when I checked that
possibility last time (with pixmaps or something like that). I think
I will find out soon enough.

Eric, Ben, do you think we should go off-list with this now? I would
prefer on-list now. There might be people following but not
responding.

Friedrich