rcParams and validation

Development matplotlib-devel
1

I've been thinking a bit about rcParams and validation. It looks like values
are currently only validated when matplolibrc is read, during the call to
rc_params. What if we define a new class (RcParams), derived from dict, which
has as an attribute, a dict, called validate. We could override __setitem__
such that

rcParams['font.size'] = 12

validates the value 12 by first calling validate_float, which is referenced in
the validate attribute:

    def __setitem__(self, i='font.size', val=12)
        try:
            self.validate[i](val)
            dict.__setitem__(self, i, val)
        except KeyError:
            raise (or warning), bad value

the validation dict and default properties would be populated during the
initial call to rc_params(), when the defaultParams dict is interpretted.
Thereafter, any attempt to set a parameter, either directly using
rcParams[i]=j or indirectly using rc(), will benefit from validation. The
behavior would otherwise be unchanged, I think. Any comments or objections?

Darren

2

This is a good idea, and it is also a good idea to validate the key.
I have wasted significant time before trying to track down a "bug"
when I thought usetex wasn't being enabled when all I had done was
mistype the rc key. I suggest a set of valid keys for fast lookup.

JDH

···

On 7/16/07, Darren Dale <dd55@...143...> wrote:

the validation dict and default properties would be populated during the
initial call to rc_params(), when the defaultParams dict is interpretted.
Thereafter, any attempt to set a parameter, either directly using
rcParams[i]=j or indirectly using rc(), will benefit from validation. The
behavior would otherwise be unchanged, I think. Any comments or objections?

3

Darren Dale wrote:

I've been thinking a bit about rcParams and validation. It looks like values are currently only validated when matplolibrc is read, during the call to rc_params. What if we define a new class (RcParams), derived from dict, which has as an attribute, a dict, called validate. We could override __setitem__ such that

rcParams['font.size'] = 12

validates the value 12 by first calling validate_float, which is referenced in the validate attribute:

    def __setitem__(self, i='font.size', val=12)
        try:
            self.validate[i](val)
            dict.__setitem__(self, i, val)
        except KeyError:
            raise (or warning), bad value

the validation dict and default properties would be populated during the initial call to rc_params(), when the defaultParams dict is interpretted. Thereafter, any attempt to set a parameter, either directly using rcParams[i]=j or indirectly using rc(), will benefit from validation. The behavior would otherwise be unchanged, I think. Any comments or objections?

I agree with John that the basic idea of validating rc keys and values regardless of whether they are set directly or via rc() is an important improvement to make. I think I understand the general idea of your proposed implementation. It looks like it could be done quite quickly and easily with no disruption elsewhere in the code. Alternative approaches that have been mentioned in the past involve properties or traits; but your proposed implementation may actually be cleaner, simpler, and more maintainable for present purposes. Have you compared alternatives? Have you looked at examples/rc_traits.py? (Or did you write it? I don't recall for sure, but I think John wrote it after one of our earlier discussions of traits/properties/neither.)

Eric

4

Darren Dale wrote:
> I've been thinking a bit about rcParams and validation. It looks like
> values are currently only validated when matplolibrc is read, during the
> call to rc_params. What if we define a new class (RcParams), derived from
> dict, which has as an attribute, a dict, called validate. We could
> override __setitem__ such that
>
> rcParams['font.size'] = 12
>
> validates the value 12 by first calling validate_float, which is
> referenced in the validate attribute:
>
> def __setitem__(self, i='font.size', val=12)
> try:
> self.validate[i](val)
> dict.__setitem__(self, i, val)
> except KeyError:
> raise (or warning), bad value
>
> the validation dict and default properties would be populated during the
> initial call to rc_params(), when the defaultParams dict is interpretted.
> Thereafter, any attempt to set a parameter, either directly using
> rcParams[i]=j or indirectly using rc(), will benefit from validation. The
> behavior would otherwise be unchanged, I think. Any comments or
> objections?

I agree with John that the basic idea of validating rc keys and values
regardless of whether they are set directly or via rc() is an important
improvement to make. I think I understand the general idea of your
proposed implementation. It looks like it could be done quite quickly
and easily with no disruption elsewhere in the code.

That was my impression as well, it looks like it would fit well with the
existing code.

Alternative
approaches that have been mentioned in the past involve properties or
traits; but your proposed implementation may actually be cleaner,
simpler, and more maintainable for present purposes. Have you compared
alternatives? Have you looked at examples/rc_traits.py? (Or did you
write it? I don't recall for sure, but I think John wrote it after one
of our earlier discussions of traits/properties/neither.)

John wrote rc_traits.py, before numpy was around, by the looks of it. Traits
seem more appealing to me than properties, but I was looking for something
that could be done outside of a chainsaw branch. If we decided on traits, we
should also try to do something about the rc file format, so we dont have to
parse and convert strings before validating. Someone had suggested python
literals, I think John or Fernando might have had some ideas about this at
one point... I dont know the details.

Darren

···

On Monday 16 July 2007 7:22:37 pm you wrote:

5

In the new ipython work, we went for pure-python configuration files.
Given how ipython (and mpl) is only of use to python programmers, we
didn't see much the benefit of shielding users from the python syntax,
and certainly lots of downsides to it.

Obviously using a full programming language in a config file opens up
potential complexities and problems, but I think for ipython it was
the right choice.

Cheers,

f

···

On 7/16/07, Darren Dale <dd55@...143...> wrote:

John wrote rc_traits.py, before numpy was around, by the looks of it. Traits
seem more appealing to me than properties, but I was looking for something
that could be done outside of a chainsaw branch. If we decided on traits, we
should also try to do something about the rc file format, so we dont have to
parse and convert strings before validating. Someone had suggested python
literals, I think John or Fernando might have had some ideas about this at
one point... I dont know the details.

6

Hey Darren,

Isn't there a potential problem here? The original validate funcs
support conversion from a string to a value, but you are proposing
using them here in a context where users will generally be supplying a
(possibly bogus) value, but in general not a string. So the existing
funcs may not work wholesale, but might be easily adapted.

As for key validation, I was only suggesting you raise a helpful error
message if the user supplies a bogus key.

As for traits, I think we are psychologically committed to them, and
if you are looking for a good summertime project, this might be a good
candidate. It's not really a chainsaw question, because you could
easily support traits in the Artist layer and rc layer w/o ripping out
the fundamental organization, and we could write setter and getter
support as thin warppers around traits to avoid significant API
breakage. There are deeper and more fundamental chainsaw like layers
where traits would help us (eg transform updates on window resizes)
but a clear cut first step would be to get the Artist properties
traitified.

If you decide to go that route, let's sync up with the latest
enthought tree first, though.

JDH

JDH

···

On 7/16/07, Darren Dale <dd55@...143...> wrote:

John wrote rc_traits.py, before numpy was around, by the looks of it. Traits
seem more appealing to me than properties, but I was looking for something
that could be done outside of a chainsaw branch. If we decided on traits, we
should also try to do something about the rc file format, so we dont have to
parse and convert strings before validating. Someone had suggested python
literals, I think John or Fernando might have had some ideas about this at
one point... I dont know the details.

7

John Hunter wrote:
[...]

Isn't there a potential problem here? The original validate funcs
support conversion from a string to a value, but you are proposing
using them here in a context where users will generally be supplying a
(possibly bogus) value, but in general not a string. So the existing
funcs may not work wholesale, but might be easily adapted.

What about Fernando's ipython strategy: parse the rc file as python code, not as strings? I don't think the changeover would be very hard; most of the present matplotlibrc file is commented out anyway, so unless users are doing lots of customization it would not be difficult for them to generate a matplotlibrc file in python form.

As for key validation, I was only suggesting you raise a helpful error
message if the user supplies a bogus key.

As for traits, I think we are psychologically committed to them, and
if you are looking for a good summertime project, this might be a good
candidate. It's not really a chainsaw question, because you could

That's why I brought it up--although every time I look at traits I pull back, worrying about the very large amount of machinery they involve. Are any real, live projects outside of enthought making major use of traits? Or would we be the first?

easily support traits in the Artist layer and rc layer w/o ripping out
the fundamental organization, and we could write setter and getter
support as thin warppers around traits to avoid significant API
breakage. There are deeper and more fundamental chainsaw like layers
where traits would help us (eg transform updates on window resizes)
but a clear cut first step would be to get the Artist properties
traitified.

Actually, it could be completely split into two phases: first rcParams handling, second Artist properties, couldn't it? Starting with rc would be nice because it would provide a gentler introduction and some experience; but I think that using traits for Artist properties can also be done piecemeal, so it shouldn't be too disruptive.

If you decide to go that route, let's sync up with the latest
enthought tree first, though.

I am happy to see this in svn:

  Revision 12335, 13.4 kB (checked in by rkern, 4 weeks ago)
Array trait updated to use numpy idioms only.

Eric

8

Are any real, live projects outside of enthought making major use of
traits? Or would we be the first?

I am happy to be the first at this point -- enthought has done a lot
to support traits. Traits has one of the most impressive pieces of
technical documentation in the scientific python community. The
enthought mailing list has become quite active of late, and they are
clearly supporting their OS code. They actively promote their product
and want a large user community -- we can provide synergy there. It
has lived in our src tree for over a year and still "just compiles".
  In fact, I am amenable too, though not committed to, requiring
traits as an *external* package rather than an included package, both
to encourage users into the ETS suite and to encourage enthought to
support us via their traits package. I'd be happy to hear from
enthought here on their preferences. With the whole Numeric, numarray
and numpy thing behind us, I'm looking for a whole new set of
compilation issues to tackle :slight_smile:

Actually, it could be completely split into two phases: first rcParams
handling, second Artist properties, couldn't it? Starting with rc would
be nice because it would provide a gentler introduction and some
experience; but I think that using traits for Artist properties can also
be done piecemeal, so it shouldn't be too disruptive.

True, but there aren't that many Artists, and they are closely tied to
the rc params. Once you get the rc done, it will be natural and easy
to do the artists. But yes, you can do them piecemeal: Line2D and
Text are natural first targets.

> If you decide to go that route, let's sync up with the latest
> enthought tree first, though.

I am happy to see this in svn:

  Revision 12335, 13.4 kB (checked in by rkern, 4 weeks ago)
Array trait updated to use numpy idioms only.

Anything that gets Robert making commits to our tree is a massive win for us.

JDH

···

On 7/16/07, Eric Firing <efiring@...229...> wrote:

9

John Hunter wrote:

Are any real, live projects outside of enthought making major use of
traits? Or would we be the first?

Yes. Most are in the somewhat formative stages, so you may not think they count
(which is fine).

I am happy to be the first at this point -- enthought has done a lot
to support traits. Traits has one of the most impressive pieces of
technical documentation in the scientific python community. The
enthought mailing list has become quite active of late, and they are
clearly supporting their OS code. They actively promote their product
and want a large user community -- we can provide synergy there. It
has lived in our src tree for over a year and still "just compiles".
  In fact, I am amenable too, though not committed to, requiring
traits as an *external* package rather than an included package, both
to encourage users into the ETS suite and to encourage enthought to
support us via their traits package. I'd be happy to hear from
enthought here on their preferences.

We've split up the main "enthought" package such that Traits can be installed
separately as "enthoguht.traits". I think we'd prefer depending on it externally
now that we've spent so much effort to make that feasible.

I am happy to see this in svn:

  Revision 12335, 13.4 kB (checked in by rkern, 4 weeks ago)
Array trait updated to use numpy idioms only.

Anything that gets Robert making commits to our tree is a massive win for us.

Flattery will get you nowhere.

···

On 7/16/07, Eric Firing <efiring@...229...> wrote:

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
  -- Umberto Eco

10

Robert Kern wrote:
[...]

We've split up the main "enthought" package such that Traits can be installed
separately as "enthoguht.traits". I think we'd prefer depending on it externally
now that we've spent so much effort to make that feasible.

When do you expect to make a release?

Eric

11

I am very happy to hear this. I have been using traits for lab work, and
it has been an absolute pleasure (ask Fernando, he has heard me express
my satisfaction over and over). I have converted a friend, here, in
France to traits for some software his company is developing, and every
body has been amazed at the results.

Traits come with added benefits than validation:

* Traits handlers, they allow nice event driven programming in a light
way. Think of it this way: you have a mappable artist, its colormap is a
traits, you have a handler triggered when you change this colormap that
automatically redraw the axes. If you use this pattern heavily, the
drawing package becomes much nicer to use we integrating in an
interactive program.

* TraitsUI. Once everything is traited, and each artist has its
properties discribed by traits, you edit these properties graphically,
traits can create automatically dialogs for them
(see http://code.enthought.com/traits/ for an example with screenshots).
Now this is Wx only, so far, so it would be very limited. However
graphical edition of figures and artists is a feature my colleagues ask
often, and having traits would make it one step closer.

My 2 cents,

Ga�l

···

On Mon, Jul 16, 2007 at 10:31:03PM -0500, John Hunter wrote:

On 7/16/07, Eric Firing <efiring@...229...> wrote:

> Are any real, live projects outside of enthought making major use of
> traits? Or would we be the first?

I am happy to be the first at this point -- enthought has done a lot
to support traits. Traits has one of the most impressive pieces of
technical documentation in the scientific python community. The
enthought mailing list has become quite active of late, and they are
clearly supporting their OS code. They actively promote their product
and want a large user community -- we can provide synergy there. It
has lived in our src tree for over a year and still "just compiles".
  In fact, I am amenable too, though not committed to, requiring
traits as an *external* package rather than an included package, both
to encourage users into the ETS suite and to encourage enthought to
support us via their traits package. I'd be happy to hear from
enthought here on their preferences. With the whole Numeric, numarray
and numpy thing behind us, I'm looking for a whole new set of
compilation issues to tackle :slight_smile:

12

Gael Varoquaux wrote:

  

I am happy to be the first at this point -- enthought has done a lot
to support traits. Traits has one of the most impressive pieces of
technical documentation in the scientific python community.
    
I am very happy to hear this. I have been using traits for lab work, and
it has been an absolute pleasure (ask Fernando, he has heard me express
my satisfaction over and over). I have converted a friend, here, in
France to traits for some software his company is developing, and every
body has been amazed at the results.
  

I am happy to see this thread as well. In a conversation with Perry Greenfield here yesterday, we both agreed that adding traits to matplotlib would be a good thing to devote some of our resources to. If this thread has already fired others up, don't let us stop you, but let's coordinate efforts if possible. The motivation from an STScI point of view is how it may make reworking the transforms system easier and more powerful. It's also good to hear the positive reports about enthought Traits.

I think the proposed roadmap makes sense: to start simple with rcParams and Artist and broaden out from there. I also see there being (at least) two phases in how it is applied: first for data validation, and second taking advantage of "trait notification" where appropriate.

TraitsUI seems really cool, but there are a couple of reasons I think that should probably be considered lower priority. For one, it would need to be generalized and ported (backend-ed) for all of matplotlib's many gui backends. Also, I wonder how it fits into matplotlib's paradigm as somewhere in between interactive and noninteractive plotting. You would almost certainly want to tweak traits with the UI and then save that back out as Python code, but code generators almost never generate code that a human being would want to edit.

Cheers,
Mike

···

On Mon, Jul 16, 2007 at 10:31:03PM -0500, John Hunter wrote:

13

I was only mentionning TraitsUI as an option when integrating matplotlib
into something larger, as a plotting engine (think something like
QME-Dev Workbench (wxSciPy) download | SourceForge.net, for instance, or some domain
specific application).

Of course I also have in mind an IDE, written in WxPython (most probably
using enthought's envisage framework), with Ipython1 as the commandline
engine, and MPL as the plotting engine. So far this is pure sci-fi,
because we don't have the resources to write it, but it is nice if pieces
start falling together.

Ga�l

···

On Tue, Jul 17, 2007 at 08:01:41AM -0400, Michael Droettboom wrote:

TraitsUI seems really cool, but there are a couple of reasons I think
that should probably be considered lower priority. For one, it would
need to be generalized and ported (backend-ed) for all of matplotlib's
many gui backends. Also, I wonder how it fits into matplotlib's
paradigm as somewhere in between interactive and noninteractive
plotting. You would almost certainly want to tweak traits with the UI
and then save that back out as Python code, but code generators almost
never generate code that a human being would want to edit.

14

I will start working on this today. I am a little surprised by how much
interest has grown since the last time it came up on the mailing list.

John, Eric, have you had a look at the way IPython1 handles config files?
Here's a taste:
------------ configfiles/ipcontroller.py --------------
from ipython1.config.api import getConfigObject

controllerConfig = getConfigObject('controller')

# Now we can configure the controller

controllerConfig.listenForEnginesOn['ip'] = '127.0.0.1'
controllerConfig.listenForEnginesOn['port'] = 20000

···

On Tuesday 17 July 2007 08:01:41 am Michael Droettboom wrote:

Gael Varoquaux wrote:
> On Mon, Jul 16, 2007 at 10:31:03PM -0500, John Hunter wrote:
>> I am happy to be the first at this point -- enthought has done a lot
>> to support traits. Traits has one of the most impressive pieces of
>> technical documentation in the scientific python community.
>
> I am very happy to hear this. I have been using traits for lab work, and
> it has been an absolute pleasure (ask Fernando, he has heard me express
> my satisfaction over and over). I have converted a friend, here, in
> France to traits for some software his company is developing, and every
> body has been amazed at the results.

I am happy to see this thread as well. In a conversation with Perry
Greenfield here yesterday, we both agreed that adding traits to
matplotlib would be a good thing to devote some of our resources to. If
this thread has already fired others up, don't let us stop you, but
let's coordinate efforts if possible. The motivation from an STScI
point of view is how it may make reworking the transforms system easier
and more powerful. It's also good to hear the positive reports about
enthought Traits.

I think the proposed roadmap makes sense: to start simple with rcParams
and Artist and broaden out from there. I also see there being (at
least) two phases in how it is applied: first for data validation, and
second taking advantage of "trait notification" where appropriate.

--------------------------------------------------------------
and then
-------- ipython1/scripts/ipcontroller.py ----------
[...]
def main():
[...]
    config.updateConfigWithFile('ipcontrollerrc.py', options.ipythondir)
[...]
--------------------------------------------------------------

I'm really impressed with how readable and well organized the code is in
ipython1. It looks like their approach to configuration has been carefully
considered. Any chance we can follow their lead? It looks like it would be a
good fit: we would define the config settings (and defaults) with traits
somewhere like config.objects, import them in the rc file to be customized. I
don't think we could automatically generate rc files with the ipython scheme,
but thats probably not too important. In ipython's scheme, the config files
are loaded using execfile, I wonder if that might appear unsafe to anyone?
Fernando, did you have any concerns about using execfile?

Darren

15

I should clarify that. I think we could still generate rc files from the
template, modifying the backend based on what toolkits are installed on the
user's system. There was a comment in the rcdefaults code mentioning the
possibility of generating the template from the defaultParams dict. That
might be tricky.

Darren

···

On Tuesday 17 July 2007 09:15:15 am Darren Dale wrote:

I'm really impressed with how readable and well organized the code is in
ipython1. It looks like their approach to configuration has been carefully
considered. Any chance we can follow their lead? It looks like it would be
a good fit: we would define the config settings (and defaults) with traits
somewhere like config.objects, import them in the rc file to be customized.
I don't think we could automatically generate rc files with the ipython
scheme, but thats probably not too important.

16

I haven't looked at it closely, but I am willing to trust Fernando on
this for the most part. I know he has put a lot of thought into it,
and it's something we've talked about for years.

I am not too fond of the dictionary usage here:

controllerConfig.listenForEnginesOn['ip'] = '127.0.0.1'
controllerConfig.listenForEnginesOn['port'] = 20000

I prefer

controllerConfig.listenForEnginesOn.ip = '127.0.0.1'
controllerConfig.listenForEnginesOn.port = 20000

Since dicts and attrs are intimately linked, eg with something like
Bunch, this should be a minor tweak. Fernando, why did you prefer
dict semantics. And are you happy with the state of your config
system in ipython1

Speaking of branches, we may need to seriously consider a branch here,
mpl1. The changes here may involve breaking a fair amount of code,
which I don't mind doing to get it right, but I'm somewhat inclined to
branch off here for matplotlib1.0, with the goal of using traits,
fixing axis handling (eg multiple y-axis with arbitrary placement),
and rewriting the transforms. I think it would be better to break a
bunch of stuff all at once for a 1.0 release, than to break things
incrementally with successive releases. here are significant pieces
of matplotlib code -- I've written some of them in past lives and I
won't be porting them to mpl1 - and we should probably try to
maintain a stable and bugfixed release of our current tree as we begin
to hack it.

We should also consider getting out a 0.91 release as soon as we can
finish the numpification, because Michael has done a lot of good work.
So much to do....

···

On 7/17/07, Darren Dale <dd55@...143...> wrote:

I'm really impressed with how readable and well organized the code is in
ipython1. It looks like their approach to configuration has been carefully
considered. Any chance we can follow their lead? It looks like it would be a

17

I'd like to add the dict-based validation scheme for 0.91. It should be quick,
and then I can focus on traits and a new config scheme, in an mpl1 branch if
we decide that is best.

···

On Tuesday 17 July 2007 09:33:47 am John Hunter wrote:

We should also consider getting out a 0.91 release as soon as we can
finish the numpification, because Michael has done a lot of good work.
So much to do....

18

Sounds like a good plan.

···

On 7/17/07, Darren Dale <dd55@...143...> wrote:

I'd like to add the dict-based validation scheme for 0.91. It should be quick,
and then I can focus on traits and a new config scheme, in an mpl1 branch if
we decide that is best.

19

As of svn 3552, rc parameters are validated. Validation occurs in the rcParams
dict itself, so settings are validated whether loaded from matplotlibrc, or
with the rc() function, or by directly accessing rcParams itself.

Validation may uncover some hidden bugs. conoutf_demo.py had an argument
hold='on', which wouldnt work before now because 'on' was not a valid boolean
(it is a valid boolean now). But backend_driver runs without errors, but I
wonder if there will be some hidden problems. For example, polar_demo.py
runs, but polar() is not plotting lines.

Darren

···

On Tuesday 17 July 2007 11:22:29 am John Hunter wrote:

On 7/17/07, Darren Dale <dd55@...143...> wrote:
> I'd like to add the dict-based validation scheme for 0.91. It should be
> quick, and then I can focus on traits and a new config scheme, in an mpl1
> branch if we decide that is best.

Sounds like a good plan.

20

Darren Dale wrote:

John, Eric, have you had a look at the way IPython1 handles config files? Here's a taste:

In ipython's scheme, the config files are loaded using execfile, I wonder if that might appear unsafe to anyone?

This is, of course, terribly unsafe, but does anyone have a use case where that matters? I don't. I use python files for config all the time, and I love it! They are very readable, and I don't have to write a parser or document the syntax myself. I usually use import, rather than execfile, though I couldn't tell you why.

I have thought about the safety issue. One idea I've had (though I never bothered with it) was to strip the input files of "import" lines first. You could do a whole lot less if you couldn't import any arbitrary modules.

I think it would be better to break a
bunch of stuff all at once for a 1.0 release, than to break things
incrementally with successive releases

+1

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...236...