rcParams and validation

I also just (r2528) updated things to use a cleaner import convention:
I don't like importing things from modules *they* imported.

[...]

The point is that instead of

-from mplconfig import MPLConfig, ConfigManager, mkConfigObj

I prefer:

+from tconfig import ConfigManager, mkConfigObj
+from mplconfig import MPLConfig

since ConfigManager and mkConfigObj really live in tconfig, I'm not
tying my top-level code to an implementation detail of an intermediate
module (what it imports and how it names it). I mention this
explicitly just because I think it's a good bit of coding practice.

I was thinking along the lines of your config/api.py file, a single entry
point to the ipython or mpl config API.

···

On Friday 20 July 2007 05:25:06 pm Fernando Perez wrote:

Ah, OK. Since we haven't really organized such an api yet, I was
still thinking of the individual components. I see your point.

Cheers,

f

···

On 7/20/07, Darren Dale <dd55@...143...> wrote:

On Friday 20 July 2007 05:25:06 pm Fernando Perez wrote:
> I also just (r2528) updated things to use a cleaner import convention:
> I don't like importing things from modules *they* imported.
[...]
> The point is that instead of
>
> -from mplconfig import MPLConfig, ConfigManager, mkConfigObj
>
> I prefer:
>
> +from tconfig import ConfigManager, mkConfigObj
> +from mplconfig import MPLConfig
>
> since ConfigManager and mkConfigObj really live in tconfig, I'm not
> tying my top-level code to an implementation detail of an intermediate
> module (what it imports and how it names it). I mention this
> explicitly just because I think it's a good bit of coding practice.

I was thinking along the lines of your config/api.py file, a single entry
point to the ipython or mpl config API.

That really depends whether you what mplconfig to be able to provide its
own abstract layer over tconfig, with possibly modified ConfigManager and
mkConfigObj (remember EZvisage). I don't think it is the case here, but I
kust wanted to point out that I am not sure this is allways good coding
practice.

My 2 euro cents,

Ga�l

···

On Fri, Jul 20, 2007 at 03:25:06PM -0600, Fernando Perez wrote:

The point is that instead of

-from mplconfig import MPLConfig, ConfigManager, mkConfigObj

I prefer:

+from tconfig import ConfigManager, mkConfigObj
+from mplconfig import MPLConfig

since ConfigManager and mkConfigObj really live in tconfig, I'm not
tying my top-level code to an implementation detail of an intermediate
module (what it imports and how it names it). I mention this
explicitly just because I think it's a good bit of coding practice.

I'm working on converting our existing rc code to tconfig this weekend. So far
so good. I just wanted to let people know to avoid duplicating effort.

Darren

···

On Friday 20 July 2007 5:13:12 pm Fernando Perez wrote:

On 7/20/07, Darren Dale <dd55@...143...> wrote:
> > This should give you a very reasonable solution for what you wanted to
> > do.
>
> Great! I'm surprised that this was possible.

And actually easy, for once.

> Im attaching a patch for the ipython1 sandbox, which adds configobj.py,
> moves the ipython and mpl config code into their own .py files, and moves
> the test code into mpltest.py and ipythontest.py.

Great, thanks! Committed as r2526. Later we'll move configobj.py
into ipython/externals, but for now it's OK to have it in, so it's
nicely self-contained for people to test this while we settle on
something long-term.

I've moved the lot into a subdirectory now, so that we can have more
than one set of toys in our sandbox, so update to 2527 and you'll have
tconfig as a self-contained project we can play with until happiness
arrives.

Excellent! Ping me if you have any problems.

I'm going to try and finish hierarchical inclusion of files (aka
ipython profiles) now. Consider for example a paper that requires
black and white only styles. You could then have in that paper's
directory a .matplotlib.conf file with:

# paper-specific tweaks
include = "$HOME/.matplotlib/matplotlib.conf"

[lines]
color = 'black'
thickness = 2

etc.

Or in your .matplotlib dir, you could keep multiple configurations
that branch off the basic one for different needs, while keeping
common defaults in one central location.

In my first implementation I'll have to give up on safe roundtripping
to files with recursion active, because it's a bit tricky to do both
(not impossible).

Cheers,

f

···

On 7/21/07, Darren Dale <dd55@...143...> wrote:

I'm working on converting our existing rc code to tconfig this weekend. So far
so good. I just wanted to let people know to avoid duplicating effort.

> I'm working on converting our existing rc code to tconfig this weekend.
> So far so good. I just wanted to let people know to avoid duplicating
> effort.

Excellent! Ping me if you have any problems.

Im attaching a patch, it includes lots of changes to mplconfig.py, a realistic
mplrc.conf file (feel free to rename it to mpl.conf or matplotlib.conf or
whatever seems standard), and a backup of that .conf file. The mpltest lets
me try your trick of modifying a well formatted file in place. Very nice!

A couple comments:

1) Comments in the config file must appear on a separate line from the
parameters, or they will be overwritten

2) It would be nice to be able to write a default file that has everything
commented out with the exception of backend.use and numerix. We comment out
all the parameters in the current matplotlibrc, which makes it easier to
identify what has been customized during troubleshooting.

3) Any key appearing in the conf file is validated, so you cant mispell a key
name or accidentally include some garbage. But after generating the MPLConfig
object mplrc, you can do something like mplrc.nonsense = 1 without generating
an error. Is there a way to prevent the creation of additional attributes
once mplrc has been instantiated? That was a useful improvement to our
current rcParams.

4) It would be nice to create a new mpl.conf file based on changes made
interactively, without having to overwrite the old file, yet still preserving
the order and comments seen in the default I attached. Maybe copying the
default into the new location, and using it as the template? Even better,
when I saved a new file, it would preserve the defaults order, and any
settings that are identical to the defaults would be commented out. Ok, I'm
getting carried away. sorry.

I'm going to try and finish hierarchical inclusion of files (aka
ipython profiles) now. Consider for example a paper that requires
black and white only styles. You could then have in that paper's
directory a .matplotlib.conf file with:

# paper-specific tweaks
include = "$HOME/.matplotlib/matplotlib.conf"

[lines]
color = 'black'
thickness = 2

etc.

Thats a nice idea.

Or in your .matplotlib dir, you could keep multiple configurations
that branch off the basic one for different needs, while keeping
common defaults in one central location.

In my first implementation I'll have to give up on safe roundtripping
to files with recursion active, because it's a bit tricky to do both
(not impossible).

I'm feeling a bit cross-eyed from coding all day, and this is a little too
abstract at the moment for my poor head. Good luck!

Darren

sandbox.patch (47.7 KB)

···

On Saturday 21 July 2007 3:12:44 pm Fernando Perez wrote:

On 7/21/07, Darren Dale <dd55@...143...> wrote:

> > I'm working on converting our existing rc code to tconfig this weekend.
> > So far so good. I just wanted to let people know to avoid duplicating
> > effort.
>
> Excellent! Ping me if you have any problems.

Im attaching a patch, it includes lots of changes to mplconfig.py, a realistic
mplrc.conf file (feel free to rename it to mpl.conf or matplotlib.conf or
whatever seems standard), and a backup of that .conf file. The mpltest lets
me try your trick of modifying a well formatted file in place. Very nice!

Is the patch against current SVN? I'm in a hurry and can't look at it
now, so if you can double-check, that would be great. For tconfig.py,
my last commit was:

Last Changed Rev: 2532
Last Changed Date: 2007-07-21 15:19:06 -0600 (Sat, 21 Jul 2007)

A couple comments:

1) Comments in the config file must appear on a separate line from the
parameters, or they will be overwritten

2) It would be nice to be able to write a default file that has everything
commented out with the exception of backend.use and numerix. We comment out
all the parameters in the current matplotlibrc, which makes it easier to
identify what has been customized during troubleshooting.

John and I just had a long conversation on this topic, and I think we
have a reasonable solution. I have to run now, but I'll try to
implement it tomorrow and will get back to you with more feedback.
Sorry but I have guests coming now :slight_smile:

3) Any key appearing in the conf file is validated, so you cant mispell a key
name or accidentally include some garbage. But after generating the MPLConfig
object mplrc, you can do something like mplrc.nonsense = 1 without generating
an error. Is there a way to prevent the creation of additional attributes
once mplrc has been instantiated? That was a useful improvement to our
current rcParams.

Yes, we can inherit from HasStrictTraits instead of HasTraits. I was
in fact doing that but had some problem, can't remember what. I'm
sure it's doable though.

4) It would be nice to create a new mpl.conf file based on changes made
interactively, without having to overwrite the old file, yet still preserving
the order and comments seen in the default I attached. Maybe copying the
default into the new location, and using it as the template? Even better,
when I saved a new file, it would preserve the defaults order, and any
settings that are identical to the defaults would be commented out. Ok, I'm
getting carried away. sorry.

> I'm going to try and finish hierarchical inclusion of files (aka
> ipython profiles) now. Consider for example a paper that requires
> black and white only styles. You could then have in that paper's
> directory a .matplotlib.conf file with:
>
> # paper-specific tweaks
> include = "$HOME/.matplotlib/matplotlib.conf"
>
> [lines]
> color = 'black'
> thickness = 2
>
> etc.

Thats a nice idea.

> Or in your .matplotlib dir, you could keep multiple configurations
> that branch off the basic one for different needs, while keeping
> common defaults in one central location.
>
> In my first implementation I'll have to give up on safe roundtripping
> to files with recursion active, because it's a bit tricky to do both
> (not impossible).

I'm feeling a bit cross-eyed from coding all day, and this is a little too
abstract at the moment for my poor head. Good luck!

Yup, I'm pretty tired too. It seems we're all in the same
carpal-tunnel-inducing marathon :slight_smile:

Thanks again for pitching in!

Cheers,

f

···

On 7/21/07, Darren Dale <dd55@...143...> wrote:

On Saturday 21 July 2007 3:12:44 pm Fernando Perez wrote:
> On 7/21/07, Darren Dale <dd55@...143...> wrote:

Im attaching a patch

Is the patch against current SVN?

It was one version out of date. Here is a new one, against 2533

sandbox_tconfig.patch (49.1 KB)

Im attaching a patch, it includes lots of changes to mplconfig.py, a realistic
mplrc.conf file (feel free to rename it to mpl.conf or matplotlib.conf or
whatever seems standard), and a backup of that .conf file. The mpltest lets
me try your trick of modifying a well formatted file in place. Very nice!

Hey Darren,

I've been following the config discussion, but am not as deep into it
as you and Fernando so I'll just make some general comments.

2) It would be nice to be able to write a default file that has everything
commented out with the exception of backend.use and numerix. We comment out
all the parameters in the current matplotlibrc, which makes it easier to
identify what has been customized during troubleshooting.

F and I talked about the "commented out" thing on the phone last
night. The motivation for this was: when we upgrade rc we don't want
users who have never seen or modified an rc file to get a rc
deprecation warning. But Fernando thought it would be impossible to
deal intelligently with comment files in the new framework, eg
persisted changes would not necessarily be written to the portion
where the commented out version was, and the commented part would
still be in. We discussed one possibility to install a full,
uncommented default config file in the system directory (eg mpl-data)
at install time which was guaranteed to be correct, and in the users
.matplotlib directory just put a stub which includes that file (so
they know where to find it and copy-and-paste from) and in the user
file also puy any build time config we detect, eg backend and
formerly, numerix. Changes made to the rc object when saved would
always go to the top level rc file, eg the one in the user dir. So
the user rc would be a pointer to the main rc with local
modifications.

3) Any key appearing in the conf file is validated, so you cant mispell a key
name or accidentally include some garbage. But after generating the MPLConfig
object mplrc, you can do something like mplrc.nonsense = 1 without generating
an error. Is there a way to prevent the creation of additional attributes
once mplrc has been instantiated? That was a useful improvement to our
current rcParams.

I believe HasStrictTraits is designed for this.

4) It would be nice to create a new mpl.conf file based on changes made
interactively, without having to overwrite the old file, yet still preserving
the order and comments seen in the default I attached. Maybe copying the
default into the new location, and using it as the template? Even better,
when I saved a new file, it would preserve the defaults order, and any
settings that are identical to the defaults would be commented out. Ok, I'm
getting carried away. sorry.

I don't know if my comments above address this..

I'm feeling a bit cross-eyed from coding all day, and this is a little too
abstract at the moment for my poor head. Good luck!

I know the feeling from my own experience over the last few days.

I think we should make sure we know what we are getting with this new
config approach, ie what is missing from our current system that
people want to do but can't. For mpl1, I don't mind doing something
different if it is simply better and cleaner. For matplotlib, I think
we need a higher bar, though not an infinitely high one. For the
latter, I think it needs to solve an existing problem, and do so in a
way that we expect to solve it "for good". I think the system you and
Fernando are discussing probably meets the second criteria, since
you've obviously put a lot of thought into what a good config system
needs. I am less convinced about the first. With your latest changes
to the current rc system, we have config time and runtime validation
of the keys and values, albeit a homegrown and not terribly elegant
version of what traits does more elegantly. Recursive includes and
runtime persistence are nice, but noone has ever asked for that as far
as I know. For ipython, which is an environment where building
hybrids of other environments is a common need, recursive includes are
clearly useful. So I'd like to make sure we have a clear rationale
here, other than shiny and new. We geeks do like our toys.

That said, as part of a general traitification of matplotlib artists,
I think it is natural and good to have a traitified rc file. If you
view this work as a first step in that direction, I think that is
sufficient justification, because having traitified artists will solve
a clear and existing need: a generalized validation and notification
framework. And the goal of having a common config system across core
packages certainly has merits as well.

JDH

···

On 7/21/07, Darren Dale <dd55@...143...> wrote:

Hi John,

> 2) It would be nice to be able to write a default file that has
> everything commented out with the exception of backend.use and numerix.
> We comment out all the parameters in the current matplotlibrc, which
> makes it easier to identify what has been customized during
> troubleshooting.

F and I talked about the "commented out" thing on the phone last
night. The motivation for this was: when we upgrade rc we don't want
users who have never seen or modified an rc file to get a rc
deprecation warning. But Fernando thought it would be impossible to
deal intelligently with comment files in the new framework, eg
persisted changes would not necessarily be written to the portion
where the commented out version was, and the commented part would
still be in. We discussed one possibility to install a full,
uncommented default config file in the system directory (eg mpl-data)
at install time which was guaranteed to be correct, and in the users
.matplotlib directory just put a stub which includes that file (so
they know where to find it and copy-and-paste from) and in the user
file also puy any build time config we detect, eg backend and
formerly, numerix. Changes made to the rc object when saved would
always go to the top level rc file, eg the one in the user dir. So
the user rc would be a pointer to the main rc with local
modifications.

I was thinking along these lines as well, having an uncommented "master
config" file that ships with mpl. I was thinking of using that file as a
template to automatically generate commented out versions, but I like
Fernando's recursive include framework much better.

> 3) Any key appearing in the conf file is validated, so you cant mispell a
> key name or accidentally include some garbage. But after generating the
> MPLConfig object mplrc, you can do something like mplrc.nonsense = 1
> without generating an error. Is there a way to prevent the creation of
> additional attributes once mplrc has been instantiated? That was a useful
> improvement to our current rcParams.

I believe HasStrictTraits is designed for this.

> 4) It would be nice to create a new mpl.conf file based on changes made
> interactively, without having to overwrite the old file, yet still
> preserving the order and comments seen in the default I attached. Maybe
> copying the default into the new location, and using it as the template?
> Even better, when I saved a new file, it would preserve the defaults
> order, and any settings that are identical to the defaults would be
> commented out. Ok, I'm getting carried away. sorry.

I don't know if my comments above address this..

They do.

> I'm feeling a bit cross-eyed from coding all day, and this is a little
> too abstract at the moment for my poor head. Good luck!

I know the feeling from my own experience over the last few days.

I think we should make sure we know what we are getting with this new
config approach, ie what is missing from our current system that
people want to do but can't. For mpl1, I don't mind doing something
different if it is simply better and cleaner. For matplotlib, I think
we need a higher bar, though not an infinitely high one. For the
latter, I think it needs to solve an existing problem, and do so in a
way that we expect to solve it "for good". I think the system you and
Fernando are discussing probably meets the second criteria, since
you've obviously put a lot of thought into what a good config system
needs. I am less convinced about the first. With your latest changes
to the current rc system, we have config time and runtime validation
of the keys and values, albeit a homegrown and not terribly elegant
version of what traits does more elegantly.

As long as matplotlib is not designed to take advantage of notification, etc,
the existing rc system could provide nearly the same functionality as the
traited config system. The validators would have to be significantly
improved. We would still have problems arising from having to parse strings,
convert values, and then validate. For example, the latex preamble is of
limited usefulness.

Recursive includes and
runtime persistence are nice, but noone has ever asked for that as far
as I know.

I imagine our more advanced users would find them extremely useful once they
knew about them. But imagine trying to explain them in the users guide.

For ipython, which is an environment where building
hybrids of other environments is a common need, recursive includes are
clearly useful. So I'd like to make sure we have a clear rationale
here, other than shiny and new. We geeks do like our toys.

The issue of configuration has come up several times on the mailing lists over
the last few years, and Fernando's tconfig feels like the right solution. On
the other hand, I can't be too enthusiastic about pushing for inclusion in
matplotlib in the near future, because I havent been able to install traits
as a separate package. At the moment, the following command

easy_install -f
http://code.enthought.com/enstaller/eggs/source/unstable "enthought.traits <
3.0a"

does not install all the required modules. Running this command:

sudo easy_install -f
http://code.enthought.com/enstaller/eggs/source/unstable "enthought.etsconfig
< 3.0a" "enthought.util <3.0a" "enthought.debug <3.0a"

will get you traits, along with debug, developer, envisage, etsconfig, help
io, logger, naming, plugins.python_shell, plugins.text_editor, pyface,
resource, sweet_pickle, traits.ui.wx, type_manager, and util. Remember that
email we got a while back from the guy complaining about the ubuntu package
manager installing too many gui toolkits? Hopefully it is in Enthought's best
interest to work out the dependency issues.

That said, as part of a general traitification of matplotlib artists,
I think it is natural and good to have a traitified rc file. If you
view this work as a first step in that direction, I think that is
sufficient justification, because having traitified artists will solve
a clear and existing need: a generalized validation and notification
framework. And the goal of having a common config system across core
packages certainly has merits as well.

I agree. I think we could give the new system a trial in svn (when we think it
is ready) by wrapping the traited rc object with an object that quacks like
an rcParams dict. That way, the new system could be turned on or off so it
wouldnt be disruptful to the broader community of svn users.

Darren

···

On Sunday 22 July 2007 9:51:05 am John Hunter wrote:

On 7/21/07, Darren Dale <dd55@...143...> wrote:

Damn it. That certainly is not normal. I am CCing the enthought-dev
mailing-list where somebody can take care of this.

Ga�l

···

On Sun, Jul 22, 2007 at 11:43:31AM -0400, Darren Dale wrote:

The issue of configuration has come up several times on the mailing
lists over the last few years, and Fernando's tconfig feels like the
right solution. On the other hand, I can't be too enthusiastic about
pushing for inclusion in matplotlib in the near future, because I
havent been able to install traits as a separate package. At the
moment, the following command

easy_install -f
http://code.enthought.com/enstaller/eggs/source/unstable "enthought.traits <
3.0a"

does not install all the required modules. Running this command:

sudo easy_install -f
http://code.enthought.com/enstaller/eggs/source/unstable "enthought.etsconfig
< 3.0a" "enthought.util <3.0a" "enthought.debug <3.0a"

will get you traits, along with debug, developer, envisage, etsconfig, help
io, logger, naming, plugins.python_shell, plugins.text_editor, pyface,
resource, sweet_pickle, traits.ui.wx, type_manager, and util. Remember that
email we got a while back from the guy complaining about the ubuntu package
manager installing too many gui toolkits? Hopefully it is in Enthought's best
interest to work out the dependency issues.

Well, we always have matplotlib.enthought.traits already installed,
and stable, but obviously we won't be able to take advantage of the
recent bugfixes and enhancements with that, and as importantly,
enthought support. On the other hand, it already ships with mpl, and
if it does everything you need, then you should be OK with it, at
least while enthought gets the install sorted out. I know this a high
priority with them. We always retain the option of pulling a new
traits version into our src tree, but hopefully it won't come to that
because it is in everyone's interest to get the working cross platform
versioned dependency manager using PYPI that they are targeting.

Please bring up your install issues on enthought dev, they are quite
responsive, weekdays and weekends, morning, noon and night, in my
experience. Since they now are a truly international operation (US,
England, India, others?), and seem to be inhabited by guys who really
like to code, someone is usually up and working at all times, it
appears :slight_smile:

https://mail.enthought.com/mailman/listinfo/enthought-dev

JDH

···

On 7/22/07, Darren Dale <dd55@...143...> wrote:

the other hand, I can't be too enthusiastic about pushing for inclusion in
matplotlib in the near future, because I havent been able to install traits
as a separate package. At the moment, the following command

Thanks, I added your more recent version. I also changed the base
class to HasStrictTraits, so that junk can't be added to the instance
post-creation via rc.junk='foo'.

There is also more support for recursive inclusion. Unfortunately I
didn't finish everything (I took a break on Sunday) and right now I
can't work on this, so it may not quite do everything you want it to.
But let's try to pinpoint what's still missing, and I'll find some
time this week in the evenings to finish it up while the logic is
still in my head.

cheers,

f

···

On 7/21/07, Darren Dale <dd55@...143...> wrote:

On Saturday 21 July 2007 3:12:44 pm Fernando Perez wrote:
> On 7/21/07, Darren Dale <dd55@...143...> wrote:
> > I'm working on converting our existing rc code to tconfig this weekend.
> > So far so good. I just wanted to let people know to avoid duplicating
> > effort.
>
> Excellent! Ping me if you have any problems.

Im attaching a patch, it includes lots of changes to mplconfig.py, a realistic
mplrc.conf file (feel free to rename it to mpl.conf or matplotlib.conf or
whatever seems standard), and a backup of that .conf file. The mpltest lets
me try your trick of modifying a well formatted file in place. Very nice!

You may want to update before touching the code further. I moved some
(not all, I didn't finish it and only meant to do enough to
illustrate) of the comments you had in mpl.conf into the class
docstrings. The TConfig __repr__ method now automatically adds these
(as comments), so that auto-generated config files, if done purely
from the TConfig class description, are still readable and possibly
even useful. You can run mpltest.py to see what I mean, I made it
just print out a purely default config at the end.

Cheers,

f

···

On 7/21/07, Darren Dale <dd55@...143...> wrote:

On Saturday 21 July 2007 3:12:44 pm Fernando Perez wrote:
> On 7/21/07, Darren Dale <dd55@...143...> wrote:
> > I'm working on converting our existing rc code to tconfig this weekend.
> > So far so good. I just wanted to let people know to avoid duplicating
> > effort.
>
> Excellent! Ping me if you have any problems.

Im attaching a patch, it includes lots of changes to mplconfig.py, a realistic
mplrc.conf file (feel free to rename it to mpl.conf or matplotlib.conf or
whatever seems standard), and a backup of that .conf file. The mpltest lets
me try your trick of modifying a well formatted file in place. Very nice!