rcParams and validation

Christopher Barker wrote:

I have thought about the safety issue. One idea I've had (though I never
bothered with it) was to strip the input files of "import" lines first.
You could do a whole lot less if you couldn't import any arbitrary modules.

Disallowing import statements won't help with that. There are simply too many
ways to get around it. See Brett Cannon's paper on securing Python:

  http://www.cs.ubc.ca/~drifty/papers/python_security.pdf

···

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
  -- Umberto Eco

I vote for a branch. Maybe we could consider an alternative directory layout in a branch, the current layout is a bit entropic. For the sake of discussion:

mpl/

configfiles/

doc/

    examples/

    users_guide/

matplotlib/

    api/

    backends/

    config/

    mpl-data/

    test/

    text/

        afm

        dviread

        fontmanager

        mathtext

        texmanager

        text

    tools/

        cbook

        dates

        numerix

        transforms

        units

    axes.py

    figure.py

    etc.py

external/ # or installed seperately

    agg/

    dateutil/

    enthought/

    pyparsing.py

    pytz/

    subprocess/

    ttconv/

    pylab.py # is that cheating?

license/

sandbox/

I would move code out of CXX, src, and swig, and into the more relevant directories, unless it is really necessary to keep it bundled like this. I have often had to hunt for _transforms, for example.

···

On Tuesday 17 July 2007 09:33:47 am John Hunter wrote:

Speaking of branches, we may need to seriously consider a branch here,

mpl1.

Sorry for double posting. Apparently the original posting was html formatted,
and looked nonsensical. Hopefully this one is more clear:

···

On Tuesday 17 July 2007 02:31:02 pm Darren Dale wrote:

On Tuesday 17 July 2007 09:33:47 am John Hunter wrote:
> Speaking of branches, we may need to seriously consider a branch here,
> mpl1.

I vote for a branch. Maybe we could consider an alternative directory
layout in a branch, the current layout is a bit entropic. For the sake of
discussion:

mpl/
configfiles/
doc/
examples/
users_guide/
matplotlib/
api/
backends/
config/
mpl-data/
test/
text/
afm
dviread
fontmanager
mathtext
texmanager
text
tools/
cbook
dates
numerix
transforms
units
axes.py
figure.py
etc.py
external/ # or installed seperately
agg/
dateutil/
enthought/
pyparsing.py
pytz/
subprocess/
ttconv/
pylab.py # is that cheating?
license/
sandbox/

I would move code out of CXX, src, and swig, and into the more relevant
directories, unless it is really necessary to keep it bundled like this. I
have often had to hunt for _transforms, for example.

I recently saw a 'technical demo' of Matlab at my university, done by
their people. I have to admit that the only thing that left me truly
impressed was the quality of their code generator for interactively
modified plots. The demo person did a fair amount of non-trivial
tweaking of a plot via their GUI, and then he hit 'save as code'. He
then opened the generated code, which was a nice function definition
with the arrays being plotted as input parameters (hence reusable
right away), was very readable overall.

I certainly thought that having such functionality would be great for
mpl/python, since it really does look extremely useful. But this was
a sales demo, perhaps others who use matlab regularly (I don't) have
a different opinion of that functionality when used in day to day
production work.

Cheers,

f

···

On 7/17/07, Michael Droettboom <mdroe@...31...> wrote:

TraitsUI seems really cool, but there are a couple of reasons I think
that should probably be considered lower priority. For one, it would
need to be generalized and ported (backend-ed) for all of matplotlib's
many gui backends. Also, I wonder how it fits into matplotlib's
paradigm as somewhere in between interactive and noninteractive
plotting. You would almost certainly want to tweak traits with the UI
and then save that back out as Python code, but code generators almost
never generate code that a human being would want to edit.

Hi all,

this is an important discussion also for us (ipython), so I'll go in
some detail into things. It would be great if out of this we got
something that both ipython and matplotlib could reuse for the long
haul, though I'm not sure (in a sense, ipython has some nastier
requirements that mpl may not need to worry about).

> I'm really impressed with how readable and well organized the code is in
> ipython1. It looks like their approach to configuration has been carefully
> considered. Any chance we can follow their lead? It looks like it would be a

I haven't looked at it closely, but I am willing to trust Fernando on
this for the most part. I know he has put a lot of thought into it,
and it's something we've talked about for years.

Thanks for the compliments. We have put much thought into it, and
Brian deserves all the implementation credit. But as you'll see in a
moment, all is not quite well; hopefully after this discussion and
feedback from you guys we'll find a good solution so we can (finally!)
consider this problem fixed and move on to other things.

I am not too fond of the dictionary usage here:

controllerConfig.listenForEnginesOn['ip'] = '127.0.0.1'
controllerConfig.listenForEnginesOn['port'] = 20000

I prefer

controllerConfig.listenForEnginesOn.ip = '127.0.0.1'
controllerConfig.listenForEnginesOn.port = 20000

Since dicts and attrs are intimately linked, eg with something like
Bunch, this should be a minor tweak. Fernando, why did you prefer
dict semantics. And are you happy with the state of your config
system in ipython1

Let's not worry about this for now: I agree with you, but it's an
implementation detail that is trivial to fix with a light wrapper,
which already basically exists in IPython.ipstruct. The real problems
lie elsewhere.

Brian and I spent a lot of time discussing this today, and here's the
summary of the whole thing. Again, some of these issues may not
affect mpl nearly as seriously as they do ipython.

Having config files be pure python has some nice benefits:

- familiar syntax for users

- the ability for users to put in complex logic when declaring their
configuration (like make decisions based on OS, hostname, etc).

- If developers want, wrapping the underlying objects in something
like Traits provides automatically validation mechanisms for
configuration, which is important.

The security issue is for *me* moot: ipython is a full-access Python
system where people are *expected* to run code. So I don't care about
letting them run real code at config time. But for other projects,
the story is completely different, obviously. I suspect MPL falls
into ipython's category here though.

So what is the problem? The main one is that we have a real difficult
time with circular import dependencies and init-time logic.
Basically, you need these configuration objects to exist at startup
for other objects to be built, but the configuration *itself* may need
to import various things if it is to work as real Python code. In
IPython (the dev branches), for example, the user-visible system is
assembled out of multiple objects, which themselves need to build
their own internal sub-objects. Some of these simply require value
parameters at init time, but in some cases even which *class* to use
is tweakable at init time (for example, so users can choose amongst
various classes that implement different network transport protocols
but with identical API as far as IPython goes).

While in any one case one can probably disentangle the various
dependency chains that arise, it's a major pain to have to deal with
this constantly. The fact that the problem is a recurring one is to
us an indicator of a design problem that we should address
differently.

There are other issues as well, besides circular imports:

A. Initialization time modification of live objects: while config
objects are obviously meant to be built *before* the 'real' objects
(who in term use the config ones to assemble themselves), there are
cases where one would like to deal with the 'real thing'. For
example, in today's IPython (not the dev branch, but the one y'all
run) you can add things like magics at init time. This is only
possible because when the rc file is parsed, the real object is
already up and alive as __IPYTHON__ (also visible via ipapi.get() ).

Obviously this is a hack, and a cleaner design would probably separate
the declaration of code objects (say magics) that are meant to be fed
into the live instance as part of the construction of the config
object, and then would have the real ipython load up its tweaks 100%
from this config object. But since ipython is also meant to be
*runtime* modifiable, it feels natural to let users access such an API
for their own tweaks, yet there is a conflict with doing this purely
on a config object that is basically just a fancy 'data bag'.

B. Automatic creation of default config files. With pure python, this
is just hard to do, since you'd be in the business of auto-generating
valid code from the structure of an object. For a pure 'class
Bunch:pass' type thing this may not be *too* hard, as long as all
values allowed are such that eval(repr(foo))==foo, which isn't really
always the case.

C. Complexity: even for Brian, who wrote the config code, it's hard to
figure out where all the details go, since declaring new flags
typically ends up requiring fishing inside the config classes
themselves. A more plain-text, purely declarative file would
summarize these things in a way that would probably be easier for both
developers and end users.

Given all this, we are currently looking at going back to a mixture of
text-based config and real code, as a two-stage process. The
ConfigObj library:

http://www.voidspace.org.uk/python/configobj.html

looks like a nice, self-contained step above hand-parsing rc files or
Python's rather primitive ConfigParser module. It has a lot of good
features, is mature and is small enough that one can ship it inside
the project (license is BSD).

Our current plan is something along these lines:

1. Expose for all configuration objects the part that is really just a
key=value API over ConfigObj. This would dispense with all the
circular import problems, and let users specify the most common things
they want to tweak with a really simple syntax (that of .ini files).
We would then load these and build all necessary objects.

2. Make sure that the part of any given object that we mean to expose
for *runtime* modification (say adding magics to a running ipython,
for example) is easily available. This would let users modify or
extend those aspects of the system that can't just be easily declared
as simple key=value settings. The mechanism for this would simply be
to have a filename in each object's ConfigObj session like:

[SomeObject]

post_init_file = "my_post_init.py"

This file would be run via execfile() *once SomeObject* were
guaranteed to be up and running. This would then allow users to have
code that manipulates SomeObject programatically, via its runtime API.

All this conversation brings us to...

3. Traits. We (Brian and I) have gone back and forth a lot on Traits,
and we've come very close to just making them a dependency. The only
real issue holding us back is that ipython so far has exactly *zero*
extension code, which is a plus in terms of ease of
installation/deployment. Having said that, as far as extension code
is concerned, Traits is light and clean, and nowhere near the kinds of
complexities that MPL deals with. But since anything is more than
zero, it is a bit of an issue for us. We may tip over though, I'm
just stating what our reasoning so far has been.

In terms of Traits, point (2) above makes them even more attractive.
The delegation aspect of Traits is a very appealing way of combining
validation with additional action for the runtime modification of an
object. For example:

ipython.color_scheme = "foo"

If color_scheme were a Trait, the above could simply:

a) validate that "foo" was acceptable as a value
b) trigger the chain of other trait updates (dependent color schemes
for exceptions, prompts, etc).

All of this can obviously be done with python properties, but before I
write Traits again, poorly (I'm no David Morrill) I'd rather ride
Enthought's back.

This approach via Traits (or something like them) also ensures that
the work done in ensuring the consitency/robustness of an object's
*runtime* configuration API becomes automatically useful to users, and
it simplifies the init-time config API, since it gives us the option
to defer more complicated tasks to runtime.

So this is a summary of where we stand. It's surprising that the
'simple question' of configuring an application can turn out to be
such a can of worms, but for us it has proven to be. Input/ideas from
you all would be very welcome, as it's quite likely we've missed
possible solutions.

I'd love to get out of this discussion some ideas, clarity and
ultimately a plan for how to proceed in the long haul. If in addition
this means that ipython, mpl and others end up uniformizing further
our approach to this problem, even better! Having our end users find
familiar configuration mechanisms across their various libraries would
only be a good thing in the long run.

Cheers,

Brian and f.

ps - I debated on having this discussion on ipython-dev, but for now
I'm going to not cross-post. The MPL team is attentive to the issue
now, so I'd rather collect your wisdom here, and I'll take it upon
myself to summarize our conclusions on ipython-dev later. I just want
to avoid list cross-posting.

···

On 7/17/07, John Hunter <jdh2358@...149...> wrote:

On 7/17/07, Darren Dale <dd55@...143...> wrote:

3. Traits. We (Brian and I) have gone back and forth a lot on Traits,
and we've come very close to just making them a dependency. The only
real issue holding us back is that ipython so far has exactly *zero*
extension code, which is a plus in terms of ease of
installation/deployment. Having said that, as far as extension code
is concerned, Traits is light and clean, and nowhere near the kinds of
complexities that MPL deals with. But since anything is more than
zero, it is a bit of an issue for us. We may tip over though, I'm
just stating what our reasoning so far has been.

In terms of Traits, point (2) above makes them even more attractive.
The delegation aspect of Traits is a very appealing way of combining
validation with additional action for the runtime modification of an
object. For example:

ipython.color_scheme = "foo"

If color_scheme were a Trait, the above could simply:

a) validate that "foo" was acceptable as a value
b) trigger the chain of other trait updates (dependent color schemes
for exceptions, prompts, etc).

At some level though, configuration is a very different thing than an
application's runtime API. While they may be related (by exposing
common functionality), not everything that can be configured would
appear in a runtime API and vice-versa. Also, some events that need
to happen when an attribute is changed at runtime can't happen at
config time as the application might not yet be up and running yet.

Using Traits in the runtime API is an entirely different ballgame than
simply using it for type validation in configuration: When using
Traits for validation+configuration, the objects that inherit from
HasTraits are simply bunch/dict like objects that do type validation -
but they don't contain any application logic. The actually
application logic is contained in classes that aren't traited
themselves, but that consume traited config objects to configure
themselves at startup.

If traited objects are exposed in a runtime API, all of a sudden the
application logic moves to the traited classes themselves. Then, the
entire application (any object that needs config or has a runtime API)
is built upon traits at its core. This is very from the previous case
where traited classes are used as a minor implementation detail of the
config system.

I am not saying that it is bad to build an application with traits at
its core, but only that that is very different from the path that
ipython and matplotlib have taken thus far. Also, it makes the
commitment level to traits much higher than if it is used merely as a
component in the config system - which could easily be swapped out if
desired.

Brian

···

All of this can obviously be done with python properties, but before I
write Traits again, poorly (I'm no David Morrill) I'd rather ride
Enthought's back.

This approach via Traits (or something like them) also ensures that
the work done in ensuring the consitency/robustness of an object's
*runtime* configuration API becomes automatically useful to users, and
it simplifies the init-time config API, since it gives us the option
to defer more complicated tasks to runtime.

So this is a summary of where we stand. It's surprising that the
'simple question' of configuring an application can turn out to be
such a can of worms, but for us it has proven to be. Input/ideas from
you all would be very welcome, as it's quite likely we've missed
possible solutions.

I'd love to get out of this discussion some ideas, clarity and
ultimately a plan for how to proceed in the long haul. If in addition
this means that ipython, mpl and others end up uniformizing further
our approach to this problem, even better! Having our end users find
familiar configuration mechanisms across their various libraries would
only be a good thing in the long run.

Cheers,

Brian and f.

ps - I debated on having this discussion on ipython-dev, but for now
I'm going to not cross-post. The MPL team is attentive to the issue
now, so I'd rather collect your wisdom here, and I'll take it upon
myself to summarize our conclusions on ipython-dev later. I just want
to avoid list cross-posting.

--
Brian E. Granger, Ph.D.
Research Scientist
Tech-X Corporation
phone: 720-974-1850
bgranger@...547...
ellisonbg@...149...

> 3. Traits. We (Brian and I) have gone back and forth a lot on Traits,
> and we've come very close to just making them a dependency. The only
> real issue holding us back is that ipython so far has exactly *zero*
> extension code, which is a plus in terms of ease of
> installation/deployment. Having said that, as far as extension code
> is concerned, Traits is light and clean, and nowhere near the kinds of
> complexities that MPL deals with. But since anything is more than
> zero, it is a bit of an issue for us. We may tip over though, I'm
> just stating what our reasoning so far has been.
>
> In terms of Traits, point (2) above makes them even more attractive.
> The delegation aspect of Traits is a very appealing way of combining
> validation with additional action for the runtime modification of an
> object. For example:
>
> ipython.color_scheme = "foo"
>
> If color_scheme were a Trait, the above could simply:
>
> a) validate that "foo" was acceptable as a value
> b) trigger the chain of other trait updates (dependent color schemes
> for exceptions, prompts, etc).

At some level though, configuration is a very different thing than an
application's runtime API. While they may be related (by exposing
common functionality), not everything that can be configured would
appear in a runtime API and vice-versa. Also, some events that need
to happen when an attribute is changed at runtime can't happen at
config time as the application might not yet be up and running yet.

[ Brian, I know we already talked about this, but I'm putting the full
reasoning here because I'm interested in this discussion going beyond
ipython to mpl and possibly other tools in this little 'universe' of
ours]

I should have provided more detail in my previous email, to address
your (valid) concerns. Here's how I view organizing things: objects
have:

1. an init-time-only configuration part, which after construction is read-only,

2. a part which can be set at init or runtime

3. an api for purely run-time behavior, that can't be exposed at init
time because it fundamentally needs the object to exist. This is the
'user public' api of the object instead of the 'developer public' one
(in the sense of not having _underscore prefixes).

1 and 2 can be expressed declaratively and the .conf format that
ConfigObj reads is very nice for that. 3 can only exist at runtime,
so it's a matter of having a way in the .conf file for the user to
declare a .py file that will be run AFTER object construction.

Using Traits in the runtime API is an entirely different ballgame than
simply using it for type validation in configuration: When using
Traits for validation+configuration, the objects that inherit from
HasTraits are simply bunch/dict like objects that do type validation -
but they don't contain any application logic. The actually
application logic is contained in classes that aren't traited
themselves, but that consume traited config objects to configure
themselves at startup.

If traited objects are exposed in a runtime API, all of a sudden the
application logic moves to the traited classes themselves. Then, the
entire application (any object that needs config or has a runtime API)
is built upon traits at its core. This is very from the previous case
where traited classes are used as a minor implementation detail of the
config system.

I am not saying that it is bad to build an application with traits at
its core, but only that that is very different from the path that
ipython and matplotlib have taken thus far. Also, it makes the
commitment level to traits much higher than if it is used merely as a
component in the config system - which could easily be swapped out if
desired.

As I said elsewhere, Traits is *really* small and self-contained, with
a single super-clean piece of C-only extension code that compiles
quickly and without a single warning. I'm still waiting for the
Enthought guys to help me with the metaclass hacks to fully disable
the GUI support at will, but I'm sure it's doable (I even already have
a solution, I just want a better one).

I'm honestly 90% convinced of just using Traits now. The reason is here:

svn co http://ipython.scipy.org/svn/ipython/ipython/branches/saw/sandbox

Just run the file tconfig.py there from within ipython, and start
playing for example with the mpl.rc object. There are a few things I
dislike:

1. Any traited object has a very dirty namespace out of the box. For
rc objects, this makes tab-completion a royal pain in the ass. We'll
see what the Enthought gurus suggest on this front.

2. The way I'm handling read-only name displays in the string
representation of these things is a bit ugly. But the mix of
declarative use of classes, multiple inheritance and traits'
metaclasses got me a bit confused and I couldn't find a cleaner
solution.

But I think this approach can be very nice. For those not interested
in downloading the code, the highlights are:

- Consider a file called mpl.conf:
# Top-level
backend = "TkAgg"
interactive = False

# Things that can only be set at init time, they become read-only afterwards
[InitOnly]
numerix = "numpy"

# Other sections
[lines]
linewidth = 2.0
linestyle = '-'

[figure]
figsize = [6.4,4.8] # figure size in inches
dpi = 100 # figure dots per inch
facecolor = 0.75 # figure facecolor; 0.75 is scalar gray
edgecolor = "white" # figure edgecolor

    [[subplot]]
    # The figure subplot parameters. All dimensions are fraction of the
    # figure width or height
    left = 0.125 # the left side of the subplots of the figure
    right = 0.9 # the right side of the subplots of the figure
    bottom = 0.1 # the bottom of the subplots of the figure
    top = 0.9 # the top of the subplots of the figure

This can be edited in any text editor, emacs even has a .conf major mode!

- Then, consider the following bit of Python code:

    standard_color = Trait ('black',
                            {'black': (0.0, 0.0, 0.0, 1.0),
                             'blue': (0.0, 0.0, 1.0, 1.0),
                             'cyan': (0.0, 1.0, 1.0, 1.0),
                             'green': (0.0, 1.0, 0.0, 1.0),
                             'magenta': (1.0, 0.0, 1.0, 1.0),
                             'orange': (0.8, 0.196, 0.196, 1.0),
                             'purple': (0.69, 0.0, 1.0, 1.0),
                             'red': (1.0, 0.0, 0.0, 1.0),
                             'violet': (0.31, 0.184, 0.31, 1.0),
                             'yellow': (1.0, 1.0, 0.0, 1.0),
                             'white': (1.0, 1.0, 1.0, 1.0),
                             'transparent': (1.0, 1.0, 1.0, 0.0) } )

    class MPLConfig(TConfig):

        # Valid backends, first is default
        backend = Trait('TkAgg','WXAgg','GTKAgg','QtAgg','Qt4Agg')
        interactive = Bool(False)

        class InitOnly(TConfig,ReadOnlyTConfig):
            """Things that can only be set at init time"""
            numerix = Str('numpy')

        class lines(TConfig):
            linewidth = Float(2.0)
            linestyle = Trait('-','=','^')

        class figure(TConfig):
            figsize = ListFloat([6.4,4.8]) # figure size in inches
            dpi = Int(100) # figure dots per inch
            facecolor = Float(0.75) # figure facecolor; 0.75 is scalar gray
            edgecolor = Trait('white',standard_color)

            class subplot(TConfig):
                """The figure subplot parameters. All dimensions are fraction
                of the figure width or height"""
                left = Float(0.125)
                right = Float(0.9)
                bottom = Float(0.1)
                top = Float(0.9)

This purely declarative code incorporates:

* The types of all parameters acceptable in the file
* The nesting hierarchy for subobjects
* The default values hardcoded in the app if users don't provide any
* The valid values for those which are choices.

An 'application' is defined via:

    class App(object):
        """A trivial 'application' class to be initialized.
        """
        def __init__(self,configClass,conf_filename):
            conf = mkConfigObj(conf_filename)
            self.rc = configClass(conf)

    mpl = App(MPLConfig,'mpl.conf')

This makes the App object load the .conf file, immediately giving
errors if any of the validation constraints implicit in the traits
spec are not met. The resulting mpl.rc object can be printed,
manipulated, modified (for non-read-only sections), even edited with a
GUI (limited at the moment to WX and not really working for
sub-sections, though that's easily fixed). For example:

In [159]: mpl.rc
Out[159]:
# Dump of MPLConfig

backend = 'TkAgg'
interactive = False

[InitOnly]
numerix = 'numpy'

[lines]
linewidth = 2.0
linestyle = '-'

[figure]
edgecolor = 'white'
figsize = [6.4000000000000004, 4.7999999999999998]
dpi = 100
facecolor = 0.75

    [[subplot]]
    top = 0.90000000000000002
    right = 0.90000000000000002
    left = 0.125
    bottom = 0.10000000000000001

These objects are automatically self-representable in a valid
roundtrip format. Editing works:

In [160]: mpl.rc.interactive
Out[160]: False

In [161]: mpl.rc.interactive = True

with validation:

In [162]: mpl.rc.interactive = "well"

···

On 7/18/07, Brian Granger <ellisonbg@...149...> wrote:
---------------------------------------------------------------------------
TraitError Traceback (most recent call last)

/home/fperez/ipython/svn/ipython/branches/saw/sandbox/<ipython

in <module>()

/home/fperez/usr/opt/lib/python2.5/site-packages/enthought/traits/trait_handlers.py
in error(self, object, name, value)
    169 another trait handler to handle to validate the value.
    170 """
--> 171 raise TraitError, ( object, name, self.info(), value )
    172
    173 def arg_error ( self, method, arg_num, object, name, value ):

TraitError: The 'interactive' trait of a MPLConfig instance must be a
value of type 'bool', but a value of well was specified.

And read-only parts can't be modified:

In [163]: mpl.rc.InitOnly.numerix = "Numeric"
---------------------------------------------------------------------------
TraitError Traceback (most recent call last)

/home/fperez/ipython/svn/ipython/branches/saw/sandbox/<ipython

in <module>()

TraitError: Cannot modify the read only 'numerix' attribute of a
'ReadOnlyTConfig' object.

In summary, I'm fairly happy with the results, and I think the benefit
is enough to convince me of falling in the embrace of the gods of
Traits. It seems John is going for Traits as well, so perhaps we can
use this little config setup across our systems, and even make it
something that others use in the future. I think there's value for
end users in having common, uniform configuration systems across the
various parts of the scientific python 'ecosystem'.

Once we have a discussion here, I'll take this over to the ipython list as well.

Cheers,

f

- Consider a file called mpl.conf:
# Top-level
backend = "TkAgg"
interactive = False

# Things that can only be set at init time, they become read-only
afterwards
[InitOnly]
numerix = "numpy"

[...]

- Then, consider the following bit of Python code:

[...]

This purely declarative code incorporates:

* The types of all parameters acceptable in the file
* The nesting hierarchy for subobjects
* The default values hardcoded in the app if users don't provide any
* The valid values for those which are choices.

An 'application' is defined via:

    class App(object):
        """A trivial 'application' class to be initialized.
        """
        def __init__(self,configClass,conf_filename):
            conf = mkConfigObj(conf_filename)
            self.rc = configClass(conf)

    mpl = App(MPLConfig,'mpl.conf')

This makes the App object load the .conf file, immediately giving
errors if any of the validation constraints implicit in the traits
spec are not met. The resulting mpl.rc object can be printed,
manipulated, modified (for non-read-only sections), even edited with a
GUI (limited at the moment to WX and not really working for
sub-sections, though that's easily fixed). For example:

In [159]: mpl.rc
Out[159]:
# Dump of MPLConfig

backend = 'TkAgg'
interactive = False

[InitOnly]
numerix = 'numpy'

[lines]
linewidth = 2.0
linestyle = '-'

[figure]
edgecolor = 'white'
figsize = [6.4000000000000004, 4.7999999999999998]
dpi = 100
facecolor = 0.75

    [[subplot]]
    top = 0.90000000000000002
    right = 0.90000000000000002
    left = 0.125
    bottom = 0.10000000000000001

These objects are automatically self-representable in a valid
roundtrip format.

Damn, that is really cool. So you can generate default config files from the
MPLConfig instance. We create a default matplotlibrc file from a template,
setting default backend and numerix values based on what is available on the
users system. It looks like it would be even easier with your scheme: import
MPLConfig in setup.py, set the attributes, dump to a default config file.

[...]

In summary, I'm fairly happy with the results, and I think the benefit
is enough to convince me of falling in the embrace of the gods of
Traits. It seems John is going for Traits as well, so perhaps we can
use this little config setup across our systems, and even make it
something that others use in the future. I think there's value for
end users in having common, uniform configuration systems across the
various parts of the scientific python 'ecosystem'.

I agree. It looks really elegant. What about the circular dependencies you
mentioned in a previous email, is that still a potential problem?

Darren

···

On Thursday 19 July 2007 8:02:53 pm Fernando Perez wrote:

Damn, that is really cool. So you can generate default config files from the
MPLConfig instance. We create a default matplotlibrc file from a template,
setting default backend and numerix values based on what is available on the
users system. It looks like it would be even easier with your scheme: import
MPLConfig in setup.py, set the attributes, dump to a default config file.

Yup. If you update, I just committed the last changes I can make to
this for now. I need to switch gears (real work calls...), but this
stuff is mostly functional. It needs a solid cleanup and my quick
tests to be moved into proper doc/unit tests, but I think it's looking
reasonably good. The examples/tests part of the file shows exactly
what you are asking for.

> In summary, I'm fairly happy with the results, and I think the benefit
> is enough to convince me of falling in the embrace of the gods of
> Traits. It seems John is going for Traits as well, so perhaps we can
> use this little config setup across our systems, and even make it
> something that others use in the future. I think there's value for
> end users in having common, uniform configuration systems across the
> various parts of the scientific python 'ecosystem'.

I agree. It looks really elegant. What about the circular dependencies you
mentioned in a previous email, is that still a potential problem?

Nope, gone (I think). Since the config file is pure text, it will
never pull the main project's objects in via code. One can allow a
config field to specify a file to execute *later*, once everything is
up and running, but the building of these config objects is going to
be strictly declarative via traits, so we should be OK.

Brian has more experience with those headaches than I, so he may still
spot issues with this, but I think this setup is looking very
reasonable.

Cheers,

f

···

On 7/19/07, Darren Dale <dd55@...143...> wrote:

Well that easy to sort out. You can put a flag to the traits handler that
simply forbids it as long as the application is not running. Something
like:

    def if_app_running(callback):
        app = get_application() # get_application could be anything that
                                    # returns the current application
        def modified_callback(self, *args, **kwargs):
            if app.running:
                callback(self, *args, **kwargs)

        return modified_callback

    class ConfigurationObject(HasTraits):

        prompt = String('[%i]')

        @if_app_running
        def _prompt_changed(self):
            do whatever you want here

Ga�l

···

On Wed, Jul 18, 2007 at 11:24:09AM -0600, Brian Granger wrote:

At some level though, configuration is a very different thing than an
application's runtime API. While they may be related (by exposing
common functionality), not everything that can be configured would
appear in a runtime API and vice-versa. Also, some events that need
to happen when an attribute is changed at runtime can't happen at
config time as the application might not yet be up and running yet.

Damn, that is really cool. So you can generate default config files from the
MPLConfig instance. We create a default matplotlibrc file from a template,
setting default backend and numerix values based on what is available on the
users system. It looks like it would be even easier with your scheme: import
MPLConfig in setup.py, set the attributes, dump to a default config file.

My thought for IPython is to have defaults set in the traited config
class itself so you wouldn't even have to do the second step. Just
import MPLConfig and dump to a file. I too think this is one of the
really nice benefits of this approach. Generating the default config
file for IPython1 was becoming a real problem.

[...]
> In summary, I'm fairly happy with the results, and I think the benefit
> is enough to convince me of falling in the embrace of the gods of
> Traits. It seems John is going for Traits as well, so perhaps we can
> use this little config setup across our systems, and even make it
> something that others use in the future. I think there's value for
> end users in having common, uniform configuration systems across the
> various parts of the scientific python 'ecosystem'.

I agree. It looks really elegant. What about the circular dependencies you
mentioned in a previous email, is that still a potential problem?

The circular dependency problem goes away in the following way. The
config files and config objects will only consist of basic types (int,
strings, float, lists, dicts, etc) - not executable python code. If
you need to specify python code (the name of a class that implements
some behavior for example) that would be specified as a string.
During the initial parts of the configuration process, that would
remain a string, thus avoiding the circular dependency problem. Only
after it is safe to execute/import, would the string be exec'd.

···

Darren

--
Brian E. Granger, Ph.D.
Research Scientist
Tech-X Corporation
phone: 720-974-1850
bgranger@...547...
ellisonbg@...149...

Although, the config files generated this way are not very well organized, due
to the seemingly random order that a dict's items are accessed. Look at the
mpl3.conf file that is generated with Fernando's proof-of-concept:

interactive = False
backend = 'TkAgg'
[figure]
    edgecolor = 'white'
    facecolor = 0.75
    dpi = 100
    figsize = [6.4000000000000004, 4.7999999999999998]
    [[subplot]]
        top = 0.90000000000000002
        right = 0.90000000000000002
        bottom = 0.10000000000000001
        left = 0.125
[lines]
    linewidth = 2.0
    linestyle = '-'
[InitOnly]
    numerix = 'numpy'

That is not too bad for small config files, but matplotlib's selection of rc
parameters is pretty large, and it is nice to have them appear in the file in
a meaningful order.

Darren

···

On Friday 20 July 2007 12:31:00 pm Brian Granger wrote:

> Damn, that is really cool. So you can generate default config files from
> the MPLConfig instance. We create a default matplotlibrc file from a
> template, setting default backend and numerix values based on what is
> available on the users system. It looks like it would be even easier with
> your scheme: import MPLConfig in setup.py, set the attributes, dump to a
> default config file.

My thought for IPython is to have defaults set in the traited config
class itself so you wouldn't even have to do the second step. Just
import MPLConfig and dump to a file. I too think this is one of the
really nice benefits of this approach. Generating the default config
file for IPython1 was becoming a real problem.

Not a whole lot we can do about that, since the write method comes
from the configobj code. Since they try to write in the same place
the original file had things (to preserve comments and such), I don't
think this is easy to fix.

However, it's not all gloom and doom: one can simply generate a config
file from the object description, edit it for ordering, and after that
keep it synced with changes to the object description fairly easily.
The current code only works in one direction: the file overrides the
object. It should take me about 5 minutes to add a method to go in
the other, since the code is really already there. With that, you can
then use the workflow of generating once, manually fixing it, and then
updating it from the class.

That way, you can always keep the default file you like as internal
MPL data, but at install time, update that file as needed from runtime
config before putting it in the user's directory. It lets you keep a
nicely commented, organized file, but update it at installation time
with proper info.

I'll code this up after I get back from lunch, and then I'll really
stop touching this code. I mean it :slight_smile:

Cheers,

f

···

On 7/20/07, Darren Dale <dd55@...143...> wrote:

Although, the config files generated this way are not very well organized, due
to the seemingly random order that a dict's items are accessed. Look at the
mpl3.conf file that is generated with Fernando's proof-of-concept:

interactive = False
backend = 'TkAgg'
[figure]
    edgecolor = 'white'
    facecolor = 0.75
    dpi = 100
    figsize = [6.4000000000000004, 4.7999999999999998]
    [[subplot]]
        top = 0.90000000000000002
        right = 0.90000000000000002
        bottom = 0.10000000000000001
        left = 0.125
[lines]
    linewidth = 2.0
    linestyle = '-'
[InitOnly]
    numerix = 'numpy'

That is not too bad for small config files, but matplotlib's selection of rc
parameters is pretty large, and it is nice to have them appear in the file in
a meaningful order.

Nah, whoo needs lunch. Done:

svn commit -m"Add option for class defaults to override files, so
projects can impose changing defaults on files. After Darren's
suggestion for MPL."

This should give you a very reasonable solution for what you wanted to do.

Now, I'll get lunch!

Cheers,

f

···

On 7/20/07, Fernando Perez <fperez.net@...149...> wrote:

However, it's not all gloom and doom: one can simply generate a config
file from the object description, edit it for ordering, and after that
keep it synced with changes to the object description fairly easily.
The current code only works in one direction: the file overrides the
object. It should take me about 5 minutes to add a method to go in
the other, since the code is really already there. With that, you can
then use the workflow of generating once, manually fixing it, and then
updating it from the class.

That way, you can always keep the default file you like as internal
MPL data, but at install time, update that file as needed from runtime
config before putting it in the user's directory. It lets you keep a
nicely commented, organized file, but update it at installation time
with proper info.

I'll code this up after I get back from lunch, and then I'll really
stop touching this code. I mean it :slight_smile:

Great! I'm surprised that this was possible.

Im attaching a patch for the ipython1 sandbox, which adds configobj.py, moves
the ipython and mpl config code into their own .py files, and moves the test
code into mpltest.py and ipythontest.py.

Darren

config.patch (94.7 KB)

···

On Friday 20 July 2007 03:01:19 pm you wrote:

On 7/20/07, Fernando Perez <fperez.net@...149...> wrote:
> However, it's not all gloom and doom: one can simply generate a config
> file from the object description, edit it for ordering, and after that
> keep it synced with changes to the object description fairly easily.
> The current code only works in one direction: the file overrides the
> object. It should take me about 5 minutes to add a method to go in
> the other, since the code is really already there. With that, you can
> then use the workflow of generating once, manually fixing it, and then
> updating it from the class.
>
> That way, you can always keep the default file you like as internal
> MPL data, but at install time, update that file as needed from runtime
> config before putting it in the user's directory. It lets you keep a
> nicely commented, organized file, but update it at installation time
> with proper info.
>
> I'll code this up after I get back from lunch, and then I'll really
> stop touching this code. I mean it :slight_smile:

Nah, whoo needs lunch. Done:

svn commit -m"Add option for class defaults to override files, so
projects can impose changing defaults on files. After Darren's
suggestion for MPL."

This should give you a very reasonable solution for what you wanted to do.

Darren Dale wrote:

That is not too bad for small config files, but matplotlib's selection of rc
parameters is pretty large, and it is nice to have them appear in the file in
a meaningful order.

You guys may want to consider iniparse instead of configobj. It will preserve
order, indentation, comments(!), and blank lines.

  GitHub - candlepin/python-iniparse: Python iniparse module with patches from Fedora project

The drawback is that it doesn't support nested sections, to my knowledge. There
might also be more special features of configobj that Fernando uses; I haven't
taken a very close look at the code, yet.

···

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
  -- Umberto Eco

Mmh, but configobj does preserve all those as well. Darren's comment
was about not being able to get a clean file when made purely from a
HasTraits description. But when there is an existing file on disk,
configobj is actually really good at preserving it.

So it seems to me that iniparse appears to be a limited subset of
configobj (we really want nested sections in ipython, and MPL has them
as well already). It does have the a.b instead of a['b'] convenience,
but we get that via traits anyway.

Am I missing something?

Cheers,

f

···

On 7/20/07, Robert Kern <robert.kern@...149...> wrote:

Darren Dale wrote:

> That is not too bad for small config files, but matplotlib's selection of rc
> parameters is pretty large, and it is nice to have them appear in the file in
> a meaningful order.

You guys may want to consider iniparse instead of configobj. It will preserve
order, indentation, comments(!), and blank lines.

  GitHub - candlepin/python-iniparse: Python iniparse module with patches from Fedora project

The drawback is that it doesn't support nested sections, to my knowledge. There
might also be more special features of configobj that Fernando uses; I haven't
taken a very close look at the code, yet.

Fernando Perez wrote:

Am I missing something?

No, I just wasn't paying close enough attention. Never mind me.

···

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
  -- Umberto Eco

> This should give you a very reasonable solution for what you wanted to do.

Great! I'm surprised that this was possible.

And actually easy, for once.

Im attaching a patch for the ipython1 sandbox, which adds configobj.py, moves
the ipython and mpl config code into their own .py files, and moves the test
code into mpltest.py and ipythontest.py.

Great, thanks! Committed as r2526. Later we'll move configobj.py
into ipython/externals, but for now it's OK to have it in, so it's
nicely self-contained for people to test this while we settle on
something long-term.

I've moved the lot into a subdirectory now, so that we can have more
than one set of toys in our sandbox, so update to 2527 and you'll have
tconfig as a self-contained project we can play with until happiness
arrives.

Cheers,

f

···

On 7/20/07, Darren Dale <dd55@...143...> wrote:

I also just (r2528) updated things to use a cleaner import convention:
I don't like importing things from modules *they* imported. To
clarify:

planck[tconfig]> svn diff -r2527:2528 mpltest.py
Index: mpltest.py

···

On 7/20/07, Fernando Perez <fperez.net@...149...> wrote:

On 7/20/07, Darren Dale <dd55@...143...> wrote:

> > This should give you a very reasonable solution for what you wanted to do.
>
> Great! I'm surprised that this was possible.

And actually easy, for once.

> Im attaching a patch for the ipython1 sandbox, which adds configobj.py, moves
> the ipython and mpl config code into their own .py files, and moves the test
> code into mpltest.py and ipythontest.py.

Great, thanks! Committed as r2526. Later we'll move configobj.py
into ipython/externals, but for now it's OK to have it in, so it's
nicely self-contained for people to test this while we settle on
something long-term.

===================================================================
--- mpltest.py (revision 2527)
+++ mpltest.py (revision 2528)
@@ -1,5 +1,11 @@
-from mplconfig import MPLConfig, ConfigManager, mkConfigObj
+# import/reload base modules for interactive testing/development
+import tconfig, mplconfig
+reload(tconfig)
+reload(mplconfig)

+from tconfig import ConfigManager, mkConfigObj
+from mplconfig import MPLConfig

The point is that instead of

-from mplconfig import MPLConfig, ConfigManager, mkConfigObj

I prefer:

+from tconfig import ConfigManager, mkConfigObj
+from mplconfig import MPLConfig

since ConfigManager and mkConfigObj really live in tconfig, I'm not
tying my top-level code to an implementation detail of an intermediate
module (what it imports and how it names it). I mention this
explicitly just because I think it's a good bit of coding practice.

The reload tricks also allow me to continue running the top-level test
codes while mucking with the stuff underneath, and have changes make
their way through consistently.

Thanks again,

f