organization of traited config system

I am working on reorganizing our config system to work with Fernando's
tconfig, and I could use some advice. Currently, a lot of the configuration
code lives in __init__.py and rcsetup.py. I am thinking of a layout like
this:

matplotlib/lib/matplotlib/
    __init__
    ...
    config/
        api # mpl's entry point: all imports come from here
        checkdep # checks for dependencies like ghostscript, dvipng, etc.
        configobj # external project, required by tconfig
        cutils # configuration utilities, like get_home, is_writable_dir, etc.
        mplconfig.py # reads new config files using tconfig
        mpltraits # defines mpl traits like colors, markers, linestyles
        rcparams # configuration using old matplotlibrc files
        rcsetup # provides defaults, setup, for rcparams
        tconfig # Fernando's traited config, requires traits and configobj
        verbose # defines the Verbose class used throughout mpl

the __init__ file would import from config.api: rcParams, which would be the
existing dict if the old config system is active, or a dict wrapping the new
config object, along with rc, rcdefaults, etc.

Does this sound reasonable, or could anyone suggest a better organization?
Should every trait used by mpl be defined in mpltraits, that is, should
mpltraits provide an interface to enthought traits? Or should mpltraits only
provide traits that are not predefined by enthought? Is config/ a bad place
for verbose?

Thanks,
Darren

Darren,
Well it wasn't one of your questions but...

Why do you need an api file at all? Why not have config be a python package and let config/__init__.py take care of importing everything? The __init__ file of a package is responsible for configuring the api of a package so it seems redundent to have another filed named api.

Ted

···

At 07:26 AM 7/26/2007, Darren Dale wrote:

I am working on reorganizing our config system to work with Fernando's
tconfig, and I could use some advice. Currently, a lot of the configuration
code lives in __init__.py and rcsetup.py. I am thinking of a layout like
this:

matplotlib/lib/matplotlib/
    __init__
    ...
    config/
        api # mpl's entry point: all imports come from here
        checkdep # checks for dependencies like ghostscript, dvipng, etc.
        configobj # external project, required by tconfig
        cutils # configuration utilities, like get_home, is_writable_dir, etc.
        mplconfig.py # reads new config files using tconfig
        mpltraits # defines mpl traits like colors, markers, linestyles
        rcparams # configuration using old matplotlibrc files
        rcsetup # provides defaults, setup, for rcparams
        tconfig # Fernando's traited config, requires traits and configobj
        verbose # defines the Verbose class used throughout mpl

the __init__ file would import from config.api: rcParams, which would be the
existing dict if the old config system is active, or a dict wrapping the new
config object, along with rc, rcdefaults, etc.

Does this sound reasonable, or could anyone suggest a better organization?
Should every trait used by mpl be defined in mpltraits, that is, should
mpltraits provide an interface to enthought traits? Or should mpltraits only
provide traits that are not predefined by enthought? Is config/ a bad place
for verbose?

Thanks,
Darren

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Ted Drain Jet Propulsion Laboratory ted.drain@...179...

Are traits going to be a dependency that I have to download and install, or will all the traits stuff be bundled with mpl?

···

--
Tom Holroyd, Ph.D.
"The fundamentally misconceived nature versus nurture debate should be
abandoned: child development is inextricably both." -- Louann Brizendine

Good suggestion. I was emulating the way traits and IPython1 are doing it, but
probably __init__ is better and cleaner.

···

On Thursday 26 July 2007 10:30:34 am Ted Drain wrote:

Darren,
Well it wasn't one of your questions but...

Why do you need an api file at all? Why not have config be a python
package and let config/__init__.py take care of importing
everything? The __init__ file of a package is responsible for
configuring the api of a package so it seems redundent to have
another filed named api.

That hasn't been determined yet.

···

On Thursday 26 July 2007 10:39:22 am Tom Holroyd (NIH/NIMH) [E] wrote:

Are traits going to be a dependency that I have to download and install, or
will all the traits stuff be bundled with mpl?

I should also have mentioned, for those of you not following the older thread:
the new system is somewhat speculative. The old system will be the default
while we investigate a new option. I don't know when the new one would be
ready to go live, or whether we will adopt it when it is ready.

···

On Thursday 26 July 2007 10:39:22 am Tom Holroyd (NIH/NIMH) [E] wrote:

Are traits going to be a dependency that I have to download and install, or
will all the traits stuff be bundled with mpl?

I don't remember the exact details, but we started using the api.py
approach instead of __init__ on Enthought's lead (I think to be more
friendly to setuptools, perhaps it had something to do with namespace
packages). There were discussions a while ago on the Enthought lists
about the reasons why they were moving away from __init__. While it
kind of bothered me a bit in the beginning out of an "If it ain't
broke, don't fix it" view, I realized that they seemed to have
thought it through.

You may want to inquire with them for the details (I was only paying
marginal attention and relied on trusting that they *did* think it
through) and then decide whether the reasons do/will/may apply to MPL
as well. But it didn't happen by accident, it was a deliberate
choice.

Cheers,

f

···

On 7/26/07, Darren Dale <dd55@...143...> wrote:

On Thursday 26 July 2007 10:30:34 am Ted Drain wrote:
> Darren,
> Well it wasn't one of your questions but...
>
> Why do you need an api file at all? Why not have config be a python
> package and let config/__init__.py take care of importing
> everything? The __init__ file of a package is responsible for
> configuring the api of a package so it seems redundent to have
> another filed named api.

Good suggestion. I was emulating the way traits and IPython1 are doing it, but
probably __init__ is better and cleaner.

Just as a data point, I think we (ipython) will ship a private copy of
traits for a while, until the dust settles. I know this goes against
Enthought's wishes, but they are still having repository
reorganizations, and I just want to make sure that for *our users*,
the existence of traits is 100% a non-issue, purely an internal
implementation detail.

In ipython1, we have an externals/ package where we shove any external
project we ship a copy of. Then we do in our code

from ipython1.externals import foo

and move on.

That centralizes things, and if at some point 'foo' becomes
stable/easy enough to rely on from upstream, the changes are trivial.

Cheers,

f

···

On 7/26/07, Darren Dale <dd55@...143...> wrote:

On Thursday 26 July 2007 10:39:22 am Tom Holroyd (NIH/NIMH) [E] wrote:
> Are traits going to be a dependency that I have to download and install, or
> will all the traits stuff be bundled with mpl?

That hasn't been determined yet.

Darren Dale wrote:

···

On Thursday 26 July 2007 10:39:22 am Tom Holroyd (NIH/NIMH) [E] wrote:

Are traits going to be a dependency that I have to download and install, or
will all the traits stuff be bundled with mpl?

That hasn't been determined yet.

Does your config system run with the old version of traits that is *already* bundled with mpl, or does it require a newer version?

Eric

Darren Dale wrote:

I am working on reorganizing our config system to work with Fernando's tconfig, and I could use some advice. Currently, a lot of the configuration code lives in __init__.py and rcsetup.py. I am thinking of a layout like this:

matplotlib/lib/matplotlib/
    __init__
    ... config/
        api # mpl's entry point: all imports come from here
        checkdep # checks for dependencies like ghostscript, dvipng, etc.
        configobj # external project, required by tconfig
        cutils # configuration utilities, like get_home, is_writable_dir, etc.
        mplconfig.py # reads new config files using tconfig
        mpltraits # defines mpl traits like colors, markers, linestyles
        rcparams # configuration using old matplotlibrc files
        rcsetup # provides defaults, setup, for rcparams
        tconfig # Fernando's traited config, requires traits and configobj
        verbose # defines the Verbose class used throughout mpl

the __init__ file would import from config.api: rcParams, which would be the existing dict if the old config system is active, or a dict wrapping the new config object, along with rc, rcdefaults, etc.

Does this sound reasonable, or could anyone suggest a better organization? Should every trait used by mpl be defined in mpltraits, that is, should mpltraits provide an interface to enthought traits? Or should mpltraits only provide traits that are not predefined by enthought? Is config/ a bad place for verbose?

I don't see any point in having mpltraits provide anything that is predefined; at most you could list in comments or the docstring the predefined traits that are being used.

The Verbose class could be more generally useful than just for mpl config, but this is true of cbook and of at least some of the things you will put in cutils. Your proposed config location is OK, but I wonder whether it would not be better to consolidate all of these utility things that have no dependencies into a single namespace that can be imported without pulling in the whole mpl configuration machinery at the same time.

I *think* that maybe this is the big advantage of using a minimal matplotlib/__init__.py and letting matplotlib/api.py be the normal entry point. It means you can do "from matplotlib import cbook", for example, and nothing gets executed other than cbook (in the extreme case where matplotlib/__init__.py is empty). If you have all sorts of things in matplotlib/__init__.py, including the global rcParams, then you can't access cbook.is_stringlike, for example, without doing the whole creation of rcParams. So, the approach of using minimal __init__.py at the base package level helps reduce the problems of circular imports and makes the whole package structure more modular.

The problem for us in making that change is that all present code (user and internal) that reads "import matplotlib as mpl", for example, would have to be changed to "import matplotlib.api as mpl". This is easy to do internally, but the breakage of user code is a problem.

I'm an amateur; I trust the experts will promptly straighten out any errors in my conjectures above.

Eric

I'm not sure if it can work with what is already in the tree or not. It might
be as simple as changing

import enthought.traits.api as traits

to

import matplotlib.enthought.traits as traits

but I havent tried yet, and wont be in a position to do so for a day or two.

···

On Thursday 26 July 2007 02:29:26 pm Eric Firing wrote:

Darren Dale wrote:
> On Thursday 26 July 2007 10:39:22 am Tom Holroyd (NIH/NIMH) [E] wrote:
>> Are traits going to be a dependency that I have to download and install,
>> or will all the traits stuff be bundled with mpl?
>
> That hasn't been determined yet.

Does your config system run with the old version of traits that is
*already* bundled with mpl, or does it require a newer version?

> > Why do you need an api file at all? Why not have config be a python
> > package and let config/__init__.py take care of importing
> > everything?

[...]

I don't remember the exact details, but we started using the api.py
approach instead of __init__ on Enthought's lead

[...]

You may want to inquire with them for the details

I did ask at enthought-dev, and they said it was primarily due to circular
dependencies. The use of __init__ is already causing us some headaches in
mpl, so I decided to stick with api.py, leaving __init__ empty.

I just uploaded my work to matplotlib svn. The idea is that matplotlib would
import the rcParams (and eventually mplConfig) from config/api, but for just
think of config/ as a sandbox. The new config system is turned on by default
in api.py by setting

USE_NEW_CONFIG = True

If it is True, you can run configtest.py in IPython, which will import the
traited mplConfig object and rcParams, a thin wrapper to mplConfig.
configtest will change rcParams['backend'] to Cairo, validate that the change
occured in mplConfig, and will then try to change the backend to something
nonsensical and should throw an error.

For now, validation in mplConfig is very similar to validation in the old
system. I have not taken advantage of the more advanced capabilities of
traits, like shadow attributes and so forth. For now I just wanted to get
something that should work with existing matplotlib.

The first time the traited config module mplconfig is loaded, it checks your
config_dir (the cwd and then probably ~/.matplotlib) for a matplotlib.conf
file. If it does not exist, it creates one, loads your old matplotlibrc,
updates mplConfig based on your matplotlibrc, saves any non-default settings
to the new matplotlib.conf. If the file does exist, it updates mplConfig
accordingly.

The config/matpltolib.conf.default file was auto-generated (comments and all,
thanks to Fernandos wizardry).

Darren

···

On Thursday 26 July 2007 11:40:23 am Fernando Perez wrote:

On 7/26/07, Darren Dale <dd55@...143...> wrote:
> On Thursday 26 July 2007 10:30:34 am Ted Drain wrote: