namespace packages in toolkits?

Development matplotlib-devel
1

Do we need namespace packages in toolkits? I recently added gtktools
and exceltools to toolkits, and got a very hard to debug error:

In [1]: import matplotlib

In [2]: matplotlib.__file__
Out[2]: '/home/titan/johnh/dev/lib/python2.4/site-packages/matplotlib/__init__.pyc'

In [3]: matplotlib.rcParams['axes.axisbelow']
Out[3]: False

In [4]: import matplotlib.toolkits.gtktools

In [5]: matplotlib.__file__
Out[5]: '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/matplotlib/__init__.pyc'

In [6]: matplotlib.rcParams['axes.axisbelow']

···

------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in ?
KeyError: 'axes.axisbelow'

Notice that the matplotlib module which was previously imported got
changed by the import of the toolkit which declared itself a namespace
package. Now this may be the result of mpl not using namespace
packages correctly (eg matplotlib/__init__.py should be empty save for
the namespace package or something like that) but I gotta tell ya,
this looks just plain wrong. Had I not known that namespace packages
were funky like this, it could have taken me a long time to trace this
bug.

I am going to comment out the toolkits namespace package code until I
hear persuasive arguments to the contrary, and until we fix the rest
of mpl to work properly with them.

JDH

2

Welcome to the wonderful world of setuptools' under-the-covers magic :slight_smile:

Cheers,

f

···

On Dec 13, 2007 11:18 AM, John Hunter <jdh2358@...149...> wrote:

Do we need namespace packages in toolkits? I recently added gtktools
and exceltools to toolkits, and got a very hard to debug error:

3

Fernando Perez wrote:

···

On Dec 13, 2007 11:18 AM, John Hunter <jdh2358@...149...> wrote:

Do we need namespace packages in toolkits? I recently added gtktools
and exceltools to toolkits, and got a very hard to debug error:

Welcome to the wonderful world of setuptools' under-the-covers magic :slight_smile:

Fernando,

Was this also the culprit that you and someone from enthought identified as causing the slow startup when using traits? This was a while back when I was complaining about a big startup time increase with Darren's traits-enabled configuration scheme.

Eric

4

Kind of, in a general sense. As best as we understood it from simple
profiling runs, the issue was that when NS packages are enabled, the
number of filesystem calls made in path searches explodes. That's
obviously always going to be a problem, but it's particularly
concerning for those of us (like John and myself) who live in
environments where most things are network-mounted disks (NFS,
typically). Tens of thousands of path searches over the network can
really cause a hit.

But setuptools is complex enough that I don't want to badmouth it
without a very precise analysis of the real root causes of the
problems, which I don't have time for right now. I just know that
using it well will require some understanding of its behavior, which
is a bit 'magic' at times.

Cheers,

f

···

On Dec 13, 2007 11:47 AM, Eric Firing <efiring@...229...> wrote:

Fernando Perez wrote:
> On Dec 13, 2007 11:18 AM, John Hunter <jdh2358@...149...> wrote:
>> Do we need namespace packages in toolkits? I recently added gtktools
>> and exceltools to toolkits, and got a very hard to debug error:
>
> Welcome to the wonderful world of setuptools' under-the-covers magic :slight_smile:

Fernando,

Was this also the culprit that you and someone from enthought identified
as causing the slow startup when using traits? This was a while back
when I was complaining about a big startup time increase with Darren's
traits-enabled configuration scheme.

5

John Hunter wrote:

Do we need namespace packages in toolkits? I recently added gtktools
and exceltools to toolkits, and got a very hard to debug error:

In [1]: import matplotlib

In [2]: matplotlib.__file__
Out[2]: '/home/titan/johnh/dev/lib/python2.4/site-packages/matplotlib/__init__.pyc'

In [3]: matplotlib.rcParams['axes.axisbelow']
Out[3]: False

In [4]: import matplotlib.toolkits.gtktools

In [5]: matplotlib.__file__
Out[5]: '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/matplotlib/__init__.pyc'

In [6]: matplotlib.rcParams['axes.axisbelow']
------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in ?
KeyError: 'axes.axisbelow'

Notice that the matplotlib module which was previously imported got
changed by the import of the toolkit which declared itself a namespace
package. Now this may be the result of mpl not using namespace
packages correctly (eg matplotlib/__init__.py should be empty save for
the namespace package or something like that)

Probably. In setuptools 0.7, they can have stuff in them, but all of them must
be identical. If you are using an unmodified matplotlib/__init__.py in the main
matplotlib package, then the "if 0:" is preventing the declaration of the
namespace package, and that may be contributing.

Also, neither matplotlib nor matplotlib.toolkits is named as a namespace_package
in matplotlib's setup.py, and both have to be. Similarly, basemap (for example)
only lists matplotlib.toolkits but not matplotlib.

but I gotta tell ya,
this looks just plain wrong. Had I not known that namespace packages
were funky like this, it could have taken me a long time to trace this
bug.

Well, when you do exactly what you were told not to do, this is what happens.
Don't make me break out the "Doctor, it hurts when I do this!" bromide. If you
don't want to bother to learn to use namespace packages correctly, that's
perfectly fine; just don't use them. But certainly don't blame the tools for
your mistake.

···

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
  -- Umberto Eco

6

John Hunter wrote:
> Do we need namespace packages in toolkits? I recently added gtktools
> and exceltools to toolkits, and got a very hard to debug error:
>
> In [1]: import matplotlib
>
> In [2]: matplotlib.__file__
> Out[2]:
> '/home/titan/johnh/dev/lib/python2.4/site-packages/matplotlib/__init__.py
>c'
>
> In [3]: matplotlib.rcParams['axes.axisbelow']
> Out[3]: False
>
> In [4]: import matplotlib.toolkits.gtktools
>
> In [5]: matplotlib.__file__
> Out[5]:
> '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/matplotlib/__ini
>t__.pyc'
>
> In [6]: matplotlib.rcParams['axes.axisbelow']
> ------------------------------------------------------------
> Traceback (most recent call last):
> File "<ipython console>", line 1, in ?
> KeyError: 'axes.axisbelow'
>
> Notice that the matplotlib module which was previously imported got
> changed by the import of the toolkit which declared itself a namespace
> package. Now this may be the result of mpl not using namespace
> packages correctly (eg matplotlib/__init__.py should be empty save for
> the namespace package or something like that)

Probably. In setuptools 0.7, they can have stuff in them, but all of them
must be identical.

How would one learn about this new feature? According to setuptools.txt from
the 0.7 svn checkout, the namespace package's __init__.py must still be empty
save the namespace declaration.

If you are using an unmodified matplotlib/__init__.py in
the main matplotlib package, then the "if 0:" is preventing the declaration
of the namespace package, and that may be contributing.

Also, neither matplotlib nor matplotlib.toolkits is named as a
namespace_package in matplotlib's setup.py, and both have to be. Similarly,
basemap (for example) only lists matplotlib.toolkits but not matplotlib.

If I understand correctly, even after fixing these two problems, namespace
packages would not work since matplotlib/__init__.py contains lots of
meaningful code. Based on the remarks at
[Distutils] setuptools development cycle and
the setuptools svn commit log, it looks like development on version 0.7 is
not a priority and a release in the next year is unlikely. So is it fair to
say that for the foreseeable future, matplotlib is not compatible with
namespace packages unless we cleaned out __init__.py?

Darren

···

On Thursday 13 December 2007 2:22:13 pm Robert Kern wrote:

7

Darren Dale wrote:

John Hunter wrote:

Do we need namespace packages in toolkits? I recently added gtktools
and exceltools to toolkits, and got a very hard to debug error:

In [1]: import matplotlib

In [2]: matplotlib.__file__
Out[2]:
'/home/titan/johnh/dev/lib/python2.4/site-packages/matplotlib/__init__.py
c'

In [3]: matplotlib.rcParams['axes.axisbelow']
Out[3]: False

In [4]: import matplotlib.toolkits.gtktools

In [5]: matplotlib.__file__
Out[5]:
'/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/matplotlib/__ini
t__.pyc'

In [6]: matplotlib.rcParams['axes.axisbelow']
------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in ?
KeyError: 'axes.axisbelow'

Notice that the matplotlib module which was previously imported got
changed by the import of the toolkit which declared itself a namespace
package. Now this may be the result of mpl not using namespace
packages correctly (eg matplotlib/__init__.py should be empty save for
the namespace package or something like that)

Probably. In setuptools 0.7, they can have stuff in them, but all of them
must be identical.

How would one learn about this new feature? According to setuptools.txt from
the 0.7 svn checkout, the namespace package's __init__.py must still be empty
save the namespace declaration.

It's a logical consequence of how namespace packages are loaded in 0.7 as has
been explained on the mailing list whenever the subject of namespace packages in
0.7 is brought up. It's still not recommended, but it can be made to work. The
remaining objection in the documentation is about installing the package using
--single-version-externally-managed for distro packagers. Using --svem, the
__init__.py files are not installed. Since a distro package would have a
mynamespace-common package (or something similar) that *just* provides this
__init__.py file for all of the mynamespace.foo packages, it can provide the
single copy of the meaningful code.

But it's still not a good idea.

If you are using an unmodified matplotlib/__init__.py in
the main matplotlib package, then the "if 0:" is preventing the declaration
of the namespace package, and that may be contributing.

Also, neither matplotlib nor matplotlib.toolkits is named as a
namespace_package in matplotlib's setup.py, and both have to be. Similarly,
basemap (for example) only lists matplotlib.toolkits but not matplotlib.

If I understand correctly, even after fixing these two problems, namespace
packages would not work since matplotlib/__init__.py contains lots of
meaningful code. Based on the remarks at
[Distutils] setuptools development cycle and
the setuptools svn commit log, it looks like development on version 0.7 is
not a priority and a release in the next year is unlikely. So is it fair to
say that for the foreseeable future, matplotlib is not compatible with
namespace packages unless we cleaned out __init__.py?

Yes. Phillip explained this in February and suggested a way around the issue:

  http://www.mail-archive.com/distutils-sig@...147.../msg02908.html

···

On Thursday 13 December 2007 2:22:13 pm Robert Kern wrote:

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
  -- Umberto Eco