Auto backend discovery at start time

Development matplotlib-devel
1

Hello guys!
Following up the discussion Benjiamin and me had about a couple of
bugs in Ubuntu[1] and Debian[2], and what Mike wrote on [1], we'd like
to explore the possibility for you to develop a "backend=Auto" mode,
that can discover automatically at runtime the backend to use from the
ones available (in case of multiple backends, let's use a priority
list [gtk, qt, tk, ...]).

Our main goal, as packagers, is to avoid the installation of gnome
dependencies (gtk2 is now the default in both sid and
lenny-near-to-be-stable, for Ubuntu is tk) for kde users (and
viceversa, or any other combination).

One way is to ship a "core" package, that contains the main
functionalities, and some "backend" packages to depends (or even
contains the real backend files, like .so & altera) on the available
gui python bindings.

Related to the above sentence, is this question: how easy is to
"split" backend files? is it enough to separate files under
/usr/lib/python*/site-packages/matplotlib/backends/* in different
packages? But that would only create 3 pkgs: agg, gtk and tk. What
about wx and qt?

Personally, I think we can even attack the problem with a different
solution: continue to ship all the mpl file in the "main" package
(python-matplotlib in Debian & Ubuntu) and then create some "dummy"
packages that simply depends on the python gui bindings (let's call
them python-matplotlib-<ui>), each of them providing a virtual
package, let's call it python-matplotlib-backend. If python-matplotlib
depends on python-matplotlib-gtk OR python-matplotlib-backend, any
backend installed can satisfy that dependency (and the default being
gtk).

Both of them has cons: the first poses problem to us for the
packaging, and both does not solve the problem of not choosing a
default (or requiring to specify another package (the backend chosen)
when installing python-matplotlib); moreover, what other packages
depending on python-matplotlib should do after this change (they
expect mpl to Just Work)?

Another solution (that would save the most of the current work done),
almost the same currently used today is: keep doing the same thing as
of now, but do not install any python gui bindings, but popup a
windows at python-matplotlib install time to ask the user what binding
to use (then create the ad-hoc /etc/matplotlirc file with that
"backend" set) and then ask to install the correct python binding for
the backend chosen. A light version is: keep choosing gtk as default
backend, and clearly document (even at install time) how to change
backend.

What you (upstream and distribution packages users) expect from us
(packagers) about these 2 questions (backend=Auto and multi backend
support)? Of course, any suggestion is very welcome :slight_smile:

Thanks in advance,
Sandro

[1] https://bugs.launchpad.net/ubuntu/+source/matplotlib/+bug/278764
[2] http://bugs.debian.org/502976

···

--
Sandro Tosi (aka morph, Morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

2

Hello guys!
Following up the discussion Benjiamin and me had about a couple of
bugs in Ubuntu[1] and Debian[2], and what Mike wrote on [1], we'd like
to explore the possibility for you to develop a "backend=Auto" mode,
that can discover automatically at runtime the backend to use from the
ones available (in case of multiple backends, let's use a priority
list [gtk, qt, tk, ...]).

This should be fairly easy to implement -- we already do something
like this at build time when we choose the default backend to put into
the template. FYI, since you are a packager, I jwant to clarify what
happens at build time. We have a file called setup.cfg.template which
is a template for people who want to override the default behavior.
You can copy this to setup.cfg and choose what default backend you
want, and the setup program will create an rc configured accordingly.
But if someone has not done this, the setup script will look (at build
time) for gtk, wx, and tk, and set the backend in order of increasing
preference: Agg, TkAgg, WXAgg, GTKAgg. The file matplotlibrc.template
is then used to generate the default matplotlibrc, with this backend
selected. This matplotlibrc is installed to matplotlib/mpl-data/ and
is used as the default config if the user does not override it.

As a debian/ubuntu packager, you will probably want to use setup.cfg
and set your backend manually. You may want to use TkAgg since in
some ways this is the simplest backend to use in the most contexts,
primarily because it does not require special threading calls to work
interactively in the python shell -- see
http://matplotlib.sf.net/installing.html

But an "Auto" feature would be useful in other contexts too. One area
is when matplotlib is embedded in a GUI IDE matlab-like application.
There are several of these that are being worked on in different user
interfaces, primarily using the new embeddable ipython, and the
concern there is that the user may be using one application which
embeds a python shell, and when users import pylab in that shell, the
ought not to have to think: "now I am using python in a wx app, so I
need to sert wxagg" but in other scenarios, "now I am using pylab in a
plain python shell so use tkagg"

The auto search algorithm should go something like the following:

* if tkinter, wx, gtk or qt has already been imported in sys.modules,
use that backend - Gael has already an implementation in the pyplot
module using the rc param 'backend_fallback'

* if backend = 'auto': try in order of preference :tkagg (most
likely to work in most contexts), gtkagg, wxagg, qtagg. This order
could easily be configurable

  * if none of the UIs are available in 'auto' mode, issue a warning
and set 'agg'

If I were packaging for ubuntu, I would select tkagg as the default,
so you don't have to wade into the GNOME vs KDE arena and because it
works out of the box in the most contexts and is a pretty small
dependency. You could modify the matplotlib runtime so that when the
.matplotlib directory is created (the first time mpl is run) it issues
a message like

  Creating config directory ~/.matplotlib. The default user interface
toolkit for matplotlib is TkInter via the "TkAgg backend" (see
  http://matplotlib.sourceforge.net/faq/installing_faq.html#id1). You
use other backends, you will need to install additional ubuntu
dependencies.
  For GTKAgg, install python-gtk, for WXAgg, install python-wxgtk, etc..."

Personally, I think we can even attack the problem with a different
solution: continue to ship all the mpl file in the "main" package
(python-matplotlib in Debian & Ubuntu) and then create some "dummy"
packages that simply depends on the python gui bindings (let's call
them python-matplotlib-<ui>), each of them providing a virtual
package, let's call it python-matplotlib-backend. If python-matplotlib
depends on python-matplotlib-gtk OR python-matplotlib-backend, any
backend installed can satisfy that dependency (and the default being
gtk).

That should work fine, but I advise installing all of mpl and use
these dummies only for dependencies.

Both of them has cons: the first poses problem to us for the
packaging, and both does not solve the problem of not choosing a
default (or requiring to specify another package (the backend chosen)
when installing python-matplotlib); moreover, what other packages
depending on python-matplotlib should do after this change (they
expect mpl to Just Work)?

Well, if the package that depends on mpl requires for example wx and
the wx backend, then it is enough for them to get a full mpl install
and handle the wx dependency in their package. They would need to
make sure that the right mpl backend is selected when they import mpl,
but they can do that with the use directive.

Another solution (that would save the most of the current work done),
almost the same currently used today is: keep doing the same thing as
of now, but do not install any python gui bindings, but popup a
windows at python-matplotlib install time to ask the user what binding
to use (then create the ad-hoc /etc/matplotlirc file with that
"backend" set) and then ask to install the correct python binding for
the backend chosen. A light version is: keep choosing gtk as default
backend, and clearly document (even at install time) how to change
backend.

This is in line with what I was suggesting, though I was suggesting it
at mpl first run time. Either way could work.

I do see that this is a problem: a colleague of mine with a new ubuntu
8.10 box installed python-matplotlib as one of his first packages, and
it brought in 280 MB of packages, including several UI toolkits and a
full tex distribution. I think the packagers are being over
inclusive. For optional dependencies like usetex, which most people
do not use, and optional backends, it would be better to have a clear
set of instructions for users who want these optional features to
simply apt-get install the optional dependencies themselves.

JDH

···

On Sat, Nov 1, 2008 at 5:37 AM, Sandro Tosi <morph@...12...> wrote:

3

Hello John & Devs :slight_smile:
first of all, let me excuse for the enormous delay of this reply, it
seems I've lost it somewhere :frowning:

Hello guys!
Following up the discussion Benjiamin and me had about a couple of
bugs in Ubuntu[1] and Debian[2], and what Mike wrote on [1], we'd like
to explore the possibility for you to develop a "backend=Auto" mode,
that can discover automatically at runtime the backend to use from the
ones available (in case of multiple backends, let's use a priority
list [gtk, qt, tk, ...]).

This should be fairly easy to implement -- we already do something
like this at build time when we choose the default backend to put into
the template. FYI, since you are a packager, I jwant to clarify what
happens at build time. We have a file called setup.cfg.template which
is a template for people who want to override the default behavior.
You can copy this to setup.cfg and choose what default backend you
want, and the setup program will create an rc configured accordingly.
But if someone has not done this, the setup script will look (at build
time) for gtk, wx, and tk, and set the backend in order of increasing
preference: Agg, TkAgg, WXAgg, GTKAgg. The file matplotlibrc.template
is then used to generate the default matplotlibrc, with this backend
selected. This matplotlibrc is installed to matplotlib/mpl-data/ and
is used as the default config if the user does not override it.

As a debian/ubuntu packager, you will probably want to use setup.cfg
and set your backend manually. You may want to use TkAgg since in
some ways this is the simplest backend to use in the most contexts,
primarily because it does not require special threading calls to work
interactively in the python shell -- see
http://matplotlib.sf.net/installing.html

yeah, we are doing (since a couple or revisions) exactly this: we
choose Tk as the default backend in setup.cfg so that even
/etc/matplotlibrc has that param set.

But an "Auto" feature would be useful in other contexts too. One area
is when matplotlib is embedded in a GUI IDE matlab-like application.
There are several of these that are being worked on in different user
interfaces, primarily using the new embeddable ipython, and the
concern there is that the user may be using one application which
embeds a python shell, and when users import pylab in that shell, the
ought not to have to think: "now I am using python in a wx app, so I
need to sert wxagg" but in other scenarios, "now I am using pylab in a
plain python shell so use tkagg"

The auto search algorithm should go something like the following:

* if tkinter, wx, gtk or qt has already been imported in sys.modules,
use that backend - Gael has already an implementation in the pyplot
module using the rc param 'backend_fallback'

* if backend = 'auto': try in order of preference :tkagg (most
likely to work in most contexts), gtkagg, wxagg, qtagg. This order
could easily be configurable

* if none of the UIs are available in 'auto' mode, issue a warning
and set 'agg'

If I were packaging for ubuntu, I would select tkagg as the default,
so you don't have to wade into the GNOME vs KDE arena and because it
works out of the box in the most contexts and is a pretty small
dependency. You could modify the matplotlib runtime so that when the
.matplotlib directory is created (the first time mpl is run) it issues
a message like

Creating config directory ~/.matplotlib. The default user interface
toolkit for matplotlib is TkInter via the "TkAgg backend" (see
http://matplotlib.sourceforge.net/faq/installing_faq.html#id1). You
use other backends, you will need to install additional ubuntu
dependencies.
For GTKAgg, install python-gtk, for WXAgg, install python-wxgtk, etc..."

We did a similar thing, but at packaging level: we added a
"notification" messages at package installation that clarify TkAgg is
the default backend and to look at a simple doc file we created to
change it to a different backend, both at machine or user level (the
file is attached, in case you would provide some
corrections/enhancements).

Personally, I think we can even attack the problem with a different
solution: continue to ship all the mpl file in the "main" package
(python-matplotlib in Debian & Ubuntu) and then create some "dummy"
packages that simply depends on the python gui bindings (let's call
them python-matplotlib-<ui>), each of them providing a virtual
package, let's call it python-matplotlib-backend. If python-matplotlib
depends on python-matplotlib-gtk OR python-matplotlib-backend, any
backend installed can satisfy that dependency (and the default being
gtk).

That should work fine, but I advise installing all of mpl and use
these dummies only for dependencies.

Both of them has cons: the first poses problem to us for the
packaging, and both does not solve the problem of not choosing a
default (or requiring to specify another package (the backend chosen)
when installing python-matplotlib); moreover, what other packages
depending on python-matplotlib should do after this change (they
expect mpl to Just Work)?

Well, if the package that depends on mpl requires for example wx and
the wx backend, then it is enough for them to get a full mpl install
and handle the wx dependency in their package. They would need to
make sure that the right mpl backend is selected when they import mpl,
but they can do that with the use directive.

Another solution (that would save the most of the current work done),
almost the same currently used today is: keep doing the same thing as
of now, but do not install any python gui bindings, but popup a
windows at python-matplotlib install time to ask the user what binding
to use (then create the ad-hoc /etc/matplotlirc file with that
"backend" set) and then ask to install the correct python binding for
the backend chosen. A light version is: keep choosing gtk as default
backend, and clearly document (even at install time) how to change
backend.

This is in line with what I was suggesting, though I was suggesting it
at mpl first run time. Either way could work.

I do see that this is a problem: a colleague of mine with a new ubuntu
8.10 box installed python-matplotlib as one of his first packages, and
it brought in 280 MB of packages, including several UI toolkits and a
full tex distribution. I think the packagers are being over
inclusive. For optional dependencies like usetex, which most people
do not use, and optional backends, it would be better to have a clear
set of instructions for users who want these optional features to
simply apt-get install the optional dependencies themselves.

Yeah, we are trying to address this problem, that I try to explain
here to receive your suggestions:

- texlive (LaTeX distribution) is brought in because of dvipng; I
think we can safely move this package to a place where it's not
automatically installed, since it's not a strict dependency
- other gui libraries (like libgtk2.0.0) are brought in because the
package contains .so files that links to those libraries, and this
situation cannot be avoided (leaving the package as it is)
- with a recent change, we avoid the installation of python gtk2
bindings if python-tk is already installed, so all that set of
packages will not be installed by default

These changes should reduce a lot the amount of packages to be
installed (in particular for texlive stuff).

Thanks for your attention,

README.debian (1.16 KB)

···

On Sat, Nov 1, 2008 at 21:14, John Hunter <jdh2358@...149...> wrote:

On Sat, Nov 1, 2008 at 5:37 AM, Sandro Tosi <morph@...12...> wrote:

--
Sandro Tosi (aka morph, Morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi