When building a manylinux wheel, we can only build against the tk libs
/ headers on the manylinux docker image, by doing `yum install -y
tk-devel` before building.
This means that the tk that matplotlib is built against, is different
from the one that Python uses with `import Tkinter`. I suppose that
is what causes the following, when building matplotlib with tk-devel
installed:
import matplotlib
matplotlib.get_backend()
u'TkAgg'
import matplotlib.pyplot as plt
plt.plot(range(10))
[<matplotlib.lines.Line2D object at 0x7f2a8e4b4390>]
plt.show()
Segmentation fault
I don't immediately see a good way to deal with this. Any thoughts?
When building a manylinux wheel, we can only build against the tk libs
/ headers on the manylinux docker image, by doing `yum install -y
tk-devel` before building.
This means that the tk that matplotlib is built against, is different
from the one that Python uses with `import Tkinter`. I suppose that
is what causes the following, when building matplotlib with tk-devel
installed:
What would be better is if we can get rid of the compilation requirement,
but from Michael's comments in the past, it seems that this isn't possible
at the moment. The problem is that the compiled portion provides the image
buffer access, which seems to not be available at the python level. Perhaps
it might make sense to pursue submitting our code to the Tk/Tcl project so
that we might be able to drop this step at some point?
Ben Root
···
On Mon, Apr 25, 2016 at 3:03 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
Hi,
On Mon, Apr 25, 2016 at 11:58 AM, Thomas Caswell <tcaswell at gmail.com> > wrote:
> My only thought is to just not build the tk backend, shipping wheels that
> are known to segfault is no good.
2016-04-25 21:19 GMT+02:00 Benjamin Root <ben.v.root at gmail.com>:
What would be better is if we can get rid of the compilation requirement,
but from Michael's comments in the past, it seems that this isn't possible
at the moment. The problem is that the compiled portion provides the image
buffer access, which seems to not be available at the python level. Perhaps
it might make sense to pursue submitting our code to the Tk/Tcl project so
that we might be able to drop this step at some point?
Wouldn't it be possible to rewrite this part using ctypes instead?
When building a manylinux wheel, we can only build against the tk libs
/ headers on the manylinux docker image, by doing `yum install -y
tk-devel` before building.
This means that the tk that matplotlib is built against, is different
from the one that Python uses with `import Tkinter`. I suppose that
is what causes the following, when building matplotlib with tk-devel
installed:
I don't immediately see a good way to deal with this. Any thoughts?
Maybe obvious, but here's a syllogism:
If:
matplotlib and python have to agree on which tk libs they are using,
and:
there are inconsistencies between tk libs on different Linux systems
such that matplotlib must be built against the same tk as it eventually
uses,
then:
we don't really have any option except to disable tk support in
matplotlib manylinux wheels
So I guess the question to focus on are whether those two premises are
actually true. I guess a backtrace on the segfault would help?
Also: how important is tk support? Do we have a fallback? I know that Qt
packaging is something of a mess right now too...
PySide might be an easier initial target -- they almost support
installing from wheels already. (You can make and post wheels, but
they have a post-install script you have to manually run. Presumably
whatever this does could be run from __init__.py on first import
instead, at least when installing into a venv...)
-n
···
On Mon, Apr 25, 2016 at 2:52 PM, Olivier Grisel <olivier.grisel at ensta.org> wrote:
It would be nice to have a manylinux1 wheel for PyQT with embedded QT libs.
On Mon, Apr 25, 2016 at 12:42 PM, Olivier Grisel <olivier.grisel at ensta.org> wrote:
2016-04-25 21:19 GMT+02:00 Benjamin Root <ben.v.root at gmail.com>:
What would be better is if we can get rid of the compilation requirement,
but from Michael's comments in the past, it seems that this isn't possible
at the moment. The problem is that the compiled portion provides the image
buffer access, which seems to not be available at the python level. Perhaps
it might make sense to pursue submitting our code to the Tk/Tcl project so
that we might be able to drop this step at some point?
Wouldn't it be possible to rewrite this part using ctypes instead?
I had the same thought - has anyone looked into that?
2016-04-25 21:19 GMT+02:00 Benjamin Root <ben.v.root at gmail.com>:
What would be better is if we can get rid of the compilation requirement,
but from Michael's comments in the past, it seems that this isn't possible
at the moment. The problem is that the compiled portion provides the image
buffer access, which seems to not be available at the python level. Perhaps
it might make sense to pursue submitting our code to the Tk/Tcl project so
that we might be able to drop this step at some point?
Wouldn't it be possible to rewrite this part using ctypes instead?
I had the same thought - has anyone looked into that?
I was thinking of looking into this. It would be very useful to
unlink matplotlib from the tk compile. It's already a pain to get
this right for OSX - for the wheel build, we have to download the
activestate tcl/tk, check it's the right version, link to it, and then
make sure we haven't shipped any of it :
I'd be open to a ctypes implementation of the tk backend as long as it's
fully optional (like the current tk backend) since using ctypes excludes us
from use in restricted environments such as Google app engine.
Submitting the buffer hook to tcl/tk and/or tkinter would be nice. All
other modern Python GUI frameworks have this feature.
Mike
···
On Thu, Apr 28, 2016, 8:45 AM Matthew Brett <matthew.brett at gmail.com> wrote:
Hi,
On Mon, Apr 25, 2016 at 4:23 PM, Matthew Brett <matthew.brett at gmail.com> > wrote:
> Hi,
>
> On Mon, Apr 25, 2016 at 12:42 PM, Olivier Grisel > > <olivier.grisel at ensta.org> wrote:
>> 2016-04-25 21:19 GMT+02:00 Benjamin Root <ben.v.root at gmail.com>:
>>> What would be better is if we can get rid of the compilation
requirement,
>>> but from Michael's comments in the past, it seems that this isn't
possible
>>> at the moment. The problem is that the compiled portion provides the
image
>>> buffer access, which seems to not be available at the python level.
Perhaps
>>> it might make sense to pursue submitting our code to the Tk/Tcl
project so
>>> that we might be able to drop this step at some point?
>>
>> Wouldn't it be possible to rewrite this part using ctypes instead?
>
> I had the same thought - has anyone looked into that?
I was thinking of looking into this. It would be very useful to
unlink matplotlib from the tk compile. It's already a pain to get
this right for OSX - for the wheel build, we have to download the
activestate tcl/tk, check it's the right version, link to it, and then
make sure we haven't shipped any of it :
I'm sure I missed the conversation, but as tkInter is included with Python
itself, why can't the wheels link against the same version that python
ships??
Submitting the buffer hook to tcl/tk and/or tkinter would be nice.
That would be the better solution, yes -- presumable to tkinter -- but
that's only going to help with Python 3.6 or greater.....
(or maybe it could be back-ported)
-CHB
···
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
I'm sure I missed the conversation, but as tkInter is included with Python
itself, why can't the wheels link against the same version that python
ships??
The problem is that we're trying to make a single build of matplotlib
that works with multiple different builds of Python, which might come
with different, incompatible builds of tkInter -- so if we build
against the version shipped with python A then it will be broken on
python B and vice-versa.
Or maybe not -- Matthew's seeing a segfault, and it *might* be due to
the existence of fundamental ABI incompatibilities between different
tk builds, but it hasn't really been characterized.
Submitting the buffer hook to tcl/tk and/or tkinter would be nice.
That would be the better solution, yes -- presumable to tkinter -- but
that's only going to help with Python 3.6 or greater.....
(or maybe it could be back-ported)
It might get backported to 3.5, maybe 3.4 if we're really quick, but
it won't get backported to 2.7, so yeah, anything involving changing
python upstream isn't going to be a short-term solution.
-n
···
On Fri, Apr 29, 2016 at 9:23 PM, Chris Barker <chris.barker at noaa.gov> wrote:
Right. Submitting the patch won't address our near-term needs, but it'll
certainly position us better a few years from now. I am thinking that for
the near-term, we could have a fail-over mechanism. First attempt to access
the proposed ik interface (to be forward-compatible), failing that, then
attempt a ctypes interface. Failing that, fall back to our current
implementation. This gives a way forward for those in the restricted
environments to keep doing whatever has been working so far, while giving
everyone else an intermediate solution -- a usable GUI backend that will
almost always be guaranteed to work no matter in what order the packages
were installed.
Cheers!
Ben Root
···
On Sat, Apr 30, 2016 at 3:43 AM, Nathaniel Smith <njs at pobox.com> wrote:
On Fri, Apr 29, 2016 at 9:23 PM, Chris Barker <chris.barker at noaa.gov> > wrote:
> I'm sure I missed the conversation, but as tkInter is included with
Python
> itself, why can't the wheels link against the same version that python
> ships??
The problem is that we're trying to make a single build of matplotlib
that works with multiple different builds of Python, which might come
with different, incompatible builds of tkInter -- so if we build
against the version shipped with python A then it will be broken on
python B and vice-versa.
Or maybe not -- Matthew's seeing a segfault, and it *might* be due to
the existence of fundamental ABI incompatibilities between different
tk builds, but it hasn't really been characterized.
>> Submitting the buffer hook to tcl/tk and/or tkinter would be nice.
>
> That would be the better solution, yes -- presumable to tkinter -- but
> that's only going to help with Python 3.6 or greater.....
>
> (or maybe it could be back-ported)
It might get backported to 3.5, maybe 3.4 if we're really quick, but
it won't get backported to 2.7, so yeah, anything involving changing
python upstream isn't going to be a short-term solution.
-n
--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel at python.org Matplotlib-devel Info Page
I'm in Cuba at the moment, so it's very difficult to run stuff remotely.
If you have access to benten.dlab.berkeley.edu, the build wheel is in
/home/mb312/dev_trees/manylinux-builds/wheelhouse - latest matplotlib
1.5.1 for Python 2.7. Otherwise, the build recipe in manylinux-builds
should give a good clue as to how to replicate.
Cheers,
Matthew
···
On Sat, Apr 30, 2016 at 3:43 AM, Nathaniel Smith <njs at pobox.com> wrote:
On Fri, Apr 29, 2016 at 9:23 PM, Chris Barker <chris.barker at noaa.gov> wrote:
I'm sure I missed the conversation, but as tkInter is included with Python
itself, why can't the wheels link against the same version that python
ships??
The problem is that we're trying to make a single build of matplotlib
that works with multiple different builds of Python, which might come
with different, incompatible builds of tkInter -- so if we build
against the version shipped with python A then it will be broken on
python B and vice-versa.
Or maybe not -- Matthew's seeing a segfault, and it *might* be due to
the existence of fundamental ABI incompatibilities between different
tk builds, but it hasn't really been characterized.
Meanwhile, I tried removing (patchelf --remove-needed) the
requirements of _tkagg.so on the vendored libtk, libtcl, and added
back the requirement (patch --add-needed) on general `libtk.so' and
'libtcl.so'. This removed the segfault and allowed me to display
`plt.range(10)'. I suppose then, that the tk / tcl ABI that we are
using is relatively stable across versions 8.4 (on the docker image)
and 8.6 (on Debian sid).
Worth experimenting with this - or too much of a hack?
Is there a way to add libraries to ignore, in `auditwheel repair` ?
Cheers,
Matthew
···
On Tue, May 3, 2016 at 5:35 AM, Olivier Grisel <olivier.grisel at ensta.org> wrote:
2016-05-03 22:47 GMT+02:00 Matthew Brett <matthew.brett at gmail.com>:
···
On Tue, May 3, 2016 at 5:35 AM, Olivier Grisel <olivier.grisel at ensta.org> wrote:
I tested it with:
import matplotlib
matplotlib.use('PyQt5')
Typo, this was supposed to read:
matplotlib.use('Qt5Agg')
Meanwhile, I tried removing (patchelf --remove-needed) the
requirements of _tkagg.so on the vendored libtk, libtcl, and added
back the requirement (patch --add-needed) on general `libtk.so' and
'libtcl.so'. This removed the segfault and allowed me to display
`plt.range(10)'. I suppose then, that the tk / tcl ABI that we are
using is relatively stable across versions 8.4 (on the docker image)
and 8.6 (on Debian sid).
Worth experimenting with this - or too much of a hack?
Is there a way to add libraries to ignore, in `auditwheel repair` ?
It sounds reasonable to add this feature as a CLI option to me.
Also maybe we could decide to amend PEP 513 to add libtk.so and
libtcl.so to the system provided white-list as they are direct
dependencies for Python via tkinter which is part of the standard
library.
I believe that they're optional dependencies, i.e. if you build python
on a system that's missing TCL/TK then it just disables those modules?
They might be available-in-fact on every system we care about, I don't
know -- but unfortunately we can't assume that they're
available-in-principle :-/.
(It looks like Debian at least splits tk off into its own python-tk package.)
-n
···
On Wed, May 4, 2016 at 12:40 AM, Olivier Grisel <olivier.grisel at ensta.org> wrote:
2016-05-03 22:47 GMT+02:00 Matthew Brett <matthew.brett at gmail.com>:
On Tue, May 3, 2016 at 5:35 AM, Olivier Grisel <olivier.grisel at ensta.org> wrote:
I tested it with:
import matplotlib
matplotlib.use('PyQt5')
Typo, this was supposed to read:
matplotlib.use('Qt5Agg')
Meanwhile, I tried removing (patchelf --remove-needed) the
requirements of _tkagg.so on the vendored libtk, libtcl, and added
back the requirement (patch --add-needed) on general `libtk.so' and
'libtcl.so'. This removed the segfault and allowed me to display
`plt.range(10)'. I suppose then, that the tk / tcl ABI that we are
using is relatively stable across versions 8.4 (on the docker image)
and 8.6 (on Debian sid).
Worth experimenting with this - or too much of a hack?
Is there a way to add libraries to ignore, in `auditwheel repair` ?
It sounds reasonable to add this feature as a CLI option to me.
Also maybe we could decide to amend PEP 513 to add libtk.so and
libtcl.so to the system provided white-list as they are direct
dependencies for Python via tkinter which is part of the standard
library.
- no good at all!