Manylinux external library test case - matplotlib, tk

Development matplotlib-devel
21

Yes, Tcl/Tk is technically optional, but it is usually included. However, I
am not entirely convinced that restricted environments like google app
engine would really benefit from having tkagg available. These sort of
environments are intended to be headless environments, anyway, so having
AGG available is sufficient.

My concern is about making it easier for everyone else, even if it will
take some time to get there. Who would be best to head up the submission of
the patch to Tkinter?

···

On Wed, May 4, 2016 at 5:30 AM, Nathaniel Smith <njs at pobox.com> wrote:

On Wed, May 4, 2016 at 12:40 AM, Olivier Grisel > <olivier.grisel at ensta.org> wrote:
> 2016-05-03 22:47 GMT+02:00 Matthew Brett <matthew.brett at gmail.com>:
>> On Tue, May 3, 2016 at 5:35 AM, Olivier Grisel < > olivier.grisel at ensta.org> wrote:
>>>> I tested it with:
>>>>
>>>> import matplotlib
>>>> matplotlib.use('PyQt5')
>>>
>>> Typo, this was supposed to read:
>>>
>>> matplotlib.use('Qt5Agg')
>>
>> Meanwhile, I tried removing (patchelf --remove-needed) the
>> requirements of _tkagg.so on the vendored libtk, libtcl, and added
>> back the requirement (patch --add-needed) on general `libtk.so' and
>> 'libtcl.so'. This removed the segfault and allowed me to display
>> `plt.range(10)'. I suppose then, that the tk / tcl ABI that we are
>> using is relatively stable across versions 8.4 (on the docker image)
>> and 8.6 (on Debian sid).
>>
>> Worth experimenting with this - or too much of a hack?
>>
>> Is there a way to add libraries to ignore, in `auditwheel repair` ?
>
> It sounds reasonable to add this feature as a CLI option to me.
>
> Also maybe we could decide to amend PEP 513 to add libtk.so and
> libtcl.so to the system provided white-list as they are direct
> dependencies for Python via tkinter which is part of the standard
> library.

I believe that they're optional dependencies, i.e. if you build python
on a system that's missing TCL/TK then it just disables those modules?
They might be available-in-fact on every system we care about, I don't
know -- but unfortunately we can't assume that they're
available-in-principle :-/.

(It looks like Debian at least splits tk off into its own python-tk
package.)

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel at python.org
Matplotlib-devel Info Page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/matplotlib-devel/attachments/20160504/7531d394/attachment.html&gt;

22

Hi,

I tested it with:

import matplotlib
matplotlib.use('PyQt5')

Typo, this was supposed to read:

matplotlib.use('Qt5Agg')

Meanwhile, I tried removing (patchelf --remove-needed) the
requirements of _tkagg.so on the vendored libtk, libtcl, and added
back the requirement (patch --add-needed) on general `libtk.so' and
'libtcl.so'. This removed the segfault and allowed me to display
`plt.range(10)'. I suppose then, that the tk / tcl ABI that we are
using is relatively stable across versions 8.4 (on the docker image)
and 8.6 (on Debian sid).

Worth experimenting with this - or too much of a hack?

I have experimented with this. It does seem that we can get away with
using the system tcl / tk libraries as installed, at least on a couple
of Debian systems I have.

Now there's a new problem, to do with linux library namespaces. We
need the _tkagg.so file to use the same libraries as the Python
Tkinter package at run time. There can be more than one tcl / tk
version installed on a given system, so, as things stand, we'd need to
inspect the Tkinter extension modules to see what they are using, and
then work out a way to use those libraries at run time. Here are the
questions for the experts out there:

* Can anyone think of a way of using the symbol namespace of the
tkinter extension modules directly, to pick up the symbols we need
rather than re-importing the libraries?
* If we do discover which libraries Tkinter is using, what would be a
good way of pulling in these libraries at run-time?

Cheers,

Matthew

···

On Tue, May 3, 2016 at 4:47 PM, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 3, 2016 at 5:35 AM, Olivier Grisel <olivier.grisel at ensta.org> wrote:

23

Hi,

>>> I tested it with:
>>>
>>> import matplotlib
>>> matplotlib.use('PyQt5')
>>
>> Typo, this was supposed to read:
>>
>> matplotlib.use('Qt5Agg')
>
> Meanwhile, I tried removing (patchelf --remove-needed) the
> requirements of _tkagg.so on the vendored libtk, libtcl, and added
> back the requirement (patch --add-needed) on general `libtk.so' and
> 'libtcl.so'. This removed the segfault and allowed me to display
> `plt.range(10)'. I suppose then, that the tk / tcl ABI that we are
> using is relatively stable across versions 8.4 (on the docker image)
> and 8.6 (on Debian sid).
>
> Worth experimenting with this - or too much of a hack?

I have experimented with this. It does seem that we can get away with
using the system tcl / tk libraries as installed, at least on a couple
of Debian systems I have.

Now there's a new problem, to do with linux library namespaces. We
need the _tkagg.so file to use the same libraries as the Python
Tkinter package at run time. There can be more than one tcl / tk
version installed on a given system, so, as things stand, we'd need to
inspect the Tkinter extension modules to see what they are using, and
then work out a way to use those libraries at run time. Here are the
questions for the experts out there:

* Can anyone think of a way of using the symbol namespace of the
tkinter extension modules directly, to pick up the symbols we need
rather than re-importing the libraries?

It would be a bit of a weird hack, but we could have _tkagg.so "link"
against the tkinter extension module. The idea would be that we don't care
about the symbols that the tkinter extension exports, but we'd take
advantage on the fact the on ELF, linking against one .so automatically
gives you not just its symbols, but also the symbols of all the libraries
that it links against, transitively.

This would probably look like:

_tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever Python calls
this module)

Before loading _tkagg.so, we use Python level introspection figure out
where tkinter.so lives

We add its directory to LD_LIBRARY_PATH

we import _tkagg.so

We take its directory back off of LD_LIBRARY_PATH

Very weird, but I can't see why it wouldn't work, and probably more
reliable than anything where we try to reimplement the dynamic loader's
search logic ourselves.

* If we do discover which libraries Tkinter is using, what would be a
good way of pulling in these libraries at run-time?

Cheers,

Matthew

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/matplotlib-devel/attachments/20160510/a01f6a9c/attachment.html&gt;

···

On May 10, 2016 09:00, "Matthew Brett" <matthew.brett at gmail.com> wrote:

On Tue, May 3, 2016 at 4:47 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> On Tue, May 3, 2016 at 5:35 AM, Olivier Grisel <olivier.grisel at ensta.org> wrote:

24

Hi,

···

On Tue, May 10, 2016 at 4:13 PM, Nathaniel Smith <njs at pobox.com> wrote:

On May 10, 2016 09:00, "Matthew Brett" <matthew.brett at gmail.com> wrote:

Hi,

On Tue, May 3, 2016 at 4:47 PM, Matthew Brett <matthew.brett at gmail.com> >> wrote:
> On Tue, May 3, 2016 at 5:35 AM, Olivier Grisel >> > <olivier.grisel at ensta.org> wrote:
>>> I tested it with:
>>>
>>> import matplotlib
>>> matplotlib.use('PyQt5')
>>
>> Typo, this was supposed to read:
>>
>> matplotlib.use('Qt5Agg')
>
> Meanwhile, I tried removing (patchelf --remove-needed) the
> requirements of _tkagg.so on the vendored libtk, libtcl, and added
> back the requirement (patch --add-needed) on general `libtk.so' and
> 'libtcl.so'. This removed the segfault and allowed me to display
> `plt.range(10)'. I suppose then, that the tk / tcl ABI that we are
> using is relatively stable across versions 8.4 (on the docker image)
> and 8.6 (on Debian sid).
>
> Worth experimenting with this - or too much of a hack?

I have experimented with this. It does seem that we can get away with
using the system tcl / tk libraries as installed, at least on a couple
of Debian systems I have.

Now there's a new problem, to do with linux library namespaces. We
need the _tkagg.so file to use the same libraries as the Python
Tkinter package at run time. There can be more than one tcl / tk
version installed on a given system, so, as things stand, we'd need to
inspect the Tkinter extension modules to see what they are using, and
then work out a way to use those libraries at run time. Here are the
questions for the experts out there:

* Can anyone think of a way of using the symbol namespace of the
tkinter extension modules directly, to pick up the symbols we need
rather than re-importing the libraries?

It would be a bit of a weird hack, but we could have _tkagg.so "link"
against the tkinter extension module. The idea would be that we don't care
about the symbols that the tkinter extension exports, but we'd take
advantage on the fact the on ELF, linking against one .so automatically
gives you not just its symbols, but also the symbols of all the libraries
that it links against, transitively.

This would probably look like:

_tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever Python calls
this module)

Before loading _tkagg.so, we use Python level introspection figure out where
tkinter.so lives

We add its directory to LD_LIBRARY_PATH

we import _tkagg.so

We take its directory back off of LD_LIBRARY_PATH

Very weird, but I can't see why it wouldn't work, and probably more reliable
than anything where we try to reimplement the dynamic loader's search logic
ourselves.

Nice - yes - it does work in a first-pass test - I'll look into automating that.

Cheers,

Matthew

25

Ah - except I do not believe it is possible to change the linker path
for Python modules, within the Python process:

Ugh.

Matthew

···

On Tue, May 10, 2016 at 5:26 PM, Matthew Brett <matthew.brett at gmail.com> wrote:

Hi,

On Tue, May 10, 2016 at 4:13 PM, Nathaniel Smith <njs at pobox.com> wrote:

On May 10, 2016 09:00, "Matthew Brett" <matthew.brett at gmail.com> wrote:

Hi,

On Tue, May 3, 2016 at 4:47 PM, Matthew Brett <matthew.brett at gmail.com> >>> wrote:
> On Tue, May 3, 2016 at 5:35 AM, Olivier Grisel >>> > <olivier.grisel at ensta.org> wrote:
>>> I tested it with:
>>>
>>> import matplotlib
>>> matplotlib.use('PyQt5')
>>
>> Typo, this was supposed to read:
>>
>> matplotlib.use('Qt5Agg')
>
> Meanwhile, I tried removing (patchelf --remove-needed) the
> requirements of _tkagg.so on the vendored libtk, libtcl, and added
> back the requirement (patch --add-needed) on general `libtk.so' and
> 'libtcl.so'. This removed the segfault and allowed me to display
> `plt.range(10)'. I suppose then, that the tk / tcl ABI that we are
> using is relatively stable across versions 8.4 (on the docker image)
> and 8.6 (on Debian sid).
>
> Worth experimenting with this - or too much of a hack?

I have experimented with this. It does seem that we can get away with
using the system tcl / tk libraries as installed, at least on a couple
of Debian systems I have.

Now there's a new problem, to do with linux library namespaces. We
need the _tkagg.so file to use the same libraries as the Python
Tkinter package at run time. There can be more than one tcl / tk
version installed on a given system, so, as things stand, we'd need to
inspect the Tkinter extension modules to see what they are using, and
then work out a way to use those libraries at run time. Here are the
questions for the experts out there:

* Can anyone think of a way of using the symbol namespace of the
tkinter extension modules directly, to pick up the symbols we need
rather than re-importing the libraries?

It would be a bit of a weird hack, but we could have _tkagg.so "link"
against the tkinter extension module. The idea would be that we don't care
about the symbols that the tkinter extension exports, but we'd take
advantage on the fact the on ELF, linking against one .so automatically
gives you not just its symbols, but also the symbols of all the libraries
that it links against, transitively.

This would probably look like:

_tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever Python calls
this module)

Before loading _tkagg.so, we use Python level introspection figure out where
tkinter.so lives

We add its directory to LD_LIBRARY_PATH

we import _tkagg.so

We take its directory back off of LD_LIBRARY_PATH

Very weird, but I can't see why it wouldn't work, and probably more reliable
than anything where we try to reimplement the dynamic loader's search logic
ourselves.

Nice - yes - it does work in a first-pass test - I'll look into automating that.

26

I was about to point that out...

On the other hand, if there is a `Tkinter` module that Python can import,
just importing it will make the symbols needed the `_tkagg` module
available, assuming Python calls `dlopen()` on import with RTLD_GLOBAL[1].

[1] dlopen(3) - Linux manual page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/matplotlib-devel/attachments/20160510/5ec91db9/attachment.html&gt;

···

On 10 May 2016 at 18:52, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 5:26 PM, Matthew Brett <matthew.brett at gmail.com> > wrote:
>> [...]
>>
>> This would probably look like:
>>
>> _tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever Python
calls
>> this module)
>>
>> Before loading _tkagg.so, we use Python level introspection figure out
where
>> tkinter.so lives
>>
>> We add its directory to LD_LIBRARY_PATH
>>
>> we import _tkagg.so
>>
>> We take its directory back off of LD_LIBRARY_PATH
>>
>> Very weird, but I can't see why it wouldn't work, and probably more
reliable
>> than anything where we try to reimplement the dynamic loader's search
logic
>> ourselves.
>
> Nice - yes - it does work in a first-pass test - I'll look into
automating that.

Ah - except I do not believe it is possible to change the linker path
for Python modules, within the Python process:

Carbon60: Managed Cloud Services

27

>> [...]
>>
>> This would probably look like:
>>
>> _tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever Python
>> calls
>> this module)
>>
>> Before loading _tkagg.so, we use Python level introspection figure out
>> where
>> tkinter.so lives
>>
>> We add its directory to LD_LIBRARY_PATH
>>
>> we import _tkagg.so
>>
>> We take its directory back off of LD_LIBRARY_PATH
>>
>> Very weird, but I can't see why it wouldn't work, and probably more
>> reliable
>> than anything where we try to reimplement the dynamic loader's search
>> logic
>> ourselves.
>
> Nice - yes - it does work in a first-pass test - I'll look into
> automating that.

Ah - except I do not believe it is possible to change the linker path
for Python modules, within the Python process:

Carbon60: Managed Cloud Services

Doh :frowning:

I was about to point that out...

On the other hand, if there is a `Tkinter` module that Python can import,
just importing it will make the symbols needed the `_tkagg` module
available, assuming Python calls `dlopen()` on import with RTLD_GLOBAL[1].

[1] dlopen(3) - Linux manual page

But it doesn't use RTLD_GLOBAL. You can make it with
sys.setdlopenflags, but this then dumps a bunch of symbols into
*every* Python extension module's namespace, which is pretty risky...

-n

···

On Tue, May 10, 2016 at 3:09 PM, Leonardo Rochael Almeida <leorochael at gmail.com> wrote:

On 10 May 2016 at 18:52, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 5:26 PM, Matthew Brett <matthew.brett at gmail.com> >> wrote:

--
Nathaniel J. Smith -- https://vorpus.org

28

Any other suggestions?

I guess this will be a problem for the external library PEP?

Matthew

···

On Tue, May 10, 2016 at 6:38 PM, Nathaniel Smith <njs at pobox.com> wrote:

On Tue, May 10, 2016 at 3:09 PM, Leonardo Rochael Almeida > <leorochael at gmail.com> wrote:

On 10 May 2016 at 18:52, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 5:26 PM, Matthew Brett <matthew.brett at gmail.com> >>> wrote:
>> [...]
>>
>> This would probably look like:
>>
>> _tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever Python
>> calls
>> this module)
>>
>> Before loading _tkagg.so, we use Python level introspection figure out
>> where
>> tkinter.so lives
>>
>> We add its directory to LD_LIBRARY_PATH
>>
>> we import _tkagg.so
>>
>> We take its directory back off of LD_LIBRARY_PATH
>>
>> Very weird, but I can't see why it wouldn't work, and probably more
>> reliable
>> than anything where we try to reimplement the dynamic loader's search
>> logic
>> ourselves.
>
> Nice - yes - it does work in a first-pass test - I'll look into
> automating that.

Ah - except I do not believe it is possible to change the linker path
for Python modules, within the Python process:

Carbon60: Managed Cloud Services

Doh :frowning:

29

>>>
>>> >> [...]
>>> >>
>>> >> This would probably look like:
>>> >>
>>> >> _tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever
Python
>>> >> calls
>>> >> this module)
>>> >>
>>> >> Before loading _tkagg.so, we use Python level introspection figure
out
>>> >> where
>>> >> tkinter.so lives
>>> >>
>>> >> We add its directory to LD_LIBRARY_PATH
>>> >>
>>> >> we import _tkagg.so
>>> >>
>>> >> We take its directory back off of LD_LIBRARY_PATH
>>> >>
>>> >> Very weird, but I can't see why it wouldn't work, and probably more
>>> >> reliable
>>> >> than anything where we try to reimplement the dynamic loader's
search
>>> >> logic
>>> >> ourselves.
>>> >
>>> > Nice - yes - it does work in a first-pass test - I'll look into
>>> > automating that.
>>>
>>> Ah - except I do not believe it is possible to change the linker path
>>> for Python modules, within the Python process:
>>>
>>> Carbon60: Managed Cloud Services
>
> Doh :frowning:

Any other suggestions?

Calling `patchelf` at runtime with the result of introspecting the
`_tkinter.so` module? :slight_smile:

On a more serious note, even though it would be dangerous to call
`sys.setdlopenflags()` with `RTLD_GLOBAL`, we could perhaps manually call:

   - ctypes.CDLL("/path/to/introspected/libtk.so", DLFCN.RTLD_GLOBAL)
   - ctypes.CDLL("/path/to/introspected/libtcl.so", DLFCN.RTLD_GLOBAL)

before importing a `_tkagg.so` with `libtk` and `libtcl` patchelf-removed
(but not re-added).

We would be dumping only libtcl/tk's symbols in the namespace of other
python extension modules.

I guess this will be a problem for the external library PEP?

Likely, yes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/matplotlib-devel/attachments/20160511/dc5711a2/attachment.html&gt;

···

On 11 May 2016 at 10:58, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 6:38 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, May 10, 2016 at 3:09 PM, Leonardo Rochael Almeida > > <leorochael at gmail.com> wrote:
>> On 10 May 2016 at 18:52, Matthew Brett <matthew.brett at gmail.com> wrote:
>>> On Tue, May 10, 2016 at 5:26 PM, Matthew Brett < > matthew.brett at gmail.com> > >>> wrote:

30

>> [...]
>>
>> This would probably look like:
>>
>> _tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever Python
>> calls
>> this module)
>>
>> Before loading _tkagg.so, we use Python level introspection figure out
>> where
>> tkinter.so lives
>>
>> We add its directory to LD_LIBRARY_PATH
>>
>> we import _tkagg.so
>>
>> We take its directory back off of LD_LIBRARY_PATH
>>
>> Very weird, but I can't see why it wouldn't work, and probably more
>> reliable
>> than anything where we try to reimplement the dynamic loader's search
>> logic
>> ourselves.
>
> Nice - yes - it does work in a first-pass test - I'll look into
> automating that.

Ah - except I do not believe it is possible to change the linker path
for Python modules, within the Python process:

Carbon60: Managed Cloud Services

Doh :frowning:

Any other suggestions?

One option is to just drop tk support from the wheels, and stick with
Qt or GTK. Super annoying because it would mean we need to sort out
the Qt-or-GTK wheel situation too, but I guess we have to do that
anyway, and this would just move it up the priority list?

Barring that, I guess the plan of converting _tkagg.so to ctypes is
the only option.

I assume the matplotlib devs are very motivated to get linux wheels --
maybe we can convince them to implement this? :slight_smile:

I guess this will be a problem for the external library PEP?

(It's not a PEP)

For the external library proposal on Linux it's annoying but I think
we're okay -- instead of adjusting LD_LIBRARY_PATH we can fall back to
using the "preload trick", like on Windows. Basically exploiting our
old friend this bug:
   19884 – Document libdl API in glibc to describe loading process in detail.
:-).

And on Windows I know we can use the preload trick, so that's okay.

What I'm really worried about now is OS X -- we need at least one of
these to work: {runtime changes to DYLD_LIBRARY_PATH, the preloading
trick}. Do they? If not then we have a serious problem :-/

-n

···

On Wed, May 11, 2016 at 6:58 AM, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 6:38 PM, Nathaniel Smith <njs at pobox.com> wrote:

On Tue, May 10, 2016 at 3:09 PM, Leonardo Rochael Almeida >> <leorochael at gmail.com> wrote:

On 10 May 2016 at 18:52, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 5:26 PM, Matthew Brett <matthew.brett at gmail.com> >>>> wrote:

--
Nathaniel J. Smith -- https://vorpus.org

31

Hi,

>> [...]
>>
>> This would probably look like:
>>
>> _tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever Python
>> calls
>> this module)
>>
>> Before loading _tkagg.so, we use Python level introspection figure out
>> where
>> tkinter.so lives
>>
>> We add its directory to LD_LIBRARY_PATH
>>
>> we import _tkagg.so
>>
>> We take its directory back off of LD_LIBRARY_PATH
>>
>> Very weird, but I can't see why it wouldn't work, and probably more
>> reliable
>> than anything where we try to reimplement the dynamic loader's search
>> logic
>> ourselves.
>
> Nice - yes - it does work in a first-pass test - I'll look into
> automating that.

Ah - except I do not believe it is possible to change the linker path
for Python modules, within the Python process:

Carbon60: Managed Cloud Services

Doh :frowning:

Any other suggestions?

One option is to just drop tk support from the wheels, and stick with
Qt or GTK. Super annoying because it would mean we need to sort out
the Qt-or-GTK wheel situation too, but I guess we have to do that
anyway, and this would just move it up the priority list?

Barring that, I guess the plan of converting _tkagg.so to ctypes is
the only option.

I assume the matplotlib devs are very motivated to get linux wheels --
maybe we can convince them to implement this? :slight_smile:

I guess this will be a problem for the external library PEP?

(It's not a PEP)

For the external library proposal on Linux it's annoying but I think
we're okay -- instead of adjusting LD_LIBRARY_PATH we can fall back to
using the "preload trick", like on Windows. Basically exploiting our
old friend this bug:
   19884 – Document libdl API in glibc to describe loading process in detail.
:-).

And on Windows I know we can use the preload trick, so that's okay.

What I'm really worried about now is OS X -- we need at least one of
these to work: {runtime changes to DYLD_LIBRARY_PATH, the preloading
trick}. Do they? If not then we have a serious problem :-/

Sorry - writing in haste - but I believe that OSX does have preload
trick. There's something called the "install name" :
http://matthew-brett.github.io/docosx/mac_runtime_link.html#the-install-name
- this is the name the library declares. I believe (but I am not
sure) that once a library is loaded with a given "install name" then
no further libraries with the same required install name get loaded...

Cheers,

Matthew

···

On Wed, May 11, 2016 at 7:26 PM, Nathaniel Smith <njs at pobox.com> wrote:

On Wed, May 11, 2016 at 6:58 AM, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 6:38 PM, Nathaniel Smith <njs at pobox.com> wrote:

On Tue, May 10, 2016 at 3:09 PM, Leonardo Rochael Almeida >>> <leorochael at gmail.com> wrote:

On 10 May 2016 at 18:52, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 5:26 PM, Matthew Brett <matthew.brett at gmail.com> >>>>> wrote:

32

Hi,

···

On Wed, May 11, 2016 at 2:26 PM, Leonardo Rochael Almeida <leorochael at gmail.com> wrote:

On 11 May 2016 at 10:58, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 6:38 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, May 10, 2016 at 3:09 PM, Leonardo Rochael Almeida >> > <leorochael at gmail.com> wrote:
>> On 10 May 2016 at 18:52, Matthew Brett <matthew.brett at gmail.com> wrote:
>>>
>>> On Tue, May 10, 2016 at 5:26 PM, Matthew Brett >> >>> <matthew.brett at gmail.com> >> >>> wrote:
>>> >> [...]
>>> >>
>>> >> This would probably look like:
>>> >>
>>> >> _tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever
>>> >> Python
>>> >> calls
>>> >> this module)
>>> >>
>>> >> Before loading _tkagg.so, we use Python level introspection figure
>>> >> out
>>> >> where
>>> >> tkinter.so lives
>>> >>
>>> >> We add its directory to LD_LIBRARY_PATH
>>> >>
>>> >> we import _tkagg.so
>>> >>
>>> >> We take its directory back off of LD_LIBRARY_PATH
>>> >>
>>> >> Very weird, but I can't see why it wouldn't work, and probably more
>>> >> reliable
>>> >> than anything where we try to reimplement the dynamic loader's
>>> >> search
>>> >> logic
>>> >> ourselves.
>>> >
>>> > Nice - yes - it does work in a first-pass test - I'll look into
>>> > automating that.
>>>
>>> Ah - except I do not believe it is possible to change the linker path
>>> for Python modules, within the Python process:
>>>
>>> Carbon60: Managed Cloud Services
>
> Doh :frowning:

Any other suggestions?

Calling `patchelf` at runtime with the result of introspecting the
`_tkinter.so` module? :slight_smile:

On a more serious note, even though it would be dangerous to call
`sys.setdlopenflags()` with `RTLD_GLOBAL`, we could perhaps manually call:

ctypes.CDLL("/path/to/introspected/libtk.so", DLFCN.RTLD_GLOBAL)
ctypes.CDLL("/path/to/introspected/libtcl.so", DLFCN.RTLD_GLOBAL)

before importing a `_tkagg.so` with `libtk` and `libtcl` patchelf-removed
(but not re-added).

We would be dumping only libtcl/tk's symbols in the namespace of other
python extension modules.

I have implemented dlopen dynamic loading of the TCL / Tk functions
from the Python tkinter extension module here :

It appears to work. I'd love any feedback y'all can offer.

Cheers,

Matthew

33

I patched matplotlib 1.5.1 with a version of the PR at [1] to build
the wheels using the script at [2] and travis-ci repo build at [3].

The built wheels are up at
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/
for testing.

They work for me, for example doing tkagg animations.

Cheers,

Matthew

[1] loading TCL / Tk symbols dynamically by matthew-brett · Pull Request #6442 · matplotlib/matplotlib · GitHub
[2] https://github.com/matthew-brett/manylinux-builds/blob/master/build_matplotlibs.sh
[3] Travis CI - Test and Deploy with Confidence

···

On Tue, May 17, 2016 at 4:23 PM, Matthew Brett <matthew.brett at gmail.com> wrote:

Hi,

On Wed, May 11, 2016 at 2:26 PM, Leonardo Rochael Almeida > <leorochael at gmail.com> wrote:

On 11 May 2016 at 10:58, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 6:38 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, May 10, 2016 at 3:09 PM, Leonardo Rochael Almeida >>> > <leorochael at gmail.com> wrote:
>> On 10 May 2016 at 18:52, Matthew Brett <matthew.brett at gmail.com> wrote:
>>>
>>> On Tue, May 10, 2016 at 5:26 PM, Matthew Brett >>> >>> <matthew.brett at gmail.com> >>> >>> wrote:
>>> >> [...]
>>> >>
>>> >> This would probably look like:
>>> >>
>>> >> _tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever
>>> >> Python
>>> >> calls
>>> >> this module)
>>> >>
>>> >> Before loading _tkagg.so, we use Python level introspection figure
>>> >> out
>>> >> where
>>> >> tkinter.so lives
>>> >>
>>> >> We add its directory to LD_LIBRARY_PATH
>>> >>
>>> >> we import _tkagg.so
>>> >>
>>> >> We take its directory back off of LD_LIBRARY_PATH
>>> >>
>>> >> Very weird, but I can't see why it wouldn't work, and probably more
>>> >> reliable
>>> >> than anything where we try to reimplement the dynamic loader's
>>> >> search
>>> >> logic
>>> >> ourselves.
>>> >
>>> > Nice - yes - it does work in a first-pass test - I'll look into
>>> > automating that.
>>>
>>> Ah - except I do not believe it is possible to change the linker path
>>> for Python modules, within the Python process:
>>>
>>> Carbon60: Managed Cloud Services
>
> Doh :frowning:

Any other suggestions?

Calling `patchelf` at runtime with the result of introspecting the
`_tkinter.so` module? :slight_smile:

On a more serious note, even though it would be dangerous to call
`sys.setdlopenflags()` with `RTLD_GLOBAL`, we could perhaps manually call:

ctypes.CDLL("/path/to/introspected/libtk.so", DLFCN.RTLD_GLOBAL)
ctypes.CDLL("/path/to/introspected/libtcl.so", DLFCN.RTLD_GLOBAL)

before importing a `_tkagg.so` with `libtk` and `libtcl` patchelf-removed
(but not re-added).

We would be dumping only libtcl/tk's symbols in the namespace of other
python extension modules.

I have implemented dlopen dynamic loading of the TCL / Tk functions
from the Python tkinter extension module here :
loading TCL / Tk symbols dynamically by matthew-brett · Pull Request #6442 · matplotlib/matplotlib · GitHub

It appears to work. I'd love any feedback y'all can offer.

34

Travis testing of these wheels, and some errors, reported at

Cheers,

Matthew

···

On Mon, May 23, 2016 at 12:53 PM, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 17, 2016 at 4:23 PM, Matthew Brett <matthew.brett at gmail.com> wrote:

Hi,

On Wed, May 11, 2016 at 2:26 PM, Leonardo Rochael Almeida >> <leorochael at gmail.com> wrote:

On 11 May 2016 at 10:58, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 6:38 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, May 10, 2016 at 3:09 PM, Leonardo Rochael Almeida >>>> > <leorochael at gmail.com> wrote:
>> On 10 May 2016 at 18:52, Matthew Brett <matthew.brett at gmail.com> wrote:
>>>
>>> On Tue, May 10, 2016 at 5:26 PM, Matthew Brett >>>> >>> <matthew.brett at gmail.com> >>>> >>> wrote:
>>> >> [...]
>>> >>
>>> >> This would probably look like:
>>> >>
>>> >> _tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever
>>> >> Python
>>> >> calls
>>> >> this module)
>>> >>
>>> >> Before loading _tkagg.so, we use Python level introspection figure
>>> >> out
>>> >> where
>>> >> tkinter.so lives
>>> >>
>>> >> We add its directory to LD_LIBRARY_PATH
>>> >>
>>> >> we import _tkagg.so
>>> >>
>>> >> We take its directory back off of LD_LIBRARY_PATH
>>> >>
>>> >> Very weird, but I can't see why it wouldn't work, and probably more
>>> >> reliable
>>> >> than anything where we try to reimplement the dynamic loader's
>>> >> search
>>> >> logic
>>> >> ourselves.
>>> >
>>> > Nice - yes - it does work in a first-pass test - I'll look into
>>> > automating that.
>>>
>>> Ah - except I do not believe it is possible to change the linker path
>>> for Python modules, within the Python process:
>>>
>>> Carbon60: Managed Cloud Services
>
> Doh :frowning:

Any other suggestions?

Calling `patchelf` at runtime with the result of introspecting the
`_tkinter.so` module? :slight_smile:

On a more serious note, even though it would be dangerous to call
`sys.setdlopenflags()` with `RTLD_GLOBAL`, we could perhaps manually call:

ctypes.CDLL("/path/to/introspected/libtk.so", DLFCN.RTLD_GLOBAL)
ctypes.CDLL("/path/to/introspected/libtcl.so", DLFCN.RTLD_GLOBAL)

before importing a `_tkagg.so` with `libtk` and `libtcl` patchelf-removed
(but not re-added).

We would be dumping only libtcl/tk's symbols in the namespace of other
python extension modules.

I have implemented dlopen dynamic loading of the TCL / Tk functions
from the Python tkinter extension module here :
loading TCL / Tk symbols dynamically by matthew-brett · Pull Request #6442 · matplotlib/matplotlib · GitHub

It appears to work. I'd love any feedback y'all can offer.

I patched matplotlib 1.5.1 with a version of the PR at [1] to build
the wheels using the script at [2] and travis-ci repo build at [3].

The built wheels are up at
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/
for testing.

They work for me, for example doing tkagg animations.

35

Thanks Olivier for testing.

Thomas Caswell gave me the pointers I needed to fix the test failures,
tests now passing at :

I think these wheels are now ready to ship - MPL issue at :

Cheers,

Matthew

···

On Mon, May 23, 2016 at 3:37 PM, Matthew Brett <matthew.brett at gmail.com> wrote:

On Mon, May 23, 2016 at 12:53 PM, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 17, 2016 at 4:23 PM, Matthew Brett <matthew.brett at gmail.com> wrote:

Hi,

On Wed, May 11, 2016 at 2:26 PM, Leonardo Rochael Almeida >>> <leorochael at gmail.com> wrote:

On 11 May 2016 at 10:58, Matthew Brett <matthew.brett at gmail.com> wrote:

On Tue, May 10, 2016 at 6:38 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, May 10, 2016 at 3:09 PM, Leonardo Rochael Almeida >>>>> > <leorochael at gmail.com> wrote:
>> On 10 May 2016 at 18:52, Matthew Brett <matthew.brett at gmail.com> wrote:
>>>
>>> On Tue, May 10, 2016 at 5:26 PM, Matthew Brett >>>>> >>> <matthew.brett at gmail.com> >>>>> >>> wrote:
>>> >> [...]
>>> >>
>>> >> This would probably look like:
>>> >>
>>> >> _tkagg.so has a DT_NEEDED entry naming tkinter.so (or whatever
>>> >> Python
>>> >> calls
>>> >> this module)
>>> >>
>>> >> Before loading _tkagg.so, we use Python level introspection figure
>>> >> out
>>> >> where
>>> >> tkinter.so lives
>>> >>
>>> >> We add its directory to LD_LIBRARY_PATH
>>> >>
>>> >> we import _tkagg.so
>>> >>
>>> >> We take its directory back off of LD_LIBRARY_PATH
>>> >>
>>> >> Very weird, but I can't see why it wouldn't work, and probably more
>>> >> reliable
>>> >> than anything where we try to reimplement the dynamic loader's
>>> >> search
>>> >> logic
>>> >> ourselves.
>>> >
>>> > Nice - yes - it does work in a first-pass test - I'll look into
>>> > automating that.
>>>
>>> Ah - except I do not believe it is possible to change the linker path
>>> for Python modules, within the Python process:
>>>
>>> Carbon60: Managed Cloud Services
>
> Doh :frowning:

Any other suggestions?

Calling `patchelf` at runtime with the result of introspecting the
`_tkinter.so` module? :slight_smile:

On a more serious note, even though it would be dangerous to call
`sys.setdlopenflags()` with `RTLD_GLOBAL`, we could perhaps manually call:

ctypes.CDLL("/path/to/introspected/libtk.so", DLFCN.RTLD_GLOBAL)
ctypes.CDLL("/path/to/introspected/libtcl.so", DLFCN.RTLD_GLOBAL)

before importing a `_tkagg.so` with `libtk` and `libtcl` patchelf-removed
(but not re-added).

We would be dumping only libtcl/tk's symbols in the namespace of other
python extension modules.

I have implemented dlopen dynamic loading of the TCL / Tk functions
from the Python tkinter extension module here :
loading TCL / Tk symbols dynamically by matthew-brett · Pull Request #6442 · matplotlib/matplotlib · GitHub

It appears to work. I'd love any feedback y'all can offer.

I patched matplotlib 1.5.1 with a version of the PR at [1] to build
the wheels using the script at [2] and travis-ci repo build at [3].

The built wheels are up at
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/
for testing.

They work for me, for example doing tkagg animations.

Travis testing of these wheels, and some errors, reported at
Test failures testing matplotlib 1.5.1 manylinux wheels · Issue #6469 · matplotlib/matplotlib · GitHub