How to make matplotlib use my own libpng over the systemwide

Hi,

As part of building matplotlib for the one python based distribution [1],
I want to always link against our own version of libpng, even if there
is some other systemwide version available. I am on linux (Ubuntu).

Currently, here is what I am doing:

CFLAGS="-I$PNG/include -I$FREETYPE/include
-I$FREETYPE/include/freetype2" LDFLAGS="-L$FREETYPE/lib -L$PNG/lib
-Wl,-rpath=$PNG/lib" $PYTHON/bin/python setup.py build
$PYTHON/bin/python setup.py install

Where $PNG and $FREETYPE points to our own versions. On a computer
with no systemwide version of png, this works great. Question:

1) If I don't specify the -rpath option, then libpng.so will fail to
be found at runtime. Is there any other recommended way to go around
this? I don't want to use LD_LIBRARY_PATH.

On a computer with systemwide version available, unfortunately the
systemwide version gets picked up instead. I think it's because of
this:

pkg-config --libs --cflags libpng
-I/usr/include/libpng12 -lpng12

so matplotlib simply calls this (right?) and uses the systemwide
version. Question:

2) How do I force matplotlib to use my own version instead?

I would highly appreciate any input, especially on 2). We have our own
issue open for this at [2] if you are interested for the background.
The merged mpl PR [3] is a little related, but doesn't answer my
questions.

Thanks,
Ondrej

[1] https://github.com/hashdist/python-hpcmp2/

[2] https://github.com/hashdist/python-hpcmp2/issues/53

[3] https://github.com/matplotlib/matplotlib/pull/1884

I meant to say "for one python distribution", not "the one"...

Ondrej

···

On Fri, May 3, 2013 at 12:41 PM, Ondřej Čertík <ondrej.certik@...287...> wrote:

Hi,

As part of building matplotlib for the one python based distribution [1],

This *should* work. Can you provide a full build log of a clean build?

Mike

···

On 05/03/2013 02:41 PM, Ondřej Čertík wrote:

Hi,

As part of building matplotlib for the one python based distribution [1],
I want to always link against our own version of libpng, even if there
is some other systemwide version available. I am on linux (Ubuntu).

Currently, here is what I am doing:

CFLAGS="-I$PNG/include -I$FREETYPE/include
-I$FREETYPE/include/freetype2" LDFLAGS="-L$FREETYPE/lib -L$PNG/lib
-Wl,-rpath=$PNG/lib" $PYTHON/bin/python setup.py build
$PYTHON/bin/python setup.py install

Sure:

https://gist.github.com/certik/5528134

The build was produced by the "build.sh" script, also in the gist.

On the line 48 (https://gist.github.com/certik/5528134#file-mpl_log-txt-L48)
you can see where our own PNG lib is:

[matplotlib] lrwxrwxrwx 1 ondrej cnls 11 May 3 11:48
/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib/libpng.so
-> libpng16.so
[matplotlib] lrwxrwxrwx 1 ondrej cnls 18 May 3 11:48
/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib/libpng16.so
-> libpng16.so.16.2.0

as printed by the line 5 in build.sh:

echo "Our PNG library:"
ls -l $PNG/lib/libpng*.so

The actual mpl build starts at the line 52
(https://gist.github.com/certik/5528134#file-mpl_log-txt-L52). As you
can see, it found the systemwide PNG lib:

[matplotlib] OPTIONAL BACKEND DEPENDENCIES
[matplotlib] libpng: 1.2.46

and just to verify this, at the line 2636
(https://gist.github.com/certik/5528134#file-mpl_log-txt-L2636) I
print:

echo "The linked PNG library:"
ldd $PYTHON/lib/python2.7/site-packages/matplotlib/_png.so

which gives:

[matplotlib] The linked PNG library:
[matplotlib] linux-vdso.so.1 => (0x00007fffd8bc1000)
[matplotlib] libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0
(0x00007f1fd0c0a000)
[matplotlib] libstdc++.so.6 =>
/usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1fd090a000)
[matplotlib] libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007f1fd06f3000)
[matplotlib] libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f1fd04d6000)
[matplotlib] libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1fd0117000)
[matplotlib] libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1fcfeff000)
[matplotlib] libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1fcfc03000)
[matplotlib] /lib64/ld-linux-x86-64.so.2 (0x00007f1fd107e000)

So the systemwide png /lib/x86_64-linux-gnu/libpng12.so.0 is linked instead:

ondrej@...4357...:~$ ls -l /lib/x86_64-linux-gnu/libpng12.so.0
lrwxrwxrwx 1 root root 18 Apr 5 2012
/lib/x86_64-linux-gnu/libpng12.so.0 -> libpng12.so.0.46.0

as you can see, it is exactly the one as advertised by the mpl build
info above. So the mpl build seems consistent,
and the bug is that it finds the systemwide version before our own version.

Ondrej

···

On Mon, May 6, 2013 at 7:18 AM, Michael Droettboom <mdroe@...86...> wrote:

On 05/03/2013 02:41 PM, Ondřej Čertík wrote:

Hi,

As part of building matplotlib for the one python based distribution [1],
I want to always link against our own version of libpng, even if there
is some other systemwide version available. I am on linux (Ubuntu).

Currently, here is what I am doing:

CFLAGS="-I$PNG/include -I$FREETYPE/include
-I$FREETYPE/include/freetype2" LDFLAGS="-L$FREETYPE/lib -L$PNG/lib
-Wl,-rpath=$PNG/lib" $PYTHON/bin/python setup.py build
$PYTHON/bin/python setup.py install

This *should* work. Can you provide a full build log of a clean build?

My understanding is that distutils builds up the commandline arguments for gcc in this order:

   1) From Python's Makefile.
   2) From environment variables
   3) From whatever was added by the setup.py script

It looks like you have some extra stuff in (1), namely

-L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu

You can find the Python Makefile that is being used to source this information here:

>>> from distutils import sysconfig
>>> sysconfig.get_makefile_filename()

You can edit that file, though obviously that's a bit of a hack.

I've run into this problem before, and there doesn't seem to be any good way around it -- i.e. there doesn't seem to be a way to insert local environment variables in front of the global Python configuration. Reason number #47 why distutils is a poor build system for C/C++ code.

Mike

···

On 05/06/2013 05:03 PM, Ondřej Čertík wrote:

On Mon, May 6, 2013 at 7:18 AM, Michael Droettboom <mdroe@...86...> wrote:

On 05/03/2013 02:41 PM, Ondřej Čertík wrote:

Hi,

As part of building matplotlib for the one python based distribution [1],
I want to always link against our own version of libpng, even if there
is some other systemwide version available. I am on linux (Ubuntu).

Currently, here is what I am doing:

CFLAGS="-I$PNG/include -I$FREETYPE/include
-I$FREETYPE/include/freetype2" LDFLAGS="-L$FREETYPE/lib -L$PNG/lib
-Wl,-rpath=$PNG/lib" $PYTHON/bin/python setup.py build
$PYTHON/bin/python setup.py install

This *should* work. Can you provide a full build log of a clean build?

Sure:

https://gist.github.com/certik/5528134

The build was produced by the "build.sh" script, also in the gist.

On the line 48 (https://gist.github.com/certik/5528134#file-mpl_log-txt-L48)
you can see where our own PNG lib is:

[matplotlib] lrwxrwxrwx 1 ondrej cnls 11 May 3 11:48
/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib/libpng.so
-> libpng16.so
[matplotlib] lrwxrwxrwx 1 ondrej cnls 18 May 3 11:48
/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib/libpng16.so
-> libpng16.so.16.2.0

as printed by the line 5 in build.sh:

echo "Our PNG library:"
ls -l $PNG/lib/libpng*.so

The actual mpl build starts at the line 52
(https://gist.github.com/certik/5528134#file-mpl_log-txt-L52). As you
can see, it found the systemwide PNG lib:

[matplotlib] OPTIONAL BACKEND DEPENDENCIES
[matplotlib] libpng: 1.2.46

and just to verify this, at the line 2636
(https://gist.github.com/certik/5528134#file-mpl_log-txt-L2636) I
print:

echo "The linked PNG library:"
ldd $PYTHON/lib/python2.7/site-packages/matplotlib/_png.so

which gives:

[matplotlib] The linked PNG library:
[matplotlib] linux-vdso.so.1 => (0x00007fffd8bc1000)
[matplotlib] libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0
(0x00007f1fd0c0a000)
[matplotlib] libstdc++.so.6 =>
/usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1fd090a000)
[matplotlib] libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007f1fd06f3000)
[matplotlib] libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f1fd04d6000)
[matplotlib] libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1fd0117000)
[matplotlib] libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1fcfeff000)
[matplotlib] libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1fcfc03000)
[matplotlib] /lib64/ld-linux-x86-64.so.2 (0x00007f1fd107e000)

So the systemwide png /lib/x86_64-linux-gnu/libpng12.so.0 is linked instead:

ondrej@...4357...:~$ ls -l /lib/x86_64-linux-gnu/libpng12.so.0
lrwxrwxrwx 1 root root 18 Apr 5 2012
/lib/x86_64-linux-gnu/libpng12.so.0 -> libpng12.so.0.46.0

as you can see, it is exactly the one as advertised by the mpl build
info above. So the mpl build seems consistent,
and the bug is that it finds the systemwide version before our own version.

Ondrej

My understanding is that distutils builds up the commandline arguments for
gcc in this order:

  1) From Python's Makefile.
  2) From environment variables
  3) From whatever was added by the setup.py script

It looks like you have some extra stuff in (1), namely

-L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu

You can find the Python Makefile that is being used to source this
information here:

from distutils import sysconfig
sysconfig.get_makefile_filename()

This gives:

In [1]: from distutils import sysconfig

In [2]: sysconfig.get_makefile_filename()
Out[2]: '/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/profile/eoul/lib/python2.7/config/Makefile'

You can edit that file, though obviously that's a bit of a hack.

It contains the lines:

CPPFLAGS= -I. -IInclude -I$(srcdir)/Include -I/usr/include/x86_64-linux-gnu
LDFLAGS= -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu

So indeed this is causing it. I think this comes from building our own
Python, which I do with:

···

On Mon, May 6, 2013 at 3:53 PM, Michael Droettboom <mdroe@...86...> wrote:

----------------------

#!/bin/bash

set -e

export arch=$(dpkg-architecture -qDEB_HOST_MULTIARCH)
export LDFLAGS="-L/usr/lib/$arch -L/lib/$arch"
export CFLAGS="-I/usr/include/$arch"
export CPPFLAGS="-I/usr/include/$arch"
# Fix for #21:
export HAS_HG="no"
./configure --prefix=${PYTHONHPC_PREFIX}

---------------------------

And this is a bit of a hack too, Ubuntu specific etc. I think I should
start fixing things here.

It just wouldn't occur to me, that remains of how I built Python would
bite me later when building
matplotlib.

So to test if modifying the Python makefile fixes it, I did:

--- Makefile.old 2013-05-06 16:26:25.426440205 -0600
+++ Makefile 2013-05-06 16:27:05.282439550 -0600
@@ -73,8 +73,8 @@
# Both CPPFLAGS and LDFLAGS need to contain the shell's value for setup.py to
# be able to build extension modules using the directories specified in the
# environment variables
-CPPFLAGS= -I. -IInclude -I$(srcdir)/Include -I/usr/include/x86_64-linux-gnu
-LDFLAGS= -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu
+CPPFLAGS= -I. -IInclude -I$(srcdir)/Include
+LDFLAGS=
LDLAST=
SGI_ABI=
CCSHARED= -fPIC

but mpl build system still shows the system one. The _png.so is built with:

[matplotlib] g++ -pthread -shared
-L/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/lib
-L/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib
-Wl,-rpath=/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib
-I/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/include
-I/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/include
-I/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/include/freetype2
build/temp.linux-x86_64-2.7/src/_png.o
build/temp.linux-x86_64-2.7/src/mplutils.o
build/temp.linux-x86_64-2.7/CXX/IndirectPythonInterface.o
build/temp.linux-x86_64-2.7/CXX/cxxsupport.o
build/temp.linux-x86_64-2.7/CXX/cxx_extensions.o
build/temp.linux-x86_64-2.7/CXX/cxxextensions.o -L/usr/local/lib
-L/usr/lib -lpng12 -lz -lstdc++ -lm -o
build/lib.linux-x86_64-2.7/matplotlib/_png.so

Which in my opinion looks good -- my own version of PNG is offered
first on the gcc command line. But the -lpng12 part spoils it --- that
forces gcc to use the systemone, because my own version is newer.

So I think that part of the problem gets fixed by modifying the Python
Makefile, but the other part of the problem is how to force distutils
to look for my PNG version before the systemwide. Any ideas?

Maybe it is something that is added by the mpl setup.py script.

I've run into this problem before, and there doesn't seem to be any good way
around it -- i.e. there doesn't seem to be a way to insert local environment
variables in front of the global Python configuration. Reason number #47
why distutils is a poor build system for C/C++ code.

This is amazingly broken.

Ondrej

Well, we could try a different approach. matplotlib will use pkg-config to find its dependencies, if available.

If you can get your local libpng to include a libpng.pc (i.e. a pkg-config information file) and then add your local pkg-config path (probably
/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/profile/eoul/lib/pkgconfig) to the PKG_CONFIG_PATH environment variable, it should pick up the right name for the library as well. If you get that working, you may be able to avoid setting CFLAGS and LDFLAGS explicitly and the Makefile modifications.

Mike

···

On 05/06/2013 06:53 PM, Ondřej Čertík wrote:

On Mon, May 6, 2013 at 3:53 PM, Michael Droettboom <mdroe@...86...> wrote:

My understanding is that distutils builds up the commandline arguments for
gcc in this order:

   1) From Python's Makefile.
   2) From environment variables
   3) From whatever was added by the setup.py script

It looks like you have some extra stuff in (1), namely

-L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu

You can find the Python Makefile that is being used to source this
information here:

from distutils import sysconfig
sysconfig.get_makefile_filename()

This gives:

In [1]: from distutils import sysconfig

In [2]: sysconfig.get_makefile_filename()
Out[2]: '/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/profile/eoul/lib/python2.7/config/Makefile'

You can edit that file, though obviously that's a bit of a hack.

It contains the lines:

CPPFLAGS= -I. -IInclude -I$(srcdir)/Include -I/usr/include/x86_64-linux-gnu
LDFLAGS= -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu

So indeed this is causing it. I think this comes from building our own
Python, which I do with:

----------------------

#!/bin/bash

set -e

export arch=$(dpkg-architecture -qDEB_HOST_MULTIARCH)
export LDFLAGS="-L/usr/lib/$arch -L/lib/$arch"
export CFLAGS="-I/usr/include/$arch"
export CPPFLAGS="-I/usr/include/$arch"
# Fix for #21:
export HAS_HG="no"
./configure --prefix=${PYTHONHPC_PREFIX}

---------------------------

And this is a bit of a hack too, Ubuntu specific etc. I think I should
start fixing things here.

It just wouldn't occur to me, that remains of how I built Python would
bite me later when building
matplotlib.

So to test if modifying the Python makefile fixes it, I did:

--- Makefile.old 2013-05-06 16:26:25.426440205 -0600
+++ Makefile 2013-05-06 16:27:05.282439550 -0600
@@ -73,8 +73,8 @@
  # Both CPPFLAGS and LDFLAGS need to contain the shell's value for setup.py to
  # be able to build extension modules using the directories specified in the
  # environment variables
-CPPFLAGS= -I. -IInclude -I$(srcdir)/Include -I/usr/include/x86_64-linux-gnu
-LDFLAGS= -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu
+CPPFLAGS= -I. -IInclude -I$(srcdir)/Include
+LDFLAGS=
  LDLAST=
  SGI_ABI=
  CCSHARED= -fPIC

but mpl build system still shows the system one. The _png.so is built with:

[matplotlib] g++ -pthread -shared
-L/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/lib
-L/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib
-Wl,-rpath=/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib
-I/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/include
-I/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/include
-I/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/include/freetype2
build/temp.linux-x86_64-2.7/src/_png.o
build/temp.linux-x86_64-2.7/src/mplutils.o
build/temp.linux-x86_64-2.7/CXX/IndirectPythonInterface.o
build/temp.linux-x86_64-2.7/CXX/cxxsupport.o
build/temp.linux-x86_64-2.7/CXX/cxx_extensions.o
build/temp.linux-x86_64-2.7/CXX/cxxextensions.o -L/usr/local/lib
-L/usr/lib -lpng12 -lz -lstdc++ -lm -o
build/lib.linux-x86_64-2.7/matplotlib/_png.so

Which in my opinion looks good -- my own version of PNG is offered
first on the gcc command line. But the -lpng12 part spoils it --- that
forces gcc to use the systemone, because my own version is newer.

So I think that part of the problem gets fixed by modifying the Python
Makefile, but the other part of the problem is how to force distutils
to look for my PNG version before the systemwide. Any ideas?

Maybe it is something that is added by the mpl setup.py script.

I've run into this problem before, and there doesn't seem to be any good way
around it -- i.e. there doesn't seem to be a way to insert local environment
variables in front of the global Python configuration. Reason number #47
why distutils is a poor build system for C/C++ code.

This is amazingly broken.

Ondrej

Well, we could try a different approach. matplotlib will use pkg-config to
find its dependencies, if available.

If you can get your local libpng to include a libpng.pc (i.e. a pkg-config
information file) and then add your local pkg-config path (probably
/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/profile/eoul/lib/pkgconfig)
to the PKG_CONFIG_PATH environment variable, it should pick up the right
name for the library as well.

It works!! Thanks a lot. Here I found the same answer as well:

http://stackoverflow.com/questions/16227285/unable-to-import-matplotlib-png-pylab

If you get that working, you may be able to
avoid setting CFLAGS and LDFLAGS explicitly and the Makefile modifications.

Indeed. I definitely don't need Python Makefile modifications and here
is my polished install script now:

export PKG_CONFIG_PATH="$PNG/lib/pkgconfig:$FREETYPE/lib/pkgconfig"
export LDFLAGS="-L$PNG/lib -Wl,-rpath=$PNG/lib"
$PYTHON/bin/python setup.py install

As you can see, I still need to setup the rpath, but that is to be expected.

For the record, here is how _png.o and _png.so end up compiled and linked:

gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall
-I/home/ondrej/repos/python-hpcmp2/opt/png/qhle/include
-I/home/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/include
-I/home/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/include/freetype2
-fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MPL_ARRAY_API -DPYCXX_ISO_CPP_LIB=1
-I/usr/local/include -I/usr/include
-I/home/ondrej/repos/python-hpcmp2/opt/png/qhle/include/libpng16
-I/usr/local/include -I/usr/include -I.
-I/home/ondrej/repos/python-hpcmp2/opt/numpy/hpbc/lib/python2.7/site-packages/numpy/core/include
-I. -I/home/ondrej/repos/python-hpcmp2/opt/python/llzf/include/python2.7
-c src/_png.cpp -o build/temp.linux-x86_64-2.7/src/_png.o

g++ -pthread -shared -L/usr/lib/x86_64-linux-gnu
-L/lib/x86_64-linux-gnu
-L/home/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/lib
-L/home/ondrej/repos/python-hpcmp2/opt/png/qhle/lib
-Wl,-rpath=/home/ondrej/repos/python-hpcmp2/opt/png/qhle/lib
-I/home/ondrej/repos/python-hpcmp2/opt/png/qhle/include
-I/home/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/include
-I/home/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/include/freetype2
build/temp.linux-x86_64-2.7/src/_png.o
build/temp.linux-x86_64-2.7/src/mplutils.o
build/temp.linux-x86_64-2.7/CXX/cxx_extensions.o
build/temp.linux-x86_64-2.7/CXX/IndirectPythonInterface.o
build/temp.linux-x86_64-2.7/CXX/cxxsupport.o
build/temp.linux-x86_64-2.7/CXX/cxxextensions.o
-L/home/ondrej/repos/python-hpcmp2/opt/png/qhle/lib -L/usr/local/lib
-L/usr/lib -lpng16 -lz -lstdc++ -lm -o
build/lib.linux-x86_64-2.7/matplotlib/_png.so

I think that this looks good. If I need to ever remove the
-L/usr/lib/x86_64-linux-gnu parts, then I just fix the Python package.

Ondrej

···

On Mon, May 6, 2013 at 6:27 PM, Michael Droettboom <mdroe@...86...> wrote: