venv, pip and Matplotlib development code

Hi,
I am trying to followup on an issue I saw a couple of days ago with trendvis
(I have opened an issue for it:
https://github.com/matplotlib/matplotlib/issues/4812 ). It was suggested to
me that instead of installing using `python setup.py develop --user`, I
should instead use virtualenv, venv or conda to manage different mpl
versions. As I am not familiar with either, I started with venv, as I use
python3.4 nowadays and it's included there.
I created a virtual environment, following the guide suggested in python
docs [1]. I created the virtual environment with --system-site-packages to
avoid manually installing all the dependencies, making sure that no system
nor user version of matplotlib were installed (import matplotlib now fails
as expected). It is suggested there that one should use pip for installing
packages. As I want to install the development version of mpl, I cloned the
mpl sources in the virtual environment and then tried to install them with
`pip install -e local/path/to/cloned/mpl/repo`, but I get an error. What am
I doing wrong? I tried searching the web for installation procedures of mpl
under virtual environments, but all I found was related to stable versions.
Can anyone point me to a working procedure?

The error I get is reproduced below:

$ pip install -e matplotlib
Obtaining file:///home/jscandal/sw/python/virtualenvs/mpl/matplotlib
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.6 in
/usr/lib/python3.4/site-packages (from matplotlib==1.5.dev1)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in
/usr/lib/python3.4/site-packages (from matplotlib==1.5.dev1)
Requirement already satisfied (use --upgrade to upgrade): pytz in
/usr/lib/python3.4/site-packages (from matplotlib==1.5.dev1)
Requirement already satisfied (use --upgrade to upgrade): pyparsing>=1.5.6
in /usr/lib/python3.4/site-packages (from matplotlib==1.5.dev1)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in
/usr/lib/python3.4/site-packages (from python-dateutil->matplotlib==1.5.dev1)
Installing collected packages: matplotlib
  Running setup.py develop for matplotlib
    Complete output from command /usr/bin/python -c "import setuptools,
tokenize;
__file__='/home/jscandal/sw/python/virtualenvs/mpl/matplotlib/setup.py';
exec(compile(getattr(tokenize, 'open',
open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop
--no-deps:

···

============================================================================
    Edit setup.cfg to change the build options

    BUILDING MATPLOTLIB
                matplotlib: yes [1.5.dev1]
                    python: yes [3.4.3 (default, Mar 25 2015, 17:13:50) [GCC
                            4.9.2 20150304 (prerelease)]]
                  platform: yes [linux]

    REQUIRED DEPENDENCIES AND EXTENSIONS
                     numpy: yes [version 1.9.2]
                  dateutil: yes [using dateutil version 2.4.2]
                      pytz: yes [using pytz version 2015.4]
                   tornado: yes [using tornado version 4.2.1]
                 pyparsing: yes [using pyparsing version 2.0.3]
                    libagg: yes [pkg-config information for 'libagg' could not
                            be found. Using local copy.]
                  freetype: yes [version 2.6.0]
                       png: yes [version 1.6.18]
                     qhull: yes [pkg-config information for 'qhull' could not be
                            found. Using local copy.]

    OPTIONAL SUBPACKAGES
               sample_data: yes [installing]
                  toolkits: yes [installing]
                     tests: yes [using nose version 1.3.7 / using unittest.mock]
            toolkits_tests: yes [using nose version 1.3.7 / using unittest.mock]

    OPTIONAL BACKEND EXTENSIONS
                    macosx: no [Mac OS-X only]
                    qt5agg: yes [installing, Qt: 5.5.0, PyQt: 5.5.0]
                    qt4agg: yes [installing, Qt: 4.8.7, PySide: 1.2.2; Qt:
                            4.8.7, PyQt: 4.8.7]

    ** (-c:13703): WARNING **: Couldn't register with accessibility bus: Did
not receive a reply. Possible causes include: the remote application did not
send a reply, the message bus security policy blocked the reply, the reply
timeout expired, or the network connection was broken.
                   gtk3agg: yes [installing, version 3.6.16]

    ** (-c:13711): WARNING **: Couldn't register with accessibility bus: Did
not receive a reply. Possible causes include: the remote application did not
send a reply, the message bus security policy blocked the reply, the reply
timeout expired, or the network connection was broken.
                 gtk3cairo: yes [installing, version 3.6.16]
                    gtkagg: no [Requires pygtk]
                     tkagg: yes [installing, version not identified]
                     wxagg: no [requires wxPython]
                       gtk: no [Requires pygtk]
                       agg: yes [installing]
                     cairo: yes [installing, cairocffi version 1.10.0]
                 windowing: no [Microsoft Windows only]

    OPTIONAL LATEX DEPENDENCIES
                    dvipng: yes [version 1.14]
               ghostscript: yes [version 9.16]
                     latex: yes [version 3.14159265]
                   pdftops: yes [version 0.33.0]

    running develop
    error: can't create or remove files in install directory

    The following error occurred while trying to add or remove files in the
    installation directory:

        [Errno 13] Permission denied:
'/usr/lib/python3.4/site-packages/test-easy-install-13667.write-test'

    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:

        /usr/lib/python3.4/site-packages/

    Perhaps your account does not have write access to this directory? If the
    installation directory is a system-owned directory, you may need to sign in
    as the administrator or "root" account. If you do not have administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.

    For information on other options, you may wish to consult the
    documentation at:

      https://pythonhosted.org/setuptools/easy_install.html

    Please make the appropriate changes for your system and try again.

    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;
__file__='/home/jscandal/sw/python/virtualenvs/mpl/matplotlib/setup.py';
exec(compile(getattr(tokenize, 'open',
open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop
--no-deps" failed with error code 1 in
/home/jscandal/sw/python/virtualenvs/mpl/matplotlib

[1]. https://packaging.python.org/en/latest/installing.html#virtual-environments

Jorge Scandaliaris <jscandaliaris at gmail.com> writes:

As I want to install the development version of mpl, I cloned the
mpl sources in the virtual environment and then tried to install them with
`pip install -e local/path/to/cloned/mpl/repo`, but I get an error.

[...]

Command "/usr/bin/python -c "import setuptools, tokenize;
__file__='/home/jscandal/sw/python/virtualenvs/mpl/matplotlib/setup.py';
exec(compile(getattr(tokenize, 'open',
open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop
--no-deps" failed with error code 1 in
/home/jscandal/sw/python/virtualenvs/mpl/matplotlib

Are you sure you are running the version of pip that is installed in the
virtualenv? I don't think it should be invoking /usr/bin/python.

···

--
Jouni K. Sepp?nen
http://www.iki.fi/jks

I can't tell from the trace back if you have activated or not. I also
would not clone the repo into the venv.

If the source for mpl is in `~/source/matplotlib` :

pyvenv --system-site-packages /tmp/test
source /tmp/test/bin/activate
cd ~/source/matplotlib
python setup.py develop

Should install matplotlib into the 'test' venv (which lives in RAM on most
modern linux machines).

Tom

···

On Tue, Aug 4, 2015 at 10:45 AM Jorge Scandaliaris <jscandaliaris at gmail.com> wrote:

Hi,
I am trying to followup on an issue I saw a couple of days ago with
trendvis
(I have opened an issue for it:
https://github.com/matplotlib/matplotlib/issues/4812 ). It was suggested
to
me that instead of installing using `python setup.py develop --user`, I
should instead use virtualenv, venv or conda to manage different mpl
versions. As I am not familiar with either, I started with venv, as I use
python3.4 nowadays and it's included there.
I created a virtual environment, following the guide suggested in python
docs [1]. I created the virtual environment with --system-site-packages to
avoid manually installing all the dependencies, making sure that no system
nor user version of matplotlib were installed (import matplotlib now fails
as expected). It is suggested there that one should use pip for installing
packages. As I want to install the development version of mpl, I cloned the
mpl sources in the virtual environment and then tried to install them with
`pip install -e local/path/to/cloned/mpl/repo`, but I get an error. What am
I doing wrong? I tried searching the web for installation procedures of mpl
under virtual environments, but all I found was related to stable versions.
Can anyone point me to a working procedure?

The error I get is reproduced below:

$ pip install -e matplotlib
Obtaining file:///home/jscandal/sw/python/virtualenvs/mpl/matplotlib
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.6 in
/usr/lib/python3.4/site-packages (from matplotlib==1.5.dev1)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil
in
/usr/lib/python3.4/site-packages (from matplotlib==1.5.dev1)
Requirement already satisfied (use --upgrade to upgrade): pytz in
/usr/lib/python3.4/site-packages (from matplotlib==1.5.dev1)
Requirement already satisfied (use --upgrade to upgrade): pyparsing>=1.5.6
in /usr/lib/python3.4/site-packages (from matplotlib==1.5.dev1)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in
/usr/lib/python3.4/site-packages (from
python-dateutil->matplotlib==1.5.dev1)
Installing collected packages: matplotlib
  Running setup.py develop for matplotlib
    Complete output from command /usr/bin/python -c "import setuptools,
tokenize;
__file__='/home/jscandal/sw/python/virtualenvs/mpl/matplotlib/setup.py';
exec(compile(getattr(tokenize, 'open',
open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop
--no-deps:

============================================================================
    Edit setup.cfg to change the build options

    BUILDING MATPLOTLIB
                matplotlib: yes [1.5.dev1]
                    python: yes [3.4.3 (default, Mar 25 2015, 17:13:50)
[GCC
                            4.9.2 20150304 (prerelease)]]
                  platform: yes [linux]

    REQUIRED DEPENDENCIES AND EXTENSIONS
                     numpy: yes [version 1.9.2]
                  dateutil: yes [using dateutil version 2.4.2]
                      pytz: yes [using pytz version 2015.4]
                   tornado: yes [using tornado version 4.2.1]
                 pyparsing: yes [using pyparsing version 2.0.3]
                    libagg: yes [pkg-config information for 'libagg' could
not
                            be found. Using local copy.]
                  freetype: yes [version 2.6.0]
                       png: yes [version 1.6.18]
                     qhull: yes [pkg-config information for 'qhull' could
not be
                            found. Using local copy.]

    OPTIONAL SUBPACKAGES
               sample_data: yes [installing]
                  toolkits: yes [installing]
                     tests: yes [using nose version 1.3.7 / using
unittest.mock]
            toolkits_tests: yes [using nose version 1.3.7 / using
unittest.mock]

    OPTIONAL BACKEND EXTENSIONS
                    macosx: no [Mac OS-X only]
                    qt5agg: yes [installing, Qt: 5.5.0, PyQt: 5.5.0]
                    qt4agg: yes [installing, Qt: 4.8.7, PySide: 1.2.2; Qt:
                            4.8.7, PyQt: 4.8.7]

    ** (-c:13703): WARNING **: Couldn't register with accessibility bus:
Did
not receive a reply. Possible causes include: the remote application did
not
send a reply, the message bus security policy blocked the reply, the reply
timeout expired, or the network connection was broken.
                   gtk3agg: yes [installing, version 3.6.16]

    ** (-c:13711): WARNING **: Couldn't register with accessibility bus:
Did
not receive a reply. Possible causes include: the remote application did
not
send a reply, the message bus security policy blocked the reply, the reply
timeout expired, or the network connection was broken.
                 gtk3cairo: yes [installing, version 3.6.16]
                    gtkagg: no [Requires pygtk]
                     tkagg: yes [installing, version not identified]
                     wxagg: no [requires wxPython]
                       gtk: no [Requires pygtk]
                       agg: yes [installing]
                     cairo: yes [installing, cairocffi version 1.10.0]
                 windowing: no [Microsoft Windows only]

    OPTIONAL LATEX DEPENDENCIES
                    dvipng: yes [version 1.14]
               ghostscript: yes [version 9.16]
                     latex: yes [version 3.14159265]
                   pdftops: yes [version 0.33.0]

    running develop
    error: can't create or remove files in install directory

    The following error occurred while trying to add or remove files in the
    installation directory:

        [Errno 13] Permission denied:
'/usr/lib/python3.4/site-packages/test-easy-install-13667.write-test'

    The installation directory you specified (via --install-dir, --prefix,
or
    the distutils default setting) was:

        /usr/lib/python3.4/site-packages/

    Perhaps your account does not have write access to this directory? If
the
    installation directory is a system-owned directory, you may need to
sign in
    as the administrator or "root" account. If you do not have
administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.

    For information on other options, you may wish to consult the
    documentation at:

      https://pythonhosted.org/setuptools/easy_install.html

    Please make the appropriate changes for your system and try again.

    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;
__file__='/home/jscandal/sw/python/virtualenvs/mpl/matplotlib/setup.py';
exec(compile(getattr(tokenize, 'open',
open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop
--no-deps" failed with error code 1 in
/home/jscandal/sw/python/virtualenvs/mpl/matplotlib

[1].
https://packaging.python.org/en/latest/installing.html#virtual-environments

_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel at python.org
https://mail.python.org/mailman/listinfo/matplotlib-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/matplotlib-devel/attachments/20150804/57f89b23/attachment-0001.html>

Thomas Caswell <tcaswell at ...> writes:

I can't tell from the trace back if you have activated or not.? I also
would not clone the repo into the venv.
If the source for mpl is in `~/source/matplotlib` :

pyvenv --system-site-packages /tmp/test
source /tmp/test/bin/activate?
cd ~/source/matplotlib
python setup.py develop

Should install matplotlib into the 'test' venv (which lives in RAM on
most modern linux machines).

Tom

@Jouni K. Sepp?nen: I used the --system-site-packages option, thus I believe
it's fine

@Thomas Caswell: thanks for your patience! I was doing it wrong it seems. I
just followed your recipe. Now matplotlib installs OK, but I get the same
error with trendvis!

Details:
  - I just moved the whole ~/.local folder somewhere else. Now I am 100%
sure nothing installed with --user can mess my tests.
  - No system matplotlib is installed, but its dependencies are system
(distro) packages.
  - I cleaned both mpl and trendvis repos with `git clean -fxd`
  - I follow your recipe to create the virtualenv and install matplotlib.
This succeeds.
  - I also do `python setup.py develop` in trendvis repo. This also succeeds.
  - Run the three examples from trendvis README.rst. The first suceeds,
although the resultant image differs slightly with the image at trendvis'
github page. The two more complex examples yield an error. Trace pasted below:

$ python two_row_ygrid_with_frame.py
Traceback (most recent call last):
  File "two_row_ygrid_with_frame.py", line 41, in <module>
    ex1.draw_frame()
  File "/home/jscandal/sw/trendvis/trendvis/trendvis/gridclass.py", line
666, in draw_frame
    ur_corner = self._convert_coords(ur_axis, (urdx, urdy))
  File "/home/jscandal/sw/trendvis/trendvis/trendvis/gridclass.py", line
921, in _convert_coords
    ax_coords = axis.transData.transform(coordinates)
  File
"/home/jscandal/sw/matplotlib/matplotlib.py3/lib/matplotlib/transforms.py",
line 1303, in transform
    values = np.asanyarray(values)
  File "/usr/lib/python3.4/site-packages/numpy/core/numeric.py", line 514,
in asanyarray
    return array(a, dtype, copy=False, order=order, subok=True)
ValueError: Expected 2-dimensional array, got 1

I also tried to remove the system-installed numpy and install it in the
virtualenv (maybe the problem is related to arch's numpy?), but that didn't
suceed. I tried to do `pip install numpy` and I got an error. I also tried
building manually, but didn't manage. The three examples run fine if I use
the system version of matplotlib (1.4.3)

Regards,

Jorge