Building documentation and matplotlibrc

Ok, I'm trying with trunk now (or those changes are on another
branch?), and it stops at

reading sources... [ 20%] examples/axes_grid/demo_axes_divider

while with RC it stops at

reading sources... [ 14%] examples/api/date_demo

6% improvements :smiley: anyhow, the demo_axes_divider needs file
axes_grid/bivariate_normal.npy that's present in
../sampledata/axes_grid/bivariate_normal.npy so I don't know. John, is
trunk working for your? I'm blocking network access to MPL SF with

sudo iptables -A OUTPUT -d matplotlib.svn.sourceforge.net -j DROP

and when you need it back

sudo iptables -F

(I flush all the rules since I have only the one above).

Regards,

···

On Wed, Jan 5, 2011 at 19:48, Sandro Tosi <morph@...12...> wrote:

ehm... no, I'm testing it the rc + your patch, sorry :slight_smile: I'm updating
my local svn copy and try again

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

We're working in the release branch. Changes have not been made to trunk.

svn co matplotlib download | SourceForge.net
mpl1

JDH

···

On Wed, Jan 5, 2011 at 1:23 PM, Sandro Tosi <morph@...12...> wrote:

On Wed, Jan 5, 2011 at 19:48, Sandro Tosi <morph@...12...> wrote:
> ehm... no, I'm testing it the rc + your patch, sorry :slight_smile: I'm updating
> my local svn copy and try again

Ok, I'm trying with trunk now (or those changes are on another
branch?), and it stops at

rc2 tarball is here:

···

On Wed, Jan 5, 2011 at 1:31 PM, John Hunter <jdh2358@...149...> wrote:

We're working in the release branch. Changes have not been made to trunk.

svn co matplotlib download | SourceForge.net
mpl1

I tried with that branch, but I had no luck :frowning: it still tries to
download stuff from the net, whatever I set as examples.directory in
matplotlibrc (yes, I'm trying to compile inside doc/ and I have
'examples.download : False' set) and/or MATPLOTLIB_SAMPLE_DATA . JFTR
the cmdline I'm using is:

morph@...914...:~/deb/upstream_checkout/mpl1/doc$
MATPLOTLIB_SAMPLE_DATA=/home/morph/deb/upstream_checkout/mpl1/sampledata/
MATPLOTLIBDATA=../lib/matplotlib/mpl-data/
PYTHONPATH=../build/lib.linux-x86_64-2.6/ ./make.py --small all

$ tail -2 matplotlibrc
examples.download : False # False to bypass downloading mechanism
examples.directory : /home/morph/deb/upstream_checkout/mpl1/sampledata/

Regards,

···

On Wed, Jan 5, 2011 at 20:31, John Hunter <jdh2358@...149...> wrote:

On Wed, Jan 5, 2011 at 1:23 PM, Sandro Tosi <morph@...12...> wrote:

On Wed, Jan 5, 2011 at 19:48, Sandro Tosi <morph@...12...> wrote:
> ehm... no, I'm testing it the rc + your patch, sorry :slight_smile: I'm updating
> my local svn copy and try again

Ok, I'm trying with trunk now (or those changes are on another
branch?), and it stops at

We're working in the release branch. Changes have not been made to trunk.

svn co matplotlib download | SourceForge.net
mpl1

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

Strange -- I'll try and test tonight with a machine disconnected from
the internet. Out of curiosity, what command/tool are you using to
monitor the internet requests? Ben, are you seeing the same problems?

JDH

···

On Wed, Jan 5, 2011 at 3:49 PM, Sandro Tosi <morph@...12...> wrote:

I tried with that branch, but I had no luck :frowning: it still tries to
download stuff from the net, whatever I set as examples.directory in
matplotlibrc (yes, I'm trying to compile inside doc/ and I have
'examples.download : False' set) and/or MATPLOTLIB_SAMPLE_DATA . JFTR
the cmdline I'm using is:

morph@...914...:~/deb/upstream_checkout/mpl1/doc$
MATPLOTLIB_SAMPLE_DATA=/home/morph/deb/upstream_checkout/mpl1/sampledata/
MATPLOTLIBDATA=../lib/matplotlib/mpl-data/
PYTHONPATH=../build/lib.linux-x86_64-2.6/ ./make.py --small all

$ tail -2 matplotlibrc
examples.download : False # False to bypass downloading mechanism
examples.directory : /home/morph/deb/upstream_checkout/mpl1/sampledata/

it's not exactly monitory, but I disable all the requests to
matplotlib.svn.sourceforge.net via iptables:

sudo iptables -A OUTPUT -d matplotlib.svn.sourceforge.net -j DROP

and you're sure nothing passes towards that address :slight_smile: and in fact,
the moment you reach an example with get_sample_data() it stops.

···

On Wed, Jan 5, 2011 at 23:00, John Hunter <jdh2358@...149...> wrote:

On Wed, Jan 5, 2011 at 3:49 PM, Sandro Tosi <morph@...12...> wrote:

I tried with that branch, but I had no luck :frowning: it still tries to
download stuff from the net, whatever I set as examples.directory in
matplotlibrc (yes, I'm trying to compile inside doc/ and I have
'examples.download : False' set) and/or MATPLOTLIB_SAMPLE_DATA . JFTR
the cmdline I'm using is:

morph@...914...:~/deb/upstream_checkout/mpl1/doc$
MATPLOTLIB_SAMPLE_DATA=/home/morph/deb/upstream_checkout/mpl1/sampledata/
MATPLOTLIBDATA=../lib/matplotlib/mpl-data/
PYTHONPATH=../build/lib.linux-x86_64-2.6/ ./make.py --small all

$ tail -2 matplotlibrc
examples.download : False # False to bypass downloading mechanism
examples.directory : /home/morph/deb/upstream_checkout/mpl1/sampledata/

Strange -- I'll try and test tonight with a machine disconnected from
the internet. Out of curiosity, what command/tool are you using to
monitor the internet requests? Ben, are you seeing the same problems?

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

That's helpful. I used to be an iptables guru, but it has been a long
time. How do you reverse the command when you are done testing?

JDH

···

On Wed, Jan 5, 2011 at 4:04 PM, Sandro Tosi <morph@...12...> wrote:

it's not exactly monitory, but I disable all the requests to
matplotlib.svn.sourceforge.net via iptables:

sudo iptables -A OUTPUT -d matplotlib.svn.sourceforge.net -j DROP

and you're sure nothing passes towards that address :slight_smile: and in fact,
the moment you reach an example with get_sample_data() it stops.

if you have only those rule, then just issue a

sudo iptables -F

to flus the whole iptables chains list rules; else, you have to:

sudo iptables -nL OUTPUT --line-numbers

take notes of the line number of those for MPL SF (the ip addresses are those in

$ host matplotlib.svn.sourceforge.net
matplotlib.svn.sourceforge.net has address 216.34.181.177
matplotlib.svn.sourceforge.net has address 216.34.181.65
)

and then remove them:

sudo iptables -D OUTPUT <line number above>

(note that if you delete a rules, the following ones have linenumber changed...)

Cheers,

···

On Wed, Jan 5, 2011 at 23:07, John Hunter <jdh2358@...149...> wrote:

On Wed, Jan 5, 2011 at 4:04 PM, Sandro Tosi <morph@...12...> wrote:

it's not exactly monitory, but I disable all the requests to
matplotlib.svn.sourceforge.net via iptables:

sudo iptables -A OUTPUT -d matplotlib.svn.sourceforge.net -j DROP

and you're sure nothing passes towards that address :slight_smile: and in fact,
the moment you reach an example with get_sample_data() it stops.

That's helpful. I used to be an iptables guru, but it has been a long
time. How do you reverse the command when you are done testing?

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

OK, I found the problem. Somehow my edit to sphinxext.plot_directive
to actually call the new function was not saved and committed. I just
fixed this, so the branch *should* work. The branch
matlpotlib/sphinxext/plot_directive.py should call rc_file_defaults
rather than rcdefaults. After an svn up, confirm that this is so
before testing.

JDH

···

On Wed, Jan 5, 2011 at 3:49 PM, Sandro Tosi <morph@...12...> wrote:

On Wed, Jan 5, 2011 at 20:31, John Hunter <jdh2358@...149...> wrote:

On Wed, Jan 5, 2011 at 1:23 PM, Sandro Tosi <morph@...12...> wrote:

On Wed, Jan 5, 2011 at 19:48, Sandro Tosi <morph@...12...> wrote:
> ehm... no, I'm testing it the rc + your patch, sorry :slight_smile: I'm updating
> my local svn copy and try again

Ok, I'm trying with trunk now (or those changes are on another
branch?), and it stops at

We're working in the release branch. Changes have not been made to trunk.

svn co matplotlib download | SourceForge.net
mpl1

I tried with that branch, but I had no luck :frowning: it still tries to
download stuff from the net, whatever I set as examples.directory in
matplotlibrc (yes, I'm trying to compile inside doc/ and I have
'examples.download : False' set) and/or MATPLOTLIB_SAMPLE_DATA . JFTR
the cmdline I'm using is:

morph@...914...:~/deb/upstream_checkout/mpl1/doc$
MATPLOTLIB_SAMPLE_DATA=/home/morph/deb/upstream_checkout/mpl1/sampledata/
MATPLOTLIBDATA=../lib/matplotlib/mpl-data/
PYTHONPATH=../build/lib.linux-x86_64-2.6/ ./make.py --small all

$ tail -2 matplotlibrc
examples.download : False # False to bypass downloading mechanism
examples.directory : /home/morph/deb/upstream_checkout/mpl1/sampledata/

$ grep -c rc_file_defaults lib/matplotlib/sphinxext/plot_directive.py
0

so it's not there :frowning:

and the only file containing rc_file_defaults is lib/matplotlib/__init__.py

of course, examples are stil stuck if no network to SF is available

Oh i think I got it, lib/matplotlib/sphinxext/plot_directive.py was
updated on trunk instead of the v1.0 branch.

Cheers,

···

On Wed, Jan 5, 2011 at 23:22, John Hunter <jdh2358@...149...> wrote:

OK, I found the problem. Somehow my edit to sphinxext.plot_directive
to actually call the new function was not saved and committed. I just
fixed this, so the branch *should* work. The branch
matlpotlib/sphinxext/plot_directive.py should call rc_file_defaults
rather than rcdefaults. After an svn up, confirm that this is so
before testing.

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

It took me a long time to finally figure out how to get matplotlib to
respect the wishes of the packaging. Ultimately I believe I got it
working but unfortunately, it's been a while, so I'm not entirely sure
what was necessary. If it helps, I believe the package in my PPA builds
correctly. Sorry I don't have more time to investigate; things are quite
hectic at the moment.

Cheers,

- Ben

···

On Wed, 5 Jan 2011 16:00:40 -0600, John Hunter <jdh2358@...149...> wrote:

Strange -- I'll try and test tonight with a machine disconnected from
the internet. Out of curiosity, what command/tool are you using to
monitor the internet requests? Ben, are you seeing the same problems?

I swear there is a gremlin in my computer. I checked and rechecked
this on my box because of all the confusion we were having, but
apparently managed to commit on the trunk (which doesn't even have the
function defined because I haven't merged) and not on the branch. We
need to get the buildbot going again, because the doc buildbot would
have caught this. I have fixed both the trunk and the branch: the
branch calls the new function and has the desired "no internet"
behavior. The trunk calls the old function which restores rc internal
defaults rather than file defaults and so is broken in the way it has
always been broken, but will be fixed when we merge.

In the command below, mpl1 is my branch (1.0.1 release) directory:

  mpl1> grep rc_file lib/matplotlib/sphinxext/plot_directive.py

Surely I have it right now!

JDH

···

On Wed, Jan 5, 2011 at 5:53 PM, Sandro Tosi <morph@...12...> wrote:

On Wed, Jan 5, 2011 at 23:22, John Hunter <jdh2358@...149...> wrote:

OK, I found the problem. Somehow my edit to sphinxext.plot_directive
to actually call the new function was not saved and committed. I just
fixed this, so the branch *should* work. The branch
matlpotlib/sphinxext/plot_directive.py should call rc_file_defaults
rather than rcdefaults. After an svn up, confirm that this is so
before testing.

$ grep -c rc_file_defaults lib/matplotlib/sphinxext/plot_directive.py
0

so it's not there :frowning:

and the only file containing rc_file_defaults is lib/matplotlib/__init__.py

of course, examples are stil stuck if no network to SF is available

Oh i think I got it, lib/matplotlib/sphinxext/plot_directive.py was
updated on trunk instead of the v1.0 branch.

Indeed, and it works fine!!! the only thing to notice is that
examples.directory must be an absolute path, but that set, the doc
build process gets files from the location specified.

Thanks a lot!!

Cheers,

···

On Thu, Jan 6, 2011 at 02:31, John Hunter <jdh2358@...149...> wrote:

mpl1> grep rc_file lib/matplotlib/sphinxext/plot_directive.py

Surely I have it right now!

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

I thought that problem was already addressed in a previous patch? John, could this be another “gremlin”? Or did I mis-understand the whole issue absolute path problem?

Ben Root

···

On Wed, Jan 5, 2011 at 10:42 PM, Sandro Tosi <morph@…12…> wrote:

On Thu, Jan 6, 2011 at 02:31, John Hunter <jdh2358@…149…> wrote:

mpl1> grep rc_file lib/matplotlib/sphinxext/plot_directive.py

Surely I have it right now!

Indeed, and it works fine!!! the only thing to notice is that

examples.directory must be an absolute path, but that set, the doc

build process gets files from the location specified.

My understanding is that because sphinx switches the current working
directory during execution, this path must be absolute. We could get
around this by processing a relative path internally in mpl at rc load
time, and converting it into an absolute path assuming the path is
relative to the directory containing the rc file.

JDH

···

On Thu, Jan 6, 2011 at 10:58 AM, Benjamin Root <ben.root@...553...> wrote:

I thought that problem was already addressed in a previous patch? John,
could this be another "gremlin"? Or did I mis-understand the whole issue
absolute path problem?

I actually think that would be a better solution. Python’s os.path module is very powerful with functions like isabs(), abspath(), expanduser(), expandvars() and realpath(). Of course, one could easily go overboard with this, but I am sure we could probably allow for something real simple like expandvars() so that packagers could utilize environment variables for the build process?

Ben Root

···

On Thu, Jan 6, 2011 at 11:15 AM, John Hunter <jdh2358@…149…> wrote:

On Thu, Jan 6, 2011 at 10:58 AM, Benjamin Root <ben.root@…553…> wrote:

I thought that problem was already addressed in a previous patch? John,

could this be another “gremlin”? Or did I mis-understand the whole issue

absolute path problem?

My understanding is that because sphinx switches the current working

directory during execution, this path must be absolute. We could get

around this by processing a relative path internally in mpl at rc load

time, and converting it into an absolute path assuming the path is

relative to the directory containing the rc file.

JDH

Here is a candidate patch...this will be processed once on module load
time. I considered doing it in the validate method of the rc params
in rcsetup, but I am not sure this is better because if users are
overriding the variable at runtime, we should probably just let them
do what they want.

I am not sure if this qualifies as a "one time hack" that makes mpl so
difficult to maintain -- Sandro and Ben can weigh in -- but at least
we are doing it to make their lives easier :slight_smile:

Index: lib/matplotlib/__init__.py

···

On Thu, Jan 6, 2011 at 11:36 AM, Benjamin Root <ben.root@...553...> wrote:

On Thu, Jan 6, 2011 at 11:15 AM, John Hunter <jdh2358@...149...> wrote:

On Thu, Jan 6, 2011 at 10:58 AM, Benjamin Root <ben.root@...553...> wrote:

> I thought that problem was already addressed in a previous patch? John,
> could this be another "gremlin"? Or did I mis-understand the whole
> issue
> absolute path problem?

My understanding is that because sphinx switches the current working
directory during execution, this path must be absolute. We could get
around this by processing a relative path internally in mpl at rc load
time, and converting it into an absolute path assuming the path is
relative to the directory containing the rc file.

JDH

I actually think that would be a better solution. Python's os.path module
is very powerful with functions like isabs(), abspath(), expanduser(),
expandvars() and realpath(). Of course, one could easily go overboard with
this, but I am sure we could probably allow for something real simple like
expandvars() so that packagers could utilize environment variables for the
build process?

===================================================================
--- lib/matplotlib/__init__.py (revision 8898)
+++ lib/matplotlib/__init__.py (working copy)
@@ -762,6 +762,13 @@

# this is the instance used by the matplotlib classes
rcParams = rc_params()
+
+if rcParams['examples.directory']:
+ if not os.path.isabs(rcParams['examples.directory']):
+ _basedir, _fname = os.path.split(matplotlib_fname())
+ _fullpath = os.path.join(_basedir, rcParams['examples.directory'])
+ rcParams['examples.directory'] = _fullpath
+
rcParamsOrig = rcParams.copy()

rcParamsDefault = RcParams([ (key, default) for key, (default, converter) in \
@@ -770,6 +777,8 @@
rcParams['ps.usedistiller'] =
checkdep_ps_distiller(rcParams['ps.usedistiller'])
rcParams['text.usetex'] = checkdep_usetex(rcParams['text.usetex'])

+
+
def rc(group, **kwargs):
     """
     Set the current rc params. Group is the grouping for the rc, eg.

I like the idea. Obviously we would need to make it very clear exactly what relative filepaths are going to be relative to.

A comment on the patch. Unless matplotlib_fname() is guaranteed to return an absolute filename, then we need to use realpath() on _basedir so that the final joined filename will also be absolute. The reason for using realpath() instead of abspath() is that in case there are any symbolic links in the name from matplotlib_fname(), these will be qualified so that the filepath matplotlib sees is the same filepath that spinx sees.

Also, a little oddity that I discovered in reading the docs for os.path.join(), turns out that if any element is already an absolute path, then it ignores all elements prior to that. In this patch, everything is ok because we already check for isabs(). Just thought I ought to point that out because I never knew that and that would certainly be a nasty bug to try and track down…

···

On Thu, Jan 6, 2011 at 11:50 AM, John Hunter <jdh2358@…149…> wrote:

On Thu, Jan 6, 2011 at 11:36 AM, Benjamin Root <ben.root@…553…> wrote:

On Thu, Jan 6, 2011 at 11:15 AM, John Hunter <jdh2358@…149…> wrote:

On Thu, Jan 6, 2011 at 10:58 AM, Benjamin Root <ben.root@…553…> wrote:

I thought that problem was already addressed in a previous patch? John,

could this be another “gremlin”? Or did I mis-understand the whole

issue

absolute path problem?

My understanding is that because sphinx switches the current working

directory during execution, this path must be absolute. We could get

around this by processing a relative path internally in mpl at rc load

time, and converting it into an absolute path assuming the path is

relative to the directory containing the rc file.

JDH

I actually think that would be a better solution. Python’s os.path module

is very powerful with functions like isabs(), abspath(), expanduser(),

expandvars() and realpath(). Of course, one could easily go overboard with

this, but I am sure we could probably allow for something real simple like

expandvars() so that packagers could utilize environment variables for the

build process?

Here is a candidate patch…this will be processed once on module load

time. I considered doing it in the validate method of the rc params

in rcsetup, but I am not sure this is better because if users are

overriding the variable at runtime, we should probably just let them

do what they want.

I am not sure if this qualifies as a “one time hack” that makes mpl so

difficult to maintain – Sandro and Ben can weigh in – but at least

we are doing it to make their lives easier :slight_smile:

Index: lib/matplotlib/init.py

===================================================================

— lib/matplotlib/init.py (revision 8898)
+++ lib/matplotlib/init.py (working copy)

@@ -762,6 +762,13 @@

this is the instance used by the matplotlib classes

rcParams = rc_params()

+if rcParams[‘examples.directory’]:

  • if not os.path.isabs(rcParams[‘examples.directory’]):

  •    _basedir, _fname = os.path.split(matplotlib_fname())
    
  •    _fullpath = os.path.join(_basedir, rcParams['examples.directory'])
    
  •    rcParams['examples.directory'] = _fullpath
    

rcParamsOrig = rcParams.copy()

rcParamsDefault = RcParams([ (key, default) for key, (default, converter) in \

@@ -770,6 +777,8 @@
rcParams[‘ps.usedistiller’] =

checkdep_ps_distiller(rcParams[‘ps.usedistiller’])

rcParams[‘text.usetex’] = checkdep_usetex(rcParams[‘text.usetex’])

def rc(group, **kwargs):

 """

 Set the current rc params.  Group is the grouping for the rc, eg.

matplotlib_fname() always returns absolute path. I have not used
realpath, but if you think there is a use for it here, feel free to
post an amended patch.

JDH

···

On Thu, Jan 6, 2011 at 12:15 PM, Benjamin Root <ben.root@...553...> wrote:

A comment on the patch. Unless matplotlib_fname() is guaranteed to return
an absolute filename, then we need to use realpath() on _basedir so that the
final joined filename will also be absolute. The reason for using
realpath() instead of abspath() is that in case there are any symbolic links
in the name from matplotlib_fname(), these will be qualified so that the
filepath matplotlib sees is the same filepath that spinx sees.

Also, a little oddity that I discovered in reading the docs for
os.path.join(), turns out that if any element is already an absolute path,
then it ignores all elements prior to that. In this patch, everything is ok
because we already check for isabs(). Just thought I ought to point that
out because I never knew that and that would certainly be a nasty bug to try
and track down..

There is an exception to this -- if MATPLOTLIBRC or MPLCONFIGDIR are
relative paths, then matplotlib_fname will return a relative path too.

···

On Thu, Jan 6, 2011 at 12:22 PM, John Hunter <jdh2358@...149...> wrote:

matplotlib_fname() always returns absolute path. I have not used
realpath, but if you think there is a use for it here, feel free to
post an amended patch.