Building documentation and matplotlibrc

Ok, I'm trying with trunk now (or those changes are on another
branch?), and it stops at

reading sources... [ 20%] examples/axes_grid/demo_axes_divider

while with RC it stops at

reading sources... [ 14%] examples/api/date_demo

6% improvements :smiley: anyhow, the demo_axes_divider needs file
axes_grid/bivariate_normal.npy that's present in
../sampledata/axes_grid/bivariate_normal.npy so I don't know. John, is
trunk working for your? I'm blocking network access to MPL SF with

sudo iptables -A OUTPUT -d matplotlib.svn.sourceforge.net -j DROP

and when you need it back

sudo iptables -F

(I flush all the rules since I have only the one above).

Regards,

路路路

On Wed, Jan 5, 2011 at 19:48, Sandro Tosi <morph@...12...> wrote:

ehm... no, I'm testing it the rc + your patch, sorry :slight_smile: I'm updating
my local svn copy and try again

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

We're working in the release branch. Changes have not been made to trunk.

svn co https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_maint
mpl1

JDH

路路路

On Wed, Jan 5, 2011 at 1:23 PM, Sandro Tosi <morph@...12...> wrote:

On Wed, Jan 5, 2011 at 19:48, Sandro Tosi <morph@...12...> wrote:
> ehm... no, I'm testing it the rc + your patch, sorry :slight_smile: I'm updating
> my local svn copy and try again

Ok, I'm trying with trunk now (or those changes are on another
branch?), and it stops at

rc2 tarball is here:

http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.0.1/matplotlib-1.0.1rc2.tar.gz/download

路路路

On Wed, Jan 5, 2011 at 1:31 PM, John Hunter <jdh2358@...149...> wrote:

We're working in the release branch. Changes have not been made to trunk.

svn co https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_maint
mpl1

I tried with that branch, but I had no luck :frowning: it still tries to
download stuff from the net, whatever I set as examples.directory in
matplotlibrc (yes, I'm trying to compile inside doc/ and I have
'examples.download : False' set) and/or MATPLOTLIB_SAMPLE_DATA . JFTR
the cmdline I'm using is:

morph@...914...:~/deb/upstream_checkout/mpl1/doc$
MATPLOTLIB_SAMPLE_DATA=/home/morph/deb/upstream_checkout/mpl1/sampledata/
MATPLOTLIBDATA=../lib/matplotlib/mpl-data/
PYTHONPATH=../build/lib.linux-x86_64-2.6/ ./make.py --small all

$ tail -2 matplotlibrc
examples.download : False # False to bypass downloading mechanism
examples.directory : /home/morph/deb/upstream_checkout/mpl1/sampledata/

Regards,

路路路

On Wed, Jan 5, 2011 at 20:31, John Hunter <jdh2358@...149...> wrote:

On Wed, Jan 5, 2011 at 1:23 PM, Sandro Tosi <morph@...12...> wrote:

On Wed, Jan 5, 2011 at 19:48, Sandro Tosi <morph@...12...> wrote:
> ehm... no, I'm testing it the rc + your patch, sorry :slight_smile: I'm updating
> my local svn copy and try again

Ok, I'm trying with trunk now (or those changes are on another
branch?), and it stops at

We're working in the release branch. Changes have not been made to trunk.

svn co https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_maint
mpl1

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

Strange -- I'll try and test tonight with a machine disconnected from
the internet. Out of curiosity, what command/tool are you using to
monitor the internet requests? Ben, are you seeing the same problems?

JDH

路路路

On Wed, Jan 5, 2011 at 3:49 PM, Sandro Tosi <morph@...12...> wrote:

I tried with that branch, but I had no luck :frowning: it still tries to
download stuff from the net, whatever I set as examples.directory in
matplotlibrc (yes, I'm trying to compile inside doc/ and I have
'examples.download : False' set) and/or MATPLOTLIB_SAMPLE_DATA . JFTR
the cmdline I'm using is:

morph@...914...:~/deb/upstream_checkout/mpl1/doc$
MATPLOTLIB_SAMPLE_DATA=/home/morph/deb/upstream_checkout/mpl1/sampledata/
MATPLOTLIBDATA=../lib/matplotlib/mpl-data/
PYTHONPATH=../build/lib.linux-x86_64-2.6/ ./make.py --small all

$ tail -2 matplotlibrc
examples.download : False # False to bypass downloading mechanism
examples.directory : /home/morph/deb/upstream_checkout/mpl1/sampledata/

it's not exactly monitory, but I disable all the requests to
matplotlib.svn.sourceforge.net via iptables:

sudo iptables -A OUTPUT -d matplotlib.svn.sourceforge.net -j DROP

and you're sure nothing passes towards that address :slight_smile: and in fact,
the moment you reach an example with get_sample_data() it stops.

路路路

On Wed, Jan 5, 2011 at 23:00, John Hunter <jdh2358@...149...> wrote:

On Wed, Jan 5, 2011 at 3:49 PM, Sandro Tosi <morph@...12...> wrote:

I tried with that branch, but I had no luck :frowning: it still tries to
download stuff from the net, whatever I set as examples.directory in
matplotlibrc (yes, I'm trying to compile inside doc/ and I have
'examples.download : False' set) and/or MATPLOTLIB_SAMPLE_DATA . JFTR
the cmdline I'm using is:

morph@...914...:~/deb/upstream_checkout/mpl1/doc$
MATPLOTLIB_SAMPLE_DATA=/home/morph/deb/upstream_checkout/mpl1/sampledata/
MATPLOTLIBDATA=../lib/matplotlib/mpl-data/
PYTHONPATH=../build/lib.linux-x86_64-2.6/ ./make.py --small all

$ tail -2 matplotlibrc
examples.download : False # False to bypass downloading mechanism
examples.directory : /home/morph/deb/upstream_checkout/mpl1/sampledata/

Strange -- I'll try and test tonight with a machine disconnected from
the internet. Out of curiosity, what command/tool are you using to
monitor the internet requests? Ben, are you seeing the same problems?

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

That's helpful. I used to be an iptables guru, but it has been a long
time. How do you reverse the command when you are done testing?

JDH

路路路

On Wed, Jan 5, 2011 at 4:04 PM, Sandro Tosi <morph@...12...> wrote:

it's not exactly monitory, but I disable all the requests to
matplotlib.svn.sourceforge.net via iptables:

sudo iptables -A OUTPUT -d matplotlib.svn.sourceforge.net -j DROP

and you're sure nothing passes towards that address :slight_smile: and in fact,
the moment you reach an example with get_sample_data() it stops.

if you have only those rule, then just issue a

sudo iptables -F

to flus the whole iptables chains list rules; else, you have to:

sudo iptables -nL OUTPUT --line-numbers

take notes of the line number of those for MPL SF (the ip addresses are those in

$ host matplotlib.svn.sourceforge.net
matplotlib.svn.sourceforge.net has address 216.34.181.177
matplotlib.svn.sourceforge.net has address 216.34.181.65
)

and then remove them:

sudo iptables -D OUTPUT <line number above>

(note that if you delete a rules, the following ones have linenumber changed...)

Cheers,

路路路

On Wed, Jan 5, 2011 at 23:07, John Hunter <jdh2358@...149...> wrote:

On Wed, Jan 5, 2011 at 4:04 PM, Sandro Tosi <morph@...12...> wrote:

it's not exactly monitory, but I disable all the requests to
matplotlib.svn.sourceforge.net via iptables:

sudo iptables -A OUTPUT -d matplotlib.svn.sourceforge.net -j DROP

and you're sure nothing passes towards that address :slight_smile: and in fact,
the moment you reach an example with get_sample_data() it stops.

That's helpful. I used to be an iptables guru, but it has been a long
time. How do you reverse the command when you are done testing?

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

OK, I found the problem. Somehow my edit to sphinxext.plot_directive
to actually call the new function was not saved and committed. I just
fixed this, so the branch *should* work. The branch
matlpotlib/sphinxext/plot_directive.py should call rc_file_defaults
rather than rcdefaults. After an svn up, confirm that this is so
before testing.

JDH

路路路

On Wed, Jan 5, 2011 at 3:49 PM, Sandro Tosi <morph@...12...> wrote:

On Wed, Jan 5, 2011 at 20:31, John Hunter <jdh2358@...149...> wrote:

On Wed, Jan 5, 2011 at 1:23 PM, Sandro Tosi <morph@...12...> wrote:

On Wed, Jan 5, 2011 at 19:48, Sandro Tosi <morph@...12...> wrote:
> ehm... no, I'm testing it the rc + your patch, sorry :slight_smile: I'm updating
> my local svn copy and try again

Ok, I'm trying with trunk now (or those changes are on another
branch?), and it stops at

We're working in the release branch. Changes have not been made to trunk.

svn co https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_maint
mpl1

I tried with that branch, but I had no luck :frowning: it still tries to
download stuff from the net, whatever I set as examples.directory in
matplotlibrc (yes, I'm trying to compile inside doc/ and I have
'examples.download : False' set) and/or MATPLOTLIB_SAMPLE_DATA . JFTR
the cmdline I'm using is:

morph@...914...:~/deb/upstream_checkout/mpl1/doc$
MATPLOTLIB_SAMPLE_DATA=/home/morph/deb/upstream_checkout/mpl1/sampledata/
MATPLOTLIBDATA=../lib/matplotlib/mpl-data/
PYTHONPATH=../build/lib.linux-x86_64-2.6/ ./make.py --small all

$ tail -2 matplotlibrc
examples.download : False # False to bypass downloading mechanism
examples.directory : /home/morph/deb/upstream_checkout/mpl1/sampledata/

$ grep -c rc_file_defaults lib/matplotlib/sphinxext/plot_directive.py
0

so it's not there :frowning:

and the only file containing rc_file_defaults is lib/matplotlib/__init__.py

of course, examples are stil stuck if no network to SF is available

Oh i think I got it, lib/matplotlib/sphinxext/plot_directive.py was
updated on trunk instead of the v1.0 branch.

Cheers,

路路路

On Wed, Jan 5, 2011 at 23:22, John Hunter <jdh2358@...149...> wrote:

OK, I found the problem. Somehow my edit to sphinxext.plot_directive
to actually call the new function was not saved and committed. I just
fixed this, so the branch *should* work. The branch
matlpotlib/sphinxext/plot_directive.py should call rc_file_defaults
rather than rcdefaults. After an svn up, confirm that this is so
before testing.

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

It took me a long time to finally figure out how to get matplotlib to
respect the wishes of the packaging. Ultimately I believe I got it
working but unfortunately, it's been a while, so I'm not entirely sure
what was necessary. If it helps, I believe the package in my PPA builds
correctly. Sorry I don't have more time to investigate; things are quite
hectic at the moment.

Cheers,

- Ben

路路路

On Wed, 5 Jan 2011 16:00:40 -0600, John Hunter <jdh2358@...149...> wrote:

Strange -- I'll try and test tonight with a machine disconnected from
the internet. Out of curiosity, what command/tool are you using to
monitor the internet requests? Ben, are you seeing the same problems?

I swear there is a gremlin in my computer. I checked and rechecked
this on my box because of all the confusion we were having, but
apparently managed to commit on the trunk (which doesn't even have the
function defined because I haven't merged) and not on the branch. We
need to get the buildbot going again, because the doc buildbot would
have caught this. I have fixed both the trunk and the branch: the
branch calls the new function and has the desired "no internet"
behavior. The trunk calls the old function which restores rc internal
defaults rather than file defaults and so is broken in the way it has
always been broken, but will be fixed when we merge.

In the command below, mpl1 is my branch (1.0.1 release) directory:

聽聽mpl1> grep rc_file lib/matplotlib/sphinxext/plot_directive.py

Surely I have it right now!

JDH

路路路

On Wed, Jan 5, 2011 at 5:53 PM, Sandro Tosi <morph@...12...> wrote:

On Wed, Jan 5, 2011 at 23:22, John Hunter <jdh2358@...149...> wrote:

OK, I found the problem. Somehow my edit to sphinxext.plot_directive
to actually call the new function was not saved and committed. I just
fixed this, so the branch *should* work. The branch
matlpotlib/sphinxext/plot_directive.py should call rc_file_defaults
rather than rcdefaults. After an svn up, confirm that this is so
before testing.

$ grep -c rc_file_defaults lib/matplotlib/sphinxext/plot_directive.py
0

so it's not there :frowning:

and the only file containing rc_file_defaults is lib/matplotlib/__init__.py

of course, examples are stil stuck if no network to SF is available

Oh i think I got it, lib/matplotlib/sphinxext/plot_directive.py was
updated on trunk instead of the v1.0 branch.

Indeed, and it works fine!!! the only thing to notice is that
examples.directory must be an absolute path, but that set, the doc
build process gets files from the location specified.

Thanks a lot!!

Cheers,

路路路

On Thu, Jan 6, 2011 at 02:31, John Hunter <jdh2358@...149...> wrote:

mpl1> grep rc_file lib/matplotlib/sphinxext/plot_directive.py

Surely I have it right now!

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

I thought that problem was already addressed in a previous patch? John, could this be another 鈥済remlin鈥? Or did I mis-understand the whole issue absolute path problem?

Ben Root

路路路

On Wed, Jan 5, 2011 at 10:42 PM, Sandro Tosi <morph@鈥12鈥> wrote:

On Thu, Jan 6, 2011 at 02:31, John Hunter <jdh2358@鈥149鈥> wrote:

mpl1> grep rc_file lib/matplotlib/sphinxext/plot_directive.py

Surely I have it right now!

Indeed, and it works fine!!! the only thing to notice is that

examples.directory must be an absolute path, but that set, the doc

build process gets files from the location specified.

My understanding is that because sphinx switches the current working
directory during execution, this path must be absolute. We could get
around this by processing a relative path internally in mpl at rc load
time, and converting it into an absolute path assuming the path is
relative to the directory containing the rc file.

JDH

路路路

On Thu, Jan 6, 2011 at 10:58 AM, Benjamin Root <ben.root@...553...> wrote:

I thought that problem was already addressed in a previous patch? John,
could this be another "gremlin"? Or did I mis-understand the whole issue
absolute path problem?

I actually think that would be a better solution. Python鈥檚 os.path module is very powerful with functions like isabs(), abspath(), expanduser(), expandvars() and realpath(). Of course, one could easily go overboard with this, but I am sure we could probably allow for something real simple like expandvars() so that packagers could utilize environment variables for the build process?

Ben Root

路路路

On Thu, Jan 6, 2011 at 11:15 AM, John Hunter <jdh2358@鈥149鈥> wrote:

On Thu, Jan 6, 2011 at 10:58 AM, Benjamin Root <ben.root@鈥553鈥> wrote:

I thought that problem was already addressed in a previous patch? John,

could this be another 鈥済remlin鈥? Or did I mis-understand the whole issue

absolute path problem?

My understanding is that because sphinx switches the current working

directory during execution, this path must be absolute. We could get

around this by processing a relative path internally in mpl at rc load

time, and converting it into an absolute path assuming the path is

relative to the directory containing the rc file.

JDH

Here is a candidate patch...this will be processed once on module load
time. I considered doing it in the validate method of the rc params
in rcsetup, but I am not sure this is better because if users are
overriding the variable at runtime, we should probably just let them
do what they want.

I am not sure if this qualifies as a "one time hack" that makes mpl so
difficult to maintain -- Sandro and Ben can weigh in -- but at least
we are doing it to make their lives easier :slight_smile:

Index: lib/matplotlib/__init__.py

路路路

On Thu, Jan 6, 2011 at 11:36 AM, Benjamin Root <ben.root@...553...> wrote:

On Thu, Jan 6, 2011 at 11:15 AM, John Hunter <jdh2358@...149...> wrote:

On Thu, Jan 6, 2011 at 10:58 AM, Benjamin Root <ben.root@...553...> wrote:

> I thought that problem was already addressed in a previous patch? John,
> could this be another "gremlin"? Or did I mis-understand the whole
> issue
> absolute path problem?

My understanding is that because sphinx switches the current working
directory during execution, this path must be absolute. We could get
around this by processing a relative path internally in mpl at rc load
time, and converting it into an absolute path assuming the path is
relative to the directory containing the rc file.

JDH

I actually think that would be a better solution. Python's os.path module
is very powerful with functions like isabs(), abspath(), expanduser(),
expandvars() and realpath(). Of course, one could easily go overboard with
this, but I am sure we could probably allow for something real simple like
expandvars() so that packagers could utilize environment variables for the
build process?

===================================================================
--- lib/matplotlib/__init__.py (revision 8898)
+++ lib/matplotlib/__init__.py (working copy)
@@ -762,6 +762,13 @@

# this is the instance used by the matplotlib classes
rcParams = rc_params()
+
+if rcParams['examples.directory']:
+ if not os.path.isabs(rcParams['examples.directory']):
+ _basedir, _fname = os.path.split(matplotlib_fname())
+ _fullpath = os.path.join(_basedir, rcParams['examples.directory'])
+ rcParams['examples.directory'] = _fullpath
+
rcParamsOrig = rcParams.copy()

rcParamsDefault = RcParams([ (key, default) for key, (default, converter) in \
@@ -770,6 +777,8 @@
rcParams['ps.usedistiller'] =
checkdep_ps_distiller(rcParams['ps.usedistiller'])
rcParams['text.usetex'] = checkdep_usetex(rcParams['text.usetex'])

+
+
def rc(group, **kwargs):
聽聽聽聽聽"""
聽聽聽聽聽Set the current rc params. Group is the grouping for the rc, eg.

I like the idea. Obviously we would need to make it very clear exactly what relative filepaths are going to be relative to.

A comment on the patch. Unless matplotlib_fname() is guaranteed to return an absolute filename, then we need to use realpath() on _basedir so that the final joined filename will also be absolute. The reason for using realpath() instead of abspath() is that in case there are any symbolic links in the name from matplotlib_fname(), these will be qualified so that the filepath matplotlib sees is the same filepath that spinx sees.

Also, a little oddity that I discovered in reading the docs for os.path.join(), turns out that if any element is already an absolute path, then it ignores all elements prior to that. In this patch, everything is ok because we already check for isabs(). Just thought I ought to point that out because I never knew that and that would certainly be a nasty bug to try and track down鈥

路路路

On Thu, Jan 6, 2011 at 11:50 AM, John Hunter <jdh2358@鈥149鈥> wrote:

On Thu, Jan 6, 2011 at 11:36 AM, Benjamin Root <ben.root@鈥553鈥> wrote:

On Thu, Jan 6, 2011 at 11:15 AM, John Hunter <jdh2358@鈥149鈥> wrote:

On Thu, Jan 6, 2011 at 10:58 AM, Benjamin Root <ben.root@鈥553鈥> wrote:

I thought that problem was already addressed in a previous patch? John,

could this be another 鈥済remlin鈥? Or did I mis-understand the whole

issue

absolute path problem?

My understanding is that because sphinx switches the current working

directory during execution, this path must be absolute. We could get

around this by processing a relative path internally in mpl at rc load

time, and converting it into an absolute path assuming the path is

relative to the directory containing the rc file.

JDH

I actually think that would be a better solution. Python鈥檚 os.path module

is very powerful with functions like isabs(), abspath(), expanduser(),

expandvars() and realpath(). Of course, one could easily go overboard with

this, but I am sure we could probably allow for something real simple like

expandvars() so that packagers could utilize environment variables for the

build process?

Here is a candidate patch鈥his will be processed once on module load

time. I considered doing it in the validate method of the rc params

in rcsetup, but I am not sure this is better because if users are

overriding the variable at runtime, we should probably just let them

do what they want.

I am not sure if this qualifies as a 鈥渙ne time hack鈥 that makes mpl so

difficult to maintain 鈥 Sandro and Ben can weigh in 鈥 but at least

we are doing it to make their lives easier :slight_smile:

Index: lib/matplotlib/init.py

===================================================================

鈥 lib/matplotlib/init.py (revision 8898)
+++ lib/matplotlib/init.py (working copy)

@@ -762,6 +762,13 @@

this is the instance used by the matplotlib classes

rcParams = rc_params()

+if rcParams[鈥榚xamples.directory鈥橾:

  • if not os.path.isabs(rcParams[鈥榚xamples.directory鈥橾):

  •    _basedir, _fname = os.path.split(matplotlib_fname())
    
  •    _fullpath = os.path.join(_basedir, rcParams['examples.directory'])
    
  •    rcParams['examples.directory'] = _fullpath
    

rcParamsOrig = rcParams.copy()

rcParamsDefault = RcParams([ (key, default) for key, (default, converter) in \

@@ -770,6 +777,8 @@
rcParams[鈥榩s.usedistiller鈥橾 =

checkdep_ps_distiller(rcParams[鈥榩s.usedistiller鈥橾)

rcParams[鈥榯ext.usetex鈥橾 = checkdep_usetex(rcParams[鈥榯ext.usetex鈥橾)

def rc(group, **kwargs):

 """

 Set the current rc params.  Group is the grouping for the rc, eg.

matplotlib_fname() always returns absolute path. I have not used
realpath, but if you think there is a use for it here, feel free to
post an amended patch.

JDH

路路路

On Thu, Jan 6, 2011 at 12:15 PM, Benjamin Root <ben.root@...553...> wrote:

A comment on the patch. Unless matplotlib_fname() is guaranteed to return
an absolute filename, then we need to use realpath() on _basedir so that the
final joined filename will also be absolute. The reason for using
realpath() instead of abspath() is that in case there are any symbolic links
in the name from matplotlib_fname(), these will be qualified so that the
filepath matplotlib sees is the same filepath that spinx sees.

Also, a little oddity that I discovered in reading the docs for
os.path.join(), turns out that if any element is already an absolute path,
then it ignores all elements prior to that. In this patch, everything is ok
because we already check for isabs(). Just thought I ought to point that
out because I never knew that and that would certainly be a nasty bug to try
and track down..

There is an exception to this -- if MATPLOTLIBRC or MPLCONFIGDIR are
relative paths, then matplotlib_fname will return a relative path too.

路路路

On Thu, Jan 6, 2011 at 12:22 PM, John Hunter <jdh2358@...149...> wrote:

matplotlib_fname() always returns absolute path. I have not used
realpath, but if you think there is a use for it here, feel free to
post an amended patch.