Doc changes

Development matplotlib-devel
1

I've been working on Debian's (Sandro Tosi's) request to produce a smaller doc tree... If only I had known how much work was involved before I started!

SUMMARY: Since so much has changed in the doc build, I need help testing it in other environments -- particularly because LaTeX docs have never built on my machine. I've done this on the branch, so the Debian guys can run with it, but it's a little risky, due to the depth of cuts I had to make. I've tested successfully with Sphinx 0.5.1 and docutils 0.5. Please clean your checkout and then build for yourself and let me know if you see anything funny.

DETAILS:

You can now do

  python make.py --small html

which will only generate low-res PNGs for the html build. "--small" has no effect for pdf build.

The payoff here is that the html tree goes from 109MB to 49MB.

There have been some fundamental changes to how files are staged in the doc build. Rather than dumping everything in _static and then having Sphinx copy them over for us, our various extensions now either a) put the files directly into the build tree under build/html/_images or build/html/pyplot_directive, or b) stage things in build/pyplot_directive, and then copy to the build tree as needed. The exception is the _static/examples directory (generated by gen_rst.py), since they will require some major work to know where the build directory is.

The plot_directive always builds all three versions of every plot (png, hires.png, pdf) into build/pyplot_directive. Depending on a user setting, only some of these may be copied to the output directory. The reason for this is so that switching between html and pdf builds works, and we don't get hundreds of false warnings from sphinx about missing files. It took a while to arrive and that procedure... :wink:

gen_gallery now runs in a sphinx callback (env-updated) after all the input files have been parsed, (and all the plots have been generated), but right before the html or latex is written out. This eliminates the need to run the build multiple times, and the need to have an autogenerated file (doc/_templates/gallery.html) in SVN. The thumbnail downsampling is now done as part of this step, rather than as part of plot_directive, since it isn't needed for LaTeX builds.

Something similar should be done for the examples, I just haven't gotten around to it yet.

Lastly, since files are in many different places, the Sourceforge site should probably be cleaned (if it isn't automatically already) to ensure we don't go over quota.

Mike

···

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

2

SUMMARY: Since so much has changed in the doc build, I need help testing
it in other environments -- particularly because LaTeX docs have never
built on my machine. I've done this on the branch, so the Debian guys
can run with it, but it's a little risky, due to the depth of cuts I had
to make. I've tested successfully with Sphinx 0.5.1 and docutils 0.5.
Please clean your checkout and then build for yourself and let me know
if you see anything funny.

DETAILS:

You can now do

python make.py --small html

which will only generate low-res PNGs for the html build. "--small" has
no effect for pdf build.

Very nice -- it seems like you've solved some of the unneeded rebuild
problems we had been having too, because successive calls to make html
seem to go much faster now.

gen_gallery now runs in a sphinx callback (env-updated) after all the
input files have been parsed, (and all the plots have been generated),
but right before the html or latex is written out. This eliminates the
need to run the build multiple times, and the need to have an
autogenerated file (doc/_templates/gallery.html) in SVN. The thumbnail
downsampling is now done as part of this step, rather than as part of
plot_directive, since it isn't needed for LaTeX builds.

I have built the pdf on a linux environment and the html on a linux
and os x environment, so it is looking good. I've built with and w/o
the --small

Lastly, since files are in many different places, the Sourceforge site
should probably be cleaned (if it isn't automatically already) to ensure
we don't go over quota.

Will do -- some time ago I emailed the sf support people and asked for
a quota increase, and they replied that there wasn't really a quota
anymore, but I will clean it so we minimize our footprint. And it
they ever do get anxious about the size we consume, we can always go
--small :slight_smile:

Thanks again, sorry that was such a bear. Hopefully the plot
directive emerges stronger from the carnage.

JDH

···

On Wed, Dec 17, 2008 at 10:52 AM, Michael Droettboom <mdroe@...31...> wrote:

3

Ok. Based on your success report, I'll go ahead and merge this to trunk.

Sandro: please let me know if these changes break anything in your package build scripts.

Mike

John Hunter wrote:

···

On Wed, Dec 17, 2008 at 10:52 AM, Michael Droettboom <mdroe@...31...> wrote:

SUMMARY: Since so much has changed in the doc build, I need help testing
it in other environments -- particularly because LaTeX docs have never
built on my machine. I've done this on the branch, so the Debian guys
can run with it, but it's a little risky, due to the depth of cuts I had
to make. I've tested successfully with Sphinx 0.5.1 and docutils 0.5.
Please clean your checkout and then build for yourself and let me know
if you see anything funny.

DETAILS:

You can now do

python make.py --small html

which will only generate low-res PNGs for the html build. "--small" has
no effect for pdf build.
    
Very nice -- it seems like you've solved some of the unneeded rebuild
problems we had been having too, because successive calls to make html
seem to go much faster now.

gen_gallery now runs in a sphinx callback (env-updated) after all the
input files have been parsed, (and all the plots have been generated),
but right before the html or latex is written out. This eliminates the
need to run the build multiple times, and the need to have an
autogenerated file (doc/_templates/gallery.html) in SVN. The thumbnail
downsampling is now done as part of this step, rather than as part of
plot_directive, since it isn't needed for LaTeX builds.
    
I have built the pdf on a linux environment and the html on a linux
and os x environment, so it is looking good. I've built with and w/o
the --small

Lastly, since files are in many different places, the Sourceforge site
should probably be cleaned (if it isn't automatically already) to ensure
we don't go over quota.
    
Will do -- some time ago I emailed the sf support people and asked for
a quota increase, and they replied that there wasn't really a quota
anymore, but I will clean it so we minimize our footprint. And it
they ever do get anxious about the size we consume, we can always go
--small :slight_smile:

Thanks again, sorry that was such a bear. Hopefully the plot
directive emerges stronger from the carnage.

JDH
  
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

4

I'll do as soon as we will have sphinx 0.5.1 packages (we're working
on it just now).

Let me thank you again for your prompt responsiveness.

Cheers,

···

On Wed, Dec 17, 2008 at 22:16, Michael Droettboom <mdroe@...31...> wrote:

Ok. Based on your success report, I'll go ahead and merge this to trunk.

Sandro: please let me know if these changes break anything in your package
build scripts.

--
Sandro Tosi (aka morph, Morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

5

Ok. Based on your success report, I'll go ahead and merge this to trunk.

Sandro: please let me know if these changes break anything in your package
build scripts.

I'll do as soon as we will have sphinx 0.5.1 packages (we're working
on it just now).

Let me thank you again for your prompt responsiveness.

Sandro -- do you distribute something like a matplotlib-devel, so that
people who want to compile mpl, and build the docs, could do so
themselves with a simple

  > apt-get install python-matplotlib-devel
  > cd ~/path/to/mpl/docs
  > python make.py html latex

That would be nice, because it would make it easier for users and
developers to contribute to the docs and test their changes. Right
now there is something of a barrier to entry for people who want to
contribute to the docs.

JDH

···

On Wed, Dec 17, 2008 at 3:22 PM, Sandro Tosi <morph@...12...> wrote:

On Wed, Dec 17, 2008 at 22:16, Michael Droettboom <mdroe@...31...> wrote:

6

Sandro -- do you distribute something like a matplotlib-devel, so that
people who want to compile mpl, and build the docs, could do so
themselves with a simple

do you mean something like the source package + debianization? of
course we do! :slight_smile: you need a "source" line in the /etc/apt/sources.list
(something like deb-src <url>) and then

apt-get source <source_package_name>

in this case matplotlib.

> apt-get install python-matplotlib-devel
> cd ~/path/to/mpl/docs
> python make.py html latex

That would be nice, because it would make it easier for users and
developers to contribute to the docs and test their changes. Right
now there is something of a barrier to entry for people who want to
contribute to the docs.

once you issued that "apt-get source" you'll have a nice "<pkg

-<version>" dir in . with the upstream tarball uncompress and with

the debianization applied. With that, you can do all you can with the
upstream source code, or use debian tools to compile the package.

I hope I answered your question,

···

On Wed, Dec 17, 2008 at 22:25, John Hunter <jdh2358@...149...> wrote:
--
Sandro Tosi (aka morph, Morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

7

Talking about that, last time I looked, the plot directive, and the other
MPL sphinx extension were not in the matplotlib namespace, and thus not
importable by other programs. This is a pity, because they can be useful
for other projects (I wrote for instance some lecture notes using sphinx
to doctest them, and I wanted to use the plot directive).

It seems it would be very easy to put them in a matplotlib submodule, and
have the sphinx conf.py import from there. Nothing urgent, but if there
is no reason no to, I would make me a happy man.

Cheers,

Gaël

···

On Wed, Dec 17, 2008 at 12:25:12PM -0600, John Hunter wrote:

Thanks again, sorry that was such a bear. Hopefully the plot
directive emerges stronger from the carnage.

8

We have provided the sphinx_template2 to serve as a starting point for
sphinx projects that want to use the plot, mathmpl, etc, extensions.
Unfortunately, we are not doing the best job of keeping the two in
sync. Fernando and I have talked of the need to have standard,
reusable components for these kinds of things. Perhaps it would be
possible to move the guts of some of these into a matplotlib.sphinxext
module.

Here is the pointer to the sphinx template:

···

On Wed, Dec 17, 2008 at 3:57 PM, Gael Varoquaux <gael.varoquaux@...427...> wrote:

On Wed, Dec 17, 2008 at 12:25:12PM -0600, John Hunter wrote:

Talking about that, last time I looked, the plot directive, and the other
MPL sphinx extension were not in the matplotlib namespace, and thus not
importable by other programs. This is a pity, because they can be useful
for other projects (I wrote for instance some lecture notes using sphinx
to doctest them, and I wanted to use the plot directive).

It seems it would be very easy to put them in a matplotlib submodule, and
have the sphinx conf.py import from there. Nothing urgent, but if there
is no reason no to, I would make me a happy man.

9

John Hunter skrev:

That would be nice, because it would make it easier for users and
developers to contribute to the docs and test their changes. Right
now there is something of a barrier to entry for people who want to
contribute to the docs.

JDH

What about building the docs on windows? The mpl_examples, mpl_data links don't work on windows, I get a file that contains:

link ../mpl_examples

Is it possible to convince sphinx to follow the path that is inside this file?

Today I saw that the new legend options are missing in the docs. At least in the docs on the front page.

/Jörgen

10

Hi!

Ok. Based on your success report, I'll go ahead and merge this to trunk.

Sandro: please let me know if these changes break anything in your package
build scripts.

I don't know if you're waiting for an input from me, but: John made me
test 0.98.5.2 preliminary version and doc build was great! the whole
set of packages (included the original tarball) is not ~62M, much
better :slight_smile:

After the off-list email exchange with John, I saw no official
announce of 0.98.5.2, so maybe you're planning to do it as xmas gift,
just to be sure I didn't miss anything :slight_smile:

Cheers and Merry Xmas guys :slight_smile:

···

On Wed, Dec 17, 2008 at 22:16, Michael Droettboom <mdroe@...31...> wrote:

--
Sandro Tosi (aka morph, Morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

11

I'll let you know, don't worry. After all the pain, I want to make
sure we have a stable build, and there is at least one fix (legend
dpi) that I will push out into 98.5.3 before I ask you to upload for
debian. After Christmas, though.

JDH

···

On Wed, Dec 24, 2008 at 3:34 AM, Sandro Tosi <morph@...12...> wrote:

Hi!

On Wed, Dec 17, 2008 at 22:16, Michael Droettboom <mdroe@...31...> wrote:

Ok. Based on your success report, I'll go ahead and merge this to trunk.

Sandro: please let me know if these changes break anything in your package
build scripts.

I don't know if you're waiting for an input from me, but: John made me
test 0.98.5.2 preliminary version and doc build was great! the whole
set of packages (included the original tarball) is not ~62M, much
better :slight_smile:

After the off-list email exchange with John, I saw no official
announce of 0.98.5.2, so maybe you're planning to do it as xmas gift,
just to be sure I didn't miss anything :slight_smile: