Sphinx custom extension mess, and patches

Hi all,

Sorry for the multiple posting, this concerns various groups, and I'd
rather the information not be lost.

While working on getting our in-lab library ready to be merged with NiPy,
I ran into some sort of 'sphinx extension mess' where various sphinx
extension would have side effects on each other, and most important, the
extensions did not work with sphinx trunk.

I got the side effects to be limited by cleaning up the generated code
from autosummary before each run: I added the following code in my
sphinx conf.py:

sphinxext.diff (2.31 KB)

···

################################################################################
# Hack: run the autosummary generation script
import shutil
if os.path.exists('generated'):
    shutil.rmtree('generated')
os.system('%s sphinxext/autosummary_generate.py -o generated *.rst' %
            sys.executable)
################################################################################

I am attaching a diff of all the modifications I made to get the various
extensions to work. I hope you can use it to get your various extensions
working on sphinx trunk quicker. For the NiPy guys, I will be committing
these changes in the fff2 tree soon, and we can go over this at the
sprint.

This does raise a problem: this extension code is all over the place, in
various repository. Some of the code cannot live in the sphinx repo, as
it introduces dependencies. However, as the extensions are not importable
from Python (I can't do 'from matplotlib.sphinxext import mathmpl'), the
different projects using them end up copying them in their repo, and thus
there are several versions floating around not updated. Some of the
extensions would do not add externa dependencies to sphinx. These should
be pushed into sphinx, with tests. That way as sphinx evolves, they do
not break.

Gaël

Gael,

You raise a very good point about the duplication of code around. As a case in point, the patches you provided no longer apply to the "canonical" (or at least original) versions of the plugins that began life in matplotlib. Recent versions of Sphinx have a proper extension API, so that "try ... except" importing is no longer necessary.

For mathmpl.py --- I will move that into the matplotlib installed code tree, so, as you suggest, "from matplotlib.sphinxext import mathmpl" will work. Obviously, that won't really provide traction until the next release of matplotlib. And it relies on only_directive.py, so I will probably have to put that in matplotlib as well for now, though that is a reasonable candidate for inclusion in Sphinx.

I have submitted inheritance_diagram.py to Sphinx already. There didn't seem to be much interest, and there were some details (particularly how it deals with files), that weren't "the Sphinx way", and there few documents and/or examples etc. at the time about how to do it right. Someone else ended up re-engineering it to use pygraphviz which felt pretty heavy weight to me. So the thing kind of stalled. But it's probably worth another push.

Mike

Gael Varoquaux wrote:

···

Hi all,

Sorry for the multiple posting, this concerns various groups, and I'd
rather the information not be lost.

While working on getting our in-lab library ready to be merged with NiPy,
I ran into some sort of 'sphinx extension mess' where various sphinx
extension would have side effects on each other, and most important, the
extensions did not work with sphinx trunk.

I got the side effects to be limited by cleaning up the generated code
from autosummary before each run: I added the following code in my
sphinx conf.py:

################################################################################
# Hack: run the autosummary generation script import shutil
if os.path.exists('generated'):
    shutil.rmtree('generated')
os.system('%s sphinxext/autosummary_generate.py -o generated *.rst' %
            sys.executable)
################################################################################

I am attaching a diff of all the modifications I made to get the various
extensions to work. I hope you can use it to get your various extensions
working on sphinx trunk quicker. For the NiPy guys, I will be committing
these changes in the fff2 tree soon, and we can go over this at the
sprint.

This does raise a problem: this extension code is all over the place, in
various repository. Some of the code cannot live in the sphinx repo, as
it introduces dependencies. However, as the extensions are not importable
from Python (I can't do 'from matplotlib.sphinxext import mathmpl'), the
different projects using them end up copying them in their repo, and thus
there are several versions floating around not updated. Some of the
extensions would do not add externa dependencies to sphinx. These should
be pushed into sphinx, with tests. That way as sphinx evolves, they do
not break.

Gaël
  ------------------------------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
------------------------------------------------------------------------

_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
matplotlib-devel List Signup and Options

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

At one point, some of us had a plan to keep all of these "scientific"
extensions collected here:

  http://sphinx.googlecode.com/svn/contrib/trunk/numpyext/

SVN-using projects could use svn:externals to include these in their
projects without diverging the code. I really don't know why this plan
changed. Pauli?

···

On Sun, Feb 15, 2009 at 08:04, Gael Varoquaux <gael.varoquaux@...427...> wrote:

Hi all,

Sorry for the multiple posting, this concerns various groups, and I'd
rather the information not be lost.

While working on getting our in-lab library ready to be merged with NiPy,
I ran into some sort of 'sphinx extension mess' where various sphinx
extension would have side effects on each other, and most important, the
extensions did not work with sphinx trunk.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

Gael Varoquaux schrieb:

Hi all,

Sorry for the multiple posting, this concerns various groups, and I'd
rather the information not be lost.

While working on getting our in-lab library ready to be merged with NiPy,
I ran into some sort of 'sphinx extension mess' where various sphinx
extension would have side effects on each other, and most important, the
extensions did not work with sphinx trunk.

Which is of course kind of my fault, because I keep changing the API :slight_smile:

But it must also be said that during 0.x, I tend to view cleanliness and
good code as more important than 100% backwards compatibility.

I got the side effects to be limited by cleaning up the generated code
from autosummary before each run: I added the following code in my
sphinx conf.py:

################################################################################
# Hack: run the autosummary generation script
import shutil
if os.path.exists('generated'):
    shutil.rmtree('generated')
os.system('%s sphinxext/autosummary_generate.py -o generated *.rst' %
            sys.executable)
################################################################################

I am attaching a diff of all the modifications I made to get the various
extensions to work. I hope you can use it to get your various extensions
working on sphinx trunk quicker. For the NiPy guys, I will be committing
these changes in the fff2 tree soon, and we can go over this at the
sprint.

This does raise a problem: this extension code is all over the place, in
various repository. Some of the code cannot live in the sphinx repo, as
it introduces dependencies. However, as the extensions are not importable
from Python (I can't do 'from matplotlib.sphinxext import mathmpl'), the
different projects using them end up copying them in their repo, and thus
there are several versions floating around not updated. Some of the
extensions would do not add externa dependencies to sphinx. These should
be pushed into sphinx, with tests. That way as sphinx evolves, they do
not break.

I'm all for it. In the case of autosummary, I'm guilty of not getting it
in sooner. This will change soon. In other cases, I don't even know of
the extension, probably because those who write it deem it as too
project-specific to contribute it.

I don't ask for too much if an extension is contributed, so by all means
do at least post about your extensions!

Georg

I am not blaming anyone, just pointing out a non ideal situation. It has
already improved a lot with the matplotlib guys and the scipy guys
merging some changes in extensions and publishing the extensions in an
importable part of their source tree.

It is true that I'll be happier when I will be able to import the
only_directive, and the auto_summary from sphinx :).

Thanks for your great work,

Gaël

···

On Tue, Feb 17, 2009 at 12:17:17AM +0100, Georg Brandl wrote:

I'm all for it. In the case of autosummary, I'm guilty of not getting it
in sooner. This will change soon. In other cases, I don't even know of
the extension, probably because those who write it deem it as too
project-specific to contribute it.

I don't ask for too much if an extension is contributed, so by all means
do at least post about your extensions!

Michael Droettboom schrieb:

Gael,

You raise a very good point about the duplication of code around. As a
case in point, the patches you provided no longer apply to the
"canonical" (or at least original) versions of the plugins that began
life in matplotlib. Recent versions of Sphinx have a proper extension
API, so that "try ... except" importing is no longer necessary.

For mathmpl.py --- I will move that into the matplotlib installed code
tree, so, as you suggest, "from matplotlib.sphinxext import mathmpl"
will work. Obviously, that won't really provide traction until the next
release of matplotlib. And it relies on only_directive.py, so I will
probably have to put that in matplotlib as well for now, though that is
a reasonable candidate for inclusion in Sphinx.

The "only" is so trivial that it will, in this or a more extended form,
certainly be part of 0.6, which will not come out later than Feb 31 :wink:

I have submitted inheritance_diagram.py to Sphinx already. There didn't
seem to be much interest, and there were some details (particularly how
it deals with files), that weren't "the Sphinx way", and there few
documents and/or examples etc. at the time about how to do it right.
Someone else ended up re-engineering it to use pygraphviz which felt
pretty heavy weight to me. So the thing kind of stalled. But it's
probably worth another push.

The mails are still in my "look at it again" folder. I kind of agree
that using pygraphviz is counterproductive, since many systems have
graphviz but no so many, I suspect, pygraphviz.

Is the current version of the inheritance_diagram.py the one in matplotlib?

cheers,
Georg

In keeping with the spirit of trying to get all of these extension
changes upstream so that we can all eventually stop carrying our own
copies, below is a tiny change I just made to the inheritance diagram
one. This is needed to ensure that the figure is separated from any
surrounding text, since otherwise you get hideous off-screen diagrams
in the rendered PDF.

This has been committed to the nipy trunk already.

Similarly (for the pymvpa crowd), the api autogen code is now a
module, and it also contains a few small fixes, in particular
regarding chapter titles. Feel free to grab and update your copy:

http://bazaar.launchpad.net/~nipy-developers/nipy/trunk/annotate/head%3A/tools/apigen.py

I've been told the gods of numpy/sphinx don't like auto-generated
docs, but I think there's a valid use case for these tools, so
hopefully in the future it will be possible to include them upstream
for us lesser mortals to use. If not, I guess we'll just continue to
carry our copies around :slight_smile:

Cheers,

f

# diff, inline because it's so trivial:

=== modified file 'doc/sphinxext/inheritance_diagram.py'
--- doc/sphinxext/inheritance_diagram.py 2009-01-30 02:00:57 +0000
+++ doc/sphinxext/inheritance_diagram.py 2009-02-20 21:11:38 +0000
@@ -370,7 +370,7 @@

     graph.run_dot(['-Tpdf', '-o%s' % pdf_path],
                   name, parts, graph_options={'size': '"6.0,6.0"'})
- return '\\includegraphics{%s}' % pdf_path
+ return '\n\\includegraphics{%s}\n\n' % pdf_path

def visit_inheritance_diagram(inner_func):
     """

···

On Mon, Feb 16, 2009 at 3:21 PM, Gael Varoquaux <gael.varoquaux@...427...> wrote:

I am not blaming anyone, just pointing out a non ideal situation. It has
already improved a lot with the matplotlib guys and the scipy guys
merging some changes in extensions and publishing the extensions in an
importable part of their source tree.

Thanks, Fernando. I've applied your patch to matplotlib (branch and trunk).

Mike

Fernando Perez wrote:

···

On Mon, Feb 16, 2009 at 3:21 PM, Gael Varoquaux > <gael.varoquaux@...427...> wrote:

I am not blaming anyone, just pointing out a non ideal situation. It has
already improved a lot with the matplotlib guys and the scipy guys
merging some changes in extensions and publishing the extensions in an
importable part of their source tree.
    
In keeping with the spirit of trying to get all of these extension
changes upstream so that we can all eventually stop carrying our own
copies, below is a tiny change I just made to the inheritance diagram
one. This is needed to ensure that the figure is separated from any
surrounding text, since otherwise you get hideous off-screen diagrams
in the rendered PDF.

This has been committed to the nipy trunk already.

Similarly (for the pymvpa crowd), the api autogen code is now a
module, and it also contains a few small fixes, in particular
regarding chapter titles. Feel free to grab and update your copy:

http://bazaar.launchpad.net/~nipy-developers/nipy/trunk/annotate/head%3A/tools/apigen.py

I've been told the gods of numpy/sphinx don't like auto-generated
docs, but I think there's a valid use case for these tools, so
hopefully in the future it will be possible to include them upstream
for us lesser mortals to use. If not, I guess we'll just continue to
carry our copies around :slight_smile:

Cheers,

f

# diff, inline because it's so trivial:

=== modified file 'doc/sphinxext/inheritance_diagram.py'
--- doc/sphinxext/inheritance_diagram.py 2009-01-30 02:00:57 +0000
+++ doc/sphinxext/inheritance_diagram.py 2009-02-20 21:11:38 +0000
@@ -370,7 +370,7 @@

     graph.run_dot(['-Tpdf', '-o%s' % pdf_path],
                   name, parts, graph_options={'size': '"6.0,6.0"'})
- return '\\includegraphics{%s}' % pdf_path
+ return '\n\\includegraphics{%s}\n\n' % pdf_path

def visit_inheritance_diagram(inner_func):
     """

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
matplotlib-devel List Signup and Options
  
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA