Problem adding a new test

Jouni K. Seppänen <jks@...278...> writes:

Oh, right. My fault: when I implemented the pdf comparison, I made it
run the test for only those formats for which a baseline image exists,

I committed a change to make it run both png and pdf tests all the time.

When we add new formats (comparing postscript files could easily be done
using the same ghostscript command as used for pdf files, and some svg
renderer could also be added) and new tests, we'll have to think about
if we want to run all tests on all backends, since the amount of data in
the repository will start growing pretty fast.

···

--
Jouni K. Seppänen
http://www.iki.fi/jks

Jouni K. Seppänen wrote:

Jouni K. Seppänen <jks@...278...> writes:

Oh, right. My fault: when I implemented the pdf comparison, I made it
run the test for only those formats for which a baseline image exists,
    
I committed a change to make it run both png and pdf tests all the time.
  
Thanks for the fix, Jouni. (My svn commit was rejected because you did
exactly the same thing as me.)

When we add new formats (comparing postscript files could easily be done
using the same ghostscript command as used for pdf files, and some svg
renderer could also be added)

"inkscape input.svg --export-png=output.png" works very well as an svg
renderer.

and new tests, we'll have to think about
if we want to run all tests on all backends, since the amount of data in
the repository will start growing pretty fast.
  

As far as the test data -- I agree this is an issue. One point in favor
of the status quo is that it's really nice to have the test data
included with the source code so there are no configuration hassles. I'm
not sure how well the buildbot infrastructure would cope with anything
else. For example, to my knowledge, there is no Buildbot precedent to
automatically pull from two branches to execute a single test run. But
in general I think this does bear thinking about.

-Andrew

Andrew Straw wrote:

Jouni K. Seppänen wrote:
  

Jouni K. Seppänen <jks@...278...> writes:
    When we add new formats (comparing postscript files could easily be done
using the same ghostscript command as used for pdf files, and some svg
renderer could also be added)
    
"inkscape input.svg --export-png=output.png" works very well as an svg
renderer.
  

I'd also like to run SVG through xmllint against the SVG schema as another sanity check. I may get to this if I can find the time.

  

and new tests, we'll have to think about
if we want to run all tests on all backends, since the amount of data in
the repository will start growing pretty fast.
  

As far as the test data -- I agree this is an issue. One point in favor
of the status quo is that it's really nice to have the test data
included with the source code so there are no configuration hassles. I'm
not sure how well the buildbot infrastructure would cope with anything
else. For example, to my knowledge, there is no Buildbot precedent to
automatically pull from two branches to execute a single test run. But
in general I think this does bear thinking about.
  

An easy improvement may be having an extra kwarg on the image_comparison decorator to select a subset of backends. For example, many of the ones in test_simplification.py only apply to the Agg backend.

While I'm sharing my wish list out loud, I think it would also be highly cool to get the native Mac OS backend in the buildbot tests, as that's one I can't test easily myself.

Cheers,
Mike

···

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Michael Droettboom wrote:

"inkscape input.svg --export-png=output.png" works very well as an svg
renderer.
  

I'd also like to run SVG through xmllint against the SVG schema as
another sanity check. I may get to this if I can find the time.

That'd be great. I just installed inkscape and xmllint on the non-bare
buildslave machine.

As far as the test data -- I agree this is an issue. One point in favor
of the status quo is that it's really nice to have the test data
included with the source code so there are no configuration hassles. I'm
not sure how well the buildbot infrastructure would cope with anything
else. For example, to my knowledge, there is no Buildbot precedent to
automatically pull from two branches to execute a single test run. But
in general I think this does bear thinking about.
  

An easy improvement may be having an extra kwarg on the image_comparison
decorator to select a subset of backends. For example, many of the ones
in test_simplification.py only apply to the Agg backend.

Done in r7863. To make use of it, do something like the following patch
(and don't forget to delete the baseline .pdf files from the repository):

-@...782...(baseline_images=['simplify_curve'])
+@image_comparison(baseline_images=['simplify_curve'],extensions=['png'])

While I'm sharing my wish list out loud, I think it would also be highly
cool to get the native Mac OS backend in the buildbot tests, as that's
one I can't test easily myself.

That would require the Mac OS X buildslave to start working again too,
as I assume the backend actually requires the OS. And that would require
building on Snow Leopard to work, as I understand it.

-Andrew

0001-don-t-test-simplification.patch (2.37 KB)

Andrew Straw wrote:

As far as the test data -- I agree this is an issue. One point in favor
of the status quo is that it's really nice to have the test data
included with the source code so there are no configuration hassles. I'm
not sure how well the buildbot infrastructure would cope with anything
else. For example, to my knowledge, there is no Buildbot precedent to
automatically pull from two branches to execute a single test run. But
in general I think this does bear thinking about.
  

An easy improvement may be having an extra kwarg on the image_comparison decorator to select a subset of backends. For example, many of the ones in test_simplification.py only apply to the Agg backend.
    
Done in r7863. To make use of it, do something like the following patch
(and don't forget to delete the baseline .pdf files from the repository):

-@...782...(baseline_images=['simplify_curve'])
+@image_comparison(baseline_images=['simplify_curve'],extensions=['png'])
  

Great!

  

While I'm sharing my wish list out loud, I think it would also be highly cool to get the native Mac OS backend in the buildbot tests, as that's one I can't test easily myself.
    
That would require the Mac OS X buildslave to start working again too,
as I assume the backend actually requires the OS. And that would require
building on Snow Leopard to work, as I understand it.
  

Oh yeah. Forgot that detail. Well -- something to think about when the other pieces fall into place.

Mike

I've committed support for comparing SVG files using Inkscape and verifying them against the official SVG DTD using xmllint.

Michael Droettboom wrote:

Andrew Straw wrote:
  

Done in r7863. To make use of it, do something like the following patch
(and don't forget to delete the baseline .pdf files from the repository):

-@...782...(baseline_images=['simplify_curve'])
+@image_comparison(baseline_images=['simplify_curve'],extensions=['png'])
  

Great!
  

This is a nice feature. However, in hindsight, I may not use it right away -- I actually found a bug in the SVG backend using one of the tests I assumed would only affect the Agg backend. :slight_smile:

A couple more comments about the test framework -- which has already paid for itself ten times over. In Numpy (and a number of local Python projects), I can 'cd' to the tests directory and do something like:

   nosetests test_simplification.py:test_hatch_simplify

and run on particular test, or a single file of tests. It's a huge time saver when trying to fix a bug. However, with matplotlib I get:

> nosetests test_simplification.py
E

···

======================================================================
ERROR: Failure: ImportError (cannot import name cbook)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/wonkabar/data1/usr/lib/python2.5/site-packages/nose-0.11.1-py2.5.egg/nose/loader.py", line 379, in loadTestsFromName
    addr.filename, addr.module)
  File "/wonkabar/data1/usr/lib/python2.5/site-packages/nose-0.11.1-py2.5.egg/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/wonkabar/data1/usr/lib/python2.5/site-packages/nose-0.11.1-py2.5.egg/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/wonkabar/data1/builds/matplotlib/build/lib.linux-i686-2.5/matplotlib/tests/test_simplification.py", line 4, in <module>
    import matplotlib.pyplot as plt
  File "/wonkabar/data1/builds/matplotlib/build/lib.linux-i686-2.5/matplotlib/pyplot.py", line 6, in <module>
    from matplotlib import docstring
  File "/wonkabar/data1/builds/matplotlib/build/lib.linux-i686-2.5/matplotlib/docstring.py", line 1, in <module>
    from matplotlib import cbook
ImportError: cannot import name cbook

----------------------------------------------------------------------
Ran 1 test in 0.009s

FAILED (errors=1)

I suspect this is something peculiar to how matplotlib gets imported.

Also, I have a quad-core machine, so I put this in my .noserc, which will run tests in parallel:

   [nosetests]
   processes=4

Unfortunately, due to however matplotlib is delegating to nose, this doesn't seem to get picked up.

I don't know if I'll get a chance to look at these things right away, but thought I'd share in case the solutions are obvious to anyone else (which I know isn't good form, but hey... :wink:

Cheers,
Mike

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Michael Droettboom wrote:

I've committed support for comparing SVG files using Inkscape and verifying them against the official SVG DTD using xmllint.
  

Man, are we standards compliant around here or what? :slight_smile: Cool.

Michael Droettboom wrote:
  

Andrew Straw wrote:
  

Done in r7863. To make use of it, do something like the following patch
(and don't forget to delete the baseline .pdf files from the repository):

-@...782...(baseline_images=['simplify_curve'])
+@image_comparison(baseline_images=['simplify_curve'],extensions=['png'])
  

Great!
  

This is a nice feature. However, in hindsight, I may not use it right away -- I actually found a bug in the SVG backend using one of the tests I assumed would only affect the Agg backend. :slight_smile:
  

I think it's good not to use the feature very much. I've already found it handy when developing against a test -- you only need to generate that test's image once.

A couple more comments about the test framework -- which has already paid for itself ten times over. In Numpy (and a number of local Python projects), I can 'cd' to the tests directory and do something like:

   nosetests test_simplification.py:test_hatch_simplify

and run on particular test, or a single file of tests. It's a huge time saver when trying to fix a bug. However, with matplotlib I get:

> nosetests test_simplification.py
E

ERROR: Failure: ImportError (cannot import name cbook)
<snip>
I suspect this is something peculiar to how matplotlib gets imported.
  
Yes, it would be very nice, I absolutely agree. I'm not sure what's going on, either, but I agree that it would be nice to fix. See below for an idea.

Also, I have a quad-core machine, so I put this in my .noserc, which will run tests in parallel:

   [nosetests]
   processes=4

Unfortunately, due to however matplotlib is delegating to nose, this doesn't seem to get picked up.

I don't know if I'll get a chance to look at these things right away, but thought I'd share in case the solutions are obvious to anyone else (which I know isn't good form, but hey... :wink:
  

My guess is that this may actually be related to the first issue. On this second issue, though, I have a specific idea -- in order for MPL to pickup the nose plugin, I had to do the song and dance in test() of matplotlib/__init__.py in which I create a nose.config.Config instance. I suspect this is why your processes argument isn't getting through -- we're completely bypassing any local nose config and creating ours programattically. I'm definitely not in favor the big song and dance, so if you can rip it out and still get the plugin to load, that would be super.

Once that is figured out, presumably the direct call to "nosetests test_simplification.py:test_hatch_simplify" will also load the nose plugins and thus not exhibit weird behavior when a known failure is encountered.

I almost certainly won't get a chance to look at these right away, so if anyone wants to go spelunking in the nose/mpl interaction, feel free.

-Andrew

Michael Droettboom wrote:

I've committed support for comparing SVG files using Inkscape and verifying them against the official SVG DTD using xmllint.

I just noticed the test_axes/hexbin_extent.svg baseline image is more than 5 MB. I wonder if we can somehow simplify the svg generated to reduce the file size or if we should perhaps just not do the svg extension on this test?

-Andrew

I suspect for that one we can just do without it. It isn't really testing anything SVG-specific.

Mike

Andrew Straw wrote:

···

Michael Droettboom wrote:

I've committed support for comparing SVG files using Inkscape and verifying them against the official SVG DTD using xmllint.

I just noticed the test_axes/hexbin_extent.svg baseline image is more than 5 MB. I wonder if we can somehow simplify the svg generated to reduce the file size or if we should perhaps just not do the svg extension on this test?

-Andrew

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Andrew Straw wrote:

Michael Droettboom wrote:

A couple more comments about the test framework -- which has already paid for itself ten times over. In Numpy (and a number of local Python projects), I can 'cd' to the tests directory and do something like:

   nosetests test_simplification.py:test_hatch_simplify

and run on particular test, or a single file of tests. It's a huge time saver when trying to fix a bug. However, with matplotlib I get:

> nosetests test_simplification.py
E

ERROR: Failure: ImportError (cannot import name cbook)
<snip>
I suspect this is something peculiar to how matplotlib gets imported.
  
Yes, it would be very nice, I absolutely agree. I'm not sure what's going on, either, but I agree that it would be nice to fix. See below for an idea.

Also, I have a quad-core machine, so I put this in my .noserc, which will run tests in parallel:

   [nosetests]
   processes=4

Unfortunately, due to however matplotlib is delegating to nose, this doesn't seem to get picked up.

I don't know if I'll get a chance to look at these things right away, but thought I'd share in case the solutions are obvious to anyone else (which I know isn't good form, but hey... :wink:
  

My guess is that this may actually be related to the first issue. On this second issue, though, I have a specific idea -- in order for MPL to pickup the nose plugin, I had to do the song and dance in test() of matplotlib/__init__.py in which I create a nose.config.Config instance. I suspect this is why your processes argument isn't getting through -- we're completely bypassing any local nose config and creating ours programattically. I'm definitely not in favor the big song and dance, so if you can rip it out and still get the plugin to load, that would be super.

Once that is figured out, presumably the direct call to "nosetests test_simplification.py:test_hatch_simplify" will also load the nose plugins and thus not exhibit weird behavior when a known failure is encountered.

I almost certainly won't get a chance to look at these right away, so if anyone wants to go spelunking in the nose/mpl interaction, feel free.

I have a partial solution to these problems.

You can now do (from any directory)

   nosetests matplotlib.tests

and this automatically picks up nose parameters, so you can do multiprocessing, pdb, coverage and other nose niceties.

Strangely, I still can't make running "nosetests" from the lib/matplotlib/tests directory work. The imports seem to get all screwed up in that case, presumably because nose is messing with sys.path. Unfortunately, nosetests -P (which is supposed to not touch sys.path) doesn't seem to help.

I made sure that "import matplotlib; matplotlib.test()" still works, so the buildbots should be unaffected.

As a side note, I disabled the xmllint testing since curl (which xmllint uses to fetch the SVG DTD) has some problems with caching in a multiprocess situation. It gives random "Operation in progress" errors.

Mike

···

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA