We could store a copy of the png output somewhere in the svn tree. Then,
whenever we change something we can do a binary comparison on all the
plots. It would avoid issues such as breakage of polar plots where the
author of the changes didn't consider that it would affect polar plot
output. Similarly for ps/pdf. Differences in fonts between platforms
might be a problem in this scheme.
- Paul
···
On Thu, Sep 06, 2007 at 08:14:19AM -0400, Michael Droettboom wrote:
On a broader note, I've been using backend_driver.py as my ersatz
"acceptance test suite." Not all of these examples are included in it,
of course. Is there good reason for that, or should I go ahead and add
these to backend_driver? As we're all well aware, matplotlib has an
unusually large configuration space, so any bit of testing automation
really helps.
VTK does this for automated testing.
Ga�l
···
On Thu, Sep 06, 2007 at 08:46:24AM -0400, Paul Kienzle wrote:
We could store a copy of the png output somewhere in the svn tree.
Then,
whenever we change something we can do a binary comparison on all the
plots. It would avoid issues such as breakage of polar plots where the
author of the changes didn't consider that it would affect polar plot
output. Similarly for ps/pdf. Differences in fonts between platforms
might be a problem in this scheme.
Paul Kienzle wrote:
On a broader note, I've been using backend_driver.py as my ersatz "acceptance test suite." Not all of these examples are included in it, of course. Is there good reason for that, or should I go ahead and add these to backend_driver? As we're all well aware, matplotlib has an unusually large configuration space, so any bit of testing automation really helps.
We could store a copy of the png output somewhere in the svn tree. Then,
whenever we change something we can do a binary comparison on all the
plots. It would avoid issues such as breakage of polar plots where the author of the changes didn't consider that it would affect polar plot
output. Similarly for ps/pdf. Differences in fonts between platforms
might be a problem in this scheme.
We could have a special flag for font_manager.py that only loads fonts from mpl-data/fonts. Then the results for Agg should theoretically be the same across platforms. The Gdk, Cairo and wxRenderer would all have slightly different results across platforms, however, as the underlying font engines could change.
Alternatively, we could sidestep this problem by producing a set of PNG images for each platform which would have to be individually verified on a per-platform basis. It (at least) triples the verification work, but cuts down on work in other areas...
Cheers,
Mike
···
On Thu, Sep 06, 2007 at 08:14:19AM -0400, Michael Droettboom wrote:
I am interested in this problem, too. A binary comparison would
probably be too sensitive. How about comparing coefficients of some
transform? I.e. the residual on certain Fourier coefficients or parts
of a wavelet transform?
Cheers
Stéfan
···
On Thu, Sep 06, 2007 at 07:55:13PM -0700, Andrew Straw wrote:
Gael Varoquaux wrote:
> On Thu, Sep 06, 2007 at 08:46:24AM -0400, Paul Kienzle wrote:
>> We could store a copy of the png output somewhere in the svn tree.
>> Then,
>> whenever we change something we can do a binary comparison on all the
>> plots. It would avoid issues such as breakage of polar plots where the
>> author of the changes didn't consider that it would affect polar plot
>> output. Similarly for ps/pdf. Differences in fonts between platforms
>> might be a problem in this scheme.
>
> VTK does this for automated testing.
Is there a URL that describes this in much detail -- a little searching
turns up the odd tidbit, but nothing I can sink my teeth into. I'm
interested because my understanding is that different OpenGL
implementations draw things (slightly) differently, so I'd be curious
how they deal with that...
OK, so I'll reply to the list (I already replied to Andrew in private).
I don't know much about this. Prabhu Ramachandran might know more.
I can however point you to where this is used in Mayavi2:
The file where the tvtk code is, is:
https://svn.enthought.com/enthought/browser/branches/enthought.mayavi_2.0/tests/
common.py
the function you are interested in is "compare_image".
It is used in tests in the directory:
https://svn.enthought.com/enthought/browser/branches/enthought.mayavi_2.0/tests
I hope you'll find what you are interested in with this info and the tvtk
code related. From the tvtk code you can go up to the vtk code (the
functions have the same name).
If you don't find what you are interested in, you will have to ask Prabhu
or someone who knows vtk better than I do.
Cheers,
Ga�l
···
On Fri, Sep 07, 2007 at 01:24:37PM +0200, Stefan van der Walt wrote:
I am interested in this problem, too. A binary comparison would
probably be too sensitive. How about comparing coefficients of some
transform? I.e. the residual on certain Fourier coefficients or parts
of a wavelet transform?