New backend API

Development matplotlib-devel
1

I think I actually got the new api working with

    > draw_markers and draw_lines in backend_ps. The transforms
    > are done by passing the results of transform.as_vec6_val
    > to postscript. I would appreciate it if someone would test
    > it out and give me feedback. In order to do so, update
    > from svn and unmask backend_ps.RendererPS._draw_markers
    > (not _draw_markers_old) by removing the leading
    > underscore. Try plotting things like

I think the best way to get people to test this is to make it the
default in backend_ps -- then all svn users will be testing it
automatically and you will hear about it when there is a problem. They
don't call it "the bleeding edge" for nothing. I made this change in
svn, revision 2173.

    > plot([0,1,2,3,4],'-o'); savefig('linear.eps')
    > plot([0,1,nan,3,4],'-o'); savefig('linear_nan.eps') # Not
    > possible with the old API semilogy([0,1,2,3,4],'-o');
    > savefig('nonlinear.eps') semilogy([0,1,nan,3,4],'-o');
    > savefig('nonlinear_nan.eps') # Not possible with the old
    > API

    > Are there other methods that need to be updated to use the
    > new API? There are a bunch of unused methods in RendererPS
    > that I will clean up once things have settled, please
    > ignore them for now.

Just those two methods for now. I'd also like to add draw_paths and
remove many of the redundant ones.

Regarding a question from your previous post:

    > Moving on, I've been trying to follow John's work in
    > __draw_lines_hide (you masked that sucker twice, is it
    > that bad?). Help me understand this, are we allowed to use
    > transform.numerix_x_y in the new API, or do we have to use
    > transform.as_vec6_val? If I can use the former:

You are certainly allowed to use it, and any other provided method,
but for nonlinear transformations it defeats the purpose of doing the
transformation in the backend. numerix_x_y transforms the entire
array with the affine plus nonlinear transform. If it fails on any
element the entire transformation fails. To prevent this problem you
would want to do an element wise transformation and on failures set
the path state for the next element to moveto, ie skip the bad point.
We have talked in the past about writing a helper routine to generate
the postscript path in extension code if performance becomes an issue.

I ran a test script to profile this before and after, making marker
plots of various sizes. I think you'll be happy with the fruits of
your labors. Because we save the effort of setting a gc for each
marker, and thus avoid of lot of redundant function calling and state
setting in the output file, the results are much better in time and space

    import time, os
    from pylab import figure, close, nx

    for i in 10,100,1000,10000,100000:
        fig = figure(1)
        ax = fig.add_subplot(111)
        x,y = nx.mlab.rand(2,i)
        tnow = time.time()
        ax.plot(x,y, 'o')
        fname = 'file%d.ps'
        fig.savefig(fname)
        elapsed = time.time() - tnow
        fsize = os.stat(fname)[6]
        print 'i=%06d time= %1.2fs size=%1.1fkb'%(i, elapsed, fsize/1e3)
        close(1)

Old API:
    > python test.py
    i=000010 time= 0.33s size=142.4kb
    i=000100 time= 0.30s size=154.9kb
    i=001000 time= 0.43s size=284.1kb
    i=010000 time= 1.79s size=1575.9kb
    i=100000 time= 15.37s size=14495.6kb
    
New API:
    > python test.py
    i=000010 time= 0.53s size=148.1kb
    i=000100 time= 0.29s size=146.5kb
    i=001000 time= 0.30s size=163.6kb
    i=010000 time= 0.44s size=334.5kb
    i=100000 time= 2.17s size=2044.5kb
    
So for largish marker plots, you have about a 7x improvement in speed
and filesize. You may want to use this script to profile and see if
you can't eke out some extra gains. Maybe another 2x <wink>. This is
using the default rc file.

Now that we have a good implementation in a pure python backend that
others can follow as an example, I will shortly deprecate the old API
entirely and begin pruning the redundant methods. Considering that
both your changes and Eric's are significant enhancements, we may wait
to do this until one release cycle down the road, so that we don't
delay getting these out to the wider community.

Thanks,
JDH

2

I just committed a fix for a minor bug, it didnt effect the performance. My
results are comparable to yours. How's this for a performance boost: set
ps.useafm : true in rc:

Old API:
i=000010 time= 0.09s size=6.2kb
i=000100 time= 0.07s size=18.0kb
i=001000 time= 0.19s size=147.2kb
i=010000 time= 1.43s size=1439.2kb
i=100000 time= 13.99s size=14358.6kb

New API:
i=000010 time= 0.16s size=11.7kb
i=000100 time= 0.07s size=9.6kb
i=001000 time= 0.06s size=26.7kb
i=010000 time= 0.13s size=197.7kb
i=100000 time= 0.75s size=1907.7kb

I dont see where I can make things any faster in draw_markers, we trimmed all
the fat when we started on this a year ago.

Here's a bug: I need isnan to create my mask. It is provided by numerix with
numpy and numarray, but not Numeric. Can this be rectified?

Darren

···

On Monday 20 March 2006 11:05, John Hunter wrote:

I ran a test script to profile this before and after, making marker
plots of various sizes. I think you'll be happy with the fruits of
your labors. Because we save the effort of setting a gc for each
marker, and thus avoid of lot of redundant function calling and state
setting in the output file, the results are much better in time and space

    import time, os
    from pylab import figure, close, nx

    for i in 10,100,1000,10000,100000:
        fig = figure(1)
        ax = fig.add_subplot(111)
        x,y = nx.mlab.rand(2,i)
        tnow = time.time()
        ax.plot(x,y, 'o')
        fname = 'file%d.ps'
        fig.savefig(fname)
        elapsed = time.time() - tnow
        fsize = os.stat(fname)[6]
        print 'i=%06d time= %1.2fs size=%1.1fkb'%(i, elapsed, fsize/1e3)
        close(1)

Old API:
    > python test.py

    i=000010 time= 0.33s size=142.4kb
    i=000100 time= 0.30s size=154.9kb
    i=001000 time= 0.43s size=284.1kb
    i=010000 time= 1.79s size=1575.9kb
    i=100000 time= 15.37s size=14495.6kb

New API:
    > python test.py

    i=000010 time= 0.53s size=148.1kb
    i=000100 time= 0.29s size=146.5kb
    i=001000 time= 0.30s size=163.6kb
    i=010000 time= 0.44s size=334.5kb
    i=100000 time= 2.17s size=2044.5kb

So for largish marker plots, you have about a 7x improvement in speed
and filesize. You may want to use this script to profile and see if
you can't eke out some extra gains. Maybe another 2x <wink>. This is
using the default rc file.

Now that we have a good implementation in a pure python backend that
others can follow as an example, I will shortly deprecate the old API
entirely and begin pruning the redundant methods. Considering that
both your changes and Eric's are significant enhancements, we may wait
to do this until one release cycle down the road, so that we don't
delay getting these out to the wider community.

3

Darren Dale wrote:

Here's a bug: I need isnan to create my mask. It is provided by numerix with
numpy and numarray, but not Numeric. Can this be rectified?

I just added the matplotlib._isnan extension module which is independent
of the numerix choice (although I think it'll be better to stick with a
numerix-given function, if available). Below is an example of its use.
Perhaps you can modify the Numeric-flavor numerix so that isnan is
exposed the same way as numarray and numpy -- I didn't do this because
you'll be more familiar with the details than I am.

#Example:
import matplotlib._isnan as n
import numpy

for val in [3.2,3,numpy.nan,'adsf']:
    print 'val',val
    print n.isnan64(val)
    print

Running displays the following:

val 3.2
False

val 3
False

val nan
True

val adsf
Traceback (most recent call last):
  File "testnan.py", line 6, in ?
    print n.isnan64(val)
TypeError: a float is required

4

Thank you, Andrew (Baker's Dozen) Straw.

Does this look about right?

def isnan(a):
    return reshape(array([isnan64(i) for i in ravel(a)],'b'), shape(a))

In [1]: a=ones((3,3,3),'d')

In [2]: a[0,0,0]=array(0.0)/0

In [3]: isnan(a)
Out[3]:
[[[1,0,0,]
  [0,0,0,]
  [0,0,0,]]
[[0,0,0,]
  [0,0,0,]
  [0,0,0,]]
[[0,0,0,]
  [0,0,0,]
  [0,0,0,]]]

···

On Monday 20 March 2006 13:40, Andrew Straw wrote:

Darren Dale wrote:
>Here's a bug: I need isnan to create my mask. It is provided by numerix
> with numpy and numarray, but not Numeric. Can this be rectified?

I just added the matplotlib._isnan extension module which is independent
of the numerix choice (although I think it'll be better to stick with a
numerix-given function, if available). Below is an example of its use.
Perhaps you can modify the Numeric-flavor numerix so that isnan is
exposed the same way as numarray and numpy -- I didn't do this because
you'll be more familiar with the details than I am.

#Example:
import matplotlib._isnan as n
import numpy

for val in [3.2,3,numpy.nan,'adsf']:
    print 'val',val
    print n.isnan64(val)
    print

5

Darren Dale wrote:

Thank you, Andrew (Baker's Dozen) Straw.

Hey, stop thanking me! :wink: Seriously, it's me who should be thanking you
for all the work you're doing on the PS and latex fronts. I'm just glad
I can do a couple of minor things to help you along the way.

Does this look about right?

def isnan(a):
   return reshape(array([isnan64(i) for i in ravel(a)],'b'), shape(a))

It looks fine to me -- it matches the behavior of numpy's isnan.

6

I modified _nc_imports.py and numerix/__init__.py. As of svn 2179, we have an
isnan for every numerix flavor (numpy and numarray use their own). That means
that the postscript backend can use nan's to mask points, if you use the new
API.

I also found a way to patch _backend_agg.cpp to make it break lines around
nan's. I havent modified extension code before, and this change would affect
all the *agg backends, so I dont want to commit before checking. Here's the
diff:

Index: src/_backend_agg.cpp

···

On Monday 20 March 2006 16:45, you wrote:

Darren Dale wrote:
>Thank you, Andrew (Baker's Dozen) Straw.

Hey, stop thanking me! :wink: Seriously, it's me who should be thanking you
for all the work you're doing on the PS and latex fronts. I'm just glad
I can do a couple of minor things to help you along the way.

>Does this look about right?
>
>
>def isnan(a):
> return reshape(array([isnan64(i) for i in ravel(a)],'b'), shape(a))

It looks fine to me -- it matches the behavior of numpy's isnan.

===================================================================
--- src/_backend_agg.cpp (revision 2178)
+++ src/_backend_agg.cpp (working copy)
@@ -23,6 +23,7 @@
#include "_backend_agg.h"
#include "_transforms.h"
#include "mplutils.h"
+#include "MPL_isnan.h"

#include "swig_runtime.h"

@@ -1324,6 +1325,7 @@

   double thisx, thisy;
   bool moveto = true;
+ bool skippoint = false;
   double heightd = height;

   double lastx(-2.0), lasty(-2.0);
@@ -1339,9 +1341,15 @@
       }
       catch (...) {
        moveto = true;
+ skippoint = true;
        continue;
       }
-
+ else
+ if (MPL_isnan64(thisx) || MPL_isnan64(thisy)) {
+ moveto = true;
+ skippoint = true;
+ }
+
     //use agg's transformer?
     xytrans.transform(&thisx, &thisy);
     thisy = heightd - thisy; //flipy
@@ -1367,8 +1375,9 @@
       path.move_to(thisx, thisy);
     else
       path.line_to(thisx, thisy);
-
- moveto = false;
+ if (!skippoint)
+ moveto = false;
+ skippoint = false;
     //std::cout << "draw lines " << thisx << " " << thisy << std::endl;
   }