bugs found

Development matplotlib-devel
1

Here is a simple illustration of the problem and

    > inconsistency, using just the first few points in
    > Tennessee's plot and a few lines from his script. With a
    > log scale for y, the Agg backends are leaving gaps (breaking
    > the line) at the points where y==0; the PS and SVG backends
    > are removing those points, but *not* breaking the line. I
    > think the Agg behavior is much better. But maybe it would
    > be better yet to raise an exception, forcing the user to
    > deal with the error explicitly. For example, the user might
    > want to change the zero values to a very small number, and
    > then explicitly set the y range to exclude the small
    > numbers. This is illustrated in the second subplot.

This issue has a long history, which I'll summarize here. In the old
days, we raised an exception when non-positive numbers were passed to
log plotting. Most people found this objectionable, and prefer to see
the valid points plotted rather than nothing (though a warning would
be nice). This is what matlab does. I needed to make some
architectural changes to support this, because the transformations
happened in the lines class which passed the transformed data to the
backend.

I looked into filtering the data in the lines class (basically what
you did for ma data) but did not think it would be fast enough,
because we potentially would have had to create a lot of line
segments.

So I changed the backend API and modified draw_lines to take the
transformation as an argument. Then the backend can do the following
(eg _backend_agg)

    if (needNonlinear)
      try {
  mpltransform->nonlinear_only_api(&thisx, &thisy);
      }
      catch (...) {
  moveto = true;
  continue;
      }

Ie if there is a nonlinear transformation that raises an exception,
the path state is changed from lineto to moveto. This is very fast,
at least an order of magnitude faster than trying to compute the
segments at the python level I suspect.

At the same time I introduced some optimizations for marker drawing
with a new backend method called draw_markers that also does the
transformations in the backend. This made marker drawing up to 25
times faster (we used to draw every marker as a separate function call
in lines.py)

We decided to support both old and new style APIs in the "transition
period" which made it easy for other backends to do ..... nothing. So
that is why the other backends haven't implemented this feature yet.

Steve worked on it a bit for Cairo and Darren and I both did for PS
but never got a working product. You can see the attempt in
backend_ps _draw_markers method (underscore hidden) and in
backend_cairo's draw_markers_OLD method. There was extensive
discussion on this list with the API and implementation details so
just search for draw_markers if you are interested.

So the backend is now in what I call a schizophrenic state, in two
ways.

  1) some backends use the old and some use the "newstyle" API

  2) Even in the newstyle API, some methods do the transformation in
     the front end (draw_rectangle) and some in the backend
     (draw_markers).

There is a need to simplify, unify and rationalize this API, but since
it mostly works, there has not been a lot of pressure to do so. And
it would break a lot of backends that may not be actively maintained.

A modest short term goal should be to get the newstyle draw_lines and
draw_markers working for the PS backend, both to fix this log problem
and because it is faster and potentially much smaller in terms of PS
file sizes. Something for a rainy day.

JDH

2

I'm playing with the new API in backend_ps today. I'm trying to make
draw_markers work:

    > Here is a simple illustration of the problem and
    > inconsistency, using just the first few points in
    > Tennessee's plot and a few lines from his script. With a
    > log scale for y, the Agg backends are leaving gaps (breaking
    > the line) at the points where y==0; the PS and SVG backends
    > are removing those points, but *not* breaking the line. I
    > think the Agg behavior is much better. But maybe it would
    > be better yet to raise an exception, forcing the user to
    > deal with the error explicitly. For example, the user might
    > want to change the zero values to a very small number, and
    > then explicitly set the y range to exclude the small
    > numbers. This is illustrated in the second subplot.

This issue has a long history, which I'll summarize here. In the old
days, we raised an exception when non-positive numbers were passed to
log plotting. Most people found this objectionable, and prefer to see
the valid points plotted rather than nothing (though a warning would
be nice). This is what matlab does. I needed to make some
architectural changes to support this, because the transformations
happened in the lines class which passed the transformed data to the
backend.

I looked into filtering the data in the lines class (basically what
you did for ma data) but did not think it would be fast enough,
because we potentially would have had to create a lot of line
segments.

So I changed the backend API and modified draw_lines to take the
transformation as an argument. Then the backend can do the following
(eg _backend_agg)

    if (needNonlinear)
      try {
  mpltransform->nonlinear_only_api(&thisx, &thisy);
      }
      catch (...) {
  moveto = true;
  continue;
      }

Ie if there is a nonlinear transformation that raises an exception,
the path state is changed from lineto to moveto. This is very fast,
at least an order of magnitude faster than trying to compute the
segments at the python level I suspect.

For backend_ps, we want to make use of the transform's numerix methods.

        if transform.need_nonlinear():
            x,y,mask = transform.nonlinear_only_numerix(x, y, returnMask=1)
        else:
            mask = ones(x.shape)
             x,y = transform.numerix_x_y(x, y)

If I do semilogy([0,1,2,3]), the mask will be [0,1,1,1]. If I do
semilogy([1,2,nan,4]), I would have expected a mask that looked like
[1,1,0,1], but it returns [1,1,1,1]. Wouldnt it make life easy if
transform.nonlinear_only_numerix and transform.numerix_x_y both returned a
mask with zeros where x or y is a nan or inf? Then I can filter out the
un-drawable markers with something like

        to_draw = izip(x[start:end],y[start:end],mask[start:end])
        ps = ['%1.3f %1.3f marker' % (xp, yp) for xp, yp, m in to_draw i m]

I could do something similar for draw_lines, using Johns suggestion above.
This would mask nans, infs and log(x<=0) without the need for masked arrays,
and would provide an opportunity to issue a warning about the bad values (if
that is actually desireable).

Darren

(There is a lot going on in the transforms that I'm still trying to wrap my
head around, so please feel free to correct me where my assumptions are
incorrect or naive.)

···

On Tuesday 14 March 2006 1:38 pm, John Hunter wrote:

3

Darren Dale wrote:

For backend_ps, we want to make use of the transform's numerix methods.

       if transform.need_nonlinear():
           x,y,mask = transform.nonlinear_only_numerix(x, y, returnMask=1)
       else:
           mask = ones(x.shape)
            x,y = transform.numerix_x_y(x, y)

If I do semilogy([0,1,2,3]), the mask will be [0,1,1,1]. If I do
semilogy([1,2,nan,4]), I would have expected a mask that looked like
[1,1,0,1], but it returns [1,1,1,1].

I just committed a patch that should make this work. nan support is done
in a case-by-case basis at the C level. So this is another case.

Wouldnt it make life easy if
transform.nonlinear_only_numerix and transform.numerix_x_y both returned a
mask with zeros where x or y is a nan or inf?

I'm not sure this is the right thing to do for inf (or -inf) -- I can
imagine transforms that accept inf and return something useful (such as
a reciprocal transform). Maybe we don't use any such transforms
currently, but that doesn't mean we should prevent that case in the
future. Is the inf case important for you?

Cheers!
Andrew

4

I don't know. For a reciprocal transform, I suppose inf shouldn't effect the
mask, but then we can plot the results of the transform. If we can't plot the
transformed value, I think the mask should reflect it.

Moving on, I've been trying to follow John's work in __draw_lines_hide (you
masked that sucker twice, is it that bad?). Help me understand this, are we
allowed to use transform.numerix_x_y in the new API, or do we have to use
transform.as_vec6_val? If I can use the former:

    print x,y
    if transform\.need\_nonlinear\(\):
        x,y,mask = transform\.nonlinear\_only\_numerix\(x, y, returnMask=1\)
        print x, y
    else:
        mask = ones\(x\.shape\)
    x, y = transform\.numerix\_x\_y\(x, y\)
    print x, y

that returns a "ValueError: Domain error on Transformation::numerix_x_y"
message when I try to do semilog plots. Linear plots look fine.

If I should be using the latter, well, I'm playing with it, but I'm spilling
blue ink all over the place.

···

On Sunday 19 March 2006 3:14 pm, Andrew Straw wrote:

Darren Dale wrote:
>For backend_ps, we want to make use of the transform's numerix methods.
>
> if transform.need_nonlinear():
> x,y,mask = transform.nonlinear_only_numerix(x, y,
> returnMask=1) else:
> mask = ones(x.shape)
> x,y = transform.numerix_x_y(x, y)
>
>If I do semilogy([0,1,2,3]), the mask will be [0,1,1,1]. If I do
>semilogy([1,2,nan,4]), I would have expected a mask that looked like
>[1,1,0,1], but it returns [1,1,1,1].

I just committed a patch that should make this work. nan support is done
in a case-by-case basis at the C level. So this is another case.

>Wouldnt it make life easy if
>transform.nonlinear_only_numerix and transform.numerix_x_y both returned a
>mask with zeros where x or y is a nan or inf?

I'm not sure this is the right thing to do for inf (or -inf) -- I can
imagine transforms that accept inf and return something useful (such as
a reciprocal transform). Maybe we don't use any such transforms
currently, but that doesn't mean we should prevent that case in the
future. Is the inf case important for you?

5

I think I actually got the new api working with draw_markers and draw_lines in
backend_ps. The transforms are done by passing the results of
transform.as_vec6_val to postscript. I would appreciate it if someone would
test it out and give me feedback. In order to do so, update from svn and
unmask backend_ps.RendererPS._draw_markers (not _draw_markers_old) by
removing the leading underscore. Try plotting things like

plot([0,1,2,3,4],'-o'); savefig('linear.eps')
plot([0,1,nan,3,4],'-o'); savefig('linear_nan.eps') # Not possible with the
old API
semilogy([0,1,2,3,4],'-o'); savefig('nonlinear.eps')
semilogy([0,1,nan,3,4],'-o'); savefig('nonlinear_nan.eps') # Not possible with
the old API

Are there other methods that need to be updated to use the new API? There are
a bunch of unused methods in RendererPS that I will clean up once things have
settled, please ignore them for now.

Darren