filled contours, and missing data

As far as keyword args go, it seems to me that they would

    > be more convenient in many cases, but as Stephen mentions,
    > may be a fair amount of work (and in essence, they are an
    > attribute of the data, so that may be where they belong).

In the context of plotting, it isn't clear that NaNess is an attribute
of the data. If the data are y = sin(2*pi*t), then NaNess enters only
under certain transformations (eg log) of the data. From my
perspective, or the perspective of a matplotlib Line, the data are
intact, it is just that under certain transformations the data are
invalid. Thus the mask is only needed under certain views
(transformations) which the Line class is mostly unaware of. I think
there are two cases to be distinguished: the case Eric mentioned where
some data points are NaN or None because the measurements are missing
or invalid, and the case where the data are valid but are nan under
certain transformations.

For the latter, it would be maximally useful to be able to do

  #x,y,xt,yt are numerix arrays; transform puts in NaN on domain error
  xt, yt = transform(x, y)
  
and the drawing routine drops NaN points and handles the connecting
segments properly. That it only works for float arrays is not a
problem since that is what we are using, eg in the line class

        self._x = asarray(x, Float)
        self._y = asarray(y, Float)

but it is my current understanding that this ain't gonna happen in a
consistent way for Numeric, Numeric3, and numarray across platforms,
because my brief forays into researching this issue turned up posts by
Tim Peters saying that there was no standard way of dealing with IEEE
754 special values across compilers. If that's true, and we can't fix
it or work around it, then I think boolean masks passed as a kwarg may
be the easiest way to handle this across various numerix
implementations, assuming that Numeric3 comes to fruition and assuming
that there isn't a consistent MA approach that is accessible at the
API level, which I don't know to be true but I get the feeling that
this is a reasonable guess.

I'd be interested in getting feedback from those of you who have
informed opinions on these matters:

  - Is it possible to handle NaN in Numeric/numarray arrays across the
    big three platforms? I'm pretty sure the answer here is no.

  - Are Numeric and numarray MAs similar enough to be used at the
    C-API level, which is where agg would be checking the mask? Does
    anyone know whether MAs will be part of the Numeric3 core? I
    haven't seen any reference to them in the PEP.

JDH

    > As far as keyword args go, it seems to me that they would
    > be more convenient in many cases, but as Stephen mentions,
    > may be a fair amount of work (and in essence, they are an
    > attribute of the data, so that may be where they belong).

In the context of plotting, it isn't clear that NaNess is an attribute
of the data. If the data are y = sin(2*pi*t), then NaNess enters only
under certain transformations (eg log) of the data. From my
perspective, or the perspective of a matplotlib Line, the data are
intact, it is just that under certain transformations the data are
invalid. Thus the mask is only needed under certain views
(transformations) which the Line class is mostly unaware of. I think
there are two cases to be distinguished: the case Eric mentioned where
some data points are NaN or None because the measurements are missing
or invalid, and the case where the data are valid but are nan under
certain transformations.

Right.

For the latter, it would be maximally useful to be able to do

  #x,y,xt,yt are numerix arrays; transform puts in NaN on domain error
  xt, yt = transform(x, y)

and the drawing routine drops NaN points and handles the connecting
segments properly. That it only works for float arrays is not a
problem since that is what we are using, eg in the line class

        self._x = asarray(x, Float)
        self._y = asarray(y, Float)

but it is my current understanding that this ain't gonna happen in a
consistent way for Numeric, Numeric3, and numarray across platforms,
because my brief forays into researching this issue turned up posts by
Tim Peters saying that there was no standard way of dealing with IEEE
754 special values across compilers. If that's true, and we can't fix
it or work around it, then I think boolean masks passed as a kwarg may
be the easiest way to handle this across various numerix
implementations, assuming that Numeric3 comes to fruition and assuming
that there isn't a consistent MA approach that is accessible at the
API level, which I don't know to be true but I get the feeling that
this is a reasonable guess.

I think you may be over-extending what Tim was saying. I believe the issue is writing C code that handles things like tests against NaN values correctly. In that, Tim is right, there is a great deal of inconsistency in how C compilers handle ieee special values. But generally speaking, the computations *are* handled consistently by the floating point processors. numarray solved this problem by not relying on the C compiler to test or set NaN values (it tests raw bit patterns instead) and so should handle this issue properly. Numeric doesn't, but Numeric3 is planned to. So I think NaNs can't be handled right now because of Numeric, but I don't think the C compiler issue that Tim mentions is a roadblock (it is a nuisance for implementation though)..

I'd be interested in getting feedback from those of you who have
informed opinions on these matters:

  - Is it possible to handle NaN in Numeric/numarray arrays across the

  big three platforms? I'm pretty sure the answer here is no.

No for Numeric, yes for numarray/Numeric3

  - Are Numeric and numarray MAs similar enough to be used at the
    C-API level, which is where agg would be checking the mask? Does
    anyone know whether MAs will be part of the Numeric3 core? I
    haven't seen any reference to them in the PEP.

We'll have to raise it. I imagine that they should be. As I mentioned, NaNs don't solve all the problems that MA does. Since MA is layered on Numeric/numarray it shouldn't be hard to do so. As far as C-API, since it is a Python implementation, I presume it comes down to whether or not the needed attributes are the same for the numeric and Numarray variants. I'm not immediately familiar with that, but Todd should be able to give a fairly quick answer on that (he did the port to numarray)

Perry

···

On Feb 22, 2005, at 4:21 PM, John Hunter wrote:

I meant masks were an attribute of the data, but not NaNs (in other words, the way MA handles it may be the most appropriate)

Perry

···

On Feb 22, 2005, at 4:21 PM, John Hunter wrote:

    > As far as keyword args go, it seems to me that they would
    > be more convenient in many cases, but as Stephen mentions,
    > may be a fair amount of work (and in essence, they are an
    > attribute of the data, so that may be where they belong).

In the context of plotting, it isn't clear that NaNess is an attribute
of the data. If the data are y = sin(2*pi*t), then NaNess enters only

As Perry said, numarray.ma is a port of MA to numarray so they're
fairly close. Both packages are pure Python layered over ordinary
numarray or Numeric numerical arrays. Accessing from C, both packages
should yield data or mask information via a method callback. The
resulting arrays are PyArrayObjects which are source compatible only:
extensions using MA component arrays will need to be compiled for either
Numeric or numarray as we do now for _image, _transforms, and _contour.

Todd

···

On Tue, 2005-02-22 at 16:46, Perry Greenfield wrote:

> - Are Numeric and numarray MAs similar enough to be used at the
> C-API level, which is where agg would be checking the mask? Does
> anyone know whether MAs will be part of the Numeric3 core? I
> haven't seen any reference to them in the PEP.
>
We'll have to raise it. I imagine that they should be. As I mentioned,
NaNs don't solve all the problems that MA does. Since MA is layered on
Numeric/numarray it shouldn't be hard to do so. As far as C-API, since
it is a Python implementation, I presume it comes down to whether or
not the needed attributes are the same for the numeric and Numarray
variants. I'm not immediately familiar with that, but Todd should be
able to give a fairly quick answer on that (he did the port to
numarray)