numerix on win32

The new transformation module I've been working on is implemented in
extension code. It offers a couple of important benefits, namely
speed and generality (affines, nonseparable xy transformations). It's
not in CVS yet because I'm still ironing out the details.

I have two methods for operating over sequences of x and y, seq_x_y
and numerix_x_y. The first uses the python sequence API and the
second uses the numerix api. I expect the latter will be a good bit
faster for long lines, since it doesn't have the python object type
coercion to deal with, but I haven't profiled it.

But there is a gotcha. When I compile the code, I use flag in
setup.py to indicate whether to compile with Numeric or numarray.
This sets a flag and that is used in the transforms module (and the
image module) that will import arrayobject.h from either Numeric or
numarray.

A problem can arise if someone compiles with one and uses a different
setting in ~/.matplotlibrc. I compiled the module with Numeric but
used numarray in my matplotlibrc. The transformation went fine,
presumably thanks to the numarray compatibility layer. I passed the
transformed arrays off to one of the backends, which does

from matplotlib.numerix import Int16

        y = int(self.height) - y.astype(Int16)

and got a

  ValueError: type must be either a 1-length string, or python type
  object

clearly because of the difference between numpy and numarray type
args.

Is there a clever workaround for this problem? It doesn't affect any
of the agg or PS backends. GTK and GD both need to convert their
arrays to ints before passing them on to their respective renderers.
One solution is to do the conversion in list comps

y = [ int(self.height-val) for val in y]

which obviously implies a performance hit but probably a bearable one
given the other speedups that the new transformation module implies
and that users have an option to use GTKAgg instead of GTK and Agg
instead of GD for optimal performance.

Another question is win32, where the user doesn't have the choice at
compile time. If we compile in numpy for win32, the user's numarrays
will be transformed into numpys at render time, but this won't affect
the user arrays (unlike in scipy where the user is getting the return
values) so I don't see it as a big deal. I think trying to compile in
both or supplying alternate binaries is doable but may not be worth
the maintenance and complexity.

In any case, there are a couple of readers here who know a bit more
about numarray and numeric than I do <wink> ; any thoughts?

JDH

John Hunter wrote:

Is there a clever workaround for this problem? It doesn't affect any
of the agg or PS backends. GTK and GD both need to convert their
arrays to ints before passing them on to their respective renderers.
One solution is to do the conversion in list comps

[...]

Another question is win32, where the user doesn't have the choice at
compile time. If we compile in numpy for win32, the user's numarrays
will be transformed into numpys at render time, but this won't affect
the user arrays (unlike in scipy where the user is getting the return
values) so I don't see it as a big deal. I think trying to compile in
both or supplying alternate binaries is doable but may not be worth
the maintenance and complexity.

[...]

In any case, there are a couple of readers here who know a bit more
about numarray and numeric than I do <wink> ; any thoughts?

Perhaps there is a clever workaround, but I wonder if the best solution
isn't to build dual versions of the extensions, one for Numeric and one
for numarray if both are present (and this would also solve the win32
issue by distributing both). I think setup.py could be setup to do this,
and a mechanism for a module that uses these to use whichever
is selected by the user to pick up the right sharable
(e.g., _numarray_mycext.so vs _Numeric_mycext.so). It's not
elegant, but it should be robust. Todd and I did talk about if
there was a way to make numarray arrays compatable at the binary
level; perhaps, but I suspect it involves a lot work and I'm not
sure it will eliminate all issues. So do you want us to try to
come up with a an example setup.py to do this? Perhaps Todd will
have some other ideas.

Perry

The new transformation module I've been working on is implemented in
extension code. It offers a couple of important benefits, namely
speed and generality (affines, nonseparable xy transformations). It's
not in CVS yet because I'm still ironing out the details.

I have two methods for operating over sequences of x and y, seq_x_y
and numerix_x_y. The first uses the python sequence API and the
second uses the numerix api. I expect the latter will be a good bit
faster for long lines, since it doesn't have the python object type
coercion to deal with, but I haven't profiled it.

But there is a gotcha. When I compile the code, I use flag in
setup.py to indicate whether to compile with Numeric or numarray.
This sets a flag and that is used in the transforms module (and the
image module) that will import arrayobject.h from either Numeric or
numarray.

A problem can arise if someone compiles with one and uses a different
setting in ~/.matplotlibrc. I compiled the module with Numeric but
used numarray in my matplotlibrc. The transformation went fine,
presumably thanks to the numarray compatibility layer. I passed the
transformed arrays off to one of the backends, which does

from matplotlib.numerix import Int16

        y = int(self.height) - y.astype(Int16)

and got a

  ValueError: type must be either a 1-length string, or python type
  object

clearly because of the difference between numpy and numarray type
args.

Yeah. numarray has full blown type objects which form a hierarchy which
can be used to test for type properties such as signed, integral, etc.

numarray uses names for its type objects that were used by Numeric to
make 1 character typecodes clearer and easier to remember. Thus, in
numarray Int16 is an object, but in Numeric Int16 is an alias for 's'.

Over time, our compatibility ambitions have grown so now numarray
generally works with Numeric typecodes wherever a type object is
expected.

So, while this feels like walking backward, you can say:

        y = int(self.height) - y.astype('s')

and expect it to work with either.

Another question is win32, where the user doesn't have the choice at
compile time. If we compile in numpy for win32, the user's numarrays
will be transformed into numpys at render time, but this won't affect
the user arrays (unlike in scipy where the user is getting the return
values) so I don't see it as a big deal. I think trying to compile in
both or supplying alternate binaries is doable but may not be worth
the maintenance and complexity.

In any case, there are a couple of readers here who know a bit more
about numarray and numeric than I do <wink> ; any thoughts?

(1) One thought I've had for handling this has been to improve the basic
binary compatibility of numarray with Numeric to the point that numerix
can be extended to the C-level. In this scheme, an extension would be
compiled against neither numarray nor Numeric but against numerix;
which array package was actually being used would be figured out at run
time much as it works at the Python level now. I'm still exploring the
idea.

(2) I could just build a numarray version of the matplotlob installer
with each release. That would offload the work from you and would keep
the matplotlib code cleaner.

Todd

···

On Fri, 2004-05-14 at 08:35, John Hunter wrote:

Todd Miller wrote:

So, while this feels like walking backward, you can say:

        y = int(self.height) - y.astype('s')

and expect it to work with either.

If this is used as the workaround, perhaps local (to the module)
versions of Int16 and such can be defined as 's' and so forth
so that it will be easier to update code rather than litter
it with such character codes.

Perry

Good idea. I'll see if I can get numerix to take care of it.

Todd

···

On Fri, 2004-05-14 at 11:27, Perry Greenfield wrote:

Todd Miller wrote:
>
> So, while this feels like walking backward, you can say:
>
> y = int(self.height) - y.astype('s')
>
> and expect it to work with either.
>
If this is used as the workaround, perhaps local (to the module)
versions of Int16 and such can be defined as 's' and so forth
so that it will be easier to update code rather than litter
it with such character codes.

Perry

I wonder what the recommended way of saving
and using different plotting defaults is.

Let me give some background first. While many users
like the current window color defaults (basically black
on white) some prefer white on black for interactive use
(for one thing, colors stand out more). And while they
may be morons for wanting to do so ;-), there is also
a very good reason to use such a color scheme for viewgraphs.
I don't know how many times I've suffered through a presentation
where the presenter showed the plots having a white background
and colored lines or symbols. In that setting, colors just don't
show well. ("well, you could see the yellow points in my version...")
On the other hand such color schemes don't print well (and
the current one does).

I could see someone wanting 2 or 3 different contexts. The
current rcParams does allow for different save color faces,
but that isn't quite enough for cases where text needs to be
white or something other than black to show up against black
bacgrounds.

Is there a means of using alternate rcParams dictionaries for
use with different backends, say as a parameter to the print_figure
method? It doesn't look like it at the moment. I'd guess to do
that now means replacing rcParams temporarily, rendering the
plot to the file, and resetting rcParams to what it was (I haven't
actually tried it), but that seems clumsier than it should be.

Perry