when to deprecate numeric and numarray?

Development matplotlib-devel
1

I think David Cournapeau's email to the -user list (included below) brings up the general issue of whether and how and when we want to go about deprecating the use of Numeric and numarray in MPL. Their continued inclusion in the core of MPL increases complexity (thereby slowing development and making bugs more likely) and limits features by introducing a least-common-denominator situation. For example, Eric Firing and I recently fixed a bug involving masked arrays being passed to quiver() that illustrated this issue. I think it's obvious that Travis Oliphant is succeeding (or is that "has succeeded"?) in creating the definitive array package for Python and people are crazy if they write new code with the older packages. That said, I'm sure there's lots of old code not yet ported, but numpy has pretty good (copy-less) support for Numeric and numarray arrays, too -- just because they won't be in the core of MPL doesn't mean they can't be used.

So, this email is just to ask the questions, not to actually propose anything concrete:

Do we add deprecation warnings for the 0.90+1 release cycle and then stop building the numarray and numeric numerix backends at some point after that? When? Do we keep the "numerix" name or just switch everything to numpy?

-Andrew

David Cournapeau wrote:

···

Hi there,

    A few months back, I complained about the slowness of the image function in matplotlib. One of the cullprit was a slow clip function; I've done a bit some work to improve the situation on numpy's side, efforts which were integrated in numpy 1.0.2. Now, when you clip a numpy array with scalar min and max values, you get a 5 to 30 fold speed-up; to get the maximum efficiency, you need inplace clipping (using the syntax a.clip(min, max, a) for a a numpy array). This makes image significantly faster (between 100 and 200 ms on recent computers), and I am sure in other functionalities of matplotlib as well.
    cheers,

    David

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

2

I agree that it is about time to begin preparing for the switch. I
was talking to Perry the other day about what an irony it was for him
when he was writing the colormap support in mpl that he had to use all
the takes and puts after spending so much effort in numarray to get
more natural indexing and other features. So I think most everyone is
ready to jettison the old stuff and move forward with the new. I've
been waiting for the green light from STScI that they are mostly
finished with their numarray->numpy migration since they have made
significant contributions to mpl (and numerix) and if I recall
correctly, I think Perry said they were mostly done, which means we
should go forward. Perry?

As for your specific points:

  * inside matplotlib we should just use numpy everywhere. We need to
agree on some import convention. I'm happy with 'import numpy as nx'
though this might be confusing for a while since people might confuse
it with the numerix layer. I like nx because numpy is too long, and N
occurs too frequently in regular code. I don't like capital
letters.... I could see ns too, since that is what we have been using
for the numpy extensions when numpy was originally discussed in the
context of the scipy core.

  * I suppose we should deprecate it for a release, but I'm inclined
just to push the thing through quickly because it is a big change and
if and when we have energy for it we should just get it done. I'm
also happy to have some sense talked into me. I suppose one
possibility is to deprecate it *now* and push out 0.91 ASAP and then
immediately pull the old support out. I'll post on user's list to get
a sense of how many people both need the latest mpl and the older
array packages. I can't imagine there are too many... I'm sure some
people need Numeric or numarray, but if they are that curmudgeonly,
surely they can live on the older mpl branch.

  * I would like to see the numerix layer live on, not for use in mpl
but for use outside it for folks who have written a lot of code around
it in external scripts. So people who have done

   from pylab import nx

or

  import matplotlib.numerix as nx

will still have working code. Of course we will lose all the mpl
extensions compiled against the other array packages, but with the
array interface I don't think this will pose a problem for people
using mpl with recent versions of Numeric or numarray

  * when we do the cleanup, we should replace all the 'from numerix
import something' with 'import numpy as nx; nx.something' as above.
Where possible when cleaning a given module for numerix, we should
standardize the other imports. Eg, instead of 'from cbook import
iterable' we should do 'import matplotlib.cbook as cbook;
cbook.iterable' Let's use this convention where we use absolute
imports renamed to relative imports, and qualify all module functions
in the code with the module names.

Anything else?

JDH

···

On 4/4/07, Andrew Straw <strawman@...36...> wrote:

Do we add deprecation warnings for the 0.90+1 release cycle and then
stop building the numarray and numeric numerix backends at some point
after that? When? Do we keep the "numerix" name or just switch
everything to numpy?

3

We are done internally for all our released software and have propagated these changes to our internal users (just this week as a matter of fact). The most convenient time to remove the support for numarray for us is when we make a public release of our software. The date isn't fixed yet but that would probably be in June sometime. The reason it is convenient for us to retain the numarray compatibility until then is that we release a bunch of things together that people can get as one download; taking numarray support out of mpl before then means that people with the existing release will have to install numpy if they want to upgrade mpl (and also face some confusion about what kind of array object they are dealing with if they use functions within mpl that create arrays).

Having said that, I've told John that I hate having held up the date that the transition to pure numpy in mpl can be accomplished by, and that if he wants to he can go ahead with it. So far he has been very kind in waiting for us to finish our transition to numpy. So to summarize on our end, the conversion to numpy has been completed and tested by our developers, and now is being tested by our institutional users, and sometime around June we will release our new software. At that point, we have no desire or need to have any further numarray option in mpl. I'll leave it to John to decide if he wants to go ahead with that conversion in mpl now. The effect on our user community probably isn't going to be great. By the time it is done in mpl and is available to our community there should only be a couple months, at most, where our users will have to deal with the issue (and they can either wait to upgrade mpl after we release, or deal with the installation/array issues that arise for the relatively short duration.

Perry

···

On Apr 4, 2007, at 9:41 AM, John Hunter wrote:

On 4/4/07, Andrew Straw <strawman@...36...> wrote:

Do we add deprecation warnings for the 0.90+1 release cycle and then
stop building the numarray and numeric numerix backends at some point
after that? When? Do we keep the "numerix" name or just switch
everything to numpy?

I agree that it is about time to begin preparing for the switch. I
was talking to Perry the other day about what an irony it was for him
when he was writing the colormap support in mpl that he had to use all
the takes and puts after spending so much effort in numarray to get
more natural indexing and other features. So I think most everyone is
ready to jettison the old stuff and move forward with the new. I've
been waiting for the green light from STScI that they are mostly
finished with their numarray->numpy migration since they have made
significant contributions to mpl (and numerix) and if I recall
correctly, I think Perry said they were mostly done, which means we
should go forward. Perry?

4

John Hunter wrote:
[...]

As for your specific points:

  * inside matplotlib we should just use numpy everywhere. We need to
agree on some import convention. I'm happy with 'import numpy as nx'
though this might be confusing for a while since people might confuse
it with the numerix layer. I like nx because numpy is too long, and N
occurs too frequently in regular code. I don't like capital
letters.... I could see ns too, since that is what we have been using
for the numpy extensions when numpy was originally discussed in the
context of the scipy core.

ns is not very mnemonic, and I think we should avoid the confusion between nx as numerix and nx as numpy, so I suggest "np" for numpy. It is mnemonic, and it will make it easier to keep track of the conversion process. An alternative would be "nu".

  * I suppose we should deprecate it for a release, but I'm inclined
just to push the thing through quickly because it is a big change and
if and when we have energy for it we should just get it done. I'm
also happy to have some sense talked into me. I suppose one

I won't be the one to talk sense into you! This change is going to take a some time, and I would like to be able to get started on it. It doesn't have to be done all at once.

possibility is to deprecate it *now* and push out 0.91 ASAP and then
immediately pull the old support out. I'll post on user's list to get

If you want to make one more release, I would like to have a few days notice to see if I can clear up at least one thing, and maybe a couple more.

a sense of how many people both need the latest mpl and the older
array packages. I can't imagine there are too many... I'm sure some
people need Numeric or numarray, but if they are that curmudgeonly,
surely they can live on the older mpl branch.

[...]

Eric

5

John Hunter wrote:

  * I suppose we should deprecate it for a release, but I'm inclined
just to push the thing through quickly

+1

You can't do it too fast for me.

>* when we do the cleanup, we should replace all the 'from numerix
>import something' with 'import numpy as nx; nx.something'

+1

> Where possible when cleaning a given module for numerix, we should
> standardize the other imports. Eg, instead of 'from cbook import
> iterable' we should do 'import matplotlib.cbook as cbook;
> cbook.iterable'

+1

all around a good plan.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...236...

6

My opinion is that all of N,np,nu,ns,nx look too much like variable
names I commonly use (nx == number of x values, etc). And none of
them looks particularly to me like the name of a big package for doing
numerical work.

So I suggest instead 'npy'. It is already used extensively in the C
API of Numpy so it has strong precedent as an abbreviation. It also
*looks* to me like an abbreviation for 'numpy' as opposed to "number
of points (np)" or something else. Having 3 letters also makes it
slightly less likely to clash with typical short user variable names.

One thing it doesn't have is a nice analogous abbreviation for scipy.
The direct analog would of course be 'spy', but that's obviously out.
Even if pylab.spy didn't exist, it still doesn't look like an
abbreviation for scipy. I guess I like sci for scipy. npy and sci.

--bb

···

On 4/5/07, Eric Firing <efiring@...229...> wrote:

John Hunter wrote:
[...]
> As for your specific points:
>
> * inside matplotlib we should just use numpy everywhere. We need to
> agree on some import convention. I'm happy with 'import numpy as nx'
> though this might be confusing for a while since people might confuse
> it with the numerix layer. I like nx because numpy is too long, and N
> occurs too frequently in regular code. I don't like capital
> letters.... I could see ns too, since that is what we have been using
> for the numpy extensions when numpy was originally discussed in the
> context of the scipy core.

ns is not very mnemonic, and I think we should avoid the confusion
between nx as numerix and nx as numpy, so I suggest "np" for numpy. It
is mnemonic, and it will make it easier to keep track of the conversion
process. An alternative would be "nu".

7

OK, let's shoot for a release next week or the week after, however
long it takes for people to get their current work into svn, tested
and stable, and (optionally) include a deprecation warning for Numeric
and numarray users. The we can get started with the numpy migration
in svn, with the view that we can take our time with it, and try and
get something out in six weeks or so. By that time, Perry and crew
will be almost ready to push out their numpy based release and their
users will be minimally inconvenienced, if at all.

JDH

···

On 4/4/07, Eric Firing <efiring@...229...> wrote:

If you want to make one more release, I would like to have a few days
notice to see if I can clear up at least one thing, and maybe a couple more.

8

+1 for npy.

···

On 4/5/07, Bill Baxter <wbaxter@...149...> wrote:

So I suggest instead 'npy'. It is already used extensively in the C
API of Numpy so it has strong precedent as an abbreviation. It also
*looks* to me like an abbreviation for 'numpy' as opposed to "number
of points (np)" or something else. Having 3 letters also makes it
slightly less likely to clash with typical short user variable names.