Preliminary patch for the (new) scipy

Development matplotlib-devel
1

I've submitted the following:

    > http://sourceforge.net/tracker/index.php?
    > func=detail&aid=1360855&group_id=80706&atid=560722

    > This patch allows one to use matplotlib with (just) the
    > new scipy.

Just for clarification, when you say "just" the new scipy, you mean
that it works with Numeric, numarray *and* the new scipy, not that it
works with the new scipy and only the new scipy.

I read over the patch and it looks like you did a very thorough job.
Thanks! In the near term, this means that mpl would compile three
shared object files for each of the three array objects for each
extension module, which of course will increase compile times and
binary distribution sizes. Travis, Perry, Todd and I have been
discussing the benefits of changing matplotlib to work *only* with the
new scipy, which include faster build times, smaller binaries, less
complexity and pushing the community to a single array object. Since
the new array interface works with Numeric 24, recent numarray or the
new scipy, *any* of these array packages would work with a matplotlib
compiled just for scipy. We decided to hold off on doing this until
scipy installations issues were sorted out even on semi-obscure
platforms -- Travis, what's your sense of this?

So my inclination is to include your patch now, during a transition
period, and then move over to a new scipy only build, retaining the
numerix layer so Numeric, numarray and (new) scipy users can continue
to use mpl transparently. In particular we need to make sure that
basemap which uses numarray.ndimage continues to work.

Does this sound like the right approach?

JDH

2

John Hunter wrote:

"daishi" == daishi <daishi@...43...> writes:

    > I've submitted the following:
    > http://sourceforge.net/tracker/index.php?
    > func=detail&aid=1360855&group_id=80706&atid=560722

    > This patch allows one to use matplotlib with (just) the
    > new scipy.

Just for clarification, when you say "just" the new scipy, you mean
that it works with Numeric, numarray *and* the new scipy, not that it
works with the new scipy and only the new scipy.

I read over the patch and it looks like you did a very thorough job.
Thanks!

Minor fix needed to avoid unpleasant surprises for users who have the old scipy on their import path:

try:
     import scipy
     if hasattr(scipy,'__core_version__'):
         NUMERIX.append('scipy')
except ImportError:
     pass

You want to make sure that you only do this for users of the _new_ scipy, not the old one.

Cheers,

f

3

John Hunter wrote:

I read over the patch and it looks like you did a very thorough job.
Thanks! In the near term, this means that mpl would compile three
shared object files for each of the three array objects for each
extension module, which of course will increase compile times and
binary distribution sizes. Travis, Perry, Todd and I have been
discussing the benefits of changing matplotlib to work *only* with the
new scipy, which include faster build times, smaller binaries, less
complexity and pushing the community to a single array object. Since
the new array interface works with Numeric 24, recent numarray or the
new scipy, *any* of these array packages would work with a matplotlib
compiled just for scipy. We decided to hold off on doing this until
scipy installations issues were sorted out even on semi-obscure
platforms -- Travis, what's your sense of this?

My sense is that scipy_core installation is working on as many platforms as Numeric did. I'd personally love to see matplotlib move to a single scipy core build as soon as possible. This would encourage even more installation and testing of the new scipy core and help us iron out any remaining issues even faster.

However, as long as matplotlib users have a sense that this is the direction matplotlib is headed, I see nothing wrong with making a release with the 3-supported arrays (especially since somebody has done all of the work already :slight_smile: ), and then making a release after that build only with new scipy core.

Just having the ability to build against new scipy will still encourage some to make the transition (and that will help with continued testing).

So my inclination is to include your patch now, during a transition
period, and then move over to a new scipy only build, retaining the
numerix layer so Numeric, numarray and (new) scipy users can continue
to use mpl transparently. In particular we need to make sure that
basemap which uses numarray.ndimage continues to work.

With the patch already submitted, I think this makes a lot of sense. I'd still like to encourage people to help us test scipy core, though. While you may still find occasional problems (which are typically fixed quickly), the system is working quite well. The sooner we can merge to one array object, the more we will be able to help each other make it a better object, and the sooner we will be able to ease the burden of third-party packages that use arrays, and eliminate the need for heroic efforts like the numerix layer.

Best regards,

-Travis

4

John Hunter wrote:

.....

So my inclination is to include your patch now, during a transition
period, and then move over to a new scipy only build, retaining the
numerix layer so Numeric, numarray and (new) scipy users can continue
to use mpl transparently. In particular we need to make sure that
basemap which uses numarray.ndimage continues to work.
  

John:

Basemap no longer uses nd_image, I've recoded the interp function in pure python, so it should work with scipy_core.

-Jeff

···

--
Jeffrey S. Whitaker Phone : (303)497-6313
Meteorologist FAX : (303)497-6449
NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@...236...
325 Broadway Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web : Jeffrey S. Whitaker: NOAA Physical Sciences Laboratory

5

The last couple of days, I have been thinking about scipy_core's atlas
dependency, which might be a big barrier for some. There is this issue with
getting the full lapack instead of the truncated version, for example. I
don't know how difficult it is to compile atlas, having been screened from
the process by gentoo's package manager. The mpl devs have put a lot of
effort into making mpl easily accessible to the newbies (remember the backend
issues circa 0.50?) so I am concerned about erecting new barriers now. Will
scipy_core ever include something like Numeric's lapack_lite?

As a staff scientist at a national lab, I will be creating tools to aid our
users in data visualization and analysis. Some of our projects involve
extensive outreach efforts, involving gradeschool, highschool, and colleges
around the country. My collaborators don't care if they are using matlab,
python or whatever, as long as it is easy to use and they can get their
analysis done.

I suggest that mpl not commit to requiring scipy *instead* of numeric or
numarray until scipy_core can be built without external math libraries (like
numarray and Numeric), and has proved to be a piece of cake to install
on the major os's/distributions known to be used in this community. For
example, scipy/distutils is still missing a site.cfg.example. I've been around
long enough to know about site.cfg from scipy-0.3.2, but even so, when I
build on Gentoo, I still have to edit system_info.py for new scipy to find
the blas libraries.

I'm impressed with scipy, and hope the numeric/numarray community will adopt
it quickly. But I don't want to advocate using MPL as a vehicle to force such
an adoption prematurely.

Respectfully,
Darren

···

On Sunday 20 November 2005 1:55 am, Travis Oliphant wrote:

John Hunter wrote:
>Travis, Perry, Todd and I have been
>discussing the benefits of changing matplotlib to work *only* with the
>new scipy, which include faster build times, smaller binaries, less
>complexity and pushing the community to a single array object. Since
>the new array interface works with Numeric 24, recent numarray or the
>new scipy, *any* of these array packages would work with a matplotlib
>compiled just for scipy. We decided to hold off on doing this until
>scipy installations issues were sorted out even on semi-obscure
>platforms -- Travis, what's your sense of this?

My sense is that scipy_core installation is working on as many platforms
as Numeric did. I'd personally love to see matplotlib move to a single
scipy core build as soon as possible. This would encourage even more
installation and testing of the new scipy core and help us iron out any
remaining issues even faster.

However, as long as matplotlib users have a sense that this is the
direction matplotlib is headed, I see nothing wrong with making a
release with the 3-supported arrays (especially since somebody has done
all of the work already :slight_smile: ), and then making a release after that
build only with new scipy core.

Just having the ability to build against new scipy will still encourage
some to make the transition (and that will help with continued testing).

6

Darren Dale wrote:

The last couple of days, I have been thinking about scipy_core's atlas
dependency, which might be a big barrier for some. There is this issue with
getting the full lapack instead of the truncated version, for example. I
don't know how difficult it is to compile atlas, having been screened from
the process by gentoo's package manager. The mpl devs have put a lot of
effort into making mpl easily accessible to the newbies (remember the backend
issues circa 0.50?) so I am concerned about erecting new barriers now. Will
scipy_core ever include something like Numeric's lapack_lite?

scipy_core does not depend on ATLAS. It already has lapack_lite.

[svk-projects]$ ls scipy_core/scipy/corelib/lapack_lite
blas_lite.c dlapack_lite.c f2c_lite.c zlapack_lite.c
dlamch.c f2c.h lapack_litemodule.c

···

--
Robert Kern
robert.kern@...149...

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
  -- Richard Harter

7

Interesting. A couple days ago I removed atlas, blas and lapack, removed my
site.cfg, and was unable to install scipy_core. I'll try again, maybe I did
something wrong.

···

On Sunday 20 November 2005 5:47 pm, Robert Kern wrote:

Darren Dale wrote:
> The last couple of days, I have been thinking about scipy_core's atlas
> dependency, which might be a big barrier for some. There is this issue
> with getting the full lapack instead of the truncated version, for
> example. I don't know how difficult it is to compile atlas, having been
> screened from the process by gentoo's package manager. The mpl devs have
> put a lot of effort into making mpl easily accessible to the newbies
> (remember the backend issues circa 0.50?) so I am concerned about
> erecting new barriers now. Will scipy_core ever include something like
> Numeric's lapack_lite?

scipy_core does not depend on ATLAS. It already has lapack_lite.

[svk-projects]$ ls scipy_core/scipy/corelib/lapack_lite
blas_lite.c dlapack_lite.c f2c_lite.c zlapack_lite.c
dlamch.c f2c.h lapack_litemodule.c

8

Sorry, I've obviously overlooked something, but maybe I have also discovered a
bug. I've just removed atlas/blas/lapack, updated scipy_core from svn,
removed my build and site-packages/scipy directories, and tried to build
scipy_core. Here is a warning message I get toward the end of the build and
install processes:

building 'scipy.lib._dotblas' extension
compiling C sources
i686-pc-linux-gnu-gcc options: '-pthread -fno-strict-aliasing -DNDEBUG -fPIC'
creating build/temp.linux-i686-2.4/scipy/corelib
creating build/temp.linux-i686-2.4/scipy/corelib/blasdot
compile options: '-DNO_ATLAS_INFO=1 -Iscipy/corelib/blasdot
-Iscipy/base/include -Ibuild/src/scipy/base -Iscipy/base/src
-I/usr/include/python2.4 -c'
i686-pc-linux-gnu-gcc: scipy/corelib/blasdot/_dotblas.c
/usr/bin/g77 -shared
build/temp.linux-i686-2.4/scipy/corelib/blasdot/_dotblas.o -L/usr/lib -lblas
-lg2c -o build/lib.linux-i686-2.4/scipy/lib/_dotblas.so
/usr/lib/gcc/i686-pc-linux-gnu/3.4.4/../../../../i686-pc-linux-gnu/bin/ld:
cannot find -lblas
collect2: ld returned 1 exit status
/usr/lib/gcc/i686-pc-linux-gnu/3.4.4/../../../../i686-pc-linux-gnu/bin/ld:
cannot find -lblas
collect2: ld returned 1 exit status
error: Command "/usr/bin/g77 -shared
build/temp.linux-i686-2.4/scipy/corelib/blasdot/_dotblas.o -L/usr/lib -lblas
-lg2c -o build/lib.linux-i686-2.4/scipy/lib/_dotblas.so" failed with exit
status 1

and here I try to import scipy:

In [1]: from scipy import *

···

On Sunday 20 November 2005 6:57 pm, Darren Dale wrote:

On Sunday 20 November 2005 5:47 pm, Robert Kern wrote:
> Darren Dale wrote:
> > Will scipy_core ever include something like Numeric's lapack_lite?
>
> scipy_core does not depend on ATLAS. It already has lapack_lite.
>
> [svk-projects]$ ls scipy_core/scipy/corelib/lapack_lite
> blas_lite.c dlapack_lite.c f2c_lite.c
> zlapack_lite.c dlamch.c f2c.h
> lapack_litemodule.c

---------------------------------------------------------------------------
exceptions.ImportError Traceback (most recent
call last)

/home/darren/<console>

ImportError: No module named scipy

Darren

9

Sorry about the delay in getting back to you
(I was away for a long weekend and just got
back in today).

    > This patch allows one to use matplotlib with (just) the
    > new scipy.

Just for clarification, when you say "just" the new scipy, you mean
that it works with Numeric, numarray *and* the new scipy, not that it
works with the new scipy and only the new scipy.

The intent was so that current users of numeric/numarray
wouldn't see any difference at all, but that matplotlib
would build against the new scipy - I believe this is the
case, mod Fernando's fix.

In the near term, this means that mpl would compile three
shared object files for each of the three array objects for each
extension module, which of course will increase compile times and
binary distribution sizes.

As it currently stands, the patch will build at most *two*
shared objects, because the build logic is basically:

if numarray:
  # numarray build
if (numeric or scipy):
  # numeric or scipy build,
  # with numeric taking precedence.

It shouldn't be difficult to flatten this out to build
three libraries, however. I submitted the patch in its
preliminary form to see whether there was interest in
going this route at all, since I saw the discussion
about the unified/new array interface plan.

Fernando wrote:

Minor fix needed to avoid unpleasant surprises for users who have the old scipy on their import path:

try:
    import scipy
    if hasattr(scipy,'__core_version__'):
        NUMERIX.append('scipy')
except ImportError:
    pass

You want to make sure that you only do this for users of the _new_ scipy, not the old one.

Thanks and sorry about that, this problem
occurred to me over the weekend too.

d

···

On Nov 19, 2005, at 8:22 AM, John Hunter wrote: