installing numpy, matplotlib, scipy from source on a Mac

belinda_thom · January 10, 2007, 7:52pm

I am posting this message to both numpy and matplotlib mailing lists because the thread relates to both.

First, Robert Kern kindly provided step-by-step instructions for Macs (PPCs and Intels) regarding how to install FROM SOURCE the packages needed to allow Python to become a viable alternative for Matlab. Details regarding the installation of matplotlib (along with wx, which is supposed to allow the WxAgg backend to work), numpy, and scipy were provided.

The reason for installing FROM SOURCE on a Mac is because various packages need to be compiled in such a way that they rely on the same underling shared libraries, etc. for things to work (I've described some problems w/installing from numpy, scipy, and matplotlib dmgs at http://www.mail-archive.com/ numpy-discussion at scipy.org/msg00481.html).

From these instructions, I've been able to acheive numpy.test() anad scipy.test() that run w/no failures -- hooray!

However, after installing wx and matplotlib, various problems result:

1) warnings about fonts
2) wx fails to work

I've appended the warnings below. These only occur the first time pylab is imported (does this make sense?).

After that, I've appended the issues I've had when trying to use wx as a backend. One reason I'd like to be able to use wx is that it appears with TkAgg, running IDLE (w/the -n flag) can hang (see: http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg02039.html).

Point that originally confused me: although wx is needed for the WxAgg backend, you don't need to install the analogous Tk package to use TkAgg PROVIDED you're using MacPython, for that comes bundled w/an Aqua-based Tk interface.

I'm on a G4 PPC, w/OS X 10.4.8, and using MacPython 2.4. Any idea what wx doesn't work?

Thanks,

--b

ONE-TIME WARNINGS:

···

--------------------
Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
history mechanism set up
>>> import pylab as P
/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/font_manager.py:455: UserWarning: Could not open font file /Library/Fonts/NISC18030.ttf
warnings.warn("Could not open font file %s"%fpath)
<snip>
/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/font_manager.py:455: UserWarning: Could not open font file /System/Library/Fonts/TimesLTMM
warnings.warn("Could not open font file %s"%fpath)
>>>

WX / MATPLOTLIB FAILURE
--------------------------

4 % python
Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
history mechanism set up
>>> import matplotlib as M
>>> import pylab as P
>>> M.rcParams['interactive']
True
>>> M.rcParams['backend']
'WXAgg'
>>> P.plot([1,2,3])
Traceback (most recent call last):
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/backends/backend_wx.py", line 1048, in _onPaint
     self.draw(repaint=False)
<snip>
     figManager.canvas.draw()
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/backends/backend_wxagg.py", line 63, in draw
     self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None)
MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create the wx.Bitmap
>>>

Robert_Kern3 · January 10, 2007, 8:08pm

belinda thom wrote:

I am posting this message to both numpy and matplotlib mailing lists
because the thread relates to both.

Actually, it's really only relevant to matplotlib.

However, after installing wx and matplotlib, various problems result:

1) warnings about fonts
2) wx fails to work

I've appended the warnings below. These only occur the first time
pylab is imported (does this make sense?).

Yes. After the first time, a cache is built and the font manager doesn't go
trawling through your fonts again. matplotlib's font library cannot parse some
of the Mac fonts (damned resource forks), so it warns you.

Personally, I think the warnings are a bit overzealous and should be silenced.
It's not as if the user is explicitly telling the font manager to load those
specific fonts. They are automatically and unavoidably attempted.

WX / MATPLOTLIB FAILURE
--------------------------

4 % python

Try running with pythonw.

···

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

belinda_thom · January 10, 2007, 8:30pm

Robert,

Try running with pythonw.

Do you know how to fix this in IDLE (it must be using python as opposed to pythonw somehow).

Thanks again for all your help,

--b

Robert_Kern3 · January 10, 2007, 8:41pm

belinda thom wrote:

Robert,

Try running with pythonw.

Do you know how to fix this in IDLE (it must be using python as
opposed to pythonw somehow).

I'm afraid that I don't know enough about IDLE to help you.

···

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

_Chris_Barker · January 10, 2007, 11:06pm

Robert Kern wrote:

Try running with pythonw.

That's probably not it -- as of MacPython 2.4, pythonw ands python are the same.

belinda thom wrote:

And running w/pythonw does what it should :-).

OK, now I'm confused:

$ ls -l /Library/Frameworks/Python.framework/Versions/2.4/bin/python2.4
-rwxrwxr-x 1 root admin 39936 Apr 7 2006 /Library/Frameworks/Python.framework/Versions/2.4/bin/python2.4

$ ls -l /Library/Frameworks/Python.framework/Versions/2.4/bin/pythonw2.4
-rwxrwxr-x 1 root admin 39936 Apr 7 2006 /Library/Frameworks/Python.framework/Versions/2.4/bin/pythonw2.4

Those two look like the same binaries to me -- and diff tells me they are.

Are you sure you're running the same python with "python" and "pythonw"? Try running them on the command line alone and see what you get.

File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/matplotlib/backends/backend_wxagg.py", line 63, in draw
self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None)
MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create the wx.Bitmap

This looks like the error we usually get when you've built the wxAgg extension against a different version of wx than the one you're running. That's easy to do, as Apple has provided an old wx with it's Python, it is often found by default by the MPL build process.

Search this list for the way to fix that, or, if you really can't find it, I'll dig it up.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...

belinda_thom · January 11, 2007, 1:35am

Hi Chris, Robert, ...?,

I'm glad you reminded me to make sure the path was right, b/c this time, I had forgotten to run the "Update Shell" command that came w/MacPython2.4. This gave me:

lrwxr-xr-x 1 root wheel 67 Jan 7 14:29 /usr/local/bin/pydoc@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/pydoc
lrwxr-xr-x 1 root wheel 70 Jan 7 14:29 /usr/local/bin/pydoc2.4@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/pydoc2.4
lrwxr-xr-x 1 root wheel 68 Jan 7 14:29 /usr/local/bin/python@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/python
lrwxr-xr-x 1 root wheel 71 Jan 7 14:29 /usr/local/bin/python2.4@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/python2.4
lrwxr-xr-x 1 root wheel 69 Jan 7 14:29 /usr/local/bin/pythonw@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/pythonw
lrwxr-xr-x 1 root wheel 72 Jan 7 14:29 /usr/local/bin/pythonw2.4@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/pythonw2.4
lrwxr-xr-x 1 root wheel 70 Jan 7 14:29 /usr/local/bin/smtpd.py@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/smtpd.py
lrwxr-xr-x 1 root wheel 73 Jan 7 14:29 /usr/local/bin/smtpd2.4.py@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/smtpd2.4.py

Nevertheless, the python I was using in the prior email looks like the same as what I'm using w/these path updates, that is, assuming version strings are enough to know:

   From prior run:
   Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
   [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin

   New run:
   Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
   [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin

As you'd suspected, it is indeed the case that:

diff pythonw python

reports no differences. So, I went back and retried the plotting w/wx as a backend and discovered that wx FAILS with PYTHONW and PYTHON (appended).

If you saw how many notes I've collected on the different installs I've tried you'd laugh (except that its not funny). Its no wonder I can't keep all this stuff straight. (When I first embarked on this exercise several months ago, I tried macports; everything seemed to work fine, but the Wx backend failed---I remember something about a bitmap, so it was likely the same error I'm having now; and unfortunately w/the macports python, the tk/tcl stuff didn't come wrapped, so TkAgg was not an option).

After an email w/Robert, I switched to MacPython. As a result, I now have TkAgg working (and its using Aqua instead of that awful, klunky looking Tk stuff).

Fortunately (ha!), I have three versions of this stuff installed on 3 different Macs (a G4, a G5, and an Intel). So I can say with safety that, using TkAgg backend, here's what's still not working:

1) Using macpython matplotlib, numpy, and scipy packages, you can't run scipy.test(); complaint: RuntimeError: module compiled against version 1000002 of C-API but this version of numpy is 1000009
2) Using superpak, everything (except ipython) worked out of the box, PROVIDED I had already installed the necessary fortran pieces (I used g77v3.4-bin.tar.gz, found somewhere on the scipy site).
3) Using Robert's install-from-source method (this install used gfortran.bin.tar.gz, with the caveat that the expected WXAgg doesn't work)

Machines 2 and 3 are enough to get me back to the real work I have to do. But it is obvious there's several real issues for Mac users who wish to use the scipy/numpy/matplotlib route. This coming summer I may have more time to devote to this stuff, and since it really irks me that there isn't a clean way to do this, I may end up figuring out how to make the needed binaries myself. In the meantime, I thought I'd never wish I was still using Windows...

Thanks to everyone for the time spent on this. And for posterity's sake, when you use matplotlib with Numeric, it also crashes. I've verified that behavior on Machines 1 and 2 above.

--b

54 % pythonw
Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
history mechanism set up
>>> import pylab as P
>>> import matplotlib as M
>>> M.rcParams['interactive']
True
>>> M.rcParams['backend']
'WXAgg'
>>> P.plot([1,2,3])
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/backends/backend_wx.py", line 1048, in _onPaint
self.draw(repaint=False)
<snip>
draw_if_interactive()
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/backends/backend_wx.py", line 1172, in draw_if_interactive
figManager.canvas.draw()
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/backends/backend_wxagg.py", line 63, in draw
self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None)
MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create the wx.Bitmap
>>

55 % python
Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
history mechanism set up
>>> import pylab as P
>>> import matplotlib as M
>>> M.rcParams['interactive']
True
>>> M.rcParams['backend']
'WXAgg'
>>> P.plot([1,2,3])
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/backends/backend_wx.py", line 1048, in _onPaint
self.draw(repaint=False)
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/backends/backend_wxagg.py", line 63, in draw
self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None)
MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create the wx.Bitmap
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/pylab.py", line 2022, in plot
<snip>
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/backends/backend_wx.py", line 1172, in draw_if_interactive
figManager.canvas.draw()
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/matplotlib/backends/backend_wxagg.py", line 63, in draw
self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None)
MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create the wx.Bitmap

···

On Jan 10, 2007, at 3:06 PM, Christopher Barker wrote:

Robert Kern wrote:

Try running with pythonw.

That's probably not it -- as of MacPython 2.4, pythonw ands python are the same.

belinda thom wrote:

And running w/pythonw does what it should :-).

OK, now I'm confused:

$ ls -l /Library/Frameworks/Python.framework/Versions/2.4/bin/python2.4
-rwxrwxr-x 1 root admin 39936 Apr 7 2006 /Library/Frameworks/Python.framework/Versions/2.4/bin/python2.4

$ ls -l /Library/Frameworks/Python.framework/Versions/2.4/bin/pythonw2.4
-rwxrwxr-x 1 root admin 39936 Apr 7 2006 /Library/Frameworks/Python.framework/Versions/2.4/bin/pythonw2.4

Those two look like the same binaries to me -- and diff tells me they are.

Are you sure you're running the same python with "python" and "pythonw"? Try running them on the command line alone and see what you get.

File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/matplotlib/backends/backend_wxagg.py", line 63, in draw
self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None)
MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create the wx.Bitmap

This looks like the error we usually get when you've built the wxAgg extension against a different version of wx than the one you're running. That's easy to do, as Apple has provided an old wx with it's Python, it is often found by default by the MPL build process.

Search this list for the way to fix that, or, if you really can't find it, I'll dig it up.

-Chris

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...

Robert_Kern3 · January 11, 2007, 1:56am

belinda thom wrote:

I went back and retried the plotting w/wx
as a backend and discovered that wx FAILS with PYTHONW and PYTHON
(appended).

Okay, what version of wxPython did you install? What version of wxPython is
actually imported (check wx.__version__)?

(And we can leave off numpy-discussion, it's not relevant there).

···

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

belinda_thom · January 11, 2007, 2:11am

Hi,

belinda thom wrote:

I went back and retried the plotting w/wx
as a backend and discovered that wx FAILS with PYTHONW and PYTHON
(appended).

Okay, what version of wxPython did you install? What version of wxPython is
actually imported (check wx.__version__)?

Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
history mechanism set up
>>> import wx
>>> wx.__version__
'2.6.3.3'

(And we can leave off numpy-discussion, it's not relevant there).

--
Robert Kern

Thanks,
--b

···

On Jan 10, 2007, at 5:56 PM, Robert Kern wrote:

Robert_Kern3 · January 11, 2007, 2:21am

belinda thom wrote:

Hi,

belinda thom wrote:

I went back and retried the plotting w/wx
as a backend and discovered that wx FAILS with PYTHONW and PYTHON
(appended).

Okay, what version of wxPython did you install? What version of
wxPython is
actually imported (check wx.__version__)?

Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
history mechanism set up
>>> import wx
>>> wx.__version__
'2.6.3.3'

Okay, let me rephrase: which binary package of wxPython did you install?

···

On Jan 10, 2007, at 5:56 PM, Robert Kern wrote:

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

_Chris_Barker · January 11, 2007, 11:12pm

belinda thom wrote:

reports no differences. So, I went back and retried the plotting w/wx as a backend and discovered that wx FAILS with PYTHONW and PYTHON (appended).

That's what I expected.

If you saw how many notes I've collected on the different installs I've tried you'd laugh (except that its not funny).

no, it's not -- but good for you for keeping notes!

3) Using Robert's install-from-source method (this install used gfortran.bin.tar.gz, with the caveat that the expected WXAgg doesn't work)

I do think I know what the problem is there.

self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None)
MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create the wx.Bitmap

the current, default, wxAgg back end uses some C++ code to efficiently create wxBitmaps from the Agg buffer. since it's at the C++ level, MPL needs to be compiled against the version of wx you want to use.

The MPL build system uses a nifty utility that comes with wx called wx-config to find the wx libs. However, Apple delivered an old version of wxPython with it's Python2.3. By default, the MPL build find the old wx-config, and you end up building the wxAgg back-end against that version of wx, which is not the one you want. Try this:

$ wx-config --version

I get "2.5.3" -- the version delivered by Apple.

now this:
$ which wx-config
/usr/bin/wx-config

It's in /usr/bin -- where apple puts stuff.

$ locate wx-config

I have a bunch -- these are a few:

/usr/bin/wx-config
/usr/local/lib/wxPython-unicode-2.6.3.3/bin/wx-config
/usr/local/lib/wxPython-unicode-2.8.0.1/bin/wx-config

The first is Apples.
The second is the 2.6 version I have installed with Python2.4
The third is the 2.8 version I have installed with Python2.5

Pick the one you want to build with, then set an environment variable to point to it:

WX_CONFIG=/usr/local/lib/wxPython-unicode-2.8.0.1/bin/wx-config

then run:

setup.py build.

If you look in setupext.py, you'll see a function:

find_wx_config()

Now that I look at it, it should be able to be beefed up to look for a wx-config that matches the default wxPython installed in the Python that you're running setup.py with. Maybe I'll do that the next time I build.

Come to think of it, if I just sat down and did all this it would probably take less time than writing all these helpful messages to the newsgroups!

-Chris

PS: I have not tested this recently, but it all should work!

PPS: Belinda, now that you've done all this work, I do hope you can go the extra mile and figure out how to make a binary package of it all for others!

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...

Robert_Kern3 · January 11, 2007, 11:47pm

Christopher Barker wrote:

The MPL build system uses a nifty utility that comes with wx called
wx-config to find the wx libs. However, Apple delivered an old version
of wxPython with it's Python2.3. By default, the MPL build find the old
wx-config, and you end up building the wxAgg back-end against that
version of wx, which is not the one you want. Try this:

Yes, thank you for figuring that out! That's the part that I forgot about.

···

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

belinda_thom · January 12, 2007, 1:46am

PPS: Belinda, now that you've done all this work, I do hope you can go the extra mile and figure out how to make a binary package of it all for others!

I certainly would like to to do this, but its gonna have to wait until after TheCurrentCrisis alleviates. Hopefully before Feb (in the worst case, mid-June), as I too am annoyed at how hard this seemed to be for the Mac. I also find it disconcerting that Vincent had different troubles than I w/the same source instructions (did they ever get resolved)? As had Erin.

At the same time, until about May I'll have access to both Intel and PPC Macs, so I have an ideal env. on which to test. I have only a vague idea how to make dmgs (have perused the incomprehensible hdutils man page), though, so may need some help. At the same time, I'm not qualified to fix the Vincent/Erin problems and have no idea how to go about making "universals" that work for everyone. In fact, these current threads lead me to wonder if it can actually be done (the sad alternative is this: anyone w/a Mac that wants to use the matscinum suite [defined below], unless you have piles of time to waste, stick w/matlab provided you can afford it). W/all this talk about numpy integrating w/python (via a PEP), it seems the best customers for such a move would be a unified matscinum community. Right now that community (at least on the Mac) is accessible to the uber-gang only (of which I barely pass the bar, but since I've gotten SOMETHING working, w/much online help, I feel I've earned the right to include myself).

I should mention a friend of mine who knows a lot more about sys stuff than I do had a heck of a time getting all this running on Linux---the entire package: matplotlib, numpy, scipy---lets call it the matscinum suite. In fact, it really seems this integration/installation issue should be addressed across these 3 respective mailing lists, for if one doesn't work, the others become unusable for on non-trivial number of people (Perhaps there should be a list that is a combo of these three?)

FWIW, this is just hot off the macpython sig list:

···

---------

Is it that hard to make a binary to put up (OK, two - one for PPC, one
for Intel), once you've gotten it all built? At least a few people have
gotten it going recently. Could someone please make them available?

How about posting them on your own Web site then? If you don't have
one, the price of commodity Web hosting is virtually zero, and there
are plenty of free open source project hosting services around, some
of them even good.

It's been repeatedly stated on this mailing list that the currently
pythonmac.org maintainer doesn't have time, and would be happy to
relinquish control to someone who did. But nobody's volunteered.

It is really rather tiresome to read about people asking for "someone"
to do something when everyone's doing this work for free.

---------

I agree with the sentiments but also recognize that the documentation at scipy (where most of this stuff is loaded) is inaccurate, e.g. the superpak doesn't work.

Well, that's it for now. I'll try your wx rec's as soon as I find the time (likely next week).

Gotta run and thanks for all the help,

--b