I've just uploaded binary wheels for OSX to pypi.
If you're on Python.org python, then this should work
pip install --upgrade pip # if you're not on latest pip already
pip install matplotlib
I mean - you should get the binary wheel for matplotlib.
If fact this wheel will also work for Mac system python or homebrew or
macports if suitably renamed, but I put this current one up now, for
Please do let me know if you have any problems.
(By the way - this should also work in a fresh install / virtualenv,
because numpy is already pip'ed up with wheels).