If you build from source, you should only need libpng and libfreetype2, in addition to Python and its headers. Agg is included in matplotlib's source distribution -- if you have a version of Agg installed elsewhere it won't be used. (The SVN trunk also requires Numpy).
The ports spec is probably pulling in one or more GUI frameworks, and thus X11 stuff (which is probably not unreasonable for most users). It may make sense for those dependencies to be specified as "optional" somehow... but that's really a BSD packaging issue.
Cheers,
Mike