matplotlib without X11 libs?

Hello matplotlib users!

I'm following a plotting turorial for the Pylons web framework[1] which
uses matplotlib (Agg backend). I tried installing matplotlib from the
ports collection on my FreeBSD web server, but it wants to install lots
of X11 related stuff. I don't want that, so I'm wondering if it's
possible to install and use matplotlib in a no-GUI environment.

Please Cc: me, as I'm not on the list. Thanks!