It would probably be considerable work to ply mathtext out of matplotlib, particularly if you consider bringing along the Ps/Pdf/Svg/Cairo backends. Just bringing the raster backend (which is really in ft2font.cpp) would be considerably less work.
But I was mainly just sharing a "wouldn't it be cool if...". Separating mathtext would be a project unto itself.
Cheers,
Mike