Here’s a proof-of-concept PR that uses fonttools to subset TrueType fonts:
The fonttools library is a nontrivial dependency, but not huge either. It could also replace our type1font parsing code. It doesn’t have code for subsetting Type-1 fonts, but the subsetting support would probably be easier to write using fonttools. It supports OpenType and CFF fonts too, so it probably wouldn’t be too much work to add some kind of support for those in the vector backends. I don’t think it can replace FreeType for the bitmap backends, though.
Do you think it would be worth it to spend more time on this? I.e. would a more polished PR have a chance of getting merged?