Exciting stuff!
The latency would be an important factor for the user experience, but this neatly sidesteps a lot of the JS issues. This would keep the main matplotlib machinery on the Python side, which is great.
We could still do simple high-speed annotations requiring very low latency such as cursors or selection rectangles on the JS side. I therefore don't foresee the requirement of sending hundreds or thousands of PNGs during e.g. a simple zoom (smooth resizing might be more intensive...).
Ironically, one of the motivations for the mplh5canvas backend is also to send *less* data over the network, as matplotlib's path simplification would effectively compress large data sets. However, path simplification does not yet apply to scatter plots, an important use case for us (although there are ways to implement that with clustering algorithms).
Obviously, a large number of vector operations could still end up being more data than a PNG file, where this idea will work better. I'm very keen to see how this pans out!
Regards,
Ludwig