I’ve used some hacky tricks to get around this, which mostly involve downsampling the image on the fly based on screen resolution. One such effort is at https://github.com/ChrisBeaumont/mpl-modest-image.
I tried your code for plotting 4kx4k image and it is another significant improvement. Originally it took 300 MB then it was reduced to 190 MB with uint8 type and using your ModestImage class it takes 70-100 MB depending on size of window.
That is much better!