Very low Performance for large interactive interfaces

I have been having some issues with very low plotting speeds for interactive usage even with blitting.
I recently realized that this the speed is heavily related to the size of the figure.
So for example using the code bellow (basically the one available here: Faster rendering by using blitting — Matplotlib 3.7.1 documentation)

import matplotlib.pyplot as plt
import numpy as np
import time
class BlitManager:
    def __init__(self, canvas, animated_artists=()):
        canvas : FigureCanvasAgg
            The canvas to work with, this only works for subclasses of the Agg
            canvas which have the `~FigureCanvasAgg.copy_from_bbox` and
            `~FigureCanvasAgg.restore_region` methods.

        animated_artists : Iterable[Artist]
            List of the artists to manage
        self.canvas = canvas
        self._bg = None
        self._artists = []

        for a in animated_artists:
        # grab the background on every draw
        self.cid = canvas.mpl_connect("draw_event", self.on_draw)

    def on_draw(self, event):
        """Callback to register with 'draw_event'."""
        cv = self.canvas
        if event is not None:
            if event.canvas != cv:
                raise RuntimeError
        self._bg = cv.copy_from_bbox(cv.figure.bbox)

    def add_artist(self, art):
        Add an artist to be managed.

        art : Artist

            The artist to be added.  Will be set to 'animated' (just
            to be safe).  *art* must be in the figure associated with
            the canvas this class is managing.

        if art.figure != self.canvas.figure:
            raise RuntimeError

    def _draw_animated(self):
        """Draw all of the animated artists."""
        fig = self.canvas.figure
        for a in self._artists:

    def update(self):
        """Update the screen with animated artists."""
        cv = self.canvas
        fig = cv.figure
        # paranoia in case we missed the draw event,
        if self._bg is None:
            # restore the background
            # draw all of the animated artists
            # update the GUI state
        # let the GUI event loop process anything it has to do

# make a new figure
fig, ax = plt.subplots()
fig.set_figheight(7), fig.set_figwidth(15)

# add a line
x = np.linspace(0, 2 * np.pi, 1000)
(ln,) = ax.plot(x, np.sin(x), animated=True)
# add a frame number
fr_number = ax.annotate(
    (0, 1),
    xycoords="axes fraction",
    xytext=(10, -10),
    textcoords="offset points",
bm = BlitManager(fig.canvas, [ln, fr_number])
# make sure our window is on the screen and drawn

last = (time.time_ns()//1_000_000)
for j in range(1000):
    # update the artists
    ln.set_ydata(np.sin(x + (j / 100) * np.pi))
    fr_number.set_text("ms since last frame: {j}".format(j=(time.time_ns()//1_000_000 - last)))
    last = (time.time_ns()//1_000_000)
    # tell the blitting manager to do its thing

By comenting or uncomenting the fig.set_figheight(7), fig.set_figwidth(15) line there is a 3x times difference in the time between frames.

In my case I get 30ms per frame with the basic image size and more than 120ms for the 7x15 size.

Is this just the natural way the plotting works? Is there any way to combat or avoid it?