High priority for 3.2: figure cleanup

In today's weekly meeting, with Hannah, Thomas, and myself, we discussed
https://github.com/matplotlib/matplotlib/issues/8519, which points to
the long-standing problem that matplotlib relies on garbage collection
to release memory after a figure is closed. With some work-flows the
automatic collection doesn't happen often enough and the user must
explicitly trigger it. It would be better if pyplot cleaned up after
itself, breaking reference cycles when a figure is closed. I think it
is time to make this change, so I marked the issue "Release-critical".
I am hoping we can get this done as a high-priority task, *early* in the
3.2 cycle.

The central point is that closing a figure should clear it, and clearing
it should recursively walk through the children, breaking connections
and reference cycles all the way down. One possibility would be to
modify or replace the figure object such that any attempt to operate on
the figure after closing would fail by raising an informative exception.

Question: should closing the gui window also close the figure? No,
because that would prevent the user from calling savefig on it. This
would interfere with a reasonable use-case, in which plt.show is used in
blocking mode, the window is closed, and the figure is saved via
savefig. Problem: if closing the gui window doesn't close the figure,
how can it ever be closed?

This leads to the suggestion that the pyplot state manager keep a list
of "latent figures" which are no longer displayed in a gui window but
have not been closed. They could then be explicitly closed by
"plt.close('all')". This also opens the possibility for an API allowing
one to re-display a figure after closing its gui window. For example,
one might do a calculation, show it in a figure in a blocking window
with "plt.show", close the window, do additional calculations, add them
to the same figure, and bring the figure back in a new gui window. This
might be done in a loop, incrementing the contents of the figure.

See also:
https://github.com/matplotlib/matplotlib/issues/6793
https://github.com/matplotlib/matplotlib/issues/6982

Eric

I think if it makes our lives easier the paradigm that a closed window means the figure is closed programmatically is pretty straightforward and standard across GUI interfaces. If folks need a workflow where they access the figure while it?s open they can turn off blocking. I vote we keep it simple and predictable rather than tie ourselves in knots with complicated ways of keeping track of what we think the user may want to do.

Cheers. Jody

···

Sent from my iPhone

On Apr 16, 2019, at 12:19 AM, Eric Firing <efiring at hawaii.edu> wrote:

In today's weekly meeting, with Hannah, Thomas, and myself, we discussed Closed figures linger in memory · Issue #8519 · matplotlib/matplotlib · GitHub, which points to the long-standing problem that matplotlib relies on garbage collection to release memory after a figure is closed. With some work-flows the automatic collection doesn't happen often enough and the user must explicitly trigger it. It would be better if pyplot cleaned up after itself, breaking reference cycles when a figure is closed. I think it is time to make this change, so I marked the issue "Release-critical". I am hoping we can get this done as a high-priority task, *early* in the 3.2 cycle.

The central point is that closing a figure should clear it, and clearing it should recursively walk through the children, breaking connections and reference cycles all the way down. One possibility would be to modify or replace the figure object such that any attempt to operate on the figure after closing would fail by raising an informative exception.

Question: should closing the gui window also close the figure? No, because that would prevent the user from calling savefig on it. This would interfere with a reasonable use-case, in which plt.show is used in blocking mode, the window is closed, and the figure is saved via savefig. Problem: if closing the gui window doesn't close the figure, how can it ever be closed?

This leads to the suggestion that the pyplot state manager keep a list of "latent figures" which are no longer displayed in a gui window but have not been closed. They could then be explicitly closed by "plt.close('all')". This also opens the possibility for an API allowing one to re-display a figure after closing its gui window. For example, one might do a calculation, show it in a figure in a blocking window with "plt.show", close the window, do additional calculations, add them to the same figure, and bring the figure back in a new gui window. This might be done in a loop, incrementing the contents of the figure.

See also:
artist.axes and .figure should never be reassignable · Issue #6793 · matplotlib/matplotlib · GitHub
cla(), clf() should unset the `.axes` and `.figure` attributes of deparented artists · Issue #6982 · matplotlib/matplotlib · GitHub

Eric
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel at python.org
Matplotlib-devel Info Page

Hi,
It's actually not clear to me how GUI-closing and really closing are
related (unless we decided that GUI closing means teardown, but I don't
think we *need* that), and what the "latent figures" concept is supposed to
achieve.
Doesn't this approximately come down to
1) when calling plt.close(fig), do fig.clf() -> ax.cla()
2) in ax.cla(), before doing ax.lines = , first do for line in ax.lines:
line.remove()? (and similarly for other artists)
Also, note that if one just wants to dump figures to your hard drive in a
loop, I would strongly suggest not bothering with pyplot as all and just do
"fig = matplotlib.figure.Figure(); ax = fig.subplots(); etc." The main
issue with this approach is that it's a bit awkward to pop up a figure if
one decides, after all, that they want to interactively play with it; a
possibility would be to allow pyplot to "adopt" figures in a post-hoc
manner via plt.show([fig1, fig2, ...]) (
Doc event loop requirements for Figure.show by timhoffm · Pull Request #13590 · matplotlib/matplotlib · GitHub).
Antony

···

On Tue, Apr 16, 2019 at 5:41 PM Jody Klymak <jklymak at uvic.ca> wrote:

I think if it makes our lives easier the paradigm that a closed window
means the figure is closed programmatically is pretty straightforward and
standard across GUI interfaces. If folks need a workflow where they access
the figure while it?s open they can turn off blocking. I vote we keep it
simple and predictable rather than tie ourselves in knots with complicated
ways of keeping track of what we think the user may want to do.

Cheers. Jody

Sent from my iPhone

> On Apr 16, 2019, at 12:19 AM, Eric Firing <efiring at hawaii.edu> wrote:
>
>
> In today's weekly meeting, with Hannah, Thomas, and myself, we discussed
Closed figures linger in memory · Issue #8519 · matplotlib/matplotlib · GitHub, which points to the
long-standing problem that matplotlib relies on garbage collection to
release memory after a figure is closed. With some work-flows the
automatic collection doesn't happen often enough and the user must
explicitly trigger it. It would be better if pyplot cleaned up after
itself, breaking reference cycles when a figure is closed. I think it is
time to make this change, so I marked the issue "Release-critical". I am
hoping we can get this done as a high-priority task, *early* in the 3.2
cycle.
>
> The central point is that closing a figure should clear it, and clearing
it should recursively walk through the children, breaking connections and
reference cycles all the way down. One possibility would be to modify or
replace the figure object such that any attempt to operate on the figure
after closing would fail by raising an informative exception.
>
> Question: should closing the gui window also close the figure? No,
because that would prevent the user from calling savefig on it. This would
interfere with a reasonable use-case, in which plt.show is used in blocking
mode, the window is closed, and the figure is saved via savefig. Problem:
if closing the gui window doesn't close the figure, how can it ever be
closed?
>
> This leads to the suggestion that the pyplot state manager keep a list
of "latent figures" which are no longer displayed in a gui window but have
not been closed. They could then be explicitly closed by
"plt.close('all')". This also opens the possibility for an API allowing
one to re-display a figure after closing its gui window. For example, one
might do a calculation, show it in a figure in a blocking window with
"plt.show", close the window, do additional calculations, add them to the
same figure, and bring the figure back in a new gui window. This might be
done in a loop, incrementing the contents of the figure.
>
> See also:
> artist.axes and .figure should never be reassignable · Issue #6793 · matplotlib/matplotlib · GitHub
> cla(), clf() should unset the `.axes` and `.figure` attributes of deparented artists · Issue #6982 · matplotlib/matplotlib · GitHub
>
> Eric
> _______________________________________________
> Matplotlib-devel mailing list
> Matplotlib-devel at python.org
> Matplotlib-devel Info Page

_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel at python.org
Matplotlib-devel Info Page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/matplotlib-devel/attachments/20190416/548ff5e6/attachment.html&gt;

I don't think the ability to save after closing the GUI window has always
worked. In fact, I have always taught students to perform a savefig prior
to doing `plt.show()` if you want to programmatically save a figure and
display it. I'd be perfectly fine with killing off that mis-feature. To me,
closing a figure window should be equivalent to calling `fig.close()`. If
you wanted to save the figure after closing the gui window, you could
easily do it from the save file button.

···

On Tue, Apr 16, 2019 at 12:02 PM Antony Lee <antony.lee at institutoptique.fr> wrote:

Hi,
It's actually not clear to me how GUI-closing and really closing are
related (unless we decided that GUI closing means teardown, but I don't
think we *need* that), and what the "latent figures" concept is supposed to
achieve.
Doesn't this approximately come down to
1) when calling plt.close(fig), do fig.clf() -> ax.cla()
2) in ax.cla(), before doing ax.lines = , first do for line in ax.lines:
line.remove()? (and similarly for other artists)
Also, note that if one just wants to dump figures to your hard drive in a
loop, I would strongly suggest not bothering with pyplot as all and just do
"fig = matplotlib.figure.Figure(); ax = fig.subplots(); etc." The main
issue with this approach is that it's a bit awkward to pop up a figure if
one decides, after all, that they want to interactively play with it; a
possibility would be to allow pyplot to "adopt" figures in a post-hoc
manner via plt.show([fig1, fig2, ...]) (
Doc event loop requirements for Figure.show by timhoffm · Pull Request #13590 · matplotlib/matplotlib · GitHub
).
Antony

On Tue, Apr 16, 2019 at 5:41 PM Jody Klymak <jklymak at uvic.ca> wrote:

I think if it makes our lives easier the paradigm that a closed window
means the figure is closed programmatically is pretty straightforward and
standard across GUI interfaces. If folks need a workflow where they access
the figure while it?s open they can turn off blocking. I vote we keep it
simple and predictable rather than tie ourselves in knots with complicated
ways of keeping track of what we think the user may want to do.

Cheers. Jody

Sent from my iPhone

> On Apr 16, 2019, at 12:19 AM, Eric Firing <efiring at hawaii.edu> wrote:
>
>
> In today's weekly meeting, with Hannah, Thomas, and myself, we
discussed Closed figures linger in memory · Issue #8519 · matplotlib/matplotlib · GitHub, which
points to the long-standing problem that matplotlib relies on garbage
collection to release memory after a figure is closed. With some
work-flows the automatic collection doesn't happen often enough and the
user must explicitly trigger it. It would be better if pyplot cleaned up
after itself, breaking reference cycles when a figure is closed. I think
it is time to make this change, so I marked the issue "Release-critical". I
am hoping we can get this done as a high-priority task, *early* in the 3.2
cycle.
>
> The central point is that closing a figure should clear it, and
clearing it should recursively walk through the children, breaking
connections and reference cycles all the way down. One possibility would
be to modify or replace the figure object such that any attempt to operate
on the figure after closing would fail by raising an informative exception.
>
> Question: should closing the gui window also close the figure? No,
because that would prevent the user from calling savefig on it. This would
interfere with a reasonable use-case, in which plt.show is used in blocking
mode, the window is closed, and the figure is saved via savefig. Problem:
if closing the gui window doesn't close the figure, how can it ever be
closed?
>
> This leads to the suggestion that the pyplot state manager keep a list
of "latent figures" which are no longer displayed in a gui window but have
not been closed. They could then be explicitly closed by
"plt.close('all')". This also opens the possibility for an API allowing
one to re-display a figure after closing its gui window. For example, one
might do a calculation, show it in a figure in a blocking window with
"plt.show", close the window, do additional calculations, add them to the
same figure, and bring the figure back in a new gui window. This might be
done in a loop, incrementing the contents of the figure.
>
> See also:
> artist.axes and .figure should never be reassignable · Issue #6793 · matplotlib/matplotlib · GitHub
> cla(), clf() should unset the `.axes` and `.figure` attributes of deparented artists · Issue #6982 · matplotlib/matplotlib · GitHub
>
> Eric
> _______________________________________________
> Matplotlib-devel mailing list
> Matplotlib-devel at python.org
> Matplotlib-devel Info Page

_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel at python.org
Matplotlib-devel Info Page

_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel at python.org
Matplotlib-devel Info Page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/matplotlib-devel/attachments/20190416/cb195e0b/attachment-0001.html&gt;

Some of the tools in

https://medium.com/zendesk-engineering/hunting-for-memory-leaks-in-python-applications-6824d0518774

May be useful for working on this.

Tom

···

On Tue, Apr 16, 2019, 20:23 Benjamin Root <ben.v.root at gmail.com> wrote:

I don't think the ability to save after closing the GUI window has always
worked. In fact, I have always taught students to perform a savefig prior
to doing `plt.show()` if you want to programmatically save a figure and
display it. I'd be perfectly fine with killing off that mis-feature. To me,
closing a figure window should be equivalent to calling `fig.close()`. If
you wanted to save the figure after closing the gui window, you could
easily do it from the save file button.

On Tue, Apr 16, 2019 at 12:02 PM Antony Lee <antony.lee at institutoptique.fr> > wrote:

Hi,
It's actually not clear to me how GUI-closing and really closing are
related (unless we decided that GUI closing means teardown, but I don't
think we *need* that), and what the "latent figures" concept is supposed to
achieve.
Doesn't this approximately come down to
1) when calling plt.close(fig), do fig.clf() -> ax.cla()
2) in ax.cla(), before doing ax.lines = , first do for line in
ax.lines: line.remove()? (and similarly for other artists)
Also, note that if one just wants to dump figures to your hard drive in a
loop, I would strongly suggest not bothering with pyplot as all and just do
"fig = matplotlib.figure.Figure(); ax = fig.subplots(); etc." The main
issue with this approach is that it's a bit awkward to pop up a figure if
one decides, after all, that they want to interactively play with it; a
possibility would be to allow pyplot to "adopt" figures in a post-hoc
manner via plt.show([fig1, fig2, ...]) (
Doc event loop requirements for Figure.show by timhoffm · Pull Request #13590 · matplotlib/matplotlib · GitHub
).
Antony

On Tue, Apr 16, 2019 at 5:41 PM Jody Klymak <jklymak at uvic.ca> wrote:

I think if it makes our lives easier the paradigm that a closed window
means the figure is closed programmatically is pretty straightforward and
standard across GUI interfaces. If folks need a workflow where they access
the figure while it?s open they can turn off blocking. I vote we keep it
simple and predictable rather than tie ourselves in knots with complicated
ways of keeping track of what we think the user may want to do.

Cheers. Jody

Sent from my iPhone

> On Apr 16, 2019, at 12:19 AM, Eric Firing <efiring at hawaii.edu> wrote:
>
>
> In today's weekly meeting, with Hannah, Thomas, and myself, we
discussed Closed figures linger in memory · Issue #8519 · matplotlib/matplotlib · GitHub, which
points to the long-standing problem that matplotlib relies on garbage
collection to release memory after a figure is closed. With some
work-flows the automatic collection doesn't happen often enough and the
user must explicitly trigger it. It would be better if pyplot cleaned up
after itself, breaking reference cycles when a figure is closed. I think
it is time to make this change, so I marked the issue "Release-critical". I
am hoping we can get this done as a high-priority task, *early* in the 3.2
cycle.
>
> The central point is that closing a figure should clear it, and
clearing it should recursively walk through the children, breaking
connections and reference cycles all the way down. One possibility would
be to modify or replace the figure object such that any attempt to operate
on the figure after closing would fail by raising an informative exception.
>
> Question: should closing the gui window also close the figure? No,
because that would prevent the user from calling savefig on it. This would
interfere with a reasonable use-case, in which plt.show is used in blocking
mode, the window is closed, and the figure is saved via savefig. Problem:
if closing the gui window doesn't close the figure, how can it ever be
closed?
>
> This leads to the suggestion that the pyplot state manager keep a list
of "latent figures" which are no longer displayed in a gui window but have
not been closed. They could then be explicitly closed by
"plt.close('all')". This also opens the possibility for an API allowing
one to re-display a figure after closing its gui window. For example, one
might do a calculation, show it in a figure in a blocking window with
"plt.show", close the window, do additional calculations, add them to the
same figure, and bring the figure back in a new gui window. This might be
done in a loop, incrementing the contents of the figure.
>
> See also:
> artist.axes and .figure should never be reassignable · Issue #6793 · matplotlib/matplotlib · GitHub
> cla(), clf() should unset the `.axes` and `.figure` attributes of deparented artists · Issue #6982 · matplotlib/matplotlib · GitHub
>
> Eric
> _______________________________________________
> Matplotlib-devel mailing list
> Matplotlib-devel at python.org
> Matplotlib-devel Info Page

_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel at python.org
Matplotlib-devel Info Page

_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel at python.org
Matplotlib-devel Info Page

_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel at python.org
Matplotlib-devel Info Page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/matplotlib-devel/attachments/20190420/9aeaf336/attachment.html&gt;

I think that this cleanup can be approached in parts, some of which
involve refactoring so as to reduce the number of circular references.
Looking at ticker.py, for example, I suspect that a cleaner
implementation would eliminate the need for a Formatter or Locator to
have an actual or dummy Axis as an attribute. Instead, the Axis
reference, or, ideally, the information actually needed from an Axis,
could be passed as kwargs to the methods that require that information.

Can we get there from here, or would the
deprecation/backwards-compatibility dance be too exhausting?

Eric

···

On 2019/04/20 12:44 PM, Thomas Caswell wrote:

Some of the tools in

Hunting for Memory Leaks in Python applications | by Wai Chee Yau | Zendesk Engineering

May be useful for working on this.

Tom

On Tue, Apr 16, 2019, 20:23 Benjamin Root <ben.v.root at gmail.com > <mailto:ben.v.root at gmail.com>> wrote:

    I don't think the ability to save after closing the GUI window has
    always worked. In fact, I have always taught students to perform a
    savefig prior to doing `plt.show()` if you want to programmatically
    save a figure and display it. I'd be perfectly fine with killing off
    that mis-feature. To me, closing a figure window should be
    equivalent to calling `fig.close()`. If you wanted to save the
    figure after closing the gui window, you could easily do it from the
    save file button.

    On Tue, Apr 16, 2019 at 12:02 PM Antony Lee > <antony.lee at institutoptique.fr > <mailto:antony.lee at institutoptique.fr>> wrote:

        Hi,
        It's actually not clear to me how GUI-closing and really closing
        are related (unless we decided that GUI closing means teardown,
        but I don't think we *need* that), and what the "latent figures"
        concept is supposed to achieve.
        Doesn't this approximately come down to
        1) when calling plt.close(fig), do fig.clf() -> ax.cla()
        2) in ax.cla(), before doing ax.lines = , first do for line in
        ax.lines: line.remove()? (and similarly for other artists)
        Also, note that if one just wants to dump figures to your hard
        drive in a loop, I would strongly suggest not bothering with
        pyplot as all and just do "fig = matplotlib.figure.Figure(); ax
        = fig.subplots(); etc."? The main issue with this approach is
        that it's a bit awkward to pop up a figure if one decides, after
        all, that they want to interactively play with it; a possibility
        would be to allow pyplot to "adopt" figures in a post-hoc manner
        via plt.show([fig1, fig2, ...])
        (Doc event loop requirements for Figure.show by timhoffm · Pull Request #13590 · matplotlib/matplotlib · GitHub).
        Antony

        On Tue, Apr 16, 2019 at 5:41 PM Jody Klymak <jklymak at uvic.ca > <mailto:jklymak at uvic.ca>> wrote:

            I think if it makes our lives easier the paradigm that a
            closed window means the figure is closed programmatically is
            pretty straightforward and standard across GUI interfaces.
            If folks need a workflow where they access the figure while
            it?s open they can turn off blocking.? I vote we keep it
            simple and predictable rather than tie ourselves in knots
            with complicated ways of keeping track of what we think the
            user may want to do.

            Cheers.? Jody

            Sent from my iPhone

             > On Apr 16, 2019, at 12:19 AM, Eric Firing > <efiring at hawaii.edu <mailto:efiring at hawaii.edu>> wrote:
             >
             >
             > In today's weekly meeting, with Hannah, Thomas, and
            myself, we discussed
            Closed figures linger in memory · Issue #8519 · matplotlib/matplotlib · GitHub, which
            points to the long-standing problem that matplotlib relies
            on garbage collection to release memory after a figure is
            closed.? With some work-flows the automatic collection
            doesn't happen often enough and the user must explicitly
            trigger it.? It would be better if pyplot cleaned up after
            itself, breaking reference cycles when a figure is closed.
            I think it is time to make this change, so I marked the
            issue "Release-critical". I am hoping we can get this done
            as a high-priority task, *early* in the 3.2 cycle.
             >
             > The central point is that closing a figure should clear
            it, and clearing it should recursively walk through the
            children, breaking connections and reference cycles all the
            way down.? One possibility would be to modify or replace the
            figure object such that any attempt to operate on the figure
            after closing would fail by raising an informative exception.
             >
             > Question: should closing the gui window also close the
            figure?? No, because that would prevent the user from
            calling savefig on it.? This would interfere with a
            reasonable use-case, in which plt.show is used in blocking
            mode, the window is closed, and the figure is saved via
            savefig.? Problem: if closing the gui window doesn't close
            the figure, how can it ever be closed?
             >
             > This leads to the suggestion that the pyplot state
            manager keep a list of "latent figures" which are no longer
            displayed in a gui window but have not been closed.? They
            could then be explicitly closed by "plt.close('all')".? This
            also opens the possibility for an API allowing one to
            re-display a figure after closing its gui window. For
            example, one might do a calculation, show it in a figure in
            a blocking window with "plt.show", close the window, do
            additional calculations, add them to the same figure, and
            bring the figure back in a new gui window.? This might be
            done in a loop, incrementing the contents of the figure.
             >
             > See also:
             > artist.axes and .figure should never be reassignable · Issue #6793 · matplotlib/matplotlib · GitHub
             > cla(), clf() should unset the `.axes` and `.figure` attributes of deparented artists · Issue #6982 · matplotlib/matplotlib · GitHub
             >
             > Eric
             > _______________________________________________
             > Matplotlib-devel mailing list
             > Matplotlib-devel at python.org
            <mailto:Matplotlib-devel at python.org>
             > Matplotlib-devel Info Page

            _______________________________________________
            Matplotlib-devel mailing list
            Matplotlib-devel at python.org <mailto:Matplotlib-devel at python.org>
            Matplotlib-devel Info Page

        _______________________________________________
        Matplotlib-devel mailing list
        Matplotlib-devel at python.org <mailto:Matplotlib-devel at python.org>
        Matplotlib-devel Info Page

    _______________________________________________
    Matplotlib-devel mailing list
    Matplotlib-devel at python.org <mailto:Matplotlib-devel at python.org>
    Matplotlib-devel Info Page

_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel at python.org
Matplotlib-devel Info Page