Where should I store custom cycler objects?

I have a package that provides a number of custom data annotation visualizers, which are more or less intended to act similarly to standard matplotlib artists (plot, scatter, etc). For example, a common use pattern looks something like:

>>> ax = plt.gca()
>>> mypackage.plot_annotation(reference_data, ax=ax)
>>> mypackage.plot_annotation(estimate_data, ax=ax)

which adds two visualizations to the same axes object. Ideally, these should automatically iterate through a property cycler (just like ax.plot() would) and not require the user to explicitly set style parameters.

In the past, I’ve done some dirty things like hijacking the axes’ line or patches_for_fill cyclers, but that’s both ugly and no longer viable. Clearly I need to be constructing a custom cycler object, but this raises the question: where should the property cycler object live?

I see two options here, and would like some input to help me figure out which is the least bad:

  1. Maintain a global (session-level) hash mapping axes to my custom cycler object.
  2. Hack the custom cycler object into the axes object as a new attribute, eg ax.my_fancy_cyler = ...

I have a slight preference toward option #2, if only because it’s one fewer thing to manage from my side. However, I also recognize that adding attributes into someone else’s objects could backfire, but I’m not sure how likely that is to happen. Any thoughts or recommendations? Is there a viable third option that I’m not seeing?

I sank probably more time into this than I should have, and have not yet come to what I would consider a viable solution here. Notes so far:

Approach 1: self-managed registry

This on the surface seems like the least bad idea, as it forces us to play by the rules.

The first problem I hit with this is that maintaining a dictionary of axes objects could lead to some pretty severe memory bloat over time (eg generating many plots in an interactive session), unless we have some way of managing garbage collection when axes are destroyed. This is a pretty common usage for the code in question, so letting the registry run wild is not a viable option.

In principle, we could use a weakref.WeakKeyDictionary to manage this, and it should clean up automatically when the key (axes object) is removed - however, this fails because axes objects are not properly hashable. (I attempted several workarounds via proxy hashes, but nothing panned out here.)

The only alternative I can see to this would be to register a callback function that removes the key from the dict when the figure is destroyed, but that would preclude any other callback functions from being attached to this signal.

Approach 2: make a sandbox in the axes object

This “works”, as far as I can tell, but as stated above, it’s obviously bad form. I was specifically worried about things like marshalling and serialization here. While these don’t seem likely to be implemented any time soon (judging from the status of MEP25 and other discussions of serialization in the issues), I’d still prefer to avoid this solution if at all possible.

Can anyone out there opine on the above, confirm if my thinking is correct, or suggest alternatives I haven’t considered?

however, this fails because axes objects are not properly hashable. (I attempted several workarounds via proxy hashes, but nothing panned out here.)

Can you say more about this? You should be able to use Axes objects as keys in a dictionary (and quick testing with 3.8.4 works).

Perhaps comment on [ENH]: Provide a standard place to manually register widgets/animations on a figure/artist/etc. · Issue #26881 · matplotlib/matplotlib · GitHub where I suggest it may be nice if artists provided a way to attach arbitrary objects to them.

Yeah, this works well enough in a vanilla dict, but that’s going to leave dangling references to axes objects after figures are destroyed. (This is what I’d like to avoid.)

The idea I had was to use a weakkeydict, keyed on weakrefs to axes objects, so that it would clean up after itself when axes objects are destroyed. This is the part that doesn’t seem to work, but perhaps I was doing something wrong. I’ll try to take another stab at this.

That does sound very related! I think for my purposes it would be more useful to attach to axes instead of artists, but the general idea is the same. I’ll try to summarize things concisely and comment over there as well

Ok, I think I’ve convinced myself that I wasn’t doing anything wrong here (took a minute to page this back into memory).

Here’s a small, self-contained example of using weakkeydict on a simple object (thanks chatgpt):

import weakref

class MyClass:
    pass

# Create an instance of MyClass
obj = MyClass()

# Create a WeakKeyDictionary
weak_dict = weakref.WeakKeyDictionary()

# Add the object as a key to the dictionary
weak_dict[obj] = "Some data"

# Check the content of the dictionary
print("Before deletion:", dict(weak_dict))

# Delete the original object
del obj

# Check the content of the dictionary again
print("After deletion:", dict(weak_dict))

Running this behaves as expected:

Before deletion: {<main.MyClass object at 0x7aa50ce64990>: ‘Some data’}
After deletion: {}

Now, if instead of obj = MyClass, I do something like:

fig, obj = plt.subplots()
...   # Same setup as above
del obj  # purge the axes object as before
del fig  # purge the figure to remove refs
plt.close('all')  # I'm paranoid :)

the result is:

Before deletion: {<Axes: >: ‘Some data’}
After deletion: {<Axes: >: ‘Some data’}

Near as I can tell, this should work, but I’ll admit that it’s quite possible that I’m missing something. Are there other refs to the axes object that could be keeping this alive? Or is there something that needs to be added to the axes teardown (or a custom finalize())?

Aha, I think I’ve got this sorted out now.

Deleting the axes object and the figure wasn’t enough, but calling fig.clear(), deleting the object, and forcing a garbage collection run does do the job and the weakkeydict is properly empty afterward:

# Create a plot
fig, ax = plt.subplots();

D = weakref.WeakKeyDictionary()
D[ax] = 'foo'
print("Before deletion: ", dict(D))
fig.clear()

del ax

gc.collect()
print("After deletion: ", dict(D))

Before deletion:
{<Axes: >: ‘foo’}
After deletion:
{}

All three steps are necessary, it seems; leaving any one out results in the weakkeydictionary holding a weak ref to ax in its key set.

I think this is now converging on an answerable question for @tacaswell then: what’s the right way to ensure proper figure cleanup? I’m okay with being lazy about garbage collection (forcing it here just makes it easy to verify that things work as expected), but it seems strange to me that deleting the figure wasn’t enough to achieve the effect, and that it also needed to be cleared.

@anntzer.lee if this weakkeydict pattern works in general (once the triggers around cleanup are sorted out), it might be an alternative way to think about the issue you linked earlier to attach sandboxes to artists and other objects.

Ah, yes weakrefs only drop when there are no hard references to the object left. If you are using pyplot, then Matplotlib holds on to a reference to the Figure object (both that plt.gcf and plt.gca work and so that the GUI objects don’t get gc’d while you are using them (which results in the widows disappearing under you!)) and because the the artist tree has oodles of circular references (Figures know what Axes they hold, Axes know what Artists they hold and Artists know both what Axes and Figure they are in). This means to get objects to fully drop you need: make sure you drop all references (either by letting them go out of scope or explicit del) make sure pyplot no longer knows about it plt.close(fig) should do the trick there), and make sure that a gc has run. See [Bug]: Memory not freed as expected after plotting heavy plot involving looping · Issue #27138 · matplotlib/matplotlib · GitHub for a much longer version of this description.

For simplifying pyplot interaciton see GitHub - matplotlib/mpl-gui: Prototype for mpl-gui module (which needs some more cycles from me to finish) and for simplify the circular references see @ksunden 's work in GitHub - matplotlib/data-prototype (which needs a lot more work).

Thanks for sharing this imformation this is helpful me @tacaswell :slightly_smiling_face:

Hello,
Thanks for sharing this is very helpful for me.