save or pickle figure object

Community matplotlib-users
1

Ted Drain wrote:

We have some experience maintaining persistent object storage over long
periods of time. The best solution we've found is to do something like
this:

- create a read/write method on each class. Every class that needs to be
stored must have this. This includes class you would store (eg Figure) and
things that are member variables of those classes.

- Each class stores a version number along with it's data which represents
the version of the persistent representation for that class. So each class
has its own, internal versioning scheme that represents a specific set of
variables with specific types.

- The read method on each class must check the version number and then read
the appropriate data for that version of itself. Whenever the persistent
representation of the class changes (usually if the member variables
change), you increment the version number. Implicit in this is that if you
change the member variables of a class, the class read method must be able
to convert the variables that existed in the older version of itself into
the new member variables (since that's what the new methods on that class
will be using)

FYI It is possible to use pickle to do this but you can't rely on pickle to
automatically save the member dictionary. You need to implement
__getstate__ and __setstate__ and have them incorporate a version number in
the dictionary they return. In addition, you shouldn't blindly save every
member variable. If member variables can be constructed in terms of other
data, it may be better to store that data and then reconstruct the member
variables in the __setstate__ method.

Using this type of system, you get a hierarchy of objects that each have
their own, internal versioning system. This lets you make changes to a
single class, increment it's version, and update its save/load methods and
it won't affect any other part of the system and still retains backwards
reading capability.

Ted

Sounds good--for some applications--but I would strongly oppose adding this additional level of complexity to mpl. It's just not worth it. If you want to be able to work with a plot, then generate it with a script, and save the data and the script. That is the user's responsibility, not mpl's.

Unless mpl is taken over by a cadre of full-time professional programmers, we have to try to keep it accessible to people who can only work on it sporadically. That means we need to try to keep it simple--indeed, to work on simplifying it and cleaning up the rough edges, and to work on maintaining a design that makes it easy to improve the real plotting capabilities and ease-of-use.

Eric

···

-----Original Message-----
From: matplotlib-users-bounces@lists.sourceforge.net
[mailto:matplotlib-users-bounces@lists.sourceforge.net] On Behalf Of
Eric Firing
Sent: Tuesday, September 16, 2008 7:04 PM
To: John Hunter
Cc: Josef Koller; matplotlib-users@lists.sourceforge.net
Subject: Re: [Matplotlib-users] save or pickle figure object

John Hunter wrote:

On Tue, Sep 16, 2008 at 5:06 PM, Josef Koller <jkoller@...652...> >> wrote:

Hi folks,
I would like to save preliminary figures for later processing and
refinement with matplotlib. Is there a way to save or pickle a

figure

object and later reload it. Matlab has a feature like that and and I

was

wondering if matplotlib has it too.

No, it doesn't exist. We've taken a stab at it once or twice, but
have been stymied because we make extensive use of a python extension
libray CXX, and these objects have resisted our attempts to pickle
them. With our recent transforms refactoring, which removes the
hairiest CXX dependency, it may be worth taking another look, but
noone is currently working on it.

My sense, based on very little experience, is that pickles of
complicated objects are very fragile, so even if we could pickle
figures, I fear it might cause more trouble ("I can't load this
absolutely critical figure I pickled 6 months ago") than it would be
worth.

Eric

-----------------------------------------------------------------------
--
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
Checked by AVG - http://www.avg.com
Version: 8.0.169 / Virus Database: 270.6.21/1674 - Release Date:
9/16/2008 8:15 AM

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

2

I agree completely - I was just pointing that it is possible. I think what
people might not be aware of is that it's really an all or nothing
proposition. You either jump in completely and pay the large cost to handle
this in a maintainable, scalable way or don't do it at all. All of the
"quick and easy" solutions have too many problems and aren't really
maintainable.

Ted

···

-----Original Message-----
From: Eric Firing [mailto:efiring@…202…]
Sent: Wednesday, September 17, 2008 12:01 PM
To: Ted Drain
Cc: matplotlib-users@lists.sourceforge.net
Subject: Re: [Matplotlib-users] save or pickle figure object

Ted Drain wrote:
> We have some experience maintaining persistent object storage over
long
> periods of time. The best solution we've found is to do something
like
> this:
>
> - create a read/write method on each class. Every class that needs
to be
> stored must have this. This includes class you would store (eg
Figure) and
> things that are member variables of those classes.
>
> - Each class stores a version number along with it's data which
represents
> the version of the persistent representation for that class. So each
class
> has its own, internal versioning scheme that represents a specific
set of
> variables with specific types.
>
> - The read method on each class must check the version number and
then read
> the appropriate data for that version of itself. Whenever the
persistent
> representation of the class changes (usually if the member variables
> change), you increment the version number. Implicit in this is that
if you
> change the member variables of a class, the class read method must be
able
> to convert the variables that existed in the older version of itself
into
> the new member variables (since that's what the new methods on that
class
> will be using)
>
> FYI It is possible to use pickle to do this but you can't rely on
pickle to
> automatically save the member dictionary. You need to implement
> __getstate__ and __setstate__ and have them incorporate a version
number in
> the dictionary they return. In addition, you shouldn't blindly save
every
> member variable. If member variables can be constructed in terms of
other
> data, it may be better to store that data and then reconstruct the
member
> variables in the __setstate__ method.
>
> Using this type of system, you get a hierarchy of objects that each
have
> their own, internal versioning system. This lets you make changes to
a
> single class, increment it's version, and update its save/load
methods and
> it won't affect any other part of the system and still retains
backwards
> reading capability.
>
> Ted

Sounds good--for some applications--but I would strongly oppose adding
this additional level of complexity to mpl. It's just not worth it.
If
you want to be able to work with a plot, then generate it with a
script,
and save the data and the script. That is the user's responsibility,
not mpl's.

Unless mpl is taken over by a cadre of full-time professional
programmers, we have to try to keep it accessible to people who can
only
work on it sporadically. That means we need to try to keep it
simple--indeed, to work on simplifying it and cleaning up the rough
edges, and to work on maintaining a design that makes it easy to
improve
the real plotting capabilities and ease-of-use.

Eric

>
>> -----Original Message-----
>> From: matplotlib-users-bounces@lists.sourceforge.net
>> [mailto:matplotlib-users-bounces@lists.sourceforge.net] On Behalf Of
>> Eric Firing
>> Sent: Tuesday, September 16, 2008 7:04 PM
>> To: John Hunter
>> Cc: Josef Koller; matplotlib-users@lists.sourceforge.net
>> Subject: Re: [Matplotlib-users] save or pickle figure object
>>
>> John Hunter wrote:
>>> On Tue, Sep 16, 2008 at 5:06 PM, Josef Koller <jkoller@...652...> > >> wrote:
>>>> Hi folks,
>>>> I would like to save preliminary figures for later processing and
>>>> refinement with matplotlib. Is there a way to save or pickle a
>> figure
>>>> object and later reload it. Matlab has a feature like that and and
I
>> was
>>>> wondering if matplotlib has it too.
>>> No, it doesn't exist. We've taken a stab at it once or twice, but
>>> have been stymied because we make extensive use of a python
extension
>>> libray CXX, and these objects have resisted our attempts to pickle
>>> them. With our recent transforms refactoring, which removes the
>>> hairiest CXX dependency, it may be worth taking another look, but
>>> noone is currently working on it.
>> My sense, based on very little experience, is that pickles of
>> complicated objects are very fragile, so even if we could pickle
>> figures, I fear it might cause more trouble ("I can't load this
>> absolutely critical figure I pickled 6 months ago") than it would be
>> worth.
>>
>> Eric
>>
>> --------------------------------------------------------------------
---
>> --
>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>> Build the coolest Linux based applications with Moblin SDK & win
great
>> prizes
>> Grand prize is a trip for two to an Open Source event anywhere in
the
>> world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> Matplotlib-users mailing list
>> Matplotlib-users@lists.sourceforge.net
>> matplotlib-users List Signup and Options
>> Checked by AVG - http://www.avg.com
>> Version: 8.0.169 / Virus Database: 270.6.21/1674 - Release Date:
>> 9/16/2008 8:15 AM
>
>
> ---------------------------------------------------------------------
----
> This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
> Build the coolest Linux based applications with Moblin SDK & win
great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the
world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Matplotlib-users mailing list
> Matplotlib-users@lists.sourceforge.net
> matplotlib-users List Signup and Options

3

Here is my (easy and maintainable) way to handle the versions for my graphics:

* I write the data creation in a python script
* I write the creation of the mpl graphics in a script (most the same)
* I manage these python file with a version-system (eg mercurial)

So I have different versions for my mpl graphics and I can modify the
graphic any time (with a new python run). If the data creation takes too
long, I could save the data in an extra file (eg pickle file) and
versioning the pickled file.

So I dont need an extra way to reedit a mpl graphic.

By,

  Friedrich

···

On Wed, Sep 17, 2008 at 12:55:40PM -0700, Ted Drain wrote:

I agree completely - I was just pointing that it is possible. I think what
people might not be aware of is that it's really an all or nothing
proposition. You either jump in completely and pay the large cost to handle
this in a maintainable, scalable way or don't do it at all. All of the
"quick and easy" solutions have too many problems and aren't really
maintainable.