free'ing the plotted data?

I'm currently trying to improve my app to render more and more data.
I'm wondering if matplotlib keeps an internal copy of the datas once the
plot function has been called?

from what I understand from python (I'm quite new to python), everything
is reference and no implicit copies are done. I'm trying to check on
matplotlib if it's the case but I must admit I got some difficulties xD

to sum it up: can I del the arrays I just passed to an axis.plot ?

thanks for your help :slight_smile:

路路路

----
This message contains confidential information and may contain information that is legally privileged. If you have received this message by mistake, please immediately notify us and delete the original message. Thank you.

Ce message contient des informations confidentielles. S'il vous est parvenu par erreur, merci de bien vouloir nous en aviser par retour, de n'en faire aucun usage et de n'en garder aucune copie.
----

Yes, Python is reference counted (with a garbage collector to handle cycles etc.). So, yes, the matplotlib plot keeps a "reference" to your data after you pass it in so it continue to redraw it as the plot is panned/zoomed/resized etc. You can safely del your local reference, but the memory itself will not be freed, since matplotlib still needs it.

Mike

guillaume ranquet wrote:

路路路

I'm currently trying to improve my app to render more and more data.
I'm wondering if matplotlib keeps an internal copy of the datas once the
plot function has been called?

from what I understand from python (I'm quite new to python), everything
is reference and no implicit copies are done. I'm trying to check on
matplotlib if it's the case but I must admit I got some difficulties xD

to sum it up: can I del the arrays I just passed to an axis.plot ?

thanks for your help :slight_smile:
----
This message contains confidential information and may contain information that is legally privileged. If you have received this message by mistake, please immediately notify us and delete the original message. Thank you.

Ce message contient des informations confidentielles. S'il vous est parvenu par erreur, merci de bien vouloir nous en aviser par retour, de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com _______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
  
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Thanks Mike,
I just wanted to be sure wether matplotlib had a copy or not.
I'll have to work on giving hints to the GC then...

guillaume.

Michael Droettboom wrote:

路路路

Yes, Python is reference counted (with a garbage collector to handle
cycles etc.). So, yes, the matplotlib plot keeps a "reference" to your
data after you pass it in so it continue to redraw it as the plot is
panned/zoomed/resized etc. You can safely del your local reference, but
the memory itself will not be freed, since matplotlib still needs it.

Mike

guillaume ranquet wrote:

I'm currently trying to improve my app to render more and more data.
I'm wondering if matplotlib keeps an internal copy of the datas once the
plot function has been called?

from what I understand from python (I'm quite new to python), everything
is reference and no implicit copies are done. I'm trying to check on
matplotlib if it's the case but I must admit I got some difficulties xD

to sum it up: can I del the arrays I just passed to an axis.plot ?

thanks for your help :slight_smile:
----
This message contains confidential information and may contain
information that is legally privileged. If you have received this
message by mistake, please immediately notify us and delete the
original message. Thank you.
Ce message contient des informations confidentielles. S'il vous est
parvenu par erreur, merci de bien vouloir nous en aviser par retour,
de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------

Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity
professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like
Barbarian Group, R/GA, & Big Spaceship.
http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
  
----
This message contains confidential information and may contain information that is legally privileged. If you have received this message by mistake, please immediately notify us and delete the original message. Thank you.

Ce message contient des informations confidentielles. S'il vous est parvenu par erreur, merci de bien vouloir nous en aviser par retour, de n'en faire aucun usage et de n'en garder aucune copie.
----

I don't follow... If you need to free the data, you have to ensure that matplotlib doesn't need it, and the only way to do that is destroy the figure that references it. It's virtually impossible to create a dangling reference in Python (from within Python).

Mike

guillaume ranquet wrote:

路路路

Thanks Mike,
I just wanted to be sure wether matplotlib had a copy or not.
I'll have to work on giving hints to the GC then...

guillaume.

Michael Droettboom wrote:
  

Yes, Python is reference counted (with a garbage collector to handle
cycles etc.). So, yes, the matplotlib plot keeps a "reference" to your
data after you pass it in so it continue to redraw it as the plot is
panned/zoomed/resized etc. You can safely del your local reference, but
the memory itself will not be freed, since matplotlib still needs it.

Mike

guillaume ranquet wrote:
    

I'm currently trying to improve my app to render more and more data.
I'm wondering if matplotlib keeps an internal copy of the datas once the
plot function has been called?

from what I understand from python (I'm quite new to python), everything
is reference and no implicit copies are done. I'm trying to check on
matplotlib if it's the case but I must admit I got some difficulties xD

to sum it up: can I del the arrays I just passed to an axis.plot ?

thanks for your help :slight_smile:
----
This message contains confidential information and may contain
information that is legally privileged. If you have received this
message by mistake, please immediately notify us and delete the
original message. Thank you. Ce message contient des informations confidentielles. S'il vous est
parvenu par erreur, merci de bien vouloir nous en aviser par retour,
de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------

Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity
professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like
Barbarian Group, R/GA, & Big Spaceship.
http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
  
----
This message contains confidential information and may contain information that is legally privileged. If you have received this message by mistake, please immediately notify us and delete the original message. Thank you.

Ce message contient des informations confidentielles. S'il vous est parvenu par erreur, merci de bien vouloir nous en aviser par retour, de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com _______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
  
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

sorry If my last message was confusing.
I'm exctracting data from an xml file (100megs of xml) which is stored
by my app as an array of dict in which I pick only some keys to plot.
I'm just trying to figure out if the gc is able to free some parts of
the dict (the un-plotted parts) by itself (and it seems not) or If I
have to give more hints to the gc.
my gig of ram is getting used, and I'd like to increase the set of data
a bit.

I can make a hardcopy of my datas, pass this copy to matplotlib and del
everything (so that it *should* be clean, from my understandings).

actually, my design is quite horrible. prolly a rewrite from scratch is
the best thing to do.

anyway, it's a bit off-topic on the matplotlib user list :slight_smile:

Michael Droettboom wrote:

路路路

I don't follow... If you need to free the data, you have to ensure that
matplotlib doesn't need it, and the only way to do that is destroy the
figure that references it. It's virtually impossible to create a
dangling reference in Python (from within Python).

Mike

guillaume ranquet wrote:

Thanks Mike,
I just wanted to be sure wether matplotlib had a copy or not.
I'll have to work on giving hints to the GC then...

guillaume.

Michael Droettboom wrote:

Yes, Python is reference counted (with a garbage collector to handle
cycles etc.). So, yes, the matplotlib plot keeps a "reference" to your
data after you pass it in so it continue to redraw it as the plot is
panned/zoomed/resized etc. You can safely del your local reference, but
the memory itself will not be freed, since matplotlib still needs it.

Mike

guillaume ranquet wrote:
   

I'm currently trying to improve my app to render more and more data.
I'm wondering if matplotlib keeps an internal copy of the datas once
the
plot function has been called?

from what I understand from python (I'm quite new to python),
everything
is reference and no implicit copies are done. I'm trying to check on
matplotlib if it's the case but I must admit I got some difficulties xD

to sum it up: can I del the arrays I just passed to an axis.plot ?

thanks for your help :slight_smile:
----
This message contains confidential information and may contain
information that is legally privileged. If you have received this
message by mistake, please immediately notify us and delete the
original message. Thank you. Ce message contient des informations
confidentielles. S'il vous est
parvenu par erreur, merci de bien vouloir nous en aviser par retour,
de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------

Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity
professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like
Barbarian Group, R/GA, & Big Spaceship.
http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
        
----
This message contains confidential information and may contain
information that is legally privileged. If you have received this
message by mistake, please immediately notify us and delete the
original message. Thank you.
Ce message contient des informations confidentielles. S'il vous est
parvenu par erreur, merci de bien vouloir nous en aviser par retour,
de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------

Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity
professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like
Barbarian Group, R/GA, & Big Spaceship.
http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
  
----
This message contains confidential information and may contain information that is legally privileged. If you have received this message by mistake, please immediately notify us and delete the original message. Thank you.

Ce message contient des informations confidentielles. S'il vous est parvenu par erreur, merci de bien vouloir nous en aviser par retour, de n'en faire aucun usage et de n'en garder aucune copie.
----

It's not off topic -- and I understand what you're trying to do much better now.

I assume at some point you convert all of the xml into a Numpy array and pass that to matplotlib? matplotlib will only keep a reference to the Numpy array, which should be far more compact, and you can safely remove the references to the xml tree and that memory should be freed.

Cheers,
Mike

guillaume ranquet wrote:

路路路

sorry If my last message was confusing.
I'm exctracting data from an xml file (100megs of xml) which is stored
by my app as an array of dict in which I pick only some keys to plot.
I'm just trying to figure out if the gc is able to free some parts of
the dict (the un-plotted parts) by itself (and it seems not) or If I
have to give more hints to the gc.
my gig of ram is getting used, and I'd like to increase the set of data
a bit.

I can make a hardcopy of my datas, pass this copy to matplotlib and del
everything (so that it *should* be clean, from my understandings).

actually, my design is quite horrible. prolly a rewrite from scratch is
the best thing to do.

anyway, it's a bit off-topic on the matplotlib user list :slight_smile:

Michael Droettboom wrote:
  

I don't follow... If you need to free the data, you have to ensure that
matplotlib doesn't need it, and the only way to do that is destroy the
figure that references it. It's virtually impossible to create a
dangling reference in Python (from within Python).

Mike

guillaume ranquet wrote:
    

Thanks Mike,
I just wanted to be sure wether matplotlib had a copy or not.
I'll have to work on giving hints to the GC then...

guillaume.

Michael Droettboom wrote:

Yes, Python is reference counted (with a garbage collector to handle
cycles etc.). So, yes, the matplotlib plot keeps a "reference" to your
data after you pass it in so it continue to redraw it as the plot is
panned/zoomed/resized etc. You can safely del your local reference, but
the memory itself will not be freed, since matplotlib still needs it.

Mike

guillaume ranquet wrote:
   

I'm currently trying to improve my app to render more and more data.
I'm wondering if matplotlib keeps an internal copy of the datas once
the
plot function has been called?

from what I understand from python (I'm quite new to python),
everything
is reference and no implicit copies are done. I'm trying to check on
matplotlib if it's the case but I must admit I got some difficulties xD

to sum it up: can I del the arrays I just passed to an axis.plot ?

thanks for your help :slight_smile:
----
This message contains confidential information and may contain
information that is legally privileged. If you have received this
message by mistake, please immediately notify us and delete the
original message. Thank you. Ce message contient des informations
confidentielles. S'il vous est
parvenu par erreur, merci de bien vouloir nous en aviser par retour,
de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------

Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity
professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like
Barbarian Group, R/GA, & Big Spaceship.
http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
        

----
This message contains confidential information and may contain
information that is legally privileged. If you have received this
message by mistake, please immediately notify us and delete the
original message. Thank you. Ce message contient des informations confidentielles. S'il vous est
parvenu par erreur, merci de bien vouloir nous en aviser par retour,
de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------

Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity
professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like
Barbarian Group, R/GA, & Big Spaceship.
http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
  
----
This message contains confidential information and may contain information that is legally privileged. If you have received this message by mistake, please immediately notify us and delete the original message. Thank you.

Ce message contient des informations confidentielles. S'il vous est parvenu par erreur, merci de bien vouloir nous en aviser par retour, de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com _______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
  
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Yes, that's obviously what I should have started this thread with :slight_smile:

this is how I convert the xml into "plottable" arrays:

def getnamefrom(name,src,what):
    """extracts timestamps and 'what' from 'src' for 'name'"""
    buffwhat =
    bufftime =
    sortedbyname =
    [sortedbyname.append(element) for
    element in src if
    element["name"] == name]
    for i in sortedbyname[:-1]:
        if i['name'] == name:
            bufftime.append(i["timestamp"]/1e9)
            buffwhat.append(i[what])
    return buffwhat,bufftime

x,y = getnamefrom("cpu0",cpustat,"usage")
plt.plot(x,y)
del cpustat

(yes, it's a hardware monitor :D)
basically, it seems cpustat is still refcounted somewhere.
I wonder if I should append(copy.copy(i[...])) instead?

or If I'm mis-interpreting what I see with top. I'm trying to get infos
directly from the gc atm.

Michael Droettboom wrote:

路路路

It's not off topic -- and I understand what you're trying to do much
better now.

I assume at some point you convert all of the xml into a Numpy array and
pass that to matplotlib? matplotlib will only keep a reference to the
Numpy array, which should be far more compact, and you can safely remove
the references to the xml tree and that memory should be freed.

Cheers,
Mike

guillaume ranquet wrote:

sorry If my last message was confusing.
I'm exctracting data from an xml file (100megs of xml) which is stored
by my app as an array of dict in which I pick only some keys to plot.
I'm just trying to figure out if the gc is able to free some parts of
the dict (the un-plotted parts) by itself (and it seems not) or If I
have to give more hints to the gc.
my gig of ram is getting used, and I'd like to increase the set of data
a bit.

I can make a hardcopy of my datas, pass this copy to matplotlib and del
everything (so that it *should* be clean, from my understandings).

actually, my design is quite horrible. prolly a rewrite from scratch is
the best thing to do.

anyway, it's a bit off-topic on the matplotlib user list :slight_smile:

Michael Droettboom wrote:

I don't follow... If you need to free the data, you have to ensure that
matplotlib doesn't need it, and the only way to do that is destroy the
figure that references it. It's virtually impossible to create a
dangling reference in Python (from within Python).

Mike

guillaume ranquet wrote:
   

Thanks Mike,
I just wanted to be sure wether matplotlib had a copy or not.
I'll have to work on giving hints to the GC then...

guillaume.

Michael Droettboom wrote:

Yes, Python is reference counted (with a garbage collector to handle
cycles etc.). So, yes, the matplotlib plot keeps a "reference" to
your
data after you pass it in so it continue to redraw it as the plot is
panned/zoomed/resized etc. You can safely del your local
reference, but
the memory itself will not be freed, since matplotlib still needs it.

Mike

guillaume ranquet wrote:
          

I'm currently trying to improve my app to render more and more data.
I'm wondering if matplotlib keeps an internal copy of the datas once
the
plot function has been called?

from what I understand from python (I'm quite new to python),
everything
is reference and no implicit copies are done. I'm trying to check on
matplotlib if it's the case but I must admit I got some
difficulties xD

to sum it up: can I del the arrays I just passed to an axis.plot ?

thanks for your help :slight_smile:
----
This message contains confidential information and may contain
information that is legally privileged. If you have received this
message by mistake, please immediately notify us and delete the
original message. Thank you. Ce message contient des informations
confidentielles. S'il vous est
parvenu par erreur, merci de bien vouloir nous en aviser par retour,
de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------

Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity
professionals. Meet
the minds behind Google Creative Lab, Visual Complexity,
Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like
Barbarian Group, R/GA, & Big Spaceship.
http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
                  

----
This message contains confidential information and may contain
information that is legally privileged. If you have received this
message by mistake, please immediately notify us and delete the
original message. Thank you. Ce message contient des informations
confidentielles. S'il vous est
parvenu par erreur, merci de bien vouloir nous en aviser par retour,
de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------

Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity
professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like
Barbarian Group, R/GA, & Big Spaceship.
http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
        
----
This message contains confidential information and may contain
information that is legally privileged. If you have received this
message by mistake, please immediately notify us and delete the
original message. Thank you.
Ce message contient des informations confidentielles. S'il vous est
parvenu par erreur, merci de bien vouloir nous en aviser par retour,
de n'en faire aucun usage et de n'en garder aucune copie.
----

------------------------------------------------------------------------------

Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity
professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like
Barbarian Group, R/GA, & Big Spaceship.
http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
  
----
This message contains confidential information and may contain information that is legally privileged. If you have received this message by mistake, please immediately notify us and delete the original message. Thank you.

Ce message contient des informations confidentielles. S'il vous est parvenu par erreur, merci de bien vouloir nous en aviser par retour, de n'en faire aucun usage et de n'en garder aucune copie.
----

guillaume ranquet wrote:

this is how I convert the xml into "plottable" arrays:

def getnamefrom(name,src,what):
    """extracts timestamps and 'what' from 'src' for 'name'"""
    buffwhat =
    bufftime =
    sortedbyname =
    [sortedbyname.append(element) for
    element in src if
    element["name"] == name]
    for i in sortedbyname[:-1]:
        if i['name'] == name:
            bufftime.append(i["timestamp"]/1e9)
            buffwhat.append(i[what])
    return buffwhat,bufftime

x,y = getnamefrom("cpu0",cpustat,"usage")
plt.plot(x,y)
del cpustat

(yes, it's a hardware monitor :D)
basically, it seems cpustat is still refcounted somewhere.

what is cpustat? ans elementtree? Anyway, it sure looks like x and y should be full of copies of the data, so you don't have any references that would keep cpustat alive.

Also, it looks like x and y are lists -- when you pass those into MPL, they will be copied to numpy arrays, so you can delete them too, jsut in case they are keeping references to cpustat items.

> I wonder if I should append(copy.copy(i[...])) instead?

rather than that, I'd copy to numpy arrays explicitly:

return np.array(buffwhat), np.array(bufftime)

That's what MPL uses internally anyway, and you'll be clear what you want. If you know how many items you'll have to begin with, you can put the data into np.arrays directly (np.arrays do not support appending).

or If I'm mis-interpreting what I see with top.

you could be -- Python does not necessarily give memory back to the system when it's done with it -- but it should be able to re-use it.

> I'm trying to get infos directly from the gc atm.

you could use sys.getrefcount() (note that it creates a reference itself, so it's always at least 2), but that may nothelp, as it may not be the elementree that has multiple references, but rather something inside it....This is very tricky business -- Python frees you from worrying about memory management almost all the time, but it does mean that you give up control.

-Chris

路路路

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...