to much points

Markus_Feldmann · June 30, 2009, 5:22pm

Hi All,

my program lets slow down my cpu. This only appears if i plot to much
points. I am not sure how many point i need to get this, normally i plot
3*14e6 + 8e3, that is round about 50million points. My system is a
dual core 2GHz cpu with 2Gbyte Ram.

Here is my method to plot,
     def drawtransientall(self,min):
         self.subplot = self.figure.add_subplot(111)
         self.subplot.grid(True)
         list_t1,list_peaks,t2,list_samples =
self.computetransientall(min,min+self.maxitems)
         offset = 0
         color = ['green','red','blue','magenta','cyan']
         markerPeaks = ['v','<','1','3','s']
         markerSamples = ['^','>','2','4','p']
         self.plots=[[],[]]
         for i in range(len(self.showBands)):
             self.plots[0] +=
self.subplot.plot(list_t1[i],list_peaks[i],color=color[i],marker=markerPeaks[i],
                                             linestyle='None')
             self.plots[1] +=
self.subplot.plot(t2,list_samples[i]+offset,color=color[i],

marker=markerSamples[i],linestyle='None')
offset +=1

self.subplot.set_xlim(t2[0]-np.abs(t2[-1]-t2[0])/100,t2[-1]+np.abs(t2[-1]-t2[0])/100)
         ymax = np.amax(list_samples)
         ymin = np.amin(list_samples)
         self.subplot.set_ylim([ymin-np.abs(ymin)*0.1, ymax*1.2 + 2])
         self.subplot.set_ylabel("abs(Sample(t)) und
abs(Peak(t)+Offset)-->",fontsize = 12)
         self.subplot.set_xlabel("Zeit in Sek. -->",fontsize = 12)

Any ideas how to avoid the slow down of my cpu ?

regards Markus

_Jae-Joon_Lee1 · July 1, 2009, 5:34pm

A snippet of code does not help much.
Please try to post a small concise standalone example that we can run and test.

A general advise is to try to reduce the number of plot call, i.e.,
plot as may points as possible with a single plot call.

However, 50million points seems to be awful a lot.
6 inch x 6 inch figure with dpi=100 has 0.36 million number of pixels.
My guess is that it makes little sense to plot 50 million points here.

Anyhow, plotting 50million points with a single plot call dies with
some segfault error in my machine. So, I feel that matplotlib may not
be suitable for your task. But, John or others may have some insight
how to deal with.

Regards,

-JJ

···

On Tue, Jun 30, 2009 at 1:22 PM, Markus Feldmann<feldmann_markus@...380...> wrote:

Hi All,

my program lets slow down my cpu. This only appears if i plot to much
points. I am not sure how many point i need to get this, normally i plot
3*14e6 + 8e3, that is round about 50million points. My system is a
dual core 2GHz cpu with 2Gbyte Ram.

Here is my method to plot,
def drawtransientall(self,min):
self.subplot = self.figure.add_subplot(111)
self.subplot.grid(True)
list_t1,list_peaks,t2,list_samples =
self.computetransientall(min,min+self.maxitems)
offset = 0
color = ['green','red','blue','magenta','cyan']
markerPeaks = ['v','<','1','3','s']
markerSamples = ['^','>','2','4','p']
self.plots=[,]
for i in range(len(self.showBands)):
self.plots[0] +=
self.subplot.plot(list_t1[i],list_peaks[i],color=color[i],marker=markerPeaks[i],
linestyle='None')
self.plots[1] +=
self.subplot.plot(t2,list_samples[i]+offset,color=color[i],

marker=markerSamples[i],linestyle='None')
offset +=1

self.subplot.set_xlim(t2[0]-np.abs(t2[-1]-t2[0])/100,t2[-1]+np.abs(t2[-1]-t2[0])/100)
ymax = np.amax(list_samples)
ymin = np.amin(list_samples)
self.subplot.set_ylim([ymin-np.abs(ymin)*0.1, ymax*1.2 + 2])
self.subplot.set_ylabel("abs(Sample(t)) und
abs(Peak(t)+Offset)-->",fontsize = 12)
self.subplot.set_xlabel("Zeit in Sek. -->",fontsize = 12)

Any ideas how to avoid the slow down of my cpu ?

regards Markus

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

Michael_Droettboom1 · July 1, 2009, 6:34pm

I agree with Jae-Joon here – try to reduce the number of points before
passing it to matplotlib.

However, I’m a little concerned about the segfault – I’d rather
matplotlib give a MemoryError exception if that’s in fact what is
happening. Jae-Joon – can you share your test that causes the
segfault?

The snippet below completely hogs my machine for a few minutes, but
then, correctly, aborts with a MemoryError.

This is on FC11 i586, Python 2.6, Numpy 1.3.

···

<feldmann_markus@…380…>Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users

_Jae-Joon_Lee1 · July 1, 2009, 7:16pm

I agree with Jae-Joon here -- try to reduce the number of points before
passing it to matplotlib.

However, I'm a little concerned about the segfault -- I'd rather matplotlib
give a MemoryError exception if that's in fact what is happening. Jae-Joon
-- can you share your test that causes the segfault?

The snippet below completely hogs my machine for a few minutes, but then,
correctly, aborts with a MemoryError.

This is on FC11 i586, Python 2.6, Numpy 1.3.

====

from matplotlib.pyplot import *
import numpy as np

points = np.random.random((50000000, 2))
plot(points)
show()

Yes, I also got MemoryError in this case during the plot() call.

But I got segfault for the code below.

x=random(50e6)
y=random(50e6)
plt.plot(x, y)
plt.show()

In this case, plot() runs fine, but segfault during show().

The segfault happens in the _path_module::affine_transform method of
src/_path.cpp.

I wonder if you can reproduce this.

-JJ

···

On Wed, Jul 1, 2009 at 2:34 PM, Michael Droettboom<mdroe@...86...> wrote:

====

Mike

On 07/01/2009 01:34 PM, Jae-Joon Lee wrote:

A snippet of code does not help much.
Please try to post a small concise standalone example that we can run and
test.

A general advise is to try to reduce the number of plot call, i.e.,
plot as may points as possible with a single plot call.

However, 50million points seems to be awful a lot.
6 inch x 6 inch figure with dpi=100 has 0.36 million number of pixels.
My guess is that it makes little sense to plot 50 million points here.

Anyhow, plotting 50million points with a single plot call dies with
some segfault error in my machine. So, I feel that matplotlib may not
be suitable for your task. But, John or others may have some insight
how to deal with.

Regards,

-JJ

On Tue, Jun 30, 2009 at 1:22 PM, Markus Feldmann<feldmann_markus@...2665....> > wrote:

Hi All,

my program lets slow down my cpu. This only appears if i plot to much
points. I am not sure how many point i need to get this, normally i plot
3*14e6 + 8e3, that is round about 50million points. My system is a
dual core 2GHz cpu with 2Gbyte Ram.

Here is my method to plot,
def drawtransientall(self,min):
self.subplot = self.figure.add_subplot(111)
self.subplot.grid(True)
list_t1,list_peaks,t2,list_samples =
self.computetransientall(min,min+self.maxitems)
offset = 0
color = ['green','red','blue','magenta','cyan']
markerPeaks = ['v','<','1','3','s']
markerSamples = ['^','>','2','4','p']
self.plots=[,]
for i in range(len(self.showBands)):
self.plots[0] +=
self.subplot.plot(list_t1[i],list_peaks[i],color=color[i],marker=markerPeaks[i],
linestyle='None')
self.plots[1] +=
self.subplot.plot(t2,list_samples[i]+offset,color=color[i],

marker=markerSamples[i],linestyle='None')
offset +=1

self.subplot.set_xlim(t2[0]-np.abs(t2[-1]-t2[0])/100,t2[-1]+np.abs(t2[-1]-t2[0])/100)
ymax = np.amax(list_samples)
ymin = np.amin(list_samples)
self.subplot.set_ylim([ymin-np.abs(ymin)*0.1, ymax*1.2 + 2])
self.subplot.set_ylabel("abs(Sample(t)) und
abs(Peak(t)+Offset)-->",fontsize = 12)
self.subplot.set_xlabel("Zeit in Sek. -->",fontsize = 12)

Any ideas how to avoid the slow down of my cpu ?

regards Markus

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

_Jae-Joon_Lee1 · July 1, 2009, 7:28pm

I tracked this down do line 962 of the _path.cpp.

double* vertex_out = (double*)PyArray_DATA(result);

My guess is that PyArray_SimpleNew at line 957 returns NULL for a
memory error instead of raising an exception, which makes result=NULL
and causes a segfault at line 962.

Since I'm not an c++ expert, I'll leave it to you, Michael.

Regards,

-JJ

···

On Wed, Jul 1, 2009 at 3:16 PM, Jae-Joon Lee<lee.j.joon@...287...> wrote:

On Wed, Jul 1, 2009 at 2:34 PM, Michael Droettboom<mdroe@...86...> wrote:

I agree with Jae-Joon here -- try to reduce the number of points before
passing it to matplotlib.

However, I'm a little concerned about the segfault -- I'd rather matplotlib
give a MemoryError exception if that's in fact what is happening. Jae-Joon
-- can you share your test that causes the segfault?

The snippet below completely hogs my machine for a few minutes, but then,
correctly, aborts with a MemoryError.

This is on FC11 i586, Python 2.6, Numpy 1.3.

====

from matplotlib.pyplot import *
import numpy as np

points = np.random.random((50000000, 2))
plot(points)
show()

Yes, I also got MemoryError in this case during the plot() call.

But I got segfault for the code below.

x=random(50e6)
y=random(50e6)
plt.plot(x, y)
plt.show()

In this case, plot() runs fine, but segfault during show().

The segfault happens in the _path_module::affine_transform method of
src/_path.cpp.

I wonder if you can reproduce this.

-JJ

====

Mike

On 07/01/2009 01:34 PM, Jae-Joon Lee wrote:

A snippet of code does not help much.
Please try to post a small concise standalone example that we can run and
test.

A general advise is to try to reduce the number of plot call, i.e.,
plot as may points as possible with a single plot call.

However, 50million points seems to be awful a lot.
6 inch x 6 inch figure with dpi=100 has 0.36 million number of pixels.
My guess is that it makes little sense to plot 50 million points here.

Anyhow, plotting 50million points with a single plot call dies with
some segfault error in my machine. So, I feel that matplotlib may not
be suitable for your task. But, John or others may have some insight
how to deal with.

Regards,

-JJ

On Tue, Jun 30, 2009 at 1:22 PM, Markus Feldmann<feldmann_markus@...2666.....> >> wrote:

Hi All,

my program lets slow down my cpu. This only appears if i plot to much
points. I am not sure how many point i need to get this, normally i plot
3*14e6 + 8e3, that is round about 50million points. My system is a
dual core 2GHz cpu with 2Gbyte Ram.

Here is my method to plot,
def drawtransientall(self,min):
self.subplot = self.figure.add_subplot(111)
self.subplot.grid(True)
list_t1,list_peaks,t2,list_samples =
self.computetransientall(min,min+self.maxitems)
offset = 0
color = ['green','red','blue','magenta','cyan']
markerPeaks = ['v','<','1','3','s']
markerSamples = ['^','>','2','4','p']
self.plots=[,]
for i in range(len(self.showBands)):
self.plots[0] +=
self.subplot.plot(list_t1[i],list_peaks[i],color=color[i],marker=markerPeaks[i],
linestyle='None')
self.plots[1] +=
self.subplot.plot(t2,list_samples[i]+offset,color=color[i],

marker=markerSamples[i],linestyle='None')
offset +=1

self.subplot.set_xlim(t2[0]-np.abs(t2[-1]-t2[0])/100,t2[-1]+np.abs(t2[-1]-t2[0])/100)
ymax = np.amax(list_samples)
ymin = np.amin(list_samples)
self.subplot.set_ylim([ymin-np.abs(ymin)*0.1, ymax*1.2 + 2])
self.subplot.set_ylabel("abs(Sample(t)) und
abs(Peak(t)+Offset)-->",fontsize = 12)
self.subplot.set_xlabel("Zeit in Sek. -->",fontsize = 12)

Any ideas how to avoid the slow down of my cpu ?

regards Markus

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

Michael_Droettboom1 · July 1, 2009, 8:02pm

This should now be fixed on the maintenance branch and trunk. A Numpy
array allocation was not being NULL-checked in
_path.cpp:affine_transform.

I know a MemoryError doesn’t help the user much more than a segfault,
but it always makes me feel better to get a real Python exception
rather than exploding

Mike

···

<mdroe@…86…><feldmann_markus@…380…>Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users

Markus_Feldmann · July 2, 2009, 8:53am

Michael Droettboom schrieb:

This should now be fixed on the maintenance branch and trunk. A Numpy array allocation was not being NULL-checked in _path.cpp:affine_transform.

I know a MemoryError doesn't help the user much more than a segfault, but it always makes me feel better to get a real Python exception rather than exploding

Mike

Hi All,

thanks for the bugfix. I also got the segfault, but forgot to wrote this. Is there a possibility to limit the maximum points shown ?
And sorry that i can not post more of my code. My program is to big.

Maybe this is a feature request. It would be nice to set up a maximum limit and compute a average. I doesnt know which way is the fastest.

I only execute the plot command three times but every plot command plots
round about 15 Mill. points.

regards Markus