Handling LARGE data sets

Hi,

We are starting to require plotting large data sets.
600K samples is one second, and we have 20 sec runs

This takes approx 5 min to draw the plots on screen.
Then if you want to zoom, ... well go get a cup of coffee, ... not very productive.

1) Is MatPlotLib the right tool for large data sets, .. should I be looking at something else?

2) Given nothing else better, I'm looking for suggestions how to handle this size data.

My initial thoughts, ... display a low res graph, then have some movable markers in the low res graph and the main graph would only show the contents between those markers.
( ala some sound editing programs )

This would mean I need to filter/decimate the data for the low res graph, .. and a quick search on the web didn't seem to turn up any python way of doing this, .. anyone know of any existing python tool set to do this, before I re-invent the wheel?

Thanks for any input
Steve

Hi Steve,

This is also interest to us - although
about 2 million data points is our maximum so far.

Here is a decimating filter module based
on the Savtsky-Golay method.

Brett McSweeney

steve george <steve_geo@…887…>

Sent by: matplotlib-users-bounces@lists.sourceforge.net
31/05/2007 06:07 AM

To

Matplotlib Users matplotlib-users@lists.sourceforge.net
cc

Subject

[Matplotlib-users] Handling LARGE
data sets

`

Hi,

We are starting to require plotting large data sets.

600K samples is one second, and we have 20 sec runs

This takes approx 5 min to draw the plots on screen.

Then if you want to zoom, … well go get a cup of coffee, … not very

productive.

  1. Is MatPlotLib the right tool for large data sets, … should I be

looking at something else?

  1. Given nothing else better, I’m looking for suggestions how to handle

this size data.

My initial thoughts, … display a low res graph, then have some movable

markers in the low res graph and the main graph would only show the

contents between those markers.

( ala some sound editing programs )

This would mean I need to filter/decimate the data for the low res

graph, … and a quick search on the web didn’t seem to turn up any

python way of doing this, … anyone know of any existing python tool set

to do this, before I re-invent the wheel?

Thanks for any input

Steve

SGfilter.py (1.81 KB)

···

This SF.net email is sponsored by DB2 Express

Download DB2 Express C - the FREE version of DB2 express and take

control of your XML. No limits. Just data. Click to get it now.

http://sourceforge.net/powerbar/db2/


Matplotlib-users mailing list

Matplotlib-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/matplotlib-users


This email has been scanned by the MessageLabs Email Security System.

For more information please visit http://www.messagelabs.com/email


`

UNITED GROUP

This email message is the property of United Group. The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorised. If you are not the intended recipient, you may not disclose, copy or distribute this email, nor take or omit to take any action in reliance on it. United Group accepts no liability for any damage caused by this email or any attachments due to viruses, interference, interception, corruption or unauthorised access.

If you have received this email in error, please notify United Group immediately by email to the sender’s email address and delete this document.

License?
Is there a presumption that code posted to this
list is public domain?

Cheers,
Alan Isaac

···

On Thu, 31 May 2007, brett.mcsweeney@...1537... apparently wrote:

Here is a decimating filter module based on the
Savtsky-Golay method.

Hi Alan,

I’m not speaking for anyone else, but
as far as I’m concerned that code is public domain.

Cheers,

Brett.

Alan G Isaac <aisaac@…310…>

Sent by: matplotlib-users-bounces@lists.sourceforge.net
31/05/2007 01:17 PM

Please respond to

aisaac@…310…

To

Matplotlib Users matplotlib-users@lists.sourceforge.net
cc

Subject

Re: [Matplotlib-users] Handling LARGE
data sets

`On Thu, 31 May 2007, brett.mcsweeney@…1537…
apparently wrote:

Here is a decimating filter module based on the

Savtsky-Golay method.

License?

Is there a presumption that code posted to this

list is public domain?

Cheers,

Alan Isaac

···

This SF.net email is sponsored by DB2 Express

Download DB2 Express C - the FREE version of DB2 express and take

control of your XML. No limits. Just data. Click to get it now.

http://sourceforge.net/powerbar/db2/


Matplotlib-users mailing list

Matplotlib-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/matplotlib-users


This email has been scanned by the MessageLabs Email Security System.

For more information please visit http://www.messagelabs.com/email


`

UNITED GROUP

This email message is the property of United Group. The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorised. If you are not the intended recipient, you may not disclose, copy or distribute this email, nor take or omit to take any action in reliance on it. United Group accepts no liability for any damage caused by this email or any attachments due to viruses, interference, interception, corruption or unauthorised access.

If you have received this email in error, please notify United Group immediately by email to the sender’s email address and delete this document.

OK, well, who wrote the code and who holds the copyright? In other
words, your concerns about the code being in the public domain may or
may not be relevant, depending on where the code came from and whether
you have any legal authority to distribute the code and under what
conditions. It would be nice to include an SG filter in scipy, for
example, but that would only be possible if it were released under a
BSD-like license.

-Andrew

···

brett.mcsweeney@...1537... wrote:

Hi Alan,

I'm not speaking for anyone else, but as far as I'm concerned that
code is public domain.

Hello Andrew,

Sorry for the delay, I have been on
holidays. I wrote the code (such as it is - but the key line is adapted
from the scipy web site).

I should have put a copy of the BSD
license at the front, I suppose, (would that be correct?) but it was never
intended as more than a quick in-house solution to our problem. We
have never “released” software to the outside world. It’s
not our focus.

Brett.

Andrew Straw <strawman@…106…>
02/06/2007 06:37 AM

To

brett.mcsweeney@…1537…
cc

Matplotlib Users matplotlib-users@lists.sourceforge.net
Subject

Re: [Matplotlib-users] Handling LARGE
data sets

Hi Alan,

I’m not speaking for anyone else, but as far as I’m concerned that

code is public domain.

OK, well, who wrote the code and who holds the copyright? In other

words, your concerns about the code being in the public domain may or

may not be relevant, depending on where the code came from and whether

you have any legal authority to distribute the code and under what

conditions. It would be nice to include an SG filter in scipy, for

example, but that would only be possible if it were released under a

BSD-like license.

-Andrew

···

`brett.mcsweeney@…1537… wrote:


This email has been scanned by the MessageLabs Email Security System.

For more information please visit http://www.messagelabs.com/email


`

UNITED GROUP

This email message is the property of United Group. The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorised. If you are not the intended recipient, you may not disclose, copy or distribute this email, nor take or omit to take any action in reliance on it. United Group accepts no liability for any damage caused by this email or any attachments due to viruses, interference, interception, corruption or unauthorised access.

If you have received this email in error, please notify United Group immediately by email to the sender’s email address and delete this document.