Plotting large data sets with Pylab

Both Python 2.6 and Python 2.7 Windows and Mac

I’ve been running a very simple plotting script with Mac OSX 10.5.8 and Windows XP Professional Version 2002, Service Pack 3 using a large, 10,000,000 element .csv file. I ran both scripts locally with the data in the same local directory as the script.

The Python version on the Mac is:
Python 2.6.6 (r266:84374, Aug 31 2010, 11:00:51)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type “copyright”, “credits” or “license()” for more information.

···
****************************************************************
Personal firewall software may warn about the connection IDLE
makes to its subprocess using this computer's internal loopback
interface.  This connection is not visible on any external
interface and no data is sent to or received from the Internet.
****************************************************************

IDLE 2.6.6

================================ RESTART ================================

Now its: Sun Feb 20 17:18:10 2011
1298251090.64
slorping
It took 414.0 seconds to slorp in an array containing 10000000 elements.

on Windows:

Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)] on win32
Type “copyright”, “credits” or “license()” for more information.

****************************************************************
Personal firewall software may warn about the connection IDLE
makes to its subprocess using this computer's internal loopback
interface.  This connection is not visible on any external
interface and no data is sent to or received from the Internet.
****************************************************************

IDLE 2.6.6

================================ RESTART ================================

Now its: Sun Feb 20 18:25:41 2011
1298255141.56
slorping
It took 169.0 seconds to slorp in an array containing 10000000 elements.

Now its: Sun Feb 20 18:28:30 2011


The script and the data are identical:

import time as ti

from math import radians

from pylab import plot, show

from numpy import loadtxt, ravel , shape

#####################################################
###Timekeeping Stuff#################################
#####################################################

print 'Now its: ',ti.asctime()

clockA = ti.time()
print clockA

#####################################################
###One Big Huge Flat Array###########################
#####################################################
column = 0

x = ()
raw = ()
normal = ()
checker = ()
trigger = ()

print ‘slorping’

raw = loadtxt(‘DataLogger.csv’, float, delimiter = ‘n,’)
x = ravel(raw)
##normal = (x+.5807)*(2/3.)

###################################################
###Just fun timing stuff############################
###################################################

clockS = ti.time()
s = shape(x)
print ‘It took’,round(clockS-clockA), ‘seconds to slorp in an array containing’,s[0], ‘elements.’
print “”
print 'Now its: ',ti.asctime()

plot(x, ‘k’)
show()


This script will plot consistently on the Mac and crashes every time on the PC. I’m data logging at 100MZ and the card is on the PC desk side machine. My Macbook Air doesn’t have a PCI slot. I developed all of the scripts on the Mac using data logged on the PC. I was hoping to integrate everything onto the PC for efficiency. Is there a workaround I can use on the PC?

Thanks,

Bruno George

The performance issue for reading those data files will be largely dependent on how fast the harddrives are on the machines.

The crashing issue is likely due to Windows XP’s memory allocation mechanism or running into the memory barrier for 32-bit machines. Is your Mac a 64-bit machine? If it is, then it is likely that getting a 64-bit Windows machine would address the crashing issue. To be sure, I would watch the memory usage of those scripts to see how high it gets.

Ben Root

···

On Sun, Feb 20, 2011 at 9:22 PM, Bruno George <bgeorge98121@…287…> wrote:

Both Python 2.6 and Python 2.7 Windows and Mac

I’ve been running a very simple plotting script with Mac OSX 10.5.8 and Windows XP Professional Version 2002, Service Pack 3 using a large, 10,000,000 element .csv file. I ran both scripts locally with the data in the same local directory as the script.

The Python version on the Mac is:
Python 2.6.6 (r266:84374, Aug 31 2010, 11:00:51)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type “copyright”, “credits” or “license()” for more information.

****************************************************************
Personal firewall software may warn about the connection IDLE
makes to its subprocess using this computer's internal loopback


interface.  This connection is not visible on any external

interface and no data is sent to or received from the Internet.
****************************************************************

IDLE 2.6.6

================================ RESTART ================================

Now its: Sun Feb 20 17:18:10 2011
1298251090.64
slorping
It took 414.0 seconds to slorp in an array containing 10000000 elements.

on Windows:

Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)] on win32

Type “copyright”, “credits” or “license()” for more information.

****************************************************************
Personal firewall software may warn about the connection IDLE



makes to its subprocess using this computer's internal loopback
interface.  This connection is not visible on any external
interface and no data is sent to or received from the Internet.
****************************************************************

IDLE 2.6.6

================================ RESTART ================================

Now its: Sun Feb 20 18:25:41 2011
1298255141.56
slorping
It took 169.0 seconds to slorp in an array containing 10000000 elements.

Now its: Sun Feb 20 18:28:30 2011


The script and the data are identical:

import time as ti

from math import radians

from pylab import plot, show

from numpy import loadtxt, ravel , shape

#####################################################
###Timekeeping Stuff#################################
#####################################################

print 'Now its: ',ti.asctime()

clockA = ti.time()
print clockA

#####################################################
###One Big Huge Flat Array###########################
#####################################################

column = 0

x = ()
raw = ()
normal = ()
checker = ()
trigger = ()

print ‘slorping’

raw = loadtxt(‘DataLogger.csv’, float, delimiter = ‘n,’)
x = ravel(raw)
##normal = (x+.5807)*(2/3.)

###################################################
###Just fun timing stuff############################
###################################################

clockS = ti.time()
s = shape(x)
print ‘It took’,round(clockS-clockA), ‘seconds to slorp in an array containing’,s[0], ‘elements.’

print “”
print 'Now its: ',ti.asctime()

plot(x, ‘k’)
show()


This script will plot consistently on the Mac and crashes every time on the PC. I’m data logging at 100MZ and the card is on the PC desk side machine. My Macbook Air doesn’t have a PCI slot. I developed all of the scripts on the Mac using data logged on the PC. I was hoping to integrate everything onto the PC for efficiency. Is there a workaround I can use on the PC?

Thanks,

Bruno George