Hi all,
I tried to plot parts of a large 3D array with each 4 float64 entries.
Loading the array with numpy.fromfile and performing a type conversion
afterwards ends up in a MemoryError.
The following code reproduces the error:
···
***********************************
import gc
from os import path
from numpy import zeros, empty, float32, float64, fromfile
# With the imports of the next three lines a memory error occurs
#import matplotlib
#matplotlib.use("AGG")
#import matplotlib.pylab as plt
_filename = "mt.dat"
if (path.exists(_filename)==False):
print "Write file ..."
_mtf = file(_filename, "wb")
_mtd = zeros( (300,300,300,4),dtype=float64)
_mtd.tofile(_mtf)
_mtf.close()
_mtd = empty((0))
gc.collect();
print "Try to read file ..."
_mtf = file(_filename, "rb")
# The memory error occurs with the type conversion to float 64,
# but only if mathplotlib is imported!
_mtd = fromfile(_mtf,float64).astype(float32)
_mtf.close()
print "Successful read:",_mtd.shape
# Here some plot stuff would be 
***********************************
If the import of matplotlib is not included, no error occures otherwise it
will not work. Perhaps anybody can help me (I tested it only on a Windows XP
32Bit).
Best regards and thank you
Stefan
--
View this message in context: http://old.nabble.com/MemoryError-with-import-matplotlib-tp30674798p30674798.html
Sent from the matplotlib - users mailing list archive at Nabble.com.
Works for me on Windows 7 64 bit with 32 bit Python. I believe you simply run out of memory. On a 32 bit Windows OS, Python can only use 2 GB, which it has to share with other all processes. zeros((300,300,300,4),dtype=float64) requires ~820MB of contiguous memory, which might not be available even if you have >1 GB free RAM. Better use a 64 bit OS and Python.
Christoph
···
On 1/14/2011 11:40 AM, sprobst wrote:
Hi all,
I tried to plot parts of a large 3D array with each 4 float64 entries.
Loading the array with numpy.fromfile and performing a type conversion
afterwards ends up in a MemoryError.
The following code reproduces the error:
***********************************
import gc
from os import path
from numpy import zeros, empty, float32, float64, fromfile
# With the imports of the next three lines a memory error occurs
#import matplotlib
#matplotlib.use("AGG")
#import matplotlib.pylab as plt
_filename = "mt.dat"
if (path.exists(_filename)==False):
print "Write file ..."
_mtf = file(_filename, "wb")
_mtd = zeros( (300,300,300,4),dtype=float64)
_mtd.tofile(_mtf)
_mtf.close()
_mtd = empty((0))
gc.collect();
print "Try to read file ..."
_mtf = file(_filename, "rb")
# The memory error occurs with the type conversion to float 64,
# but only if mathplotlib is imported!
_mtd = fromfile(_mtf,float64).astype(float32)
_mtf.close()
print "Successful read:",_mtd.shape
# Here some plot stuff would be 
***********************************
If the import of matplotlib is not included, no error occures otherwise it
will not work. Perhaps anybody can help me (I tested it only on a Windows XP
32Bit).
Best regards and thank you
Stefan
Stefan,
I don’t know if it would make a difference, but I see that you are importing matplotlib.pylab as plt. This is not a typical way of importing matplotlib. Try instead:
import matplotlib.pyplot as plt
Ben Root
···
On Fri, Jan 14, 2011 at 1:40 PM, sprobst <StefanProbst@…361…> wrote:
Hi all,
I tried to plot parts of a large 3D array with each 4 float64 entries.
Loading the array with numpy.fromfile and performing a type conversion
afterwards ends up in a MemoryError.
The following code reproduces the error:
import gc
from os import path
from numpy import zeros, empty, float32, float64, fromfile
With the imports of the next three lines a memory error occurs
#import matplotlib
#matplotlib.use(“AGG”)
#import matplotlib.pylab as plt
_filename = “mt.dat”
if (path.exists(_filename)==False):
print "Write file ..."
_mtf = file(_filename, "wb")
_mtd = zeros( (300,300,300,4),dtype=float64)
_mtd.tofile(_mtf)
_mtf.close()
_mtd = empty((0))
gc.collect();
print “Try to read file …”
_mtf = file(_filename, “rb”)
The memory error occurs with the type conversion to float 64,
but only if mathplotlib is imported!
_mtd = fromfile(_mtf,float64).astype(float32)
_mtf.close()
print “Successful read:”,_mtd.shape
Here some plot stuff would be
If the import of matplotlib is not included, no error occures otherwise it
will not work. Perhaps anybody can help me (I tested it only on a Windows XP
32Bit).
Best regards and thank you
Stefan
Dear Christoph,
It is not a memory problem, because before the programm starts I have more
than 1.3 GB free memory left. And on the other hand it depends on the import
line! There seems to be some interference of the packages. My Python version
is 2.6 (needed by another programm), perhaps that also makes the difference.
But thank you for the quick answer!
Stefan
···
--
View this message in context: http://old.nabble.com/MemoryError-with-import-matplotlib-tp30674798p30675089.html
Sent from the matplotlib - users mailing list archive at Nabble.com.
The zeros(...) statement needs a contiguous block of > 800 MB RAM and the conversion another contiguous block of > 400 MB. Memory allocated during the matplotlib import statement could easily fragment the available memory such that no 800 + 400 MB blocks exist. Try monitoring your memory usage and import matplotlib after allocating/converting the array. Anyway, you'll be better off using a 64 bit system if you deal with such amount of data.
Christoph
···
On 1/14/2011 12:17 PM, sprobst wrote:
Dear Christoph,
It is not a memory problem, because before the programm starts I have more
than 1.3 GB free memory left. And on the other hand it depends on the import
line! There seems to be some interference of the packages. My Python version
is 2.6 (needed by another programm), perhaps that also makes the difference.
But thank you for the quick answer!
Stefan
Problem solved ... even as orginally C++ or Java programmer it is strange,
but a interpreted language makes it possible to obtain with an import at a
different position a complety different behaviour during run-time.
I just putted the import of mathplotlib and pylab after the file reading and
after the type conversion and it works. :jumping:
Comment to just switch to another system with 64bit. Sometimes if your are
not admin of a machine, than it is hard to change the system 
Thank you anyhow, based on your comments I found the solution. Do you
perhaps know if there is something like a memdefrag :-/?
Have a nice weekend
Stefan
···
--
View this message in context: http://old.nabble.com/MemoryError-with-import-matplotlib-tp30674798p30676114.html
Sent from the matplotlib - users mailing list archive at Nabble.com.