Thanks a lot Chris for the detailed answer. I had the same doubts about 64bits and multiple cores. I just found out about Gdal and yes, Jeff's grib tools seem to be exactly what I needed.
I'll stay with a fast dual core and 32 bit os and I'll get a 10000 rpm hdd. That should be more than enough especially that this machine will not do anything else but processing this.
···
Sent from my iPhone
On Apr 9, 2009, at 10:12 AM, Christopher Barker <Chris.Barker@...259...> wrote:
Eric Firing wrote:
The biggest bottleneck is happening because I'm unpacking grib files to csv
files using Degrib in command line. That operation is usually around half an
disk speed -- you might want to try SATA RAID 0 (striping) -- I"d get a
good hardware vendor's advise in maximizing your disk IO.
You can also multi-task that process easily, but if you're disk-bound,
that won't help anyway.
Instead of going to csv files--which are *very* inefficient to write,
store, and then read in again--why not convert directly to netcdf,
Or HDF, via PyTables. Or even direct binary numpy arrays, with either
fromfile / to file, or, more robustly, with numpy.save and numpy.load.
direct numpy-enabled access to the grib files might be
even better, eliminating the translation phase entirely. Have you
looked into http://www.pyngl.ucar.edu/Nio.shtml?
Also, I think GDAL support GRIB, and can directly give you numpy arrays.
I have noticed also that on a lower spec AMD desktop this runs
faster than on my P4 Intel Laptop, my guess being that the laptop hdd is
5400 rpm and the desktop is 7200 rpm.
yup, those laptop hard drives are SLOW -- you culd look into a Solic
State drive, if you have some money to spend.
Next step is to take all those csv files and make images from them. For this
one I haven't dug too deep to see what is happening but it seems to be the
other way, using the cpu a lot more while keeping the memory usage high too.
mulit-cores aren't going to help here, unless yuo run a few separate
processes -- also, how much memory? All 64 bits will buy you is more
memory, which you may or may not need.
Also, as for Windows 64 bits -- is numpy supported there yet? I'd make
sure, there are issues, as there is no MingGW for 64 bit Windows.
antonv wrote:
I know that using the csv files is very slow but I have no knowledge of
working with the netcdf format and I was in a bit of a rush when I wrote
this. I will take a look again at it. How would you translate a grib in
netcdf?
See if degrib supports any binary formats (I now, I'm form NOAA, I
should know...). Otherewise yuo could use the hGDAL command-line tools
to translate into something else binary that may be easier to deal with.
Though it looks like Jeff may have solved this problem for you (One
NOAA, Jeff!)
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@...259...
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users