Plotting NOAA data...

Community matplotlib-users
1

Sorry, I meant the griddata module as part of mlab.
Kurt

···

From: peterskurt@…1954…
To: matplotlib-users@…642…ts.sourceforge.net
Subject: RE: Plotting NOAA data…
Date: Tue, 23 Dec 2008 09:28:49 -0700

I don’t seem to have a mlab module as part of matplotlib. Is that something new?
Kurt

2

KURT PETERS wrote:

Sorry, I meant the griddata module as part of mlab.
Kurt

It was first released in 0.98.3.

Ryan

···

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

3

Hi Jeff, here is a link to the csv file:
http://downloads.75ive.com/testdata.csv

Thanks again for all your help guys! Really appreciated!

Ryan May-3 wrote:

···

KURT PETERS wrote:

Sorry, I meant the griddata module as part of mlab.
Kurt

It was first released in 0.98.3.

Ryan

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

--
View this message in context: http://www.nabble.com/Plotting-NOAA-data…-tp21139727p21148270.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

4

antonv wrote:

Hi Jeff, here is a link to the csv file:
http://downloads.75ive.com/testdata.csv

Thanks again for all your help guys! Really appreciated!

I meant the original GRIB file ...

-Jeff

···

Ryan May-3 wrote:
  

KURT PETERS wrote:
    

Sorry, I meant the griddata module as part of mlab.
Kurt

It was first released in 0.98.3.

Ryan

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

5

Here is the link to the folder that has the grib files. They are updated
every 3 hours:
ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/wave/prod/

Any of the files there would have the same info only at different times of
day.

···

–
View this message in context: http://www.nabble.com/Plotting-NOAA-data…-tp21139727p21148738.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

6

Better yet, here is the file that I am using on my testing but keep in mind
that it's a 33mb file:
http://downloads.75ive.com/enp.t18z.grib.grib2

antonv wrote:

···

Here is the link to the folder that has the grib files. They are updated
every 3 hours:
ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/wave/prod/

Any of the files there would have the same info only at different times of
day.

--
View this message in context: http://www.nabble.com/Plotting-NOAA-data…-tp21139727p21148866.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

7

antonv wrote:

Better yet, here is the file that I am using on my testing but keep in mind
that it's a 33mb file:
http://downloads.75ive.com/enp.t18z.grib.grib2
  
Anton: I had already downloaded a file from the ftp site, so here's an example script:

import Nio
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import numpy as np
f = Nio.open_file('akw.t00z.grib.grib2')
print f.variables.keys()
lons = f.variables['lon_0'][:]
# flip latitudes so data goes S-->N
lats = f.variables['lat_0'][::-1]
times = f.variables['forecast_time0'][:]
datavar = f.variables['WWSWHGT_P0_L1_GLL0']
ntime = 10
data = datavar[ntime,::-1]
print f.variables['WWSWHGT_P0_L1_GLL0']
print data.min(), data.max()
m = Basemap(projection='cyl',llcrnrlat=lats[0],llcrnrlon=lons[0],\
            urcrnrlat=lats[-1],urcrnrlon=lons[-1],resolution='l')
x, y = m(*np.meshgrid(lons, lats))
levels = np.arange(0,9.1,0.5)
m.contourf(x,y,data,levels)
m.drawcoastlines()
m.fillcontinents()
m.drawparallels(np.arange(40,81,10),labels=[1,0,0,0])
m.drawmeridians(np.arange(150,241,10),labels=[0,0,0,1])
m.drawparallels(np.arange(40,81,10),labels=[1,0,0,0])
m.drawmeridians(np.arange(150,241,10),labels=[0,0,0,1])
plt.title(datavar.long_name+' %s hr fcst'%(times[ntime]),fontsize=12)
plt.colorbar(orientation='horizontal',shrink=0.9,format="%g")
plt.show()

The resulting plot is attached.

Note you will need the Basemap toolkit (available from the matplotlib download site). If you don't want to plot the coastlines, you can just use plt.contourf. You'll also need PyNIO (The Nio module for reading and writing supported data formats) to read the GRIB file.

The png produced by this script is attached.

Let me know if you have any questions.

-Jeff

testgrib.png

···

antonv wrote:
  

Here is the link to the folder that has the grib files. They are updated
every 3 hours:
ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/wave/prod/

Any of the files there would have the same info only at different times of
day.

8

Dear Anton,

2008/12/23 antonv <vasilescu_anton@...9...>:

Also, because I figured out the data I need and already have the scripts in place
to extract the CSV files I would really like to keep it that way. Would it be possible to
just show me how to get from the csv file to the plot?

Here is a short recipe:

import numpy as np

f = open("file.csv", "r")
coords = np.loadtxt(f, delimiter=",", skiprows=1)
lon = coords[:,0]
lat = coords[:,1]
dat = coords[:,2]

where "file.csv" is a regular comma-separated values file in the format:

Lat,Lon,Dat
-61.05,10.4,20
-79.43,9.15,50
-70.66,9.53,10
-63.11,7.91,40
...

Hope this helps!

Best regards,

···

--
Dr. Mauro J. Cavalcanti
Ecoinformatics Studio
P.O. Box 46521, CEP 20551-970
Rio de Janeiro, RJ, BRASIL
E-mail: maurobio@...287...
Web: http://studio.infobio.net
Linux Registered User #473524 * Ubuntu User #22717
"Life is complex. It consists of real and imaginary parts."

9

Mauro Cavalcanti wrote:

Dear Anton,

2008/12/23 antonv <vasilescu_anton@...9...>:
  

Also, because I figured out the data I need and already have the scripts in place
to extract the CSV files I would really like to keep it that way. Would it be possible to
just show me how to get from the csv file to the plot?
    
Here is a short recipe:

import numpy as np

f = open("file.csv", "r")
coords = np.loadtxt(f, delimiter=",", skiprows=1)
lon = coords[:,0]
lat = coords[:,1]
dat = coords[:,2]

where "file.csv" is a regular comma-separated values file in the format:

Lat,Lon,Dat
-61.05,10.4,20
-79.43,9.15,50
-70.66,9.53,10
-63.11,7.91,40
...

Hope this helps!

Best regards,

Since the arrays are 2D (for gridded data), a reshape is also needed, i.e.

lon.shape = (nlats,nlons)
lat.shape = (nlats,nlons)
data.shape = (nlats,nlons)

You'll need to know what the grid dimensons (nlats,nlons) are.

-Jeff

10

It seems that I just cannot grasp the way the data needs to be formatted for
this to work...
I've used the griddata sample that James posted but it takes about 10
minutes to prep the data for plotting so that solution seems to be out of
discussion.

I guess my issue is that I don't know what type of data is required by
contourf function. Also as Jeff was saying earlier, the data is read from a
grib file so supposedly it's already gridded. I've also looked at the
basemap demo
(http://matplotlib.sourceforge.net/users/screenshots.html#basemap-demo) and
the data is read from 3 files, one for Lat one for Long and the Last for Z
Data. Is there a way to automatically extract the data from the grib file to
a format similar to the one used in the basemap example?

Jeff Whitaker wrote:

···

Mauro Cavalcanti wrote:

Dear Anton,

2008/12/23 antonv <vasilescu_anton@...9...>:
  

Also, because I figured out the data I need and already have the
scripts in place
to extract the CSV files I would really like to keep it that way. Would
it be possible to
just show me how to get from the csv file to the plot?
    
Here is a short recipe:

import numpy as np

f = open("file.csv", "r")
coords = np.loadtxt(f, delimiter=",", skiprows=1)
lon = coords[:,0]
lat = coords[:,1]
dat = coords[:,2]

where "file.csv" is a regular comma-separated values file in the format:

Lat,Lon,Dat
-61.05,10.4,20
-79.43,9.15,50
-70.66,9.53,10
-63.11,7.91,40
...

Hope this helps!

Best regards,

Since the arrays are 2D (for gridded data), a reshape is also needed, i.e.

lon.shape = (nlats,nlons)
lat.shape = (nlats,nlons)
data.shape = (nlats,nlons)

You'll need to know what the grid dimensons (nlats,nlons) are.

-Jeff

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

--
View this message in context: http://www.nabble.com/Plotting-NOAA-data…-tp21139727p21190078.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

11

antonv wrote:

It seems that I just cannot grasp the way the data needs to be formatted for
this to work...
I've used the griddata sample that James posted but it takes about 10
minutes to prep the data for plotting so that solution seems to be out of
discussion.

I guess my issue is that I don't know what type of data is required by
contourf function. Also as Jeff was saying earlier, the data is read from a
grib file so supposedly it's already gridded. I've also looked at the
basemap demo
(http://matplotlib.sourceforge.net/users/screenshots.html#basemap-demo) and
the data is read from 3 files, one for Lat one for Long and the Last for Z
Data. Is there a way to automatically extract the data from the grib file to
a format similar to the one used in the basemap example?
  
Anton: I just looked at your csv file and I think I know what the problem is. Whatever program you used to dump the grib data did not write all the data - the missing land values were skipped. That means you don't have the full rectangular array of data. I think you have two choices:

1) insert the missing land values into the array, either in the csv file or into the array after it is read in from the csv file. What program did you use to dump the GRIB data to a CSV file?

2) use a python grib interface. If you're on Windows, PyNIO won't work. I've written my own module (pygrib2 - Google Code Archive - Long-term storage for Google Code Project Hosting.) which you should be able to compile on windows. You'll need the png and jasper (jpeg2000) libraries, however.

I recommend (2) - in the time you've already spent messing with that csv file, you could have already gotten a real python grib reader working!

-Jeff

···

Jeff Whitaker wrote:
  

Mauro Cavalcanti wrote:
    

Dear Anton,

2008/12/23 antonv <vasilescu_anton@...9...>:
  

Also, because I figured out the data I need and already have the
scripts in place
to extract the CSV files I would really like to keep it that way. Would
it be possible to
just show me how to get from the csv file to the plot?
    

Here is a short recipe:

import numpy as np

f = open("file.csv", "r")
coords = np.loadtxt(f, delimiter=",", skiprows=1)
lon = coords[:,0]
lat = coords[:,1]
dat = coords[:,2]

where "file.csv" is a regular comma-separated values file in the format:

Lat,Lon,Dat
-61.05,10.4,20
-79.43,9.15,50
-70.66,9.53,10
-63.11,7.91,40
...

Hope this helps!

Best regards,

Since the arrays are 2D (for gridded data), a reshape is also needed, i.e.

lon.shape = (nlats,nlons)
lat.shape = (nlats,nlons)
data.shape = (nlats,nlons)

You'll need to know what the grid dimensons (nlats,nlons) are.

-Jeff

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

12

Hey Jeff,

I've got it sorted out a bit now. You're right the data was an output from
Degrib and I had the option to output the csv's with or without data in the
land areas. As before I was using a program that was placing the pixels in
an image based on the X and Y columns it didn't create an issue. That was an
easy fix by switching the option in the Degrib export.

Also by looking at your example I realized the way the contourf function
requests the Z data and by just reshaping the array I was able to make all
the stuff work properly. Numpy is amazing by the way! I had no idea how easy
you can work with huge arrays!

My new issue is that I need to mask the land areas in the Z array so I would
have a clean plot over the basemap. Any ideas on how to achieve that?

Thanks,
Anton

Jeff Whitaker wrote:

···

antonv wrote:

It seems that I just cannot grasp the way the data needs to be formatted
for
this to work...
I've used the griddata sample that James posted but it takes about 10
minutes to prep the data for plotting so that solution seems to be out of
discussion.

I guess my issue is that I don't know what type of data is required by
contourf function. Also as Jeff was saying earlier, the data is read from
a
grib file so supposedly it's already gridded. I've also looked at the
basemap demo
(http://matplotlib.sourceforge.net/users/screenshots.html#basemap-demo)
and
the data is read from 3 files, one for Lat one for Long and the Last for
Z
Data. Is there a way to automatically extract the data from the grib file
to
a format similar to the one used in the basemap example?
  
Anton: I just looked at your csv file and I think I know what the
problem is. Whatever program you used to dump the grib data did not
write all the data - the missing land values were skipped. That means
you don't have the full rectangular array of data. I think you have two
choices:

1) insert the missing land values into the array, either in the csv file
or into the array after it is read in from the csv file. What program
did you use to dump the GRIB data to a CSV file?

2) use a python grib interface. If you're on Windows, PyNIO won't
work. I've written my own module (pygrib2 -
Google Code Archive - Long-term storage for Google Code Project Hosting.) which you should be able to compile on
windows. You'll need the png and jasper (jpeg2000) libraries, however.

I recommend (2) - in the time you've already spent messing with that csv
file, you could have already gotten a real python grib reader working!

-Jeff

Jeff Whitaker wrote:
  

Mauro Cavalcanti wrote:
    

Dear Anton,

2008/12/23 antonv <vasilescu_anton@...9...>:
  

Also, because I figured out the data I need and already have the
scripts in place
to extract the CSV files I would really like to keep it that way.
Would
it be possible to
just show me how to get from the csv file to the plot?
    

Here is a short recipe:

import numpy as np

f = open("file.csv", "r")
coords = np.loadtxt(f, delimiter=",", skiprows=1)
lon = coords[:,0]
lat = coords[:,1]
dat = coords[:,2]

where "file.csv" is a regular comma-separated values file in the
format:

Lat,Lon,Dat
-61.05,10.4,20
-79.43,9.15,50
-70.66,9.53,10
-63.11,7.91,40
...

Hope this helps!

Best regards,

Since the arrays are 2D (for gridded data), a reshape is also needed,
i.e.

lon.shape = (nlats,nlons)
lat.shape = (nlats,nlons)
data.shape = (nlats,nlons)

You'll need to know what the grid dimensons (nlats,nlons) are.

-Jeff

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

--
View this message in context: http://www.nabble.com/Plotting-NOAA-data…-tp21139727p21199108.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

13

antonv wrote:

Hey Jeff,

I've got it sorted out a bit now. You're right the data was an output from
Degrib and I had the option to output the csv's with or without data in the
land areas. As before I was using a program that was placing the pixels in
an image based on the X and Y columns it didn't create an issue. That was an
easy fix by switching the option in the Degrib export.

Also by looking at your example I realized the way the contourf function
requests the Z data and by just reshaping the array I was able to make all
the stuff work properly. Numpy is amazing by the way! I had no idea how easy
you can work with huge arrays!

My new issue is that I need to mask the land areas in the Z array so I would
have a clean plot over the basemap. Any ideas on how to achieve that?
  
Create a masked array. Say the values in the Z array set to 1.e30 over land areas in the CSV file.

from numpy import ma
Z = ma.masked_values(Z,1.e30)

Then plot with contourf as before and the land areas will not be contoured.

-Jeff

···

Thanks,
Anton

Jeff Whitaker wrote:
  

antonv wrote:
    

It seems that I just cannot grasp the way the data needs to be formatted
for
this to work...
I've used the griddata sample that James posted but it takes about 10
minutes to prep the data for plotting so that solution seems to be out of
discussion.

I guess my issue is that I don't know what type of data is required by
contourf function. Also as Jeff was saying earlier, the data is read from
a
grib file so supposedly it's already gridded. I've also looked at the
basemap demo
(http://matplotlib.sourceforge.net/users/screenshots.html#basemap-demo)
and
the data is read from 3 files, one for Lat one for Long and the Last for
Z
Data. Is there a way to automatically extract the data from the grib file
to
a format similar to the one used in the basemap example?
  

Anton: I just looked at your csv file and I think I know what the problem is. Whatever program you used to dump the grib data did not write all the data - the missing land values were skipped. That means you don't have the full rectangular array of data. I think you have two choices:

1) insert the missing land values into the array, either in the csv file or into the array after it is read in from the csv file. What program did you use to dump the GRIB data to a CSV file?

2) use a python grib interface. If you're on Windows, PyNIO won't work. I've written my own module (pygrib2 - Google Code Archive - Long-term storage for Google Code Project Hosting.) which you should be able to compile on windows. You'll need the png and jasper (jpeg2000) libraries, however.

I recommend (2) - in the time you've already spent messing with that csv file, you could have already gotten a real python grib reader working!

-Jeff
    

Jeff Whitaker wrote:
  

Mauro Cavalcanti wrote:
    

Dear Anton,

2008/12/23 antonv <vasilescu_anton@...9...>:
  

Also, because I figured out the data I need and already have the
scripts in place
to extract the CSV files I would really like to keep it that way.
Would
it be possible to
just show me how to get from the csv file to the plot?
    

Here is a short recipe:

import numpy as np

f = open("file.csv", "r")
coords = np.loadtxt(f, delimiter=",", skiprows=1)
lon = coords[:,0]
lat = coords[:,1]
dat = coords[:,2]

where "file.csv" is a regular comma-separated values file in the
format:

Lat,Lon,Dat
-61.05,10.4,20
-79.43,9.15,50
-70.66,9.53,10
-63.11,7.91,40
...

Hope this helps!

Best regards,

Since the arrays are 2D (for gridded data), a reshape is also needed,
i.e.

lon.shape = (nlats,nlons)
lat.shape = (nlats,nlons)
data.shape = (nlats,nlons)

You'll need to know what the grid dimensons (nlats,nlons) are.

-Jeff

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

14

So my beginner saga continues with another question:
I am trying to create a custom colormap using ListedColormap. The custom
color map is for a range of values between 0 and 20 while the data in my
file is between 0 and 8. Now my issue is that when plotting the graph the
colormap is using the 0 to 8 values from the file. How can I force it to use
all the 21 values in the colormap?

Thanks,
Anton

Jeff Whitaker wrote:

···

antonv wrote:

Hey Jeff,

I've got it sorted out a bit now. You're right the data was an output
from
Degrib and I had the option to output the csv's with or without data in
the
land areas. As before I was using a program that was placing the pixels
in
an image based on the X and Y columns it didn't create an issue. That was
an
easy fix by switching the option in the Degrib export.

Also by looking at your example I realized the way the contourf function
requests the Z data and by just reshaping the array I was able to make
all
the stuff work properly. Numpy is amazing by the way! I had no idea how
easy
you can work with huge arrays!

My new issue is that I need to mask the land areas in the Z array so I
would
have a clean plot over the basemap. Any ideas on how to achieve that?
  
Create a masked array. Say the values in the Z array set to 1.e30 over
land areas in the CSV file.

from numpy import ma
Z = ma.masked_values(Z,1.e30)

Then plot with contourf as before and the land areas will not be
contoured.

-Jeff

Thanks,
Anton

Jeff Whitaker wrote:
  

antonv wrote:
    

It seems that I just cannot grasp the way the data needs to be
formatted
for
this to work...
I've used the griddata sample that James posted but it takes about 10
minutes to prep the data for plotting so that solution seems to be out
of
discussion.

I guess my issue is that I don't know what type of data is required by
contourf function. Also as Jeff was saying earlier, the data is read
from
a
grib file so supposedly it's already gridded. I've also looked at the
basemap demo
(http://matplotlib.sourceforge.net/users/screenshots.html#basemap-demo)
and
the data is read from 3 files, one for Lat one for Long and the Last
for
Z
Data. Is there a way to automatically extract the data from the grib
file
to
a format similar to the one used in the basemap example?
  

Anton: I just looked at your csv file and I think I know what the
problem is. Whatever program you used to dump the grib data did not
write all the data - the missing land values were skipped. That means
you don't have the full rectangular array of data. I think you have two
choices:

1) insert the missing land values into the array, either in the csv file
or into the array after it is read in from the csv file. What program
did you use to dump the GRIB data to a CSV file?

2) use a python grib interface. If you're on Windows, PyNIO won't
work. I've written my own module (pygrib2 -
Google Code Archive - Long-term storage for Google Code Project Hosting.) which you should be able to compile on
windows. You'll need the png and jasper (jpeg2000) libraries, however.

I recommend (2) - in the time you've already spent messing with that csv
file, you could have already gotten a real python grib reader working!

-Jeff
    

Jeff Whitaker wrote:
  

Mauro Cavalcanti wrote:
    

Dear Anton,

2008/12/23 antonv <vasilescu_anton@...9...>:
  

Also, because I figured out the data I need and already have the
scripts in place
to extract the CSV files I would really like to keep it that way.
Would
it be possible to
just show me how to get from the csv file to the plot?
    

Here is a short recipe:

import numpy as np

f = open("file.csv", "r")
coords = np.loadtxt(f, delimiter=",", skiprows=1)
lon = coords[:,0]
lat = coords[:,1]
dat = coords[:,2]

where "file.csv" is a regular comma-separated values file in the
format:

Lat,Lon,Dat
-61.05,10.4,20
-79.43,9.15,50
-70.66,9.53,10
-63.11,7.91,40
...

Hope this helps!

Best regards,

Since the arrays are 2D (for gridded data), a reshape is also needed,
i.e.

lon.shape = (nlats,nlons)
lat.shape = (nlats,nlons)
data.shape = (nlats,nlons)

You'll need to know what the grid dimensons (nlats,nlons) are.

-Jeff

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

------------------------------------------------------------------------------
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

--
View this message in context: http://www.nabble.com/Plotting-NOAA-data…-tp21139727p21244624.html
Sent from the matplotlib - users mailing list archive at Nabble.com.