Warning: converting a masked element to nan

Hi All,
I am trying to plot time against mean daily temperature values. The problem is temperature contains no data in a few areas. I want to be able to ignore this and continue plotting.
When I run the below script on my data that has all normal numbers it works fine but when I run the script on my data that has sections of ‘no data’ I receive the below error and the graph will not plot the trendline. Any suggestions on how I could fix this.
Thanks
Warning (from warnings module):
File “C:\Python27\lib\site-packages\numpy\ma\core.py”, line 3785
warnings.warn(“Warning: converting a masked element to nan.”)
UserWarning: Warning: converting a masked element to nan.

from netCDF4 import Dataset
import matplotlib.pyplot as plt
import numpy as N
from mpl_toolkits.basemap import Basemap
from netcdftime import utime
from datetime import datetime
import os
from numpy import *
import matplotlib.dates as mdates
from numpy import ma as MA

TSFCmeanall=[]
timeall=[]
time_intall=[]

MainFolder=r"E:/GriddedData/T_SFC/1987/"
for (path, dirs, files) in os.walk(MainFolder):
for dir in dirs:
print dir
path=path+’/’

                    for ncfile in files:
                            if ncfile[-3:]=='.nc':
                                ncfile=os.path.join(path,ncfile)
                                ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
                                TSFC=ncfile.variables['T_SFC'][0:20]
                                TIME=ncfile.variables['time'][0:20]
                                fillvalue=ncfile.variables['T_SFC']._FillValue
                                TSFC=MA.masked_values(TSFC, fillvalue)
                                ncfile.close()

                                for TSFC, TIME in zip((TSFC[:]),(TIME[:])):
                                    cdftime=utime('seconds since 1970-01-01 00:00:00')
                                    ncfiletime=cdftime.num2date(TIME)
                                    timestr=str(ncfiletime)
                                    d = datetime.strptime(timestr, '%Y-%m-%d %H:%M:%S')
                                    date_string = d.strftime('%Y%m%d%H')
                                    time_int=int(date_string)

                                    TSFCmean=N.mean(TSFC)

                                    TSFCmeanall.append(TSFCmean)
                                    timeall.append(ncfiletime)
                                    time_intall.append(time_int)

x=timeall
y=TSFCmeanall
x2=time_intall

fig, ax=plt.subplots(1)

z=N.polyfit(x2,y,1)
p=N.poly1d(z)

plt.plot(x,y)
plt.plot(x,p(x2),‘r–’) #add trendline to plot

fig.autofmt_xdate()
ax.fmt_xdata=mdates.DateFormatter(’%Y-%m-%d %H:%M:%S’)
plt.ylabel(“Temperature C”)
plt.title(“Mean Daily Temp”)
plt.show()

···

matplotlib should handle both masked arrays and arrays with NaNs and
treat both the same. Can you reduce the script to something that
can be run independently without data? It’s not clear to me yet why
this is failing.

Mike
···

http://p.sf.net/sfu/rsa-sfdev2dev1Matplotlib-users@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/matplotlib-users

I have seen situations when using datetimes for the x axis where if all the data being viewed is masked (or nans, or whatever) then it errors out because the axis range can’t be determined. Maybe that is what is happening here?

Ben Root

···

On Wed, Nov 2, 2011 at 11:48 AM, Michael Droettboom <mdroe@…86…> wrote:

matplotlib should handle both masked arrays and arrays with NaNs and

treat both the same. Can you reduce the script to something that
can be run independently without data? It’s not clear to me yet why
this is failing.

Mike
On 11/01/2011 05:24 PM, questions anon wrote:

Hi All,
I am trying to plot time against mean daily temperature values.
The problem is temperature contains no data in a few areas. I want
to be able to ignore this and continue plotting.
When I run the below script on my data that has all normal numbers
it works fine but when I run the script on my data that has
sections of ‘no data’ I receive the below error and the graph will
not plot the trendline. Any suggestions on how I could fix this.
Thanks

  *Warning (from warnings module):
      File "C:\Python27\lib\site-packages\numpy\ma\core.py", line

3785
warnings.warn(“Warning: converting a masked element to
nan.”)
UserWarning: Warning: converting a masked element to nan.*

  from netCDF4 import Dataset

  import matplotlib.pyplot as plt

  import numpy as N

  from mpl_toolkits.basemap import Basemap

  from netcdftime import utime

  from datetime import datetime

  import os

  from numpy import *

  import matplotlib.dates as mdates

  from numpy import ma as MA



  TSFCmeanall=[]

  timeall=[]

  time_intall=[]



  MainFolder=r"E:/GriddedData/T_SFC/1987/"

  for (path, dirs, files) in os.walk(MainFolder):

                          for dir in dirs:

                                  print dir

                          path=path+'/'

                  

                          for ncfile in files:

                                  if ncfile[-3:]=='.nc':

ncfile=os.path.join(path,ncfile)

                                      ncfile=Dataset(ncfile, 'r+',

‘NETCDF4’)

TSFC=ncfile.variables[‘T_SFC’][0:20]

TIME=ncfile.variables[‘time’][0:20]

fillvalue=ncfile.variables[‘T_SFC’]._FillValue

                                      TSFC=MA.masked_values(TSFC,

fillvalue)

                                      ncfile.close()



                                      for TSFC, TIME in

zip((TSFC[:]),(TIME[:])):

                                          cdftime=utime('seconds

since 1970-01-01 00:00:00’)

ncfiletime=cdftime.num2date(TIME)

                                          timestr=str(ncfiletime)

                                          d =

datetime.strptime(timestr, ‘%Y-%m-%d %H:%M:%S’)

                                          date_string =

d.strftime(‘%Y%m%d%H’)

                                          time_int=int(date_string)

   

                                          TSFCmean=N.mean(TSFC)

TSFCmeanall.append(TSFCmean)

                                          timeall.append(ncfiletime)

time_intall.append(time_int)

  x=timeall

  y=TSFCmeanall

  x2=time_intall



  fig, ax=plt.subplots(1)



  z=N.polyfit(x2,y,1)

  p=N.poly1d(z)



  plt.plot(x,y)

  plt.plot(x,p(x2),'r--') #add trendline to plot



  fig.autofmt_xdate()

  ax.fmt_xdata=mdates.DateFormatter('%Y-%m-%d %H:%M:%S')

  plt.ylabel("Temperature C")

  plt.title("Mean Daily Temp")

  plt.show()  
------------------------------------------------------------------------------
RSA&#174; Conference 2012
Save $700 by Nov 18
Register now&#33;
[http://p.sf.net/sfu/rsa-sfdev2dev1](http://p.sf.net/sfu/rsa-sfdev2dev1)
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
[https://lists.sourceforge.net/lists/listinfo/matplotlib-users](https://lists.sourceforge.net/lists/listinfo/matplotlib-users)

RSA® Conference 2012

Save $700 by Nov 18

Register now!

http://p.sf.net/sfu/rsa-sfdev2dev1


Matplotlib-users mailing list

Matplotlib-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/matplotlib-users

Thanks, I think you are right about the datetimes for the x axis causing the problem.
Does anyone have any ideas how to resolve this?

···

On Thu, Nov 3, 2011 at 4:11 AM, Benjamin Root <ben.root@…1304…> wrote:

I have seen situations when using datetimes for the x axis where if all the data being viewed is masked (or nans, or whatever) then it errors out because the axis range can’t be determined. Maybe that is what is happening here?

Ben Root

On Wed, Nov 2, 2011 at 11:48 AM, Michael Droettboom <mdroe@…86…> wrote:

matplotlib should handle both masked arrays and arrays with NaNs and

treat both the same. Can you reduce the script to something that
can be run independently without data? It’s not clear to me yet why
this is failing.

Mike
On 11/01/2011 05:24 PM, questions anon wrote:

Hi All,
I am trying to plot time against mean daily temperature values.
The problem is temperature contains no data in a few areas. I want
to be able to ignore this and continue plotting.
When I run the below script on my data that has all normal numbers
it works fine but when I run the script on my data that has
sections of ‘no data’ I receive the below error and the graph will
not plot the trendline. Any suggestions on how I could fix this.
Thanks

  *Warning (from warnings module):
      File "C:\Python27\lib\site-packages\numpy\ma\core.py", line

3785
warnings.warn(“Warning: converting a masked element to
nan.”)
UserWarning: Warning: converting a masked element to nan.*

  from netCDF4 import Dataset

  import matplotlib.pyplot as plt

  import numpy as N

  from mpl_toolkits.basemap import Basemap

  from netcdftime import utime

  from datetime import datetime

  import os

  from numpy import *

  import matplotlib.dates as mdates

  from numpy import ma as MA



  TSFCmeanall=[]

  timeall=[]

  time_intall=[]



  MainFolder=r"E:/GriddedData/T_SFC/1987/"

  for (path, dirs, files) in os.walk(MainFolder):

                          for dir in dirs:

                                  print dir

                          path=path+'/'

                  

                          for ncfile in files:

                                  if ncfile[-3:]=='.nc':

ncfile=os.path.join(path,ncfile)

                                      ncfile=Dataset(ncfile, 'r+',

‘NETCDF4’)

TSFC=ncfile.variables[‘T_SFC’][0:20]

TIME=ncfile.variables[‘time’][0:20]

fillvalue=ncfile.variables[‘T_SFC’]._FillValue

                                      TSFC=MA.masked_values(TSFC,

fillvalue)

                                      ncfile.close()



                                      for TSFC, TIME in

zip((TSFC[:]),(TIME[:])):

                                          cdftime=utime('seconds

since 1970-01-01 00:00:00’)

ncfiletime=cdftime.num2date(TIME)

                                          timestr=str(ncfiletime)

                                          d =

datetime.strptime(timestr, ‘%Y-%m-%d %H:%M:%S’)

                                          date_string =

d.strftime(‘%Y%m%d%H’)

                                          time_int=int(date_string)

   

                                          TSFCmean=N.mean(TSFC)

TSFCmeanall.append(TSFCmean)

                                          timeall.append(ncfiletime)

time_intall.append(time_int)

  x=timeall

  y=TSFCmeanall

  x2=time_intall



  fig, ax=plt.subplots(1)



  z=N.polyfit(x2,y,1)

  p=N.poly1d(z)



  plt.plot(x,y)

  plt.plot(x,p(x2),'r--') #add trendline to plot



  fig.autofmt_xdate()

  ax.fmt_xdata=mdates.DateFormatter('%Y-%m-%d %H:%M:%S')

  plt.ylabel("Temperature C")

  plt.title("Mean Daily Temp")

  plt.show()  
------------------------------------------------------------------------------
RSA&#174; Conference 2012
Save $700 by Nov 18
Register now&#33;
[http://p.sf.net/sfu/rsa-sfdev2dev1](http://p.sf.net/sfu/rsa-sfdev2dev1)
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
[https://lists.sourceforge.net/lists/listinfo/matplotlib-users](https://lists.sourceforge.net/lists/listinfo/matplotlib-users)

RSA® Conference 2012

Save $700 by Nov 18

Register now!

http://p.sf.net/sfu/rsa-sfdev2dev1


Matplotlib-users mailing list

Matplotlib-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/matplotlib-users


RSA® Conference 2012

Save $700 by Nov 18

Register now!

http://p.sf.net/sfu/rsa-sfdev2dev1


Matplotlib-users mailing list

Matplotlib-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/matplotlib-users

From: questions anon [mailto:questions.anon@…1896…]
Sent: Wednesday, November 02, 2011 17:17

Thanks, I think you are right about the datetimes for the x axis causing the problem.
Does anyone have any ideas how to resolve this?

Does it help to call ax.xaxis_date() before your calls to plt.plot()?

no that didn’t work and I am back to thinking it is not the dates fault, this is because if I only choose a section of my array that I know doesn’t have any NANs it works fine. Is there a way to tell is to skip/ignore these?

···

On Sat, Nov 5, 2011 at 8:50 AM, Stan West <stan.west@…3836…> wrote:

From: questions anon [mailto:questions.anon@…287… ]
Sent: Wednesday, November 02, 2011 17:17

Thanks, I think you are right about the datetimes for the x axis causing the problem.
Does anyone have any ideas how to resolve this?

Does it help to call ax.xaxis_date() before your calls to plt.plot()?