problem with (poor) quiver performance & X, Y, U, V arrays

Community matplotlib-users
1

Im experiencing very poor performance when using the 'quiver' function over relatively large grids.
Im using quiver to plot wind 'u,v' data over a lat/lon grid using basemap.

quiver performs decently over small lat/lon ranges, such as a bounding box of lat(0-30),lon(-120- -100), but when I try to plot larger areas (i.e. the entire globe), quiver causes a very large pause. through some debugging of quiver.py, I narrowed down the performance lag to the "set_vertices" function call within the "draw()" function in the quiver class. as a test, I printed 'len(vert)' in order to see the vertice array length that was causing problems...it seems that Im getting vertice counts in the high thousands, and quiver seems to struggle with this.

should quiver be able to easily handle such a large amount of vertices?

A secondary question:
the method Im using to create my X,Y,U,V arrays is creating 'larger-than-necessary' X,Y arrays; i.e., Im not plotting the U,V vectors at each lat&lon point, Im 'skipping' over 'every-nth-point', so my U,V arrays are equal in size to my X,Y arrays, but have many empty elements.

Unfortunately, the array data is being imported from an external program (named 'GRADS'), so I cannot prevent these arrays from being 'oversized' upon their creation.

example:
for a (10 degree lat)x(10 degree lon) area, I might have arrays like this:

X='[0,0.5,1.0,1.5,2.0,2.5,...10.0],[0,0.5,...10],...[0.0,0.5,...10.0]'
Y='[0,0.5,1.0,1.5,2.0,2.5,...10.0],[0,0.5,...10],...[0.0,0.5,...10.0]'
U='[--,--,0.40,--,--,0.15,...0.30],[--,--,0.25,--,--...,0.12],...[--,--,0.50,...10.0]'
V='[--,--,0.30,--,--,0.25,...0.40],[--,--,0.25,--,--...,0.12],...[--,--,0.50,...10.0]'

these values are completely inaccurate and are meant just to illustrate the fact that I have many 'skipped' values in my U,V arrays, and I have oversized X,Y arrays that cover the 10x10 lat/lon grid...

So, what I want to do is create NEW X,Y,U,V arrays or remove elements from these existing X,Y,U,V arrays, so that:
1) only valid, "non-empty" values will exist in my U,V arrays..and
2) the only values that exist in X,Y are those that correspond to valid points in the U,V arrays...

Im not very experienced with python/matplotlib, so I dont know what would be the best way to iterate over these 4 arrays and remove the empty invalid elements (or copy the valid elements into new arrays). How can I go about shortening these arrays?

Im hoping that with less points in my X,Y arrays, that quiver will perform faster since it isnt wasting time trying to process X,Y points where U,V are empty).

Please help,
P.Romero

···

_________________________________________________________________
Hotmail® is up to 70% faster. Now good news travels really fast.
http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_HM_70faster_032009

2

Pablo Romero wrote:

Im experiencing very poor performance when using the 'quiver' function over relatively large grids.
Im using quiver to plot wind 'u,v' data over a lat/lon grid using basemap.
quiver performs decently over small lat/lon ranges, such as a bounding box of lat(0-30),lon(-120- -100), but when I try to plot larger areas (i.e. the entire globe), quiver causes a very large pause. through some debugging of quiver.py, I narrowed down the performance lag to the "set_vertices" function call within the "draw()" function in the quiver class. as a test, I printed 'len(vert)' in order to see the vertice array length that was causing problems...it seems that Im getting vertice counts in the high thousands, and quiver seems to struggle with this.
should quiver be able to easily handle such a large amount of vertices?

What version of matplotlib are you using, and on what platform?

How long is the pause, and for how many vectors?

I presume you are referring to the set_verts() method which is inherited from collections.PolyCollection. It is using a list comprehension to loop over the vectors, making a path for each, so I would not expect it to be particularly quick; but whether it is unreasonable, or whether it can be sped up reasonably easily, I don't know.

the method Im using to create my X,Y,U,V arrays is creating 'larger-than-necessary' X,Y arrays; i.e., Im not plotting the U,V vectors at each lat&lon point, Im 'skipping' over 'every-nth-point', so my U,V arrays are equal in size to my X,Y arrays, but have many empty elements.
Unfortunately, the array data is being imported from an external program (named 'GRADS'), so I cannot prevent these arrays from being 'oversized' upon their creation.
example:
for a (10 degree lat)x(10 degree lon) area, I might have arrays like this:
X='[0,0.5,1.0,1.5,2.0,2.5,...10.0],[0,0.5,...10],...[0.0,0.5,...10.0]'
Y='[0,0.5,1.0,1.5,2.0,2.5,...10.0],[0,0.5,...10],...[0.0,0.5,...10.0]'
U='[--,--,0.40,--,--,0.15,...0.30],[--,--,0.25,--,--...,0.12],...[--,--,0.50,...10.0]'
V='[--,--,0.30,--,--,0.25,...0.40],[--,--,0.25,--,--...,0.12],...[--,--,0.50,...10.0]'
these values are completely inaccurate and are meant just to illustrate the fact that I have many 'skipped' values in my U,V arrays, and I have oversized X,Y arrays that cover the 10x10 lat/lon grid...
So, what I want to do is create NEW X,Y,U,V arrays or remove elements from these existing X,Y,U,V arrays, so that:
1) only valid, "non-empty" values will exist in my U,V arrays..and
2) the only values that exist in X,Y are those that correspond to valid points in the U,V arrays...
Im not very experienced with python/matplotlib, so I dont know what would be the best way to iterate over these 4 arrays and remove the empty invalid elements (or copy the valid elements into new arrays). How can I go about shortening these arrays?

Assuming your X, Y, U, V are all masked arrays or ndarrays of the same shape, you can use

from matplotlib.cbook import delete_masked_points

x,y,u,v = delete_masked_points(X.ravel(), Y.ravel(). U.ravel(), V.ravel())

The Barbs class in quiver.py uses this function; Quiver could be modified to use it, with the loss of a bit of functionality that I suspect no one is using anyway.

All the .ravel() method calls are needed only if the arrays are 2-D or higher.

Im hoping that with less points in my X,Y arrays, that quiver will perform faster since it isnt wasting time trying to process X,Y points where U,V are empty).

Let us know how much difference it makes, and tell us what the initial and modified number of vectors is.

Eric

3

Eric,
version matplotlib-0.98.5.2
running on fedora 9, i386.

the pause was significant, up to about 30 seconds to a minute.

HOWEVER, your delete_masked_points() solution worked incredibly, reducing the amount of vertices by a giant amount (in one case, from 6,000+ to 180 vertices), and the rendering time to almost nothing.

Im still not clear on how the 'delete_masked_points()' or 'X.ravel()' functions work; I need to read up on these.

One thing that *IS* clear is that my previous X,Y arrays were 2D arrays, and the resulting x,y arrays after using your solution are only 1D arrays. The new 1D X,Y arrays do work with the quiver() function, which I dont understand, since I was under the impression that these X,Y arrays needed to be created using a method like this:

m=basemap(...)
X,Y=m(*numpy.meshgrid(lat_points_array,lon_points_array))

and that this should, in theory, create the 2D X,Y arrays that quiver requires, correct?

am I missing something here?

the new 1D arrays are more than twice the size than the original arrays as reported by 'len()', but they are obviously much smaller due to the fact that they are 1D arrays.

Anyway, thanks again for the help,
P.Romero

···

----------------------------------------

Date: Sat, 14 Mar 2009 17:57:27 -1000
From: efiring@...202...
Subject: Re: [Matplotlib-users] problem with (poor) quiver performance & X, Y, U, V arrays
To: romero619@...32...
CC: matplotlib-users@lists.sourceforge.net

Pablo Romero wrote:

Im experiencing very poor performance when using the 'quiver' function over relatively large grids.
Im using quiver to plot wind 'u,v' data over a lat/lon grid using basemap.

quiver performs decently over small lat/lon ranges, such as a bounding box of lat(0-30),lon(-120- -100), but when I try to plot larger areas (i.e. the entire globe), quiver causes a very large pause. through some debugging of quiver.py, I narrowed down the performance lag to the "set_vertices" function call within the "draw()" function in the quiver class. as a test, I printed 'len(vert)' in order to see the vertice array length that was causing problems...it seems that Im getting vertice counts in the high thousands, and quiver seems to struggle with this.

should quiver be able to easily handle such a large amount of vertices?

What version of matplotlib are you using, and on what platform?

How long is the pause, and for how many vectors?

I presume you are referring to the set_verts() method which is inherited
from collections.PolyCollection. It is using a list comprehension to
loop over the vectors, making a path for each, so I would not expect it
to be particularly quick; but whether it is unreasonable, or whether it
can be sped up reasonably easily, I don't know.

A secondary question:
the method Im using to create my X,Y,U,V arrays is creating 'larger-than-necessary' X,Y arrays; i.e., Im not plotting the U,V vectors at each lat&lon point, Im 'skipping' over 'every-nth-point', so my U,V arrays are equal in size to my X,Y arrays, but have many empty elements.

Unfortunately, the array data is being imported from an external program (named 'GRADS'), so I cannot prevent these arrays from being 'oversized' upon their creation.

example:
for a (10 degree lat)x(10 degree lon) area, I might have arrays like this:

X='[0,0.5,1.0,1.5,2.0,2.5,...10.0],[0,0.5,...10],...[0.0,0.5,...10.0]'
Y='[0,0.5,1.0,1.5,2.0,2.5,...10.0],[0,0.5,...10],...[0.0,0.5,...10.0]'
U='[--,--,0.40,--,--,0.15,...0.30],[--,--,0.25,--,--...,0.12],...[--,--,0.50,...10.0]'
V='[--,--,0.30,--,--,0.25,...0.40],[--,--,0.25,--,--...,0.12],...[--,--,0.50,...10.0]'

these values are completely inaccurate and are meant just to illustrate the fact that I have many 'skipped' values in my U,V arrays, and I have oversized X,Y arrays that cover the 10x10 lat/lon grid...

So, what I want to do is create NEW X,Y,U,V arrays or remove elements from these existing X,Y,U,V arrays, so that:
1) only valid, "non-empty" values will exist in my U,V arrays..and
2) the only values that exist in X,Y are those that correspond to valid points in the U,V arrays...

Im not very experienced with python/matplotlib, so I dont know what would be the best way to iterate over these 4 arrays and remove the empty invalid elements (or copy the valid elements into new arrays). How can I go about shortening these arrays?

Assuming your X, Y, U, V are all masked arrays or ndarrays of the same
shape, you can use

from matplotlib.cbook import delete_masked_points

x,y,u,v = delete_masked_points(X.ravel(), Y.ravel(). U.ravel(), V.ravel())

The Barbs class in quiver.py uses this function; Quiver could be
modified to use it, with the loss of a bit of functionality that I
suspect no one is using anyway.

All the .ravel() method calls are needed only if the arrays are 2-D or
higher.

Im hoping that with less points in my X,Y arrays, that quiver will perform faster since it isnt wasting time trying to process X,Y points where U,V are empty).

Let us know how much difference it makes, and tell us what the initial
and modified number of vectors is.

Eric

_________________________________________________________________
Windows Live™ Groups: Create an online spot for your favorite groups to meet.
http://windowslive.com/online/groups?ocid=TXT_TAGLM_WL_groups_032009