Performance problems

Hi, I upgraded to matplotlib0.82 from the previous version. Now my
markers plot fast. I plotted 30,000 points without too much complaining
attached the sample code I was using. If I set marker = 'None' it
plotted just fine. If I set marker = 'o' is would seize up on me.

Thanks for the help!

sample.py (3.92 KB)

···

from my computer. Not sure what the problem was but it works now. I

-----Original Message-----
From: matplotlib-users-admin@lists.sourceforge.net
[mailto:matplotlib-users-admin@lists.sourceforge.net] On Behalf Of
matplotlib-users-request@lists.sourceforge.net
Sent: Tuesday, July 12, 2005 11:50 AM
To: matplotlib-users@lists.sourceforge.net
Subject: Matplotlib-users digest, Vol 1 #698 - 14 msgs

Send Matplotlib-users mailing list submissions to
  matplotlib-users@lists.sourceforge.net

To subscribe or unsubscribe via the World Wide Web, visit
  matplotlib-users List Signup and Options
or, via email, send a message with subject or body 'help' to
  matplotlib-users-request@lists.sourceforge.net

You can reach the person managing the list at
  matplotlib-users-admin@lists.sourceforge.net

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Matplotlib-users digest..."

Today's Topics:

   1. Re: plot with nans (Andrew Straw)
   2. Re: difference between imshow and pcolor (Norbert Nemec)
   3. Re: plot with nans (Norbert Nemec)
   4. Re: plot with nans (Eric Firing)
   5. Re: plot with nans (Jeff Whitaker)
   6. Re: plot with nans (Andrew Straw)
   7. Re: RE: Performance problems (Matt Newville)
   8. Re: plot with nans (Mark Bakker)
   9. Re: Re: plot with nans (Darren Dale)
  10. imcrop (Brice Thurin)
  11. Re: imcrop (John Hunter)
  12. Re: legend fontsize in rc file (John Hunter)
  13. Re: performance problems (Donour Sizemore)
  14. Re: performance problems (John Hunter)

--__--__--

Message: 1
From: Andrew Straw <strawman@...106...>
Subject: Re: [Matplotlib-users] plot with nans
Date: Mon, 11 Jul 2005 22:46:53 -0700
To: matplotlib-users matplotlib-users
<matplotlib-users@lists.sourceforge.net>

Hi Alan,

It seems MaskedArrays are the preferred way to do this. See the
masked_demo.py example.

And while I'm at it, does anyone if there is simple function call which
returns only the unmasked values of a masked array (without filling the
masked values in, but simply returning a shorter sequence)? What is
it? I have my own, but it's not pretty:

import matplotlib.numerix as nx
import matplotlib.numerix.ma as M

def unmasked_values(b):
     indices = nx.where(b.mask()==0)
     result = M.take(b,indices).filled()[0]
     return result

(tested only in numarray)

Cheers!
Andrew

On Jul 11, 2005, at 6:21 PM, Alan G Isaac wrote:

Is the default handling of nan by plot the "right" one?
I am accustomed (from GAUSS) to missing values being
treated as "gaps".
E.g.,
    x=[nan,2,3,4,5]
    plot(x)
fails completely. I expect the last 4 numbers to be
plotted.
    x=[1,2,nan,4,5]
    plot(x)
plots a single line. I expect two segments and a gap.

fwiw,
Alan Isaac

-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar
happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in
dual
core and dual graphics technology at this free one hour event hosted
by HP,
AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

--__--__--

Message: 2
Date: Tue, 12 Jul 2005 09:39:58 +0200
From: Norbert Nemec <Norbert.Nemec.list@...380...>
To: Matplotlib-users@lists.sourceforge.net
CC: Danny Shevitz <shevitz@...652...>
Subject: Re: [Matplotlib-users] difference between imshow and pcolor

Danny Shevitz wrote:

Howdy, I'm struggling to understand the different intents between
imshow and pcolor. They have different
return types but both seem to do pretty much the same thing. Pcolor
seems to do non-rectangular domains
and bordered pixels if you want. Is there more to the story?

As I understand it, there is a fundamental internal difference: imshow
produces embedded png, while pcolor produces a collection of rectangles.
The first is less flexible, the second less efficient. I believe it is
correct to have two distinct functions for that, but it certainly could
be documented more clearly.

--__--__--

Message: 3
Date: Tue, 12 Jul 2005 09:43:51 +0200
From: Norbert Nemec <Norbert.Nemec.list@...380...>
To: matplotlib-users matplotlib-users
<matplotlib-users@lists.sourceforge.net>
CC: Andrew Straw <strawman@...106...>
Subject: Re: [Matplotlib-users] plot with nans

Andrew Straw wrote:

It seems MaskedArrays are the preferred way to do this. See the
masked_demo.py example.

Why is that? Couldn't both be handled in the same way?

The only advantage of MaskedArray is the portability to non IEEE
systems. However, 99% of the systems in use are IEEE conformant, and
there NaN is by far more powerful.

--__--__--

Message: 4
Date: Mon, 11 Jul 2005 22:07:14 -1000
From: Eric Firing <efiring@...202...>
Organization: University of Hawaii
To: Alan G Isaac <aisaac@...310...>
CC: matplotlib-users@lists.sourceforge.net
Subject: [Matplotlib-users] Re: plot with nans

Alan,

Is the default handling of nan by plot the "right" one?
I am accustomed (from GAUSS) to missing values being
treated as "gaps".
E.g.,
    x=[nan,2,3,4,5]
    plot(x)
fails completely. I expect the last 4 numbers to be
plotted.
    x=[1,2,nan,4,5]
    plot(x)
plots a single line. I expect two segments and a gap.

Slightly modifying a reply to a similar question not long ago:

At present, you need to use masked arrays instead of NaN. NaN is not
uniformly supported in Python and numerix, but masked arrays, in both
numarray and Numeric, can accomplish the same thing and more. See
masked_demo.py in the mpl examples subdirectory, and see
http://stsdas.stsci.edu/numarray/numarray-1.3.html/module-numarray.ma.ht
ml
for an explanation of masked arrays.

Numeric is in the midst of a major revision (including renaming), which
I think will include support of NaNs similar to that in numarray. When
this new version becomes common, then it may be both possible and
desirable for matplotlib to handle NaNs as smoothly as Matlab, Octave,
and evidently Gauss do.

Eric

--__--__--

Message: 5
Date: Tue, 12 Jul 2005 06:23:33 -0600
From: Jeff Whitaker <jswhit@...146...>
To: Andrew Straw <strawman@...106...>
CC: matplotlib-users matplotlib-users
<matplotlib-users@lists.sourceforge.net>
Subject: Re: [Matplotlib-users] plot with nans

Andrew Straw wrote:

Hi Alan,

It seems MaskedArrays are the preferred way to do this. See the
masked_demo.py example.

And while I'm at it, does anyone if there is simple function call
which returns only the unmasked values of a masked array (without
filling the masked values in, but simply returning a shorter
sequence)? What is it? I have my own, but it's not pretty:

Andrew:

If x is a masked array, x.compressed() returns the unmasked contented in

a ravelled (i.e. rank-1) array. For example:

>>> x = ma.array(arange(5), mask = [0,0,1,0,0])
>>> print x.compressed()
[0,1,3,4,]

-Jeff

--
Jeffrey S. Whitaker Phone : (303)497-6313
NOAA/OAR/CDC R/CDC1 FAX : (303)497-6449
325 Broadway Web : http://www.cdc.noaa.gov/~jsw
Boulder, CO, USA 80305-3328 Office: Skaggs Research Cntr 1D-124

--__--__--

Message: 6
Date: Tue, 12 Jul 2005 08:49:10 -0700
From: Andrew Straw <strawman@...106...>
To: Norbert Nemec <Norbert.Nemec.list@...380...>,
  matplotlib-users matplotlib-users
<matplotlib-users@lists.sourceforge.net>
Subject: Re: [Matplotlib-users] plot with nans

Norbert Nemec wrote:

Andrew Straw wrote:

It seems MaskedArrays are the preferred way to do this. See the
masked_demo.py example.

Why is that? Couldn't both be handled in the same way?

The only advantage of MaskedArray is the portability to non IEEE
systems. However, 99% of the systems in use are IEEE conformant, and
there NaN is by far more powerful.

The reason is compatibility across platforms, and the issue is (at
least) at the level of the Python interpreter. Handling of nans and infs
depends on the hardware-specific representation and C library support.
Have you ever tried struct.pack('>f',nan)?

Maybe someday these issues will get resolved in a cross-platform manner.
Or maybe not. For now, masked arrays are the way to go.

A couple threads of interest:

http://groups-beta.google.com/group/comp.lang.python/msg/16dbf848c050405
a

Cheers!
Andrew

--__--__--

Message: 7
Date: Tue, 12 Jul 2005 12:04:52 -0500 (CDT)
From: Matt Newville <newville@...189...>
To: Jeff Peery <jeffpeery@...420...>
cc: matplotlib-users@lists.sourceforge.net
Subject: Re: [Matplotlib-users] RE: Performance problems

Jeff, Donour,

Hm, this seems like it's a recurring complaint, but WXAgg and
wxPython are plenty fast for me. I routinely get better than 10
plots per second with ~500 data points on WinXP / WXAgg, and can
plot 25000 point arrays in under 0.5 second with any symbol or
line type.

Actually, that's with my own wrapper around matplotlib/wxPython,
so raw matplotlib may go even faster. Example code is included
as 'examples/test.py' in MPlot at
   http://cars.uchicago.edu/~newville/Python/MPlot

If you're having performance trouble, please post code next
time, and/or try the test.py from MPlot and report drawing
times. Also, make sure you know what you're timing and don't
include the creation of the data or the creation of the wx
Window that contains the plot frame.

Cheers,

--Matt

--__--__--

Message: 8
Date: Tue, 12 Jul 2005 13:03:52 -0400
From: Mark Bakker <markbak@...287...>
Reply-To: Mark Bakker <markbak@...287...>
To: aisaac@...310...
Cc: matplotlib-users@lists.sourceforge.net
Subject: [Matplotlib-users] Re: plot with nans

Plotting with 'nans' doesn't work as far as I know.
Read the discussion on using masked arrays posted in the last 2 weeks.
They work well in my experience,
Mark

--__--__--

Message: 9
From: Darren Dale <dd55@...163...>
Reply-To: dd55@...163...
Organization: Cornell University
To: matplotlib-users@lists.sourceforge.net
Subject: Re: [Matplotlib-users] Re: plot with nans
Date: Tue, 12 Jul 2005 13:17:08 -0400

On Tuesday 12 July 2005 01:03 pm, Mark Bakker wrote:

Plotting with 'nans' doesn't work as far as I know.
Read the discussion on using masked arrays posted in the last 2 weeks.

I had asked about plotting with NaNs a while back. John says the problem
is
with the way the Agg backend deals with them. plot([0,nan,2,3],'o')
works,
plot([nan,1,2,3],'o') does not.

--

Darren

--__--__--

Message: 10
To: Matplotlib-users@lists.sourceforge.net
From: Brice Thurin <B.P.S.Thurin@...648...>
Date: Tue, 12 Jul 2005 19:10:54 +0100
Subject: [Matplotlib-users] imcrop

Dear All,

I am new to matplotlib and I was wondering if there is any functions
like imcrop in matlab which allow to interactively select the zone of
the image.

Many thanks for your answers

Brice

--__--__--

Message: 11
To: Brice Thurin <B.P.S.Thurin@...648...>
Cc: Matplotlib-users@lists.sourceforge.net
Subject: Re: [Matplotlib-users] imcrop
From: John Hunter <jdhunter@...4...>
Date: Tue, 12 Jul 2005 13:27:15 -0500

    > Dear All, I am new to matplotlib and I was wondering if
    > there is any functions like imcrop in matlab which allow to
    > interactively select the zone of the image.

I do not currently have matlab installed on my system. Could you
describe what imcrop does. Not knowing, I'll venture and answer

1) you can use the zoom-to-rectangle feature on the toolbar to zoom
    to the rectangle of interest. Equivalently, you can set the axis
    xlim and ylim to view only part of the rectangle

2) you can extract the region of interest from you numerix array
    using array slicing before passing it to matplotlib for plotting.

JDH

--__--__--

Message: 12
To: Ryan Krauss <ryanfedora@...614...>
Cc: matplotlib-users@lists.sourceforge.net
Subject: Re: [Matplotlib-users] legend fontsize in rc file
From: John Hunter <jdhunter@...4...>
Date: Tue, 12 Jul 2005 13:37:21 -0500

    > A while back I asked whether or not I could set the legend
    > fontsize from the rc file and John asked me to make it
    > possible. I think I have it working, but don't know how to
    > check it in to CVS and need some one else to try it. I have
    > made two changes to legend.py: 1. added from matplotlib
    > import rcParams (added at line 29) 2. changed the default
    > parameters to the __init__ method to read from the rc: prop
    > = FontProperties(size=rcParams['legend.fontsize']), (change
    > line 116)

Hi Ryan,

Thanks for looking into this. This is the right approach, but if we
are going to add the fontsize I think it would make more sense to
expose the other legend properties that currently have default values.

In particular, the various values in the Legend class init function

    isaxes=True,
    numpoints = 4, # the number of points in the legend line
    prop = FontProperties(size='smaller'),
    pad = 0.2, # the fractional whitespace inside the legend
border
    markerscale = 0.6, # the relative size of legend markers vs.
original

    # the following dimensions are in axes coords
    labelsep = 0.005, # the vertical space between the legend
entries
    handlelen = 0.05, # the length of the legend lines
    handletextsep = 0.02, # the space between the legend line and legend
text
    axespad = 0.02, # the border between the axes and legend edge

    shadow=False,

The only other comment I have is that in the legend __init__
*signature* you do

  def __init__(self,
               prop = FontProperties(size=rcParams['legend.fontsize']),
               ...snip):
      self.prop = prop

This sets the rc parameter at *module load time*.

Compare this with

  def __init__(self,
               prop = None,
               ...snip):
      if prop is None: prop =
FontProperties(size=rcParams['legend.fontsize']),
      self.prop = prop

This set the rc value at the time the legend is created, and thus
allows for dynamic, script based configuration of the rc properties.
The latter is what is used throughout matplotlib for rc properties.

If you are still interested in pursuing this, you can send me a diff
or your updated files and I'll check them in.

Thanks!
JDH

--__--__--

Message: 13
Cc: matplotlib-users@lists.sourceforge.net
From: Donour Sizemore <donour@...676...>
Subject: Re: [Matplotlib-users] performance problems
Date: Tue, 12 Jul 2005 13:42:54 -0500
To: John Hunter <jdhunter@...8...>

On Jul 12, 2005, at 1:25 PM, John Hunter wrote:

Can you post a complete script that is slow for you?

    > Sure thing.

To assist in timing, I modified your script to call SystemExit on idle
-- that way we can get a pretty good measurement of how much time it
takes to load the libs and plot the figure. You can pass plot or
noplot or the script to compare the all the wx overhead with wx +
matplotlib.

While this is important, it's not the start up costs that concern me. I
know well the overhead incurred with an import. My problem is with the
time to actually perform the WX window painting as I've embedded matplot
lib into my own custom wxpanel.

Here are three runs on my system with and w/o plotting.

time python slow_example.py noplot

    0.401u 0.044s 0:00.61 72.1% 0+0k 0+0io 0pf+0w

time python slow_example.py noplot

    0.391u 0.048s 0:00.54 79.6% 0+0k 0+0io 0pf+0w

time python slow_example.py noplot

    0.394u 0.049s 0:00.51 84.3% 0+0k 0+0io 0pf+0w

time python slow_example.py plot

    0.612u 0.074s 0:01.04 65.3% 0+0k 0+0io 5pf+0w

time python slow_example.py plot

    0.628u 0.057s 0:00.81 82.7% 0+0k 0+0io 0pf+0w

time python slow_example.py plot

    update
    0.624u 0.057s 0:00.81 82.7% 0+0k 0+0io 0pf+0w

For comparable CPU utilization, the numbers are 0.51s w/o mpl and
0.81s w/ mpl.

Notice that the first run with plotting is considerably slower than
subsequent runs and that CPU utilization is lower. This suggests that
a large part of the runtime is being dominated by the load time of
the many large matplotlib modules.

I would argue that this isn't terribly useful without the addition
of profile data. The time _might_ be going to python namespace
conversion
and dyload, but in my case it isn't. As per my first post, I noticed
an inordinate amount of time spent in the axes drawing functions. I can
provide more details shortly.

Here is another example that simply does 'from pylab import *'

cat test.py

    from pylab import *

time python test.py

    0.245u 0.074s 0:01.96 15.8% 0+0k 0+0io 72pf+0w

time python test.py

    0.248u 0.034s 0:00.30 90.0% 0+0k 0+0io 0pf+0w

time python test.py

    0.249u 0.030s 0:00.30 90.0% 0+0k 0+0io 0pf+0w

My guess is that your matplotlib runtime is being dominated by module
load time. This explains why Matt is getting 10 frames per second
(you only pay load costs once) and other users are complaining about
sluggishness when scripts launch.

Again, launch time is of no concern to me, only the window paint time.

I'm on the chicago campus, so I've you'd like to see my slow as molasses
code in action, that can be arranged. :slight_smile:

thanks

donour

Donour Sizemore donour@...677...|
Technical Programmer & Numerical Analyst |
Economics Research Center Ph: 773-834-4399 |
University of Chicago Office: Walker 303-a |

--__--__--

Message: 14
To: Donour Sizemore <donour@...676...>
Cc: matplotlib-users@lists.sourceforge.net
Subject: Re: [Matplotlib-users] performance problems
From: John Hunter <jdhunter@...4...>
Date: Tue, 12 Jul 2005 13:46:46 -0500

    > Again, launch time is of no concern to me, only the window
    > paint time.

Well that simplifies things :slight_smile:

    > I'm on the chicago campus, so I've you'd like to see my
    > slow as molasses code in action, that can be arranged. :slight_smile:

Sure, I could swing by tomorrow. When and where?

JDH

--__--__--

_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net

End of Matplotlib-users Digest