plotting speed

I have developed a wxPython GUI that uses matplotlib to plot line traces (using plot()). If I were to plot about 250,000 data points, it takes nearly 1 minute to load the plot (using a intel dual core), although once this plot is loaded, it is easy to interact with it - changing the x/y scale & scrolling along an x-axis, etc goes smoothly. Is there a strategy for loading these plots faster? I basically just call a plot() function with the 250,000 datapoints… I’ve plotted as little as ~15,000 datapoints, which loads much quicker. One strategy I can think of would be to only load a segment of data and add more points to the plot as you scroll down the x-axis. But that would require appending to the current plot() instance, which I assume isn’t possible. Thanks to anyone who has some insights.

Are you using the wxAgg backend or just wx? Which version of matplotlib?

If you're using 0.98.x with the wx backend, things are *very* slow. The performance of wx.GraphicsContext (that is uses under the hood) is terrible. You may want to try switching to the WxAgg backend, which, among other things, will automatically simplify the line so it doesn't plot points whose effect would be invisible.

Hope that helps.

Cheers,
Mike

Paul Hartley wrote:

···

I have developed a wxPython GUI that uses matplotlib to plot line traces (using plot()). If I were to plot about 250,000 data points, it takes nearly 1 minute to load the plot (using a intel dual core), although once this plot is loaded, it is easy to interact with it - changing the x/y scale & scrolling along an x-axis, etc goes smoothly. Is there a strategy for loading these plots faster? I basically just call a plot() function with the 250,000 datapoints... I've plotted as little as ~15,000 datapoints, which loads much quicker. One strategy I can think of would be to only load a segment of data and add more points to the plot as you scroll down the x-axis. But that would require appending to the current plot() instance, which I assume isn't possible. Thanks to anyone who has some insights.
------------------------------------------------------------------------

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
------------------------------------------------------------------------

_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options
  
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Michael Droettboom wrote:

If you're using 0.98.x with the wx backend, things are *very* slow. The performance of wx.GraphicsContext (that is uses under the hood) is terrible.

For what it's worth, apparently you can speed up GraphicsContext code substantially if you use it right -- i.e. create and manipulate paths, rather than using GCDC.

I haven't looked at the MPL code though, so I can't make any recommendations (I use wxAGG myself).

I just thought I'd post this, so if someone does want to improve the wx back-end performance, they'll know it's likely that it can be done.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...

Christopher Barker wrote:

Michael Droettboom wrote:
  

If you're using 0.98.x with the wx backend, things are *very* slow. The performance of wx.GraphicsContext (that is uses under the hood) is terrible.
    
For what it's worth, apparently you can speed up GraphicsContext code substantially if you use it right -- i.e. create and manipulate paths, rather than using GCDC.
  

Can you be more specific about this or provide a reference? Matplotlib doesn't use the wx.GCDC class. Eventually you do have to draw something to the screen, right? Is there another way to do that besides going through a DC? It doesn't seem possible to create a wx.GraphicsContext without providing a DC (at least in Python). In my testing, it seemed that clipping of any kind caused a major decrease in speed, not the kind of DC being used.

Cheers,
Mike

···

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Michael Droettboom wrote:

Christopher Barker wrote:

For what it's worth, apparently you can speed up GraphicsContext code substantially if you use it right --

Can you be more specific about this

no. I haven't done much with it myself.

or provide a reference?

A number of threads on the wxPython list. Here's a couple:

a quote from that:

""
An interesting phenomena
I've noticed is that the rendering speed is directly related to the
number of pixels that get pushed - scale the tiger up to 500x500, with
it all visible on screen, and it's slower. Scale it down to 10x10 and
the rendering is lightning fast. Move it off screen so that only a
portion shows, and it's even faster.
"""

This seems to indicate that clipping is efficient -- hmm.

Here is another one:

http://lists.wxwidgets.org/pipermail/wxpython-users/2007-August/067098.html

Eventually you do have to draw something to the screen, right?

yes.

Is there another way to do that besides going through a DC? It
doesn't seem possible to create a wx.GraphicsContext
without providing a DC (at least in Python).

No, I think you've got that right. There is:

"""
wxGraphicsContext::CreateFromNative

Creates a wxGraphicsContext from a native context. This native context must be eg a CGContextRef for Core Graphics, a Graphics pointer for GDIPlus or a cairo_t pointer for cairo.
"""

But I doubt we'd want to get into that for MPL.

In my testing, it seemed
that clipping of any kind caused a major decrease in speed, not the
kind of DC being used.

If you can come up with an self-contained example of this, a post tot he wxPython list may yield results.

Sorry I can't be more help.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...

Christopher Barker wrote:

Michael Droettboom wrote:
  

Christopher Barker wrote:
    

A number of threads on the wxPython list. Here's a couple:

ActiveState Community - Boosting coder and team productivity with ready-to-use open source languages and tools.
  

Thanks. I had seen these threads already -- I was hoping you had some more.

a quote from that:

""
An interesting phenomena
I've noticed is that the rendering speed is directly related to the
number of pixels that get pushed - scale the tiger up to 500x500, with
it all visible on screen, and it's slower. Scale it down to 10x10 and
the rendering is lightning fast. Move it off screen so that only a
portion shows, and it's even faster.
"""

This seems to indicate that clipping is efficient -- hmm.
  

That doesn't seem at all surprising, unless the time to render increases faster than the number of pixels. That's an interesting question to answer.

It also is different from what I was talking about in the sense of clipping. The effect I see is that when an explicit clipping rectangle is applied (e.g. the axes), speed decreases about 8x. I don't get the same effect using Cairo directly, so that leads me to think there's something in the wx abstraction layer that's causing this.

Here is another one:

http://lists.wxwidgets.org/pipermail/wxpython-users/2007-August/067098.html

Eventually you do have to draw something to the screen, right?
    
yes.

Is there another way to do that besides going through a DC? It
doesn't seem possible to create a wx.GraphicsContext
without providing a DC (at least in Python).
    
No, I think you've got that right. There is:

"""
wxGraphicsContext::CreateFromNative

Creates a wxGraphicsContext from a native context. This native context must be eg a CGContextRef for Core Graphics, a Graphics pointer for GDIPlus or a cairo_t pointer for cairo.
"""

But I doubt we'd want to get into that for MPL.
  

Yes. But it is not usable from Python. In the "new" docs (which have Python-specific annotations):

"""

Creates a wx.GraphicsContext from a native context. This native context must be eg a CGContextRef for Core Graphics, a Graphics pointer for GDIPlus or a cairo_t pointer for Cairo. NOTE: For wxPython we still need a way to make this value usable.
"""

http://wxpython.org/docs/api/wx.GraphicsContext-class.html#CreateFromNative

In my testing, it seemed
that clipping of any kind caused a major decrease in speed, not the
kind of DC being used.
    
If you can come up with an self-contained example of this, a post tot he wxPython list may yield results.
  

It might be worth it just to get to the bottom of this. Like you, I'm not a Wx backend user either, so I haven't been terribly motivated. The change to wx.GraphicsContext was really only made to have something that sorta-kinda-works with the new "everything is a polycurve" refactoring that went into 0.98. But I would just as soon deprecate it in favor of wxAgg. I think the best we'll be able to do performance-wise with wx.GraphicsContext is the same as the Cairo backend (since wx.GraphicsContext is built on top of Cairo on X11, at least), which is already slower than Agg, so there's no compelling reason for the Wx backend to exist, IMHO, *unless* it needs to integrate with other code that draws to a wx.GraphicsContext.

(To avoid a flamewar: I didn't say "Cairo is slower than Agg", I said the "Cairo backend is slower than the Agg backend", and the reason for that is somewhat by "cheating". The Agg backend makes a number of optimizations, such as line simplification and marker blitting, that the Cairo backend doesn't do. Perhaps they could be done, but I haven't looked very deeply there. Plus, the agg backend is primarily C++ that we can control and optimize for matplotlib, whereas with Cairo mpl uses the stock pycairo wrappers.)

Cheers,
Mike

···

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Michael Droettboom wrote:

Thanks. I had seen these threads already -- I was hoping you had some more.

sorry :frowning:

It also is different from what I was talking about in the sense of clipping. The effect I see is that when an explicit clipping rectangle is applied (e.g. the axes), speed decreases about 8x.

wow! that is a lot.

Creates a wx.GraphicsContext from a native context. This native context must be eg a CGContextRef for Core Graphics, a Graphics pointer for GDIPlus or a cairo_t pointer for Cairo. NOTE: For wxPython we still need a way to make this value usable.

If we really think that would help, and we would use it for MPL, we could try to get it done -- but I doubt that it's worth it -- we have enough trouble keeping the wx back-ends maintained as it is.

If you can come up with an self-contained example of this, a post tot he wxPython list may yield results.
  

It might be worth it just to get to the bottom of this.

so are you working on an example? Or should I?

I think the best we'll be able to do performance-wise with wx.GraphicsContext is the same as the Cairo backend (since wx.GraphicsContext is built on top of Cairo on X11, at least)

so, does Cairo render a bitmap, and then push that to the X Server? If so, that would explain why the new wx backend isn't any better than wxAgg with remote X connections -- and that's was the only reason I know to use it.

already slower than Agg, so there's no compelling reason for the Wx backend to exist, IMHO, *unless* it needs to integrate with other code that draws to a wx.GraphicsContext.

I cant really imagine why anyone would try to do that -- and if you really wanted to, you could still transfer the Agg buffer to a wxBitmap, and draw to that with GraphicsContext anyway.

So, like you, I'm curious (and I need GraphicsContext for other projects), so I'd still like to know what the deal is.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...

Christopher Barker wrote:

Michael Droettboom wrote:

Creates a wx.GraphicsContext from a native context. This native context must be eg a CGContextRef for Core Graphics, a Graphics pointer for GDIPlus or a cairo_t pointer for Cairo. NOTE: For wxPython we still need a way to make this value usable.

If we really think that would help, and we would use it for MPL, we could try to get it done -- but I doubt that it's worth it -- we have enough trouble keeping the wx back-ends maintained as it is.

Yes --- there is something to be said for keeping the number of backends to a minimum. Each one needs to have a reason to exist :wink:

If you can come up with an self-contained example of this, a post tot he wxPython list may yield results.
  

It might be worth it just to get to the bottom of this.

so are you working on an example? Or should I?

I'm happy to do it, but may not get to it for a few days. My own test was to run "simple_plot_fps.py" with "handle_clip_rectangle" (in backend_wx.py) turned on and off. But obviously the wxPython folks will want a more standalone example.

I think the best we'll be able to do performance-wise with wx.GraphicsContext is the same as the Cairo backend (since wx.GraphicsContext is built on top of Cairo on X11, at least)

so, does Cairo render a bitmap, and then push that to the X Server? If so, that would explain why the new wx backend isn't any better than wxAgg with remote X connections -- and that's was the only reason I know to use it.

I believe it does from what I've read (assuming one isn't using the OpenGL backend), but don't know that first hand.

already slower than Agg, so there's no compelling reason for the Wx backend to exist, IMHO, *unless* it needs to integrate with other code that draws to a wx.GraphicsContext.

I cant really imagine why anyone would try to do that -- and if you really wanted to, you could still transfer the Agg buffer to a wxBitmap, and draw to that with GraphicsContext anyway.

True.

So, like you, I'm curious (and I need GraphicsContext for other projects), so I'd still like to know what the deal is.

Thanks for all the info so far... Hopefully we can get to the bottom of this.

Cheers,
Mike

···

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

I am happy to keep backends around if someone cares enough about it to
step up and maintain it. Michael you are stretched pretty thin w/ all
the other stuff you are doing for matplotlib, and I know you don't use
wx yourself or at work, so I suggest we let Ken or Chris or someone
who has a vested interest in this backend pursue it. As it stands,
wxagg is superior for almost every use case, and we want to encourage
people to use it. Any effort we spend optimizing would be much better
put into agg, since that would help almost every user, rather than the
few who are using native wx.

JDH

···

On Tue, Jun 17, 2008 at 7:16 AM, Michael Droettboom <mdroe@...86...> wrote:

Christopher Barker wrote:

If we really think that would help, and we would use it for MPL, we
could try to get it done -- but I doubt that it's worth it -- we have
enough trouble keeping the wx back-ends maintained as it is.

Yes --- there is something to be said for keeping the number of backends
to a minimum. Each one needs to have a reason to exist :wink:

If you can come up with an self-contained example of this, a post
tot he wxPython list may yield results.

It might be worth it just to get to the bottom of this.

so are you working on an example? Or should I?

I'm happy to do it, but may not get to it for a few days. My own test
was to run "simple_plot_fps.py" with "handle_clip_rectangle" (in
backend_wx.py) turned on and off. But obviously the wxPython folks will
want a more standalone example.

Michael Droettboom wrote:

so are you working on an example? Or should I?

I'm happy to do it, but may not get to it for a few days. My own test was to run "simple_plot_fps.py" with "handle_clip_rectangle" (in backend_wx.py) turned on and off. But obviously the wxPython folks will want a more standalone example.

I've made a standalone example (enclosed). It simply makes a call to

GraphicsContext.DrawLines()

with and without clipping. In this case, it's actually a bit faster clipped (on OS-X). Maybe it's different with Paths -- I haven't dug into the MPL code to see how it's being used there.

Also, the whole thing almost hangs (134 seconds to draw) if I up the number of points to 5000!

Could you alter this to use the drawing calls MPL is actually using, then we can send it on the wxPython list.

-Chris

GraphicsContextSpeed.py (2.7 KB)

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...

Ok -- well, I'm genuinely sorry for wasting your time. The benchmarks I was referring to are here:

http://sourceforge.net/mailarchive/message.php?msg_id=4734927C.3060601%40stsci.edu

but it seems on further inspection that they no longer apply.

Paul Kienzle made a change back in February that changed where wx.Yield() gets called, that by side-effect seems to have fixed the clipping slowness.

The new results for simple_plot_fps.py are:

wx (with clipping) -----
wallclock: 6.20115208626
user: 6.19
fps: 16.1260357122

wx (without clipping) ----
wallclock: 6.1539440155
user: 6.14
fps: 16.2497415882

wxagg ----
wallclock: 3.52209401131
user: 2.71
fps: 28.3922006849

So, wxAgg is still clearly faster, but not by orders of magnitude, and clipping makes no significant difference (the difference shown here is probably noise).

One inherent slowness between Wx and WxAgg is that Wx needs to create a wx.GraphicsPath using a Python loop over the data. With the Agg backend, we just pass NumPy arrays (without any copies) to the Agg backend. Perhaps wxPython needs to grow a similar interface...

An interesting side effect of this is that wx and Cairo backends slow down sooner as the number of points increases (see attached plot).

As for your toy example, I don't see it getting significantly slower as the number of points increases, but it does crash completely when I plot more than about 11000 points (this is on RHEL4 with a locally-built wxPython-2.8.6.1.) I have updated it to emulate what matplotlib does more closely (use CreatePath etc.)

Lastly, be sure to do an SVN update on matplotlib -- there was a clipping bug in the Wx backend that I have fixed.

Thanks for your help.

Cheers,
Mike

Christopher Barker wrote:

GraphicsContextSpeed.py (2.75 KB)

···

Michael Droettboom wrote:

so are you working on an example? Or should I?

I'm happy to do it, but may not get to it for a few days. My own test was to run "simple_plot_fps.py" with "handle_clip_rectangle" (in backend_wx.py) turned on and off. But obviously the wxPython folks will want a more standalone example.

I've made a standalone example (enclosed). It simply makes a call to

GraphicsContext.DrawLines()

with and without clipping. In this case, it's actually a bit faster clipped (on OS-X). Maybe it's different with Paths -- I haven't dug into the MPL code to see how it's being used there.

Also, the whole thing almost hangs (134 seconds to draw) if I up the number of points to 5000!

Could you alter this to use the drawing calls MPL is actually using, then we can send it on the wxPython list.

-Chris

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Michael Droettboom wrote:

Ok -- well, I'm genuinely sorry for wasting your time.

No waste. As I said, I'm investigating Graphics Context for other things anyway.

Paul Kienzle made a change back in February that changed where wx.Yield() gets called, that by side-effect seems to have fixed the clipping slowness.

cool!

One inherent slowness between Wx and WxAgg is that Wx needs to create a wx.GraphicsPath using a Python loop over the data.

Yes, I suspect that is the biggest problem.

With the Agg backend, we just pass NumPy arrays (without any copies) to the Agg backend. Perhaps wxPython needs to grow a similar interface...

I think it does, and with the numpy array protocol, it may. There is a Google Summer of Code project that may address this. If it doesn't get done there, it may get done for another SoC project that I"m mentoring, where we need better GraphicsContext performance with numpy arrays.

As for your toy example, I don't see it getting significantly slower as the number of points increases, but it does crash completely when I plot more than about 11000 points (this is on RHEL4 with a locally-built wxPython-2.8.6.1.) I have updated it to emulate what matplotlib does more closely (use CreatePath etc.)

Does is it still crash with your version? 11000 points really isn't that many.

Lastly, be sure to do an SVN update on matplotlib -- there was a clipping bug in the Wx backend that I have fixed.

thanks for the tip, and even more so, all your work on this.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...