HTML5 Matplotlib Backend

Hello,

I've been following the updates to mplh5canvas for some time now, since its debut at SciPy 2010. I am interested in working to extend mplh5canvas into a cloud-based web service as a thesis project. I have not found anybody whose is currently working on this path. Are there any active projects working on this? Specifically, I am referring to Michael Droettboom's comments of a possible web application and Simon Ratcliffe's comments about using Flex. If anyone is interested I would be happy to talk in person at SciPy next month.

Cheers,
Brian

I did a little exploratory work with this at the Sage Days sprint in Seattle back in March.

My (humble) assessment is that HTML 5 Canvas doesn't make much sense for any of the use cases I could dream up, and that in fact SVG is a more appropriate choice for interactive graphics. The browser support for each technology is fairly equivalent at this point, with IE9 finally getting on the SVG bandwagon. The main problem with Canvas (from the point of view of matplotlib) is that it doesn't support persistence, (without building such a layer on top in JavaScript), so if you want to update the figure, you have to send the whole thing over the wire each time. SVG, on the other hand, maintains a tree of objects that can be tweaked at any time (and the performance in the current generation of browsers is stunning). One could send all of the large data objects as SVG from matplotlib to the browser and using XML ids to maintain relationships between the client and the server. Then, do scale the data (in many common cases), it is just a matter of updating the affiine transform on that object, (as well as updating the ticks etc, but that's peanuts), which requires very little bandwidth. I have some hackish "proof of concept" code doing this kind of thing, but it's a long way from there to something that truly works.

This all glosses over the path simplification stuff that matplotlib does -- the assumption here is that the browser would have access to *all* of the data, and there are probably practical limits on how big that data can be.

I recently did a lot of cleanup to the SVG backend to pave the way for having persistent objects etc. -- though there is no client/server code at all in master at the moment. All of that is "to be written", perhaps by you if you're interested.

Cheers,
Mike

···

On 06/06/2011 02:06 PM, Brian Refsdal wrote:

Hello,

I've been following the updates to mplh5canvas for some time now, since
its debut at SciPy 2010. I am interested in working to extend
mplh5canvas into a cloud-based web service as a thesis project. I have
not found anybody whose is currently working on this path. Are there
any active projects working on this? Specifically, I am referring to
Michael Droettboom's comments of a possible web application and Simon
Ratcliffe's comments about using Flex. If anyone is interested I would
be happy to talk in person at SciPy next month.

--
Michael Droettboom
Science Software Branch
Space Telescope Science Institute
Baltimore, Maryland, USA

Hey Mike,

Thanks for your informative assessment of Canvas vs SVG. Indeed it
encapsulates much of the thinking and horse trading
done when we decided on Canvas as a delivery technology for mplh5canvas.

I particularly agree with you that sending smaller encapsulated
updates is less bandwidth intensive (although not necessarily faster
to re-rasterise the
client display). We have also spent a lot of time tearing our hair out
at the primitive drawing commands, the lack of dashed lines in
particular
is pretty painful.

For us the plus points for Canvas came down to our perception of
better future browser support, and that most of the browsers seemed to
be putting
a large push into Canvas performance. Coupled with the use of
WebSockets, we felt an all HTML5 solution may be more universally
supported
in the long run. (although who can tell what the browser vendors are
actually thinking :slight_smile:

Our initial testing showed that DOM manipulation was a surprising
bottleneck in SVG animations and that, particularly as the SVG object
count increased,
Canvas performance (time to raster) was better. Also the rendering of
the imshow inline png's was a lot quicker than the SVG equivalent in
our initial testing.
Obviously the vector strengths of SVG are highlighted with higher
output DPI, and this is certainly an area of interest.

This mail is mostly to give others a bit of background about our
choice, I think it could easily have gone either way, and perhaps it
should.

Most of the client/server code could easily be factored out with a
common API that accepts either an SVG or Canvas drawing object for
display on the client side.
This would give us the best of both worlds (raster and vector) in a
number of situations (i.e. lower DPI / high animation rate using
Canvas, high DPI with lots of interactivity using SVG etc...).
Interactivity would need a tweak on the browser side (much easier to
do with SVG than Canvas).

As suggested, perhaps Brian Refsdal would be interested in looking at
producing a more generic mplweb backend.

How about an imshow hybrid, with a canvas for the bitmap and SVG axes
and surrounds :slight_smile:

Cheers,

Simon Ratcliffe / Ludwig Schwardt

SKA South Africa
www.ska.ac.za

···

My (humble) assessment is that HTML 5 Canvas doesn't make much sense for
any of the use cases I could dream up, and that in fact SVG is a more
appropriate choice for interactive graphics. The browser support for
each technology is fairly equivalent at this point, with IE9 finally
getting on the SVG bandwagon. The main problem with Canvas (from the
point of view of matplotlib) is that it doesn't support persistence,
(without building such a layer on top in JavaScript), so if you want to
update the figure, you have to send the whole thing over the wire each
time. SVG, on the other hand, maintains a tree of objects that can be
tweaked at any time (and the performance in the current generation of
browsers is stunning). One could send all of the large data objects as
SVG from matplotlib to the browser and using XML ids to maintain
relationships between the client and the server. Then, do scale the
data (in many common cases), it is just a matter of updating the affiine
transform on that object, (as well as updating the ticks etc, but that's
peanuts), which requires very little bandwidth. I have some hackish
"proof of concept" code doing this kind of thing, but it's a long way
from there to something that truly works.

This all glosses over the path simplification stuff that matplotlib does
-- the assumption here is that the browser would have access to *all* of
the data, and there are probably practical limits on how big that data
can be.

I recently did a lot of cleanup to the SVG backend to pave the way for
having persistent objects etc. -- though there is no client/server code
at all in master at the moment. All of that is "to be written", perhaps
by you if you're interested.

Cheers,
Mike

--
Michael Droettboom
Science Software Branch
Space Telescope Science Institute
Baltimore, Maryland, USA

------------------------------------------------------------------------------
EditLive Enterprise is the world's most technically advanced content
authoring tool. Experience the power of Track Changes, Inline Image
Editing and ensure content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Hey Mike,

Thanks for your informative assessment of Canvas vs SVG. Indeed it
encapsulates much of the thinking and horse trading
done when we decided on Canvas as a delivery technology for mplh5canvas.

For us the plus points for Canvas came down to our perception of
better future browser support, and that most of the browsers seemed to
be putting
a large push into Canvas performance. Coupled with the use of
WebSockets, we felt an all HTML5 solution may be more universally
supported
in the long run. (although who can tell what the browser vendors are
actually thinking :slight_smile:

Yeah. I'll start implementing the "crystal ball" backend so we can answer that question :wink:

Our initial testing showed that DOM manipulation was a surprising
bottleneck in SVG animations and that, particularly as the SVG object
count increased,
Canvas performance (time to raster) was better.

Interesting result. I wonder if browsers differ in that respect. I've been using Firefox 4 primarily, but testing with Chrome as well, and have been nothing but impressed with SVG performance -- but I'm only doing getElementById and then tweaking attributes, not actually changing tree morphology.

  Also the rendering of
the imshow inline png's was a lot quicker than the SVG equivalent in
our initial testing.

Have you tried non-inline PNGs? The only need for inline PNGs is to have a self-contained SVG file. When one has a webapp server available that is no longer necessary. I suspect the encoding to/from base64 adds some overhead.

Obviously the vector strengths of SVG are highlighted with higher
output DPI, and this is certainly an area of interest.

True. Of course, given that we already have a static SVG backend that works, it should be simple enough to have a "Print" button that generates SVG only when printing (much like Google Docs generates a PDF when printing).

This mail is mostly to give others a bit of background about our
choice, I think it could easily have gone either way, and perhaps it
should.

Yes -- perhaps still an open question.

I should probably also add: I have sort of a bias toward a smart server/dumb client approach to continue to leverage the existing matplotlib code as much as possible -- which is kind of at odds with the ideal arrangement for best interactive performance, which would be to move a bunch of logic into Javascript. I don't have anything against plotting with Javascript -- there are some great packages out there -- but then it becomes a very different beast. I think your work has that assumption behind it as well, but speaking to some folks at Sage Days, it's not always implied when people start drafting a plan to do this.

Cheers,
Mike

···

On 06/13/2011 10:07 AM, Simon Ratcliffe wrote:

SKA South Africa
www.ska.ac.za

My (humble) assessment is that HTML 5 Canvas doesn't make much sense for
any of the use cases I could dream up, and that in fact SVG is a more
appropriate choice for interactive graphics. The browser support for
each technology is fairly equivalent at this point, with IE9 finally
getting on the SVG bandwagon. The main problem with Canvas (from the
point of view of matplotlib) is that it doesn't support persistence,
(without building such a layer on top in JavaScript), so if you want to
update the figure, you have to send the whole thing over the wire each
time. SVG, on the other hand, maintains a tree of objects that can be
tweaked at any time (and the performance in the current generation of
browsers is stunning). One could send all of the large data objects as
SVG from matplotlib to the browser and using XML ids to maintain
relationships between the client and the server. Then, do scale the
data (in many common cases), it is just a matter of updating the affiine
transform on that object, (as well as updating the ticks etc, but that's
peanuts), which requires very little bandwidth. I have some hackish
"proof of concept" code doing this kind of thing, but it's a long way
from there to something that truly works.

This all glosses over the path simplification stuff that matplotlib does
-- the assumption here is that the browser would have access to *all* of
the data, and there are probably practical limits on how big that data
can be.

I recently did a lot of cleanup to the SVG backend to pave the way for
having persistent objects etc. -- though there is no client/server code
at all in master at the moment. All of that is "to be written", perhaps
by you if you're interested.

Cheers,
Mike

--
Michael Droettboom
Science Software Branch
Space Telescope Science Institute
Baltimore, Maryland, USA

------------------------------------------------------------------------------
EditLive Enterprise is the world's most technically advanced content
authoring tool. Experience the power of Track Changes, Inline Image
Editing and ensure content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

--
Michael Droettboom
Science Software Branch
Space Telescope Science Institute
Baltimore, Maryland, USA

Hey Mike,

Thanks for your informative assessment of Canvas vs SVG. Indeed it
encapsulates much of the thinking and horse trading
done when we decided on Canvas as a delivery technology for mplh5canvas.

I particularly agree with you that sending smaller encapsulated
updates is less bandwidth intensive (although not necessarily faster
to re-rasterise the
client display). We have also spent a lot of time tearing our hair out
at the primitive drawing commands, the lack of dashed lines in
particular
is pretty painful.

For us the plus points for Canvas came down to our perception of
better future browser support, and that most of the browsers seemed to
be putting
a large push into Canvas performance. Coupled with the use of
WebSockets, we felt an all HTML5 solution may be more universally
supported
in the long run. (although who can tell what the browser vendors are
actually thinking :slight_smile:

Our initial testing showed that DOM manipulation was a surprising
bottleneck in SVG animations and that, particularly as the SVG object
count increased,
Canvas performance (time to raster) was better. Also the rendering of
the imshow inline png's was a lot quicker than the SVG equivalent in
our initial testing.
Obviously the vector strengths of SVG are highlighted with higher
output DPI, and this is certainly an area of interest.

This mail is mostly to give others a bit of background about our
choice, I think it could easily have gone either way, and perhaps it
should.

Most of the client/server code could easily be factored out with a
common API that accepts either an SVG or Canvas drawing object for
display on the client side.
This would give us the best of both worlds (raster and vector) in a
number of situations (i.e. lower DPI / high animation rate using
Canvas, high DPI with lots of interactivity using SVG etc...).
Interactivity would need a tweak on the browser side (much easier to
do with SVG than Canvas).

As suggested, perhaps Brian Refsdal would be interested in looking at
producing a more generic mplweb backend.

Great! I am interested in developing some of these ideas. I particularly like the idea to keep the transport design independent of the payload. I am also looking to expand on the collaborative element in mplh5canvas, and one of my first thoughts is to move the server code to the cloud. This way all communication can travel over port 80 or 443 for maximum compatibility between networks/ISPs and the instance running matplotlib does not have to serve up pages (latency is a concern, have you found multiple ports to be superior for throughput?). I am also looking to reduce UI request contention with multiple users and develop a session space. Something akin to doodle or imgur where users can anonymously generate unique URLs to share.

I have written some prototype code that separates the canvas backend from the web server and websocket server to test for latency issues, but I am open to new ideas.

-Brian

···

On 06/13/2011 10:07 AM, Simon Ratcliffe wrote:

How about an imshow hybrid, with a canvas for the bitmap and SVG axes
and surrounds :slight_smile:

Cheers,

Simon Ratcliffe / Ludwig Schwardt

SKA South Africa
www.ska.ac.za

My (humble) assessment is that HTML 5 Canvas doesn't make much sense for
any of the use cases I could dream up, and that in fact SVG is a more
appropriate choice for interactive graphics. The browser support for
each technology is fairly equivalent at this point, with IE9 finally
getting on the SVG bandwagon. The main problem with Canvas (from the
point of view of matplotlib) is that it doesn't support persistence,
(without building such a layer on top in JavaScript), so if you want to
update the figure, you have to send the whole thing over the wire each
time. SVG, on the other hand, maintains a tree of objects that can be
tweaked at any time (and the performance in the current generation of
browsers is stunning). One could send all of the large data objects as
SVG from matplotlib to the browser and using XML ids to maintain
relationships between the client and the server. Then, do scale the
data (in many common cases), it is just a matter of updating the affiine
transform on that object, (as well as updating the ticks etc, but that's
peanuts), which requires very little bandwidth. I have some hackish
"proof of concept" code doing this kind of thing, but it's a long way
from there to something that truly works.

This all glosses over the path simplification stuff that matplotlib does
-- the assumption here is that the browser would have access to *all* of
the data, and there are probably practical limits on how big that data
can be.

I recently did a lot of cleanup to the SVG backend to pave the way for
having persistent objects etc. -- though there is no client/server code
at all in master at the moment. All of that is "to be written", perhaps
by you if you're interested.

Cheers,
Mike

--
Michael Droettboom
Science Software Branch
Space Telescope Science Institute
Baltimore, Maryland, USA

------------------------------------------------------------------------------
EditLive Enterprise is the world's most technically advanced content
authoring tool. Experience the power of Track Changes, Inline Image
Editing and ensure content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

------------------------------------------------------------------------------
EditLive Enterprise is the world's most technically advanced content
authoring tool. Experience the power of Track Changes, Inline Image
Editing and ensure content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel