OpenGL backend with Galry

Hi all,

I am developing a high-performance interactive visualization package in Python based on PyOpenGL (http://rossant.github.com/galry/). It is primarily meant to be used as a framework for developing complex interactive GUIs (in QT) that deal with very large amounts of data (tens of millions of points). But it may also be used, like matplotlib, as a high-level interactive library to plot and visualize data.

The low-level interface is mostly done at this point (the code is still in an experimental stage though), and I’m now focusing on my current research project which is to write a scientific GUI based on this interface. However, I think people (including myself!) may be interested in a matplotlib-like high-level interface. I was first thinking about writing such an interface from scratch, by implementing a very small fraction of the matplotlib interface (basic commands like figure(), plot(), subplot(), show(), etc.). One could then quickly visualize huge datasets with the same commands than matplotlib.

Another solution would be to write a matplotlib backend based on this library. I am not familar enough with the internals of matplotlib to know how complicated it could be. I may do it myself, but it would probably take a long time since it is currently not my highest priority. I would be glad if someone experienced in writing backends was interested in working on it. Actually I could do everything that is specific to my library, which already provides commands to plot points, lines, textures, etc. The canvas is based on QT and may be similar to what is already implemented in the QT backend.

Of course, it would already be great if only the most basic plotting features were available in the backend. A first step could be for example to have a simplistic example “plot(x, sin(x))” working (with interactive navigation).

I am looking forward to your feedback.

Best,

Cyrille Rossant

Great to hear another person interested in bringing opengl to matplotlib! Another project you might be interested in collaborating with is Glumpy: http://code.google.com/p/glumpy/

Cheers!
Ben Root

···

On Thu, Nov 15, 2012 at 2:24 PM, Cyrille Rossant <cyrille.rossant@…149…> wrote:

Hi all,

I am developing a high-performance interactive visualization package in Python based on PyOpenGL (http://rossant.github.com/galry/). It is primarily meant to be used as a framework for developing complex interactive GUIs (in QT) that deal with very large amounts of data (tens of millions of points). But it may also be used, like matplotlib, as a high-level interactive library to plot and visualize data.

The low-level interface is mostly done at this point (the code is still in an experimental stage though), and I’m now focusing on my current research project which is to write a scientific GUI based on this interface. However, I think people (including myself!) may be interested in a matplotlib-like high-level interface. I was first thinking about writing such an interface from scratch, by implementing a very small fraction of the matplotlib interface (basic commands like figure(), plot(), subplot(), show(), etc.). One could then quickly visualize huge datasets with the same commands than matplotlib.

Another solution would be to write a matplotlib backend based on this library. I am not familar enough with the internals of matplotlib to know how complicated it could be. I may do it myself, but it would probably take a long time since it is currently not my highest priority. I would be glad if someone experienced in writing backends was interested in working on it. Actually I could do everything that is specific to my library, which already provides commands to plot points, lines, textures, etc. The canvas is based on QT and may be similar to what is already implemented in the QT backend.

Of course, it would already be great if only the most basic plotting features were available in the backend. A first step could be for example to have a simplistic example “plot(x, sin(x))” working (with interactive navigation).

I am looking forward to your feedback.

Best,

Cyrille Rossant

From my limited knowledge of OpenGL, what my vision is that any of the existing backends have support for an OpenGL object, so we just need to be able to instantiate the opengl object in any figure object, and know how to send it the appropriate commands and data. So, it is not exactly a backend, more of a “middling”. Anyway, I think the dev at Glumpy would be happy to have help, and probably have much more developed ideas on how to integrate with matplotlib.

Yep, I'm still developing some OpenGL technics to provide both nice and fast rendering and I hope to be able to help the writing of a GL backend for matplotlib next summer (provided we get a GSoC student for the project).

So far, my main concern is that for efficient rendering using OpenGL, you need to consider that drawing means to create objects on the graphic card (line, curve, image, points, etc) that can be later manipulated (scaling/rotating/coloring/properties change, etc.). From what I remember in my early attempts at writing an OpenGL backend, I did not find the proper way to enforce such framework. Said differently, the backend is supposed to implement drawing operations while I would need to know if the drawing operations actually relates to something that is already on the graphic card or not. I'm not sure I'm very clear but I can develop the point if necessary. Having read the post by Michael (http://mdboom.github.com/blog/2012/08/06/matplotlib-client-side/) on client-side rendering, I think the proposed three-way split might be a solution but I do not know how advanced are the ideas.

To date, I've been working on different things:

Text/font : http://code.google.com/p/freetype-gl/ (c code)
Stroke/dash/paths: http://code.google.com/p/gl-agg/ (python)
Images: http://code.google.com/p/glumpy/ (python)

If you want to get a feel of how nice and fast rendering could be, have a look at 'demo-lines.py' from the gl-agg repository (and play with mouse). From these experiments, I think it is possible to achieve AGG quality using OpenGL. What is really exciting is the perspective of having a opengl/webgl backend that could be used with ipython (there has been a recent post on ipython list that show such integration for a molecule viewer).

Anyway, you're more than welcome to contribute to glumpy, but in the long run, I hope it will disappear in favor of a matplolib GL backend.

Nicolas

···

On Nov 15, 2012, at 22:03 , Benjamin Root wrote:

On Thu, Nov 15, 2012 at 2:24 PM, Cyrille Rossant <cyrille.rossant@...761.....> wrote:
Hi all,

I am developing a high-performance interactive visualization package in Python based on PyOpenGL (http://rossant.github.com/galry/). It is primarily meant to be used as a framework for developing complex interactive GUIs (in QT) that deal with very large amounts of data (tens of millions of points). But it may also be used, like matplotlib, as a high-level interactive library to plot and visualize data.

The low-level interface is mostly done at this point (the code is still in an experimental stage though), and I'm now focusing on my current research project which is to write a scientific GUI based on this interface. However, I think people (including myself!) may be interested in a matplotlib-like high-level interface. I was first thinking about writing such an interface from scratch, by implementing a very small fraction of the matplotlib interface (basic commands like figure(), plot(), subplot(), show(), etc.). One could then quickly visualize huge datasets with the same commands than matplotlib.

Another solution would be to write a matplotlib backend based on this library. I am not familar enough with the internals of matplotlib to know how complicated it could be. I may do it myself, but it would probably take a long time since it is currently not my highest priority. I would be glad if someone experienced in writing backends was interested in working on it. Actually I could do everything that is specific to my library, which already provides commands to plot points, lines, textures, etc. The canvas is based on QT and may be similar to what is already implemented in the QT backend.

Of course, it would already be great if only the most basic plotting features were available in the backend. A first step could be for example to have a simplistic example "plot(x, sin(x))" working (with interactive navigation).

I am looking forward to your feedback.

Best,
Cyrille Rossant

Great to hear another person interested in bringing opengl to matplotlib! Another project you might be interested in collaborating with is Glumpy: http://code.google.com/p/glumpy/

From my limited knowledge of OpenGL, what my vision is that any of the existing backends have support for an OpenGL object, so we just need to be able to instantiate the opengl object in any figure object, and know how to send it the appropriate commands and data. So, it is not exactly a backend, more of a "middling". Anyway, I think the dev at Glumpy would be happy to have help, and probably have much more developed ideas on how to integrate with matplotlib.

Cheers!
Ben Root

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

OK so it seems that integrating any efficient OpenGL rendering code in matplotlib as a backend is much more complicated than what I thought.

I’m guessing with galry, you push the user-coordinates to the graphics

card, then as the user is interacting, you’re changing the transforms

and re-rendering, but don’t need to push the vertex data itself over

and over again, and with with shaders, you could do arbitrary
transforms. hence high performance.

You’re absolutely right, that’s the way high performance is achieved in Galry. In fact, the low-level interface provides a way to write custom shaders to implement really anything (interactive transformations or any rendering effect). Whereas it is possible to push data on the GPU at any time, the most efficient way of rendering stuff in Galry is to push everything at the beginning, and then only update uniform shader variables to implement transforms and dynamic effects.

···

That does not seem very compatible with the way backends appear to work then. I fear that in order to have an efficient GL backend, either this backend system would need to be updated so that it can be directly connected to transformation events, or one would need to get around the backend system.

Anyway, it looks like the difficulty would come more from the matplotlib backend system than the OpenGL part (by the way, your rendering demos are really cool Nicolas!). It would be great if a GSoC student could work on this, and I would be happy to help if necessary. In the meantime, I might write sometime an extremely basic high-level matplotlib-like interface for Galry, with support in particular for scatter plots, continuous-time signals, textures, maybe other plot types if anyone asks. It may be useful until a fully working GL backend becomes available.

Cyrille

2012/11/15 Nicolas Rougier <Nicolas.Rougier@…922…>

Yep, I’m still developing some OpenGL technics to provide both nice and fast rendering and I hope to be able to help the writing of a GL backend for matplotlib next summer (provided we get a GSoC student for the project).

So far, my main concern is that for efficient rendering using OpenGL, you need to consider that drawing means to create objects on the graphic card (line, curve, image, points, etc) that can be later manipulated (scaling/rotating/coloring/properties change, etc.). From what I remember in my early attempts at writing an OpenGL backend, I did not find the proper way to enforce such framework. Said differently, the backend is supposed to implement drawing operations while I would need to know if the drawing operations actually relates to something that is already on the graphic card or not. I’m not sure I’m very clear but I can develop the point if necessary. Having read the post by Michael (http://mdboom.github.com/blog/2012/08/06/matplotlib-client-side/) on client-side rendering, I think the proposed three-way split might be a solution but I do not know how advanced are the ideas.

To date, I’ve been working on different things:

Text/font : http://code.google.com/p/freetype-gl/ (c code)

Stroke/dash/paths: http://code.google.com/p/gl-agg/ (python)

Images: http://code.google.com/p/glumpy/ (python)

If you want to get a feel of how nice and fast rendering could be, have a look at ‘demo-lines.py’ from the gl-agg repository (and play with mouse). From these experiments, I think it is possible to achieve AGG quality using OpenGL. What is really exciting is the perspective of having a opengl/webgl backend that could be used with ipython (there has been a recent post on ipython list that show such integration for a molecule viewer).

Anyway, you’re more than welcome to contribute to glumpy, but in the long run, I hope it will disappear in favor of a matplolib GL backend.

Nicolas

On Nov 15, 2012, at 22:03 , Benjamin Root wrote:

On Thu, Nov 15, 2012 at 2:24 PM, Cyrille Rossant <cyrille.rossant@…149…> wrote:

Hi all,

I am developing a high-performance interactive visualization package in Python based on PyOpenGL (http://rossant.github.com/galry/). It is primarily meant to be used as a framework for developing complex interactive GUIs (in QT) that deal with very large amounts of data (tens of millions of points). But it may also be used, like matplotlib, as a high-level interactive library to plot and visualize data.

The low-level interface is mostly done at this point (the code is still in an experimental stage though), and I’m now focusing on my current research project which is to write a scientific GUI based on this interface. However, I think people (including myself!) may be interested in a matplotlib-like high-level interface. I was first thinking about writing such an interface from scratch, by implementing a very small fraction of the matplotlib interface (basic commands like figure(), plot(), subplot(), show(), etc.). One could then quickly visualize huge datasets with the same commands than matplotlib.

Another solution would be to write a matplotlib backend based on this library. I am not familar enough with the internals of matplotlib to know how complicated it could be. I may do it myself, but it would probably take a long time since it is currently not my highest priority. I would be glad if someone experienced in writing backends was interested in working on it. Actually I could do everything that is specific to my library, which already provides commands to plot points, lines, textures, etc. The canvas is based on QT and may be similar to what is already implemented in the QT backend.

Of course, it would already be great if only the most basic plotting features were available in the backend. A first step could be for example to have a simplistic example “plot(x, sin(x))” working (with interactive navigation).

I am looking forward to your feedback.

Best,

Cyrille Rossant

Great to hear another person interested in bringing opengl to matplotlib! Another project you might be interested in collaborating with is Glumpy: http://code.google.com/p/glumpy/

From my limited knowledge of OpenGL, what my vision is that any of the existing backends have support for an OpenGL object, so we just need to be able to instantiate the opengl object in any figure object, and know how to send it the appropriate commands and data. So, it is not exactly a backend, more of a “middling”. Anyway, I think the dev at Glumpy would be happy to have help, and probably have much more developed ideas on how to integrate with matplotlib.

Cheers!

Ben Root


Monitor your physical, virtual and cloud infrastructure from a single

web console. Get in-depth insight into apps, servers, databases, vmware,

SAP, cloud infrastructure, etc. Download 30-day Free Trial.

Pricing starts from $795 for 25 servers or applications!

http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________

Matplotlib-devel mailing list

Matplotlib-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Hi Cyrille,

I am developing a high-performance interactive visualization package in
Python based on PyOpenGL (http://rossant.github.com/galry/). It is primarily
meant to be used as a framework for developing complex interactive GUIs (in
QT) that deal with very large amounts of data (tens of millions of points).
But it may also be used, like matplotlib, as a high-level interactive
library to plot and visualize data.

quick question: how easy/feasible is WebGL integration? I ask b/c
we're starting to get the necessary machinery for easy WebGL
visualization in the ipython notebook, see e.g.:

http://www.flickr.com/photos/47156828@...1107.../8183294725

so bringing galry to the notebook with minimal code duplication would
be great. I just mention it now in case it helps you make design
decisions as you go along.

Cheers,

f

···

On Thu, Nov 15, 2012 at 11:24 AM, Cyrille Rossant <cyrille.rossant@...149...> wrote:

Hi Fernando,

It would be really great if galry could be integrated in the notebook indeed. Is the code of this demo available somewhere, so that I can get an idea about how this integration works?

In theory, galry should be compatible with WebGL because one of the main components of galry is a shader code generator that can produce OpenGL ES-compatible GLSL code. Apart from that, I suppose you have some way of making Javascript and Python communicate? The interaction system of Galry, which is based on QT but with an abstraction layer, could then be plugged to Javascript somehow… Anyway, if I could take a look to the code of this demo, I should be able to evaluate how complicated this integration would be.

···

Cyrille

2012/11/16 Fernando Perez <fperez.net@…272…149…>

Hi Cyrille,

On Thu, Nov 15, 2012 at 11:24 AM, Cyrille Rossant

<cyrille.rossant@…149…> wrote:

I am developing a high-performance interactive visualization package in

Python based on PyOpenGL (http://rossant.github.com/galry/). It is primarily

meant to be used as a framework for developing complex interactive GUIs (in

QT) that deal with very large amounts of data (tens of millions of points).

But it may also be used, like matplotlib, as a high-level interactive

library to plot and visualize data.

quick question: how easy/feasible is WebGL integration? I ask b/c

we’re starting to get the necessary machinery for easy WebGL

visualization in the ipython notebook, see e.g.:

http://www.flickr.com/photos/47156828@…1107…/8183294725

so bringing galry to the notebook with minimal code duplication would

be great. I just mention it now in case it helps you make design

decisions as you go along.

Cheers,

f

Hi Cyrille,

Hi Fernando,

It would be really great if galry could be integrated in the notebook
indeed. Is the code of this demo available somewhere, so that I can get an
idea about how this integration works?

In theory, galry should be compatible with WebGL because one of the main
components of galry is a shader code generator that can produce OpenGL
ES-compatible GLSL code. Apart from that, I suppose you have some way of
making Javascript and Python communicate? The interaction system of Galry,
which is based on QT but with an abstraction layer, could then be plugged to
Javascript somehow... Anyway, if I could take a look to the code of this
demo, I should be able to evaluate how complicated this integration would
be.

Yup, it's a bit of a hack right now b/c you need to merge several
branches and tools that are still in review, but it's not too bad.

You need to start from this branch:

https://github.com/ellisonbg/ipython/tree/jsonhandlers

and then grab this repo:

https://github.com/ipython/jsplugins

I would start by testing the d3graph plugin and verify that you can do
what I show here (watch ~ 40 seconds):

http://www.youtube.com/watch?v=F4rFuIb1Ie4&t=40m0s

That should give you the basics. Then the webgl visualizer example is here:

https://github.com/RishiRamraj/seepymol

Cheers,

f

···

On Fri, Nov 16, 2012 at 1:00 PM, Cyrille Rossant <cyrille.rossant@...149...> wrote:

Yup, it's a bit of a hack right now b/c you need to merge several
branches and tools that are still in review, but it's not too bad.

You need to start from this branch:

https://github.com/ellisonbg/ipython/tree/jsonhandlers

and then grab this repo:

https://github.com/ipython/jsplugins

I would start by testing the d3graph plugin and verify that you can do
what I show here (watch ~ 40 seconds):

http://www.youtube.com/watch?v=F4rFuIb1Ie4&t=40m0s

That should give you the basics. Then the webgl visualizer example is
here:

https://github.com/RishiRamraj/seepymol

Cheers,

f

Great, thanks! I'll take a look to that and get back to you when I have
something.

Cheers,
Cyrille

Yup, it's a bit of a hack right now b/c you need to merge several

branches and tools that are still in review, but it's not too bad.

You need to start from this branch:

https://github.com/ellisonbg/ipython/tree/jsonhandlers

and then grab this repo:

https://github.com/ipython/jsplugins

I would start by testing the d3graph plugin and verify that you can do
what I show here (watch ~ 40 seconds):

http://www.youtube.com/watch?v=F4rFuIb1Ie4&t=40m0s

That should give you the basics. Then the webgl visualizer example is
here:

https://github.com/RishiRamraj/seepymol

OK so I now have a very experimental proof of concept of how integrating
Galry in the IPython notebook. There's a short demo here:
http://www.youtube.com/watch?v=taN4TobRS-E

I'll put the code on github but there's of course much more to do.

I'll also work on a basic matplotlib-like high-level interface that will
work in both standard python/ipython consoles, and in the IPython notebook.

Cheers,
Cyrille

Awesome! This is really great to see, before long we'll sort out
these APIs so all this can be made available easily to end users.

Great job!

Cheers,

f

···

On Sat, Nov 17, 2012 at 4:07 PM, Cyrille Rossant <cyrille.rossant@...149...> wrote:

OK so I now have a very experimental proof of concept of how integrating
Galry in the IPython notebook. There's a short demo here:
http://www.youtube.com/watch?v=taN4TobRS-E

I'll put the code on github but there's of course much more to do.

I'll also work on a basic matplotlib-like high-level interface that will
work in both standard python/ipython consoles, and in the IPython notebook.