Cocoa backend for embedding in Cocoa application

I've written the start of a Cocoa-native backend for matplotlib and
would like to submit feedback on the code and on the possibility of
including it in the standard matplotlib distribution. The backend
implementation is not complete (image rendering and mathtext rendering
are currently no implemented, nor are the print_* methods of the
FigureCanvas). Image rendering is trivial once I figure out how to get
the pixel data out of a matplotlib image (I just haven't investigated
the API yet). The print_* methods are also trivial (see point 1
below). I'm not sure how to handle mathtext yet. This backend has two
major feature differences from CocoaAgg:

1. All drawing is done via native Quartz drawing calls. This has the
advantage of making all of the native rendering functionality
(including native font rendering and output formats (all those
supported by Cocoa-eps,pdf,tiff,etc.) available to matplotlib. It has
the disadvantage that the translation from matplotlib paths to Cocoa
NSBezierPaths is done in python and is much slower than the compiled
version of this operation in the Agg backend (this is a solvable
problem, of course).

2. The reason I wrote the backend was so that matplotlib could be used
seemlesslly from within a Cocoa application. Thus this backend *will
not work* without an existing NSRunLoop. It won't work from the
terminal or an IPython session. It will work from the in-progress
Cocoa frontend for IPython or from any other Cocoa application. Again
there are tradeoffs. On the positive side, figure windows are treated
like any other application window, selectable from the Window menu
etc. and matplotlib becomes a seemless part of the application.
Existing backends designed to create their own runloop (e.g. CocoaAgg
or TkAgg) cause menubar and run loop problems when used from within an
existing application. It would be possible to merge the CocoaAgg and
Cocoa backends in this regard to use the existing run loop if present.

I've attached the backend_cocoa.py.

cheers,
Barry

backend_cocoa.py (23.5 KB)

I've written the start of a Cocoa-native backend for matplotlib and
would like to submit feedback on the code and on the possibility of
including it in the standard matplotlib distribution. The backend

Hey Barry,

This is great and it is something we are interested in. I haven't had
a chance to test it and will be away next week but I can give you some
initial thoughts. In general, we want to pare down the number of
backends but OS X is an important platform and there are certainly
some advantages to doing native rendering. For us to accept a new
backend, we need to meet the following criteria:

  - someone volunteers to support it. This is a multi-year
commitment. Presumably this would be you.

  - feature complete - images, mathtext, rotated text, etc... It
sounds like the only part you are unsure about is mathtext so perhaps
Michael can advise. I don't think it will be too hard since Michael
has designed this to work with unicode fonts if requested.

  - gui backends: toolbar support

If you think these are doable, that would be great. If not, I think
we can rework the backend infrastructure a bit to support external
backend, eg an rc backend setting which points to a file that contains
the backend implementation. This latter is worth doing in any case,
so if you want to have a look at it....

implementation is not complete (image rendering and mathtext rendering
are currently no implemented, nor are the print_* methods of the
FigureCanvas). Image rendering is trivial once I figure out how to get
the pixel data out of a matplotlib image (I just haven't investigated
the API yet). The print_* methods are also trivial (see point 1
below). I'm not sure how to handle mathtext yet. This backend has two
major feature differences from CocoaAgg:

2. The reason I wrote the backend was so that matplotlib could be used
seemlesslly from within a Cocoa application. Thus this backend *will
not work* without an existing NSRunLoop. It won't work from the
terminal or an IPython session. It will work from the in-progress
Cocoa frontend for IPython or from any other Cocoa application. Again
there are tradeoffs. On the positive side, figure windows are treated
like any other application window, selectable from the Window menu
etc. and matplotlib becomes a seemless part of the application.
Existing backends designed to create their own runloop (e.g. CocoaAgg
or TkAgg) cause menubar and run loop problems when used from within an
existing application. It would be possible to merge the CocoaAgg and
Cocoa backends in this regard to use the existing run loop if present.

I know nothing about cocoa apps, but from the way you describe this I
think there may be some confusion in how the backends should work. I
will speak in terms of gtk since this is the toolkit I know best. The
FigureCanvas should be a widget which is embeddable in an app (in a
container or window, etc) -- the gtk figure canvas is such a widget
and can be used in a gtk app and mpl knows nothing about the
application or mainloop -- that part is up to the application
developer. The FigureManager is basically a pylab helper class and is
not meant to be used by the application developer, though I think some
backends may have been designed differently (eg wx). The
FigureManager creates a canvas, a toolbar and a window, and puts all
the pieces together. "show" is a pylab construct that raises the
active windows and starts the mainloop.

So your backend *should* be designed so that it can be run from the
shell, basically you need to create an application within the backend
and start the mainloop with show -- this app will only be created in
pylab mode on calls to new_figure_manager and show. I don't know if
any of this makes sense for cocoa apps, but it is a nice feature for
backends because then those of us who do not know how to build a cocoa
app (and maybe don't want to learn) can still test the backend on
pylab scripts and regular os x users can use your backend w/ pylab
scripts.

JDH

···

On Wed, Jul 2, 2008 at 3:41 PM, Barry Wark <barrywark@...287...> wrote:

I've written the start of a Cocoa-native backend for matplotlib and
would like to submit feedback on the code and on the possibility of
including it in the standard matplotlib distribution. The backend

Hey Barry,

This is great and it is something we are interested in. I haven't had
a chance to test it and will be away next week but I can give you some
initial thoughts. In general, we want to pare down the number of
backends but OS X is an important platform and there are certainly
some advantages to doing native rendering. For us to accept a new
backend, we need to meet the following criteria:

- someone volunteers to support it. This is a multi-year
commitment. Presumably this would be you.

Yes.

- feature complete - images, mathtext, rotated text, etc... It
sounds like the only part you are unsure about is mathtext so perhaps
Michael can advise. I don't think it will be too hard since Michael
has designed this to work with unicode fonts if requested.

Using unicode fonts should make it easy.

- gui backends: toolbar support

Currently not implemented, but not a problem.

If you think these are doable, that would be great. If not, I think
we can rework the backend infrastructure a bit to support external
backend, eg an rc backend setting which points to a file that contains
the backend implementation. This latter is worth doing in any case,
so if you want to have a look at it....

This would be _very_ useful, whether or not you decide to include a
Cocoa native backend in the distribution. Perhaps also making it
possible to delegate to an other package instead of just a file so
that backends could be installed as separate eggs would be useuful
(e.g. delegate the backend to my.package.backend).

implementation is not complete (image rendering and mathtext rendering
are currently no implemented, nor are the print_* methods of the
FigureCanvas). Image rendering is trivial once I figure out how to get
the pixel data out of a matplotlib image (I just haven't investigated
the API yet). The print_* methods are also trivial (see point 1
below). I'm not sure how to handle mathtext yet. This backend has two
major feature differences from CocoaAgg:

2. The reason I wrote the backend was so that matplotlib could be used
seemlesslly from within a Cocoa application. Thus this backend *will
not work* without an existing NSRunLoop. It won't work from the
terminal or an IPython session. It will work from the in-progress
Cocoa frontend for IPython or from any other Cocoa application. Again
there are tradeoffs. On the positive side, figure windows are treated
like any other application window, selectable from the Window menu
etc. and matplotlib becomes a seemless part of the application.
Existing backends designed to create their own runloop (e.g. CocoaAgg
or TkAgg) cause menubar and run loop problems when used from within an
existing application. It would be possible to merge the CocoaAgg and
Cocoa backends in this regard to use the existing run loop if present.

I know nothing about cocoa apps, but from the way you describe this I
think there may be some confusion in how the backends should work. I
will speak in terms of gtk since this is the toolkit I know best. The
FigureCanvas should be a widget which is embeddable in an app (in a
container or window, etc) -- the gtk figure canvas is such a widget
and can be used in a gtk app and mpl knows nothing about the
application or mainloop -- that part is up to the application
developer. The FigureManager is basically a pylab helper class and is
not meant to be used by the application developer, though I think some
backends may have been designed differently (eg wx). The
FigureManager creates a canvas, a toolbar and a window, and puts all
the pieces together. "show" is a pylab construct that raises the
active windows and starts the mainloop.

No, in fact you've done a very good job of communicating the
architecture and it appears that it was my mistake in explaining my
work that's led to confusion. backend_cocoa.FigureCanvasView (a
FigureCanvas subclass) is an NSView subclass that can be embedded in a
Cocoa application to render matplotlib figures.

We have a special use case in mind for the
backend_cocoa.FigureManagerCocoa (the FigureManager subclass),
however. I'm not sure if you've been following discussions on the
ipython-dev regarding GUI frontends for IPython, so let me briefly
recap. The new Twisted-based architecture of IPython now allows easy
development of GUI frontends for the IPython engine because the engine
is decoupled from readline. Thus a Cocoa-based frontend for IPython
might behave like a terminal but be implemented as a native Cocoa
widget that can be embedded in other applications. We want the user of
this GUI IPython frontend to be able to use matplotlib as if in a
pylab session at the terminal. Because there's already an NSRunLoop
running, however, the existing backends that attempt to start their on
run loop cause problems. I imagine GUI frontends (on OS X) doing a
matplotlib.use('Cocoa') at startup but the user keeping their existing
backend for terminal usage. I haven't yet explored the
FigureManagerCocoa creating its own runloop if none exists. If you
want to try the new IPython frontend (still very much a work in
progress), have a look in the IPython/frontend/cocoa/examples
directory of IPython trunk (launchpad.net/ipython).

Thus there are really two different aspects to consider for backend_cocoa
1. Native rendering. this may or may not be valuable. If valuable, we
could merge this renderer into the existing CocoaAgg backend,
replacing the FiureCanvas that blits from the Agg renderer.
2. A FigureManager for use *within* a running cocoa application. I
will investigate using the FigureManagerCocoa from terminal as well,
but that's not the use case I wrote it for.

Although I personally think (2) (in conjunciton with a native forntend
for IPython) is the way of the future on OS X, it may still be too
small a use case for you to feel its inclusion in matplotlib is
valuable. In that case, being able to delegate the backend to an other
package as discussed above would meet our needs.

So your backend *should* be designed so that it can be run from the
shell, basically you need to create an application within the backend
and start the mainloop with show -- this app will only be created in
pylab mode on calls to new_figure_manager and show. I don't know if
any of this makes sense for cocoa apps, but it is a nice feature for
backends because then those of us who do not know how to build a cocoa
app (and maybe don't want to learn) can still test the backend on
pylab scripts and regular os x users can use your backend w/ pylab
scripts.

backend_cocoaagg already does this. if there's a compelling reason to
move to the native renderer, we could merge that into
backend_cocoaagg. backend_cocoa provides a separate functionality --
simulating a pylab session from within a native GUI frontend for
IPython.

···

On Thu, Jul 3, 2008 at 8:41 AM, John Hunter <jdh2358@...287...> wrote:

On Wed, Jul 2, 2008 at 3:41 PM, Barry Wark <barrywark@...287...> wrote:

JDH

Work is currently going in the wrong direction for me to be able to
implement toolbars etc. in a timely manner. Would you be willing to
add the rc parameter you discussed so that backend could be defined in
a separate package? This would allow me to continue working on (and
deploying internally) the Cocoa native backend. As soon as it meets
the requirements, I will be happy to resubmit it for inclusion with
the matplotlib distribution (including supporting it going forward).

Barry

···

On Thu, Jul 3, 2008 at 12:14 PM, Barry Wark <barrywark@...287...> wrote:

On Thu, Jul 3, 2008 at 8:41 AM, John Hunter <jdh2358@...287...> wrote:

On Wed, Jul 2, 2008 at 3:41 PM, Barry Wark <barrywark@...287...> wrote:

I've written the start of a Cocoa-native backend for matplotlib and
would like to submit feedback on the code and on the possibility of
including it in the standard matplotlib distribution. The backend

Hey Barry,

This is great and it is something we are interested in. I haven't had
a chance to test it and will be away next week but I can give you some
initial thoughts. In general, we want to pare down the number of
backends but OS X is an important platform and there are certainly
some advantages to doing native rendering. For us to accept a new
backend, we need to meet the following criteria:

- someone volunteers to support it. This is a multi-year
commitment. Presumably this would be you.

Yes.

- feature complete - images, mathtext, rotated text, etc... It
sounds like the only part you are unsure about is mathtext so perhaps
Michael can advise. I don't think it will be too hard since Michael
has designed this to work with unicode fonts if requested.

Using unicode fonts should make it easy.

- gui backends: toolbar support

Currently not implemented, but not a problem.

If you think these are doable, that would be great. If not, I think
we can rework the backend infrastructure a bit to support external
backend, eg an rc backend setting which points to a file that contains
the backend implementation. This latter is worth doing in any case,
so if you want to have a look at it....

This would be _very_ useful, whether or not you decide to include a
Cocoa native backend in the distribution. Perhaps also making it
possible to delegate to an other package instead of just a file so
that backends could be installed as separate eggs would be useuful
(e.g. delegate the backend to my.package.backend).

implementation is not complete (image rendering and mathtext rendering
are currently no implemented, nor are the print_* methods of the
FigureCanvas). Image rendering is trivial once I figure out how to get
the pixel data out of a matplotlib image (I just haven't investigated
the API yet). The print_* methods are also trivial (see point 1
below). I'm not sure how to handle mathtext yet. This backend has two
major feature differences from CocoaAgg:

2. The reason I wrote the backend was so that matplotlib could be used
seemlesslly from within a Cocoa application. Thus this backend *will
not work* without an existing NSRunLoop. It won't work from the
terminal or an IPython session. It will work from the in-progress
Cocoa frontend for IPython or from any other Cocoa application. Again
there are tradeoffs. On the positive side, figure windows are treated
like any other application window, selectable from the Window menu
etc. and matplotlib becomes a seemless part of the application.
Existing backends designed to create their own runloop (e.g. CocoaAgg
or TkAgg) cause menubar and run loop problems when used from within an
existing application. It would be possible to merge the CocoaAgg and
Cocoa backends in this regard to use the existing run loop if present.

I know nothing about cocoa apps, but from the way you describe this I
think there may be some confusion in how the backends should work. I
will speak in terms of gtk since this is the toolkit I know best. The
FigureCanvas should be a widget which is embeddable in an app (in a
container or window, etc) -- the gtk figure canvas is such a widget
and can be used in a gtk app and mpl knows nothing about the
application or mainloop -- that part is up to the application
developer. The FigureManager is basically a pylab helper class and is
not meant to be used by the application developer, though I think some
backends may have been designed differently (eg wx). The
FigureManager creates a canvas, a toolbar and a window, and puts all
the pieces together. "show" is a pylab construct that raises the
active windows and starts the mainloop.

No, in fact you've done a very good job of communicating the
architecture and it appears that it was my mistake in explaining my
work that's led to confusion. backend_cocoa.FigureCanvasView (a
FigureCanvas subclass) is an NSView subclass that can be embedded in a
Cocoa application to render matplotlib figures.

We have a special use case in mind for the
backend_cocoa.FigureManagerCocoa (the FigureManager subclass),
however. I'm not sure if you've been following discussions on the
ipython-dev regarding GUI frontends for IPython, so let me briefly
recap. The new Twisted-based architecture of IPython now allows easy
development of GUI frontends for the IPython engine because the engine
is decoupled from readline. Thus a Cocoa-based frontend for IPython
might behave like a terminal but be implemented as a native Cocoa
widget that can be embedded in other applications. We want the user of
this GUI IPython frontend to be able to use matplotlib as if in a
pylab session at the terminal. Because there's already an NSRunLoop
running, however, the existing backends that attempt to start their on
run loop cause problems. I imagine GUI frontends (on OS X) doing a
matplotlib.use('Cocoa') at startup but the user keeping their existing
backend for terminal usage. I haven't yet explored the
FigureManagerCocoa creating its own runloop if none exists. If you
want to try the new IPython frontend (still very much a work in
progress), have a look in the IPython/frontend/cocoa/examples
directory of IPython trunk (launchpad.net/ipython).

Thus there are really two different aspects to consider for backend_cocoa
1. Native rendering. this may or may not be valuable. If valuable, we
could merge this renderer into the existing CocoaAgg backend,
replacing the FiureCanvas that blits from the Agg renderer.
2. A FigureManager for use *within* a running cocoa application. I
will investigate using the FigureManagerCocoa from terminal as well,
but that's not the use case I wrote it for.

Although I personally think (2) (in conjunciton with a native forntend
for IPython) is the way of the future on OS X, it may still be too
small a use case for you to feel its inclusion in matplotlib is
valuable. In that case, being able to delegate the backend to an other
package as discussed above would meet our needs.

So your backend *should* be designed so that it can be run from the
shell, basically you need to create an application within the backend
and start the mainloop with show -- this app will only be created in
pylab mode on calls to new_figure_manager and show. I don't know if
any of this makes sense for cocoa apps, but it is a nice feature for
backends because then those of us who do not know how to build a cocoa
app (and maybe don't want to learn) can still test the backend on
pylab scripts and regular os x users can use your backend w/ pylab
scripts.

backend_cocoaagg already does this. if there's a compelling reason to
move to the native renderer, we could merge that into
backend_cocoaagg. backend_cocoa provides a separate functionality --
simulating a pylab session from within a native GUI frontend for
IPython.

JDH

Work is currently going in the wrong direction for me to be able to
implement toolbars etc. in a timely manner. Would you be willing to
add the rc parameter you discussed so that backend could be defined in
a separate package? This would allow me to continue working on (and
deploying internally) the Cocoa native backend. As soon as it meets
the requirements, I will be happy to resubmit it for inclusion with
the matplotlib distribution (including supporting it going forward).

I just committed some changes to svn to support the syntax
module://some_backend to load the backend from a module in the
pythonpath. Give it a test drive and see if it works in your use
cases. The matplotlibrc file, the use directive and the -d argument
to pylab will all respect this syntax, eg

  matplotlib.use('module://backend_cocoa')

or

  > python simple_plot.py -dmodule://my_backend

but backend_cocoa doesn't currently work with pylab...

Outside of of pylab, it doesn't matter much because you could already
import directly from your own modules if you are building apps, but at
least the backend validation machinery won't complain if it sees
something starting with 'module://'

Let me know if you had something different in mind....

JDH

···

On Sat, Jul 12, 2008 at 3:09 PM, Barry Wark <barrywark@...287...> wrote:

This looks like exactly what I had in mind. Thanks! A quick test seems
to work and I will ping you if I find issues upon digging deeper.
Thanks!

Barry

···

On Sat, Jul 12, 2008 at 2:35 PM, John Hunter <jdh2358@...287...> wrote:

On Sat, Jul 12, 2008 at 3:09 PM, Barry Wark <barrywark@...287...> wrote:

Work is currently going in the wrong direction for me to be able to
implement toolbars etc. in a timely manner. Would you be willing to
add the rc parameter you discussed so that backend could be defined in
a separate package? This would allow me to continue working on (and
deploying internally) the Cocoa native backend. As soon as it meets
the requirements, I will be happy to resubmit it for inclusion with
the matplotlib distribution (including supporting it going forward).

I just committed some changes to svn to support the syntax
module://some_backend to load the backend from a module in the
pythonpath. Give it a test drive and see if it works in your use
cases. The matplotlibrc file, the use directive and the -d argument
to pylab will all respect this syntax, eg

matplotlib.use('module://backend_cocoa')

or

> python simple_plot.py -dmodule://my_backend

but backend_cocoa doesn't currently work with pylab...

Outside of of pylab, it doesn't matter much because you could already
import directly from your own modules if you are building apps, but at
least the backend validation machinery won't complain if it sees
something starting with 'module://'

Let me know if you had something different in mind....