advice on writing an application

Here's what I think the fundamental differences are. Maybe Ken can
give an opinion too.

My take on things is that WxMpl and MPlot solve two different

problems.

WxMpl is intended to be a FigureCanvasWxAgg that has some

pointy-clicky

user interactions (e.g. zoom) built in. I gather that Matt considers
it "focused on the programmer/script writer, not on the end user
of an application" because it doesn't provide end-users with any means
to edit the plot. MPlot lets users interactively change the title,
axes labels, etc.

I wrote WxMpl after Matt sent me MPlot 0.7 because I felt that his
approach wasn't solving my problem: I was thrilled what matplotlib
and wanted *all* of it to Just Work with wxPython. Allowing users
to edit plots was less important to me than supporting as many kinds
of plots as possible. By focusing on adding user interaction to
FigureCanvasWxAgg I was able to support all of matplotlib's OO API
in one fell swoop.

Sorry to horn in here, but this is a topic in which I am very
interested. What I want is both of these solutions combined! I'd like
to have the full power of MatPlotLib when I'm developing my
applications, but, I'd also like to have (nearly) that full power
presented to the user for further customization and interaction. I'd
like to have many more features available to the user than are currently
provided by Mplot, but when I'm coding I'd like to be able to drop in a
complete panel without having to do the coding that I'd need to do with
WxMPL.

My idea is for an interaction model similar to most spreadsheet graph
objects (e.g. Excel), where context menus control the individual
components of the plot(s). Perhaps with introspection, many of the
customizable features could be automatically extracted and presented to
the user, so that menus and dialog boxes wouldn't have to be completely
coded by hand.

I'd like the plot to appear simple (by default), but with suitable
access (perhaps triggered by mouseover?) to hidden features like a
toolbar, scrollable axes, crosshair cursors (with coordinate readout),
and zoom controls.

I'm certainly willing and able to contribute to this development. I'm
conflicted as to which code base to begin with, however.

Andrew Henshaw
Georgia Tech Research Institute

···

On 12/01/05 16:13, Matt Newville wrote:

Sorry to horn in here, but this is a topic in which I am very
interested. What I want is both of these solutions combined! I'd like
to have the full power of MatPlotLib when I'm developing my
applications, but, I'd also like to have (nearly) that full power
presented to the user for further customization and interaction. I'd
like to have many more features available to the user than are currently
provided by Mplot, but when I'm coding I'd like to be able to drop in a
complete panel without having to do the coding that I'd need to do with
WxMPL.
[...] I'm certainly willing and able to contribute to this development. I'm
conflicted as to which code base to begin with, however.

Tell me what do you think too about the "stack model" I proposed a couple of posts above. I'd like to help with it, but I'm still very much in the learning phase...

Massimo

···

--
Massimo Sandal
University of Bologna
Department of Biochemistry "G.Moruzzi"

snail mail:
Via Irnerio 48, 40126 Bologna, Italy

email:
massimo.sandal@...898...

tel: +39-051-2094388
fax: +39-051-2094387

I'd like to have many more features available to the user than are
currently provided by Mplot, but when I'm coding I'd like to be able to
drop in a complete panel without having to do the coding that I'd need to
do with WxMPL.

It sounds you'd like to see a library with a simplified API that supports most kinds of plots while letting users edit things like the axes title, line width, number of histogram bins, etc. Am I correct?

Perhaps with introspection, many of the customizable features could be
automatically extracted and presented to the user, so that menus and dialog
boxes wouldn't have to be completely coded by hand.

I would definitely go with introspection and declarative programming. First, come up with a reasonable object picking algorithm (see the `object_picker.py' example). Let matplot manage all of the plot objects' data (axes title, line style, edge color, etc). Use declarative programming to define the editable parameters for each object. Finally, use introspection to map between type of the object and the list of editable parameters. PEP 246 adaption might be useful here, since it would let you break the tight coupling between types and their parameters:

  def OnPickObject(self, evt):
    obj = evt.object
    params = adapt(obj, IObjectParameters, None)
    if params is not None:
      frame = ObjectEditorFrame(self, params)
      frame.Show()

  # ...imagine that ObjectEditorFrame looks something like right hand
  # side of the window on page 15 of
  # http://www.python.org/pycon/papers/chaco.pdf
  # except that is automagically builds its contents based on the
  # its IObjectParameters argument.

It might be a good idea to bug John about matplotlib transitioning to using
the Traits package from Enthough, which would take care of managing the object
parameters for you. They developed it for use in their Chaco plotting library,
which went MIA shortly after release.

I'd like the plot to appear simple (by default), but with suitable
access (perhaps triggered by mouseover?) to hidden features like a
toolbar, scrollable axes, crosshair cursors (with coordinate readout),
and zoom controls.

A toolbar that pops up via mouseover would be pretty impressive, but I can't think of a way to implement it off the top of my head. I'm not sure what you mean by "scrollable axes". Do you mean pannning, the way pylab plots work? Crosshair cursors are easy and coordinate readout are pretty easy and I believe they already work in both MPlot and WxMpl. Likewise zoom controls.

I'm certainly willing and able to contribute to this development. I'm
conflicted as to which code base to begin with, however.

I'm not sure what to recommend. Merging MPlot and WxMpl so that MPlot uses WxMpl for rendering and plot-level user interactions like zooming would be one way to start. I think you'd probably be better off going with a more dynamic approach to parameter editing, since it would be a bit more work up front but a lot less work to add plot types. If done right, the object picking and parameters stuff could even be backend independent.

Ken

···

On 12/02/05 09:26, Andy.Henshaw@...905... wrote:

HI,

I agree with Ken -- this is definitely not a flamewar. I have no
ill-will toward anyone here, or any of the codes in discussion. In
fact, quite the opposite. I also do not have a lot of time to devote
to this discussion or MPlot itself, mostly due to running experiments
and working on and supporting about other analysis software. I have
very little allegiance to MPlot, take no pride in it and in no way
'want it to win' over anything else. I do want something that has the
end-user level flexibility of MPlot, and would happily use something
else if available. I am committed to wxPython because, while a
long-time and confirmed Unix (now linux/OSX) person, about 90% of the
'scientific customers' for my software use Windows (and at my age and
baseline crankiness, I don't convert easily to anything).

···

==================
Massimo wrote:

I have only tried a little wxmpl, and still didn't try MPlot, but I was
wondering: can MPlot be rewritten using WxMpl?

Definitely. Volunteers welcome. My only advice here would be to
remember that bridging wxPython and matplotlib should use the best of
both and that the maintainers will have to know both, but the users of
the library should probably not have to know anything about
matplotlib, because they're going to be asking "I'm writing this wx
Application and want to include a graph -- what should I use?".

In this way we wouldn't have to "choose" and to fork the goal of a
unified wxPython/MPL solution. Instead, we would have a flexible stack
of solutions at different levels. WxMpl would stay less or more as it
is, providing a quite low level interface for people who need/like it (I
feel I'm among these people, although it's still very possible to change
my mind). On top of it, MPlot would extend WxMpl, creating advanced
wrappers and controls to ease things for who needs "intermediate"
scripting (a need that I think is very real). And possibly in future
Pylab would stay on top of this stack, using MPlot and WxMpl to deal
with interactive plotting.

I agree.

==================
John wrote:

Ken, Matt: would either or both of you like to bundle these packages
into the mpl releases so they would be easier for users to find and
use? We could either put them in the main release or in a "toolbox".

This is OK with me, but I can go either way. I sent MPlot to you over
a year ago and suggested that it could be included with MPL. At the
time, I think you had no 'toolbox' model for that kind of thing.

==================
Andrew wrote:

My idea is for an interaction model similar to most spreadsheet graph
objects (e.g. Excel), where context menus control the individual
components of the plot(s). Perhaps with introspection, many of the
customizable features could be automatically extracted and presented to
the user, so that menus and dialog boxes wouldn't have to be completely
coded by hand.

I'd like the plot to appear simple (by default), but with suitable
access (perhaps triggered by mouseover?) to hidden features like a
toolbar, scrollable axes, crosshair cursors (with coordinate readout),
and zoom controls.

I'm certainly willing and able to contribute to this development. I'm
conflicted as to which code base to begin with, however.

There's potentially a lot of flexibility and actions to trigger there.
With BLT (and Pmw.BLT) clicking on a component (say, a trace or label)
could cause a 'configure' box to modify its attributes (line color,
etc). I found this could confuse users who would accidentally click
on things, but it's not a terrible model. For MPlot, I made
Menu->Option->Configure or
right-click->pop-up->Configure bring up a form for many graph customizations.

To do this kind of interactivity, I believe that mpl would have to
support it, and that might be tricky using the wxAgg backend (that
is, I'm not sure how easy it would be to convert coordinates on the
agg-image back to the Artist that rendered it). So I think clickable
plot objects is a reasonable way to go, but I might not make it the
highest priority.

==================
Ken wrote:

> So, out of curiousity which of the *all* the features do you actually
> use that can't be wrapped away to a higher level interface?

I can't think of any features that can't be wrapped away into a higher level
interface, at least hypothetically. Things like creating two subplots that
share an X axis (so they zoom together) would be a bit hairy, but I'm sure it
could be done.

Oh, well I think this would get so application specific that it would
be just as easy to do it at the GUI level than at the MPL level.

That being said, it would be frustrating if you had to periodically rewrite
the user interaction code to add support new things, like subplots and
overlapping Axes. Providing this kind of support initially was an important
goal while writing WxMpl.

Oh the 'user-level support code' is definitely the larger part of the
work for adding any plot type. With imshow(), you'd need to provide a
GUI to choose color tables and scalebars to adjust levels and
contrast, and have a place on the GUI for displaying coordinates and
intensities. And of course, zooming, flipping, etc. For the
application we both happen to most need false-color-maps for (x-ray
fluorescence), I'd also want a way to put overlay one map in red and
another in green (with user-selectable colors of course). So adding
'axes.imshow()' is the easy part. Thanks to matplotlib. I did this
once with PIL and the Tk canvas -- and I'm using that program today,
as it turns out.

> I've seen the matplotlib code, and have used a fair number of plotting
> libraries in my day. The matplotlib results are wonderful, but the API is a
> bit goofy, don't you think? Does FigureCanvas->Figure->Axes make sense to
> you? I'd be OK with two of those, but I don't understand how three possibly
> helps.

I don't think it's goofy at all. You have FigureCanvas because seperating out
the thing that "draws" the figure is the best way to implement backend
neutrality. You have Figure and Axes (rather than just Axes) because that's
the best way to implement things like subplots, plotting overlapped axes, and
figimage(), colorbars, and figure legends.

I think we agree: the hierarchy is implementation driven, not use driven.

> A grid of multiple plots is certainly possible by packing
> panels together.
That's not quite what I meant:
       http://matplotlib.sourceforge.net/examples/shared_axis_demo.py

All I'm saying is that this _could_ be handled at a GUI level. The
advantage there is that you absolutely know someone using a wxPython
plotting widget will know wxPython, whereas their understanding of
matplotlib may be zero.

==================
Fernando said:

> >I'm also really very concerned about the priority given to pylab and ipython.
>
> I'm really concerned about it too. As a MPL newbie, I found it very
> misleading. I'd like to see a full programming stack on top of MPL, with
> pylab, MPlot and wxMpl being integrated components of it, and with a
> nice documentation about it.

Chill out, there is no dark plan here :slight_smile: It's simply a matter of usage
patterns: my personal audience is one of other programmers, I'm mostly an
algorithms developer and the code I write is used by other mathematicians,
physicists, etc. to write programs.

I think Fernando is not quite understanding the concern or complaint.
I also don't think anyone is 'hot' about it either. And I know that
Ken and I both "understand how free software works". As it turns out,
I'm mostly 'an algorithm developer' too and write open source
applications that other scientists use as well.

Certainly IPython is very nice, and has an important niche -- I've
even used it a couple of times myself. The issue (for mpl) is that it
may appear to be the "normal" way one uses mpl. I do find that to
similar to confusing python with IPython. IPython is one way to use
python, but it's not only way .... and it's not driving decisions
about python.

For mpl, I believe that the 'make a wx mainloop for a command-line
program' did drive some fairly ugly business in backend_wx.py that
ended up making it harder to use backend_wx for GUIs. I tried to
patch this to be favor GUI writers, but the conflict was resolved in
favor of the wx-mainloop-in-commandline. I believe this may be fixed
now.

More importantly, when discussing matplotlib the usage is generally
given in terms of pylab, and often 'from pylab import *', which
conveys the idea that matplotlib is intended for writing quick-n-dirty
code. And, as you can see from this discussion, the best way to use
it in real GUI applications is not always well resolved <wink>. So
it is easy to conclude that pylab and ipython -pylab are given
priority.

I stand by my concern about this, and I think you're missing an
important point if you easily dismiss the fact that Massimo's found
this to be misleading.

Finally, I'll be out of town and won't be able to respond for at least
a week. Sorry this was so long, and thanks again.

-Matt

Matt Newville wrote:

I think Fernando is not quite understanding the concern or complaint. I also don't think anyone is 'hot' about it either. And I know that
Ken and I both "understand how free software works". As it turns out,
I'm mostly 'an algorithm developer' too and write open source
applications that other scientists use as well.

Certainly IPython is very nice, and has an important niche -- I've
even used it a couple of times myself. The issue (for mpl) is that it
may appear to be the "normal" way one uses mpl. I do find that to
similar to confusing python with IPython. IPython is one way to use
python, but it's not only way .... and it's not driving decisions
about python.

For mpl, I believe that the 'make a wx mainloop for a command-line
program' did drive some fairly ugly business in backend_wx.py that
ended up making it harder to use backend_wx for GUIs. I tried to
patch this to be favor GUI writers, but the conflict was resolved in
favor of the wx-mainloop-in-commandline. I believe this may be fixed
now.

More importantly, when discussing matplotlib the usage is generally
given in terms of pylab, and often 'from pylab import *', which
conveys the idea that matplotlib is intended for writing quick-n-dirty
code. And, as you can see from this discussion, the best way to use
it in real GUI applications is not always well resolved <wink>. So
it is easy to conclude that pylab and ipython -pylab are given
priority.

I stand by my concern about this, and I think you're missing an
important point if you easily dismiss the fact that Massimo's found
this to be misleading.

OK, I don't have the time to get into a long-winded discussion here. My simple point is that the reasons for the ipython/pylab presence in the public docs are mostly historical: over one year ago, I needed this, I wrote it and it got integrated (both mpl and ipython needed changes for this to work). I should even add that I use Tk, not WX/GTK/Qt at all, so I could have lived without this support, but I figured that it would be a good thing to have a well-rounded interactive support.

Now, those using mpl as a library for handling plotting in GUI apps are becoming a growing group: I think that's GREAT! I will probably benefit myself from these developments in the future, as I may need WX support for other things.

I am not _dismissing_ anything. I am simply saying that if this is a need of a group of users, then the mechanisms exist to address it. But saying that the docs 'have' to present one aspect vs another is, in a free software world, misplaced. No free software project 'has' to do anything: it does what those writing the code make it do, end of story.

When 'pitching' mpl to other users, I always stress how the fact that it's a _library_ (instead of a program like gnuplot) is one of its greatest architectural assets. While I personally don't use it as such (I could still get by with gnuplot and would be fine, given my needs), I use mpl because I like to know that, were I to need a more complex usage mode, the library allows me to do so. So I am fully aware of the value and importance of this aspect of mpl.

To summarize: there's a need for easy, interactive plotting in python. ipython/pylab address that reasonably well. There's also a need for easier usage of mpl in a more complex WX context: you guys are working on improving the situation here, and that's an excellent development.

The only sense in which the first aspect was given 'priority' was historical: we wrote that code over a year ago. As your solutions mature and become publicly documented, I'm sure they will be given equal space, since they address a _different_ problem than pylab/ipython do. Not more important, not less important: different.

I'm afraid that this is drifting off-topic, so I'll probably close it here. It seems I wasn't very successful in clarifying a simple fact of the evolution of matplotlib I was involved with, without generating more noise than necessary. Sorry about that.

Cheers,

f

Ken wrote:

I can't think of any features that can't be wrapped away into a higher level
interface, at least hypothetically. Things like creating two subplots that
share an X axis (so they zoom together) would be a bit hairy, but I'm sure it
could be done.

Oh, well I think this would get so application specific that it would
be just as easy to do it at the GUI level than at the MPL level.

That's possible, but there is already a quick and well-documented way to do it at the MPL level. Situations like that, where there's already a backend-portable way to do things, are what WxMpl tries to cater.

That being said, it would be frustrating if you had to periodically rewrite
the user interaction code to add support new things, like subplots and
overlapping Axes. Providing this kind of support initially was an important
goal while writing WxMpl.

Oh the 'user-level support code' is definitely the larger part of the
work for adding any plot type.

I should have been more specific and said "plot-level user interaction code, like zooming". For example, adding support for subplots means you need to add support for picking out which Axes object the user has drawn the selection over and rejecting the selection if it overlaps multiple Axes.

Ken

···

On 12/02/05 14:11, Matt Newville wrote:

Ken,

That being said, it would be frustrating if you had to periodically rewrite
the user interaction code to add support new things, like subplots and
overlapping Axes. Providing this kind of support initially was an important
goal while writing WxMpl.

Oh the 'user-level support code' is definitely the larger part of the
work for adding any plot type.

I should have been more specific and said "plot-level user interaction code,
like zooming". For example, adding support for subplots means you need to add
support for picking out which Axes object the user has drawn the selection
over and rejecting the selection if it overlaps multiple Axes.

I think that's just the tip of the iceberg. If you have multiple
'Axes' -- I'd call them multiple graphs --- in an application, the
application writer may very well want to let the user decide which
plot goes to which graph. How would this be handled? Well, at the
wxMPL/MPlot layer --- which means it does not have to be at the mpl
layer.

If multiple graphs were handled as each one being an instance of a
Plotting Panel, that would probably seem quite natural to a wxPython
author. They could be layed out anyway you'd like (they wouldn't
have to be contiguous or even on the same frame) and the wxPython
author could handle wehre 'the blue trace in graph 1' was the same as
'the green trace in graph 2' or scales of two graphs were somehow
related.

When you take the view of a Toolkit-aware programmer trying to add
graphics to their application, the details of the mpl implementation
seem less important and a few of its more whizbang features seem
pointless,

For multi-valued maps as with x-ray fluorescence (that is, a map that
is L x N x M for L
elements and N x M spatial pixels with L ~= 10, N ~= M ~= 100) which
I look at all the time, I very much want a LxL grid (or pick a subset
of the L elements and have an L' x L' grid) of graphs that W.S.
Cleveland (in 'The Elements of Graphing Data') called "ScatterPlots"
(that term has a different connotation now): At each cell along the
diagonals of the grid is a NxM false color map for an element's
intensity, the off-diagonal cells have the correlation plots for each
i,j pair of elements. The user can highlight any patch on any
sub-graph and the data points falling in that region are highlighted
on all the other graphs. So one can pick out the all the points with
highest As levels, and see immediately what the Fe and Zn levels are
at those points.

I humbly submit that this would be best handled at the GUI level.

--Matt

Matt Newville wrote:

the users of
the library should probably not have to know anything about
matplotlib, because they're going to be asking "I'm writing this wx
Application and want to include a graph -- what should I use?".

Huh? to put a graph in your app, you'd need to know the API to create and manipulate that graph, That API should be MPL's API, why write and document another one?

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer
                                         
NOAA/OR&R/HAZMAT (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...

<snip>

At each cell along the diagonals of the grid is a NxM false color map for
an element's intensity, the off-diagonal cells have the correlation plots
for each i,j pair of elements.

How are you planning on implementing the correlation plots?

<snip>

I humbly submit that this would be best handled at the GUI level.

I disagree, and am willing to put my money where my mouth is! :slight_smile:

Ken

P.S. Watch out: to unzoom, you have to right-click on the same axes you zoomed in on. This is a bug.

scatter.py (2.03 KB)

···

On 12/02/05 15:20, Matt Newville wrote:

Hi Ken,

That's nice, and coordinated zooming of separate maps is certainly a
good feature,
but that's not what I had in mind. Sorry I was imprecise and probably hurried.

The case is to help visualize L maps of 'element' values that share
the same pixel grid (NxM). You make a LxL (in your case 3x3) grid and
in each diagonal grid i = 1,...L, you draw the map for the ith
element. On the off-diagonal cells you draw a 2d plot of pixel
intensities for a pair of element: value in element j v. value in
element i for all NxM pixels. If elemetns i and j are highly
correlated, this plot is a straight line, if they are uncorrelated
it's a blob. There may be multiple trends that show (several points
with high value for i and lower value for j, say). I think each of
the correlation plots is what mpl calls a scatterplot with parameter
's' a scalar.

The correlation plot shows trends between element at the expense of
spatial information. The maps of individual element intensities shows
spatial information at the expense of showing trends between elements.
But if you could highlight or mask out selected pixels on a map or
regions on a correlation plot (with a rubberband box) and have those
those pixels light up or change color on other the maps and 2-element
correlation plots, then you can better understand the spatial and
correlation data. Does that make sense?

I say I believe this would be better handled by GUI code than directly
in mpl because someone has to keep track of the pixels for the
correlation plot data, then change some set of plot attribute (color,
symbol, etc) for those pixels on _all_ the correlation plots and
false-color maps. That is, the series of plots has to be managed
and altered as a unit in a way that is depends on the data, not by the
plot characteristics alone. And presumably the GUI would allow you
to change the color used to 'highlight' the selected points. I'm sure
it _could_ be done other ways, and perhaps I'm missing something.
Anyway, what's wrong with having multiple wx.Panel each of which has a
Figure.Axes on it?

That's one example.... What about linking together plots in different
frames or having draggable panels so that a user can orgranize them on
the desktop as they choose?
This doesn't have a lot to do with the 'should I use MPlot or wxMPL?'
question, and all I'm saying is that as plots get more complicated and
have more complex user interaction, some interactions will most
naturally be handled at the GUI level.

Sorry to stir up flames and run.....

--Matt

PS: Chris Barker asked:

Matt Newville wrote:
> the users of
> the library should probably not have to know anything about
> matplotlib, because they're going to be asking "I'm writing this wx
> Application and want to include a graph -- what should I use?".

Huh? to put a graph in your app, you'd need to know the API to create
and manipulate that graph, That API should be MPL's API, why write and
document another one?

Well MPlot and wxMPL both did just that (create other APIs). wxMPL is
very close (derived from) MPL but it's not identical. I do think the
MPL API is a little complicated if all you want to do is make a 2D
line plot. With MPlot, it's:
        plotter = MPlot.PlotPanel(...)
        plotter.plot(x2,y2)
        plotter.oplot(x2,y2)
and MPlot somehow neglects to make the user turn on or select how
zooming works, turn on or place the legend (that's a
user-configuration) or reveal that a PlotPanel has a FigureCanvas has
a Figure which has an Axes which plot()s. Yes, it is less general
than MPL, but it's simpler to use and includes functionality that MPL
does not have. It's simply a different thing: it's not MPL, but it
does use MPL to achieve its ends.

So a wxPython plotting widget based on MPL doesn't need to expose the
MPL API. If you want the MPL API, it already exists. I used this
for MPlot (nothing else existed) -- it works fine. One of the
questions here is whether it might be worth it to use the additional
layer of wxMPL in MPlot. I could be convinced, and would not mind if
someone else did this, but I'm not sure how big the gain would be.