interacting with graph objects

I am interested in creating graphs where I can interact with
the data plotted in the graph.

For example, suppose I plot the heat of formation of ~ 80 different
alloy structures (which each have a name) against the composition of each
structure. What I would like is to hold the mouse over a data point and see a
balloon pop up that displays the crystal structure name, or to left click on a
data point to launch a structure viewer for the structure at that point, or
right click to get a menu of different options, or to press “f” to
see what filename the data for that point is saved in, etc…

Another example would be a graph of many lines, where each
line corresponds to a structure. Then I would want to click on the lines to do
various things.

Sometimes the data is very dense, so it would also be nice
if you could move the focus to different points by tabbing or using the arrow

It would be interesting to be able to select portions of
data points to do some kind of analysis on, or to modify their appearance
someway, maybe change symbol the size according to atomic volume, or color
according to some property, etc…

Could these kinds of things be done in/with matplotlib? Or
more importantly, does a framework in matplotlib exist that this kind of thing
could be developed? I am interested in talking to anyone who has thoughts about