Mouse input occasionally apparently loses mouse events. The effect is a sometimes "sticky" quality to the mouse. I believe this is due to incorrect handling of the mouse input queue in the main loop.
Getting a mouse input queue right is a bit tricky in the presence of latency since you can't be looking at the mouse state at every nanosecond, and therefore inevitably miss some events. However, it should be possible to guarantee the following absolutes with a correct implementation, regardless of factors such as cpu load associated with the handlers:
1. If the mouse is currently lingering at some location, the most recent move event should be at this final position.
2. If the mouse button is currently lingering in an up state, there must be a mouse up that is more recent than mouse down.
3. If the mouse button is currently lingering in a down state, there must be a mouse down that is more recent than mouse up.
4. The mouse button state for any mouse event should be consistent with the most recent mouse up/down event.
Currently, none of the first three are guaranteed, but I'm not certain about the fourth. I think this should be considered a high priority defect because it impacts the feel of all matplotlib applications that use the mouse (i.e. although it's not a show stopper for most apps, it is important because it affects a very large number of apps).
How to reproduce:
Make an application with a dragable object, and add some heavy duty computation in the mouse handlers to create extra latency.
Item 1 can be demonstrated by moving the mouse rapidly back and forth and then stopping. Occasionally the object will not be where the mouse settles. Sometimes it appears that the mouse events are queued up in the wrong order (i.e. the object jumps back to a previous mouse position).
Items 2 and 3 are very intermittent, but can be achieved by lots of jerky motion while clicking. Sometimes the object will "stick" to the mouse (i.e. the final mouse up was lost).
Note that the issue is not simply the jumpy quality, as that is obviously to be expected when the handler is slow. Rather the issue is that the mouse state does not always "settle" into the correct final state after motion. Be sure that you understand this point clearly before responding.
Fixing this would result is a much smoother mouse feel. 
Does the Matplotlib project have a public bug tracking system somewhere? I can't seem to find it.