Ok, here's something of a weird patch, because I don't know how to use subversion properly. It has changes to axes.py which update quiver so that it accepts arbitrary X,Y data; it doesn't demand the data be on a grid anymore.
The other changes are to collections.py; I updated LineCollection so it inherits from ScalarMappable. I did this just by copying what looked like the relevant code from PatchCollection and then I tested it with my LineCollection based version of quiver. I think I got it right, but I don't really know what I'm doing here so someone should check it over.
quiverpatch (6.11 KB)