Anchoring text to middle of arrow

So I’m plotting a ton of diagrams with arrows, using connection patch because I’m crossing Axes, and wondering if there’s a way to position the text inside/inline with the arrow so that I can rescale the image without having to reposition everything:
The arrow demo more or less hard codes everything.

So my best guess is I can grab the path from the patch and compute slopes and the like, but is there anything already implemented that I’m overlooking?