Transforms and datetime axis

Hello, I am trying to help a user with adjustText to use it together with a time series plot: python - Timeline of events -- setting annotation location - Stack Overflow

Recently adjust_text got some improvements to work with arbitrary transforms, which helps it work with, I thought, any kind of matplotlib axes, and it definitely helped with maps and projections, for example.
However it seems to be broken when working with datetime axes. Older versions, I think, would give some random chaotic results, while currently it just breaks. Unfortunately, I personally have zero experience working with this sort of data, and it’s quite confusing. Here is a simple example that produces the same error as what I get when using adjust_text:

import matplotlib.pyplot as plt
import numpy as np
import matplotlib.dates as mdates
from datetime import datetime
names = ['v2.2.4', 'v3.0.3', 'v3.0.2', 'v3.0.1', 'v3.0.0', 'v2.2.3',
        'v2.2.2', 'v2.2.1', 'v2.2.0', 'v2.1.2', 'v2.1.1', 'v2.1.0',
        'v2.0.2', 'v2.0.1', 'v2.0.0', 'v1.5.3', 'v1.5.2', 'v1.5.1',
        'v1.5.0', 'v1.4.3', 'v1.4.2', 'v1.4.1', 'v1.4.0']

dates = ['2019-02-26', '2019-02-26', '2018-11-10', '2018-11-10',
            '2018-09-18', '2018-08-10', '2018-03-17', '2018-03-16',
            '2018-03-06', '2018-01-18', '2017-12-10', '2017-10-07',
            '2017-05-10', '2017-05-02', '2017-01-17', '2016-09-09',
            '2016-07-03', '2016-01-10', '2015-10-29', '2015-02-16',
            '2014-10-26', '2014-10-18', '2014-08-26']
dates = [datetime.strptime(d, "%Y-%m-%d") for d in dates]
levels = np.tile([-5, 5, -3, 3, -1, 1],

f, ax = plt.subplots()
ax.scatter(dates, levels)
texts = []
for x, y, l in zip(dates, levels, names):
    texts.append(ax.text(x, y, l))

Which produces the plot as expected, but the last line gives an error that I don’t understand:

TypeError                                 Traceback (most recent call last)
<ipython-input-10-36886e03c6a3> in <module>
      4 for x, y, l in zip(dates, levels, names):
      5     texts.append(ax.text(x, y, l))
----> 6 texts[0].get_transform().transform(texts[0].get_position())

~/miniconda/envs/adjustText/lib/python3.7/site-packages/matplotlib/ in transform(self, values)
   1404         # Transform the values
-> 1405         res = self.transform_affine(self.transform_non_affine(values))
   1407         # Convert the result back to the shape of the input values.

~/miniconda/envs/adjustText/lib/python3.7/site-packages/matplotlib/ in transform_affine(self, points)
   2363     def transform_affine(self, points):
   2364         # docstring inherited
-> 2365         return self.get_affine().transform(points)
   2367     def transform_non_affine(self, points):

~/miniconda/envs/adjustText/lib/python3.7/site-packages/matplotlib/ in transform(self, values)
   1714     def transform(self, values):
   1715         # docstring inherited
-> 1716         return self.transform_affine(values)
   1718     def transform_affine(self, values):

~/miniconda/envs/adjustText/lib/python3.7/site-packages/matplotlib/ in transform_affine(self, points)
   1796             tpoints = affine_transform(, mtx)
   1797             return,
-> 1798         return affine_transform(points, mtx)
   1800     if DEBUG:

TypeError: Cannot cast array data from dtype('O') to dtype('float64') according to the rule 'safe'

Is this a wrong way to use datetimes when plotting? Or is there a different way to use transforms when there are datetimes in play? Thanks!