LaTeX Fonts ONLY in x and ylabels and title, but not in ticklabels

I tried using usetex=True in rcParams but that makes all the text, including the tick labels use LaTeX fonts, but I only want LaTeX in the x-y/labels and in the title. How can I do this?

Thanks!

Put usetex=True as an argument to set_xlabel etc?