rc font sizes do not apply to math.text font sizes?

Hi all,

I am using the following rc params for the text font size
    plt.rc('font', size=10.0)
    plt.rc('xtick', labelsize='small')
    plt.rc('ytick', labelsize='small')
    plt.rc('legend', fontsize='medium', numpoints=1)
    plt.rc('mathtext', fontset='custom', it='serif:regular')

But font settings don't seem to be applied when I use
    plt.ylabel(r'$\Delta\mu \ \left[\mu_{B}\right]$')

Everytime I need to change the fontsize when the depending on the size
of the figure. Can this be avoided or am I missing something? It would
be nice if the font settings also apply math.text.