Mathtext improvements (merging into trunk)

The new mathtext with an underlying TeX-like box model is passing all unit tests for all backends. I'm now at a point where I'd like to merge this back into the trunk, but I'd like some feedback as to how to best deal with the backward-incompatible change wrt font changes.

(This was discussed in an earlier thread, but I don't believe a conclusion was reached, so I'm summarizing again here.) In the existing mathtext implementation, font style changes affect the following group. For example, in $\cal{R} x$, only the R will be in calligraphic face. In TeX, however, font style markers stay in effect until the next font style marker or the end of the enclosing group. Therefore, the same example would be rendered with both the R and the x in calligraphic face. The new mathtext adds LaTeX-style font macros (\mathcal, \mathrm etc.) that behave as the old mathtext ones. So the example expression could be re-written $\mathcal{R} x$. It's difficult to trap the old case and correct for it, since both are still valid.

Becoming more TeX-like is a good thing, but does breaking existing matplotlib plots warrant some additional care here? Should this be merged into trunk, or onto some other branch where we're going to be breaking a lot of things. It is probably feasible to have a "compatibility" mode for this, but is that (in this particular case) worth the extra maintenance burden?

Cheers,
Mike

The new mathtext with an underlying TeX-like box model is passing all
unit tests for all backends. I'm now at a point where I'd like to merge
this back into the trunk, but I'd like some feedback as to how to best
deal with the backward-incompatible change wrt font changes.

This is really, really cool stuff. On a mostly unrelated note, do
you thing the TeX box layout algorithms could be used to solve more
general figure layout problems, eg positioning axes and tick labels,
etc?

(This was discussed in an earlier thread, but I don't believe a
conclusion was reached, so I'm summarizing again here.) In the existing
mathtext implementation, font style changes affect the following group.
For example, in $\cal{R} x$, only the R will be in calligraphic face.
In TeX, however, font style markers stay in effect until the next font
style marker or the end of the enclosing group. Therefore, the same
example would be rendered with both the R and the x in calligraphic
face. The new mathtext adds LaTeX-style font macros (\mathcal, \mathrm
etc.) that behave as the old mathtext ones. So the example expression
could be re-written $\mathcal{R} x$. It's difficult to trap the old
case and correct for it, since both are still valid.

Becoming more TeX-like is a good thing, but does breaking existing
matplotlib plots warrant some additional care here? Should this be
merged into trunk, or onto some other branch where we're going to be
breaking a lot of things. It is probably feasible to have a
"compatibility" mode for this, but is that (in this particular case)
worth the extra maintenance burden?

I think we should strive for TeX correctness and should not support
the incorrect usage in a
compatibility mode. The improvements you've made are sufficiently
important that mathtext users will gladly accept a little breakage.
We can put up a news flash on the main site, and note the changes in
the CHANGELOG and API_CHANGES file, as well as in the mathtext_demo.

I have a few questions about the mathtext_demo.py. A number of these
are probably just related to the bakoma fonts, which are just so
horrific it pains me to look at them (like why is the s in sin so much
lighter?)

* The i in the {i+1} subscript looks too small. It also looks to
small compared to the j in the \Delta_i^j example. Are the i and j
coming from the same file, and do you have any idea why it is small?

* There is a lot of space between the \prod symbol and the rest of
the expression and between the \mathcal{R} and the \prod symbol --
what controls this? It looks like it is being affected by the wide
\prod subscript {i=\alpha_{i+1}} -- is this correct and is this how
TeX handles it? Since the subscript is under the prod, it seems like
it's width should not affect the spacing of the R and the sin

* Have you given any thought to the "cross font" kerning problem.
Eg, if the parens comes from one file, and the 2 from another and the
\pi from a 3rd, how to we kern "( 2 \pi )?

* have you succeeded in get any non bakoma fonts to work, eg with the
UnicodeFont code?

Anyway, this is really great stuff. Thanks!
JDH

···

On 7/25/07, Michael Droettboom <mdroe@...31...> wrote:

AFAIK yes. I correct this with a lot of "\!\!". This is a hack, but it
gives good results.

Ga�l

···

On Wed, Jul 25, 2007 at 11:36:17AM -0500, John Hunter wrote:

* There is a lot of space between the \prod symbol and the rest of
the expression and between the \mathcal{R} and the \prod symbol --
what controls this? It looks like it is being affected by the wide
\prod subscript {i=\alpha_{i+1}} -- is this correct and is this how
TeX handles it?

John Hunter wrote:

The new mathtext with an underlying TeX-like box model is passing all
unit tests for all backends. I'm now at a point where I'd like to merge
this back into the trunk, but I'd like some feedback as to how to best
deal with the backward-incompatible change wrt font changes.

This is really, really cool stuff. On a mostly unrelated note, do
you thing the TeX box layout algorithms could be used to solve more
general figure layout problems, eg positioning axes and tick labels,
etc?

That seems like it would be worth some thought, but nothing springs to mind.

I think we should strive for TeX correctness and should not support
the incorrect usage in a
compatibility mode. The improvements you've made are sufficiently
important that mathtext users will gladly accept a little breakage.
We can put up a news flash on the main site, and note the changes in
the CHANGELOG and API_CHANGES file, as well as in the mathtext_demo.

Sounds good.

I have a few questions about the mathtext_demo.py. A number of these
are probably just related to the bakoma fonts, which are just so
horrific it pains me to look at them (like why is the s in sin so much
lighter?)

I think that's due to lack of hinting in the font. If you generate a .ps and zoom in enough, they do look much better.

* The i in the {i+1} subscript looks too small. It also looks to
small compared to the j in the \Delta_i^j example. Are the i and j
coming from the same file, and do you have any idea why it is small?

Hmmm. Both i's look identical here. Perhaps an optical illusion from the rotation? I agree, in general, though that there is some tweaking to font sizes and positioning that still needs to be done. Since mathtext always uses the same glyph outlines regardless of font size, unlike TeX which changes the actual shapes in response to font size, some compromises had to be made in that regard.

* There is a lot of space between the \prod symbol and the rest of
the expression and between the \mathcal{R} and the \prod symbol --
what controls this? It looks like it is being affected by the wide
\prod subscript {i=\alpha_{i+1}} -- is this correct and is this how
TeX handles it? Since the subscript is under the prod, it seems like
it's width should not affect the spacing of the R and the sin

Yes, as others have pointed out, this is normal TeX behavior. The subscript affects the width of the entire vertical list. If it didn't, it could bump into subscripts before and after. (Think of $R_i \prod_{i=\alpha_{i+1} $-- the subscripts would run together). The TeX box model only supports a strict hierarchy of boxes within boxes (excepting shifting and kerning), and doesn't ever do any checks for boxes overlapping boxes with other parents. (It's really a lot less smart than many, myself included, assume.) The hacks that Gael and Jouni suggest may help, but aren't currently supported by the mathtext parser (though they could be fairly easily). Exposing some of the low-level commands for TeX gurus might be useful in general, though I worry (somewhat abstractly) that since they will probably never behave 100% like TeX, they will just disappoint.

* Have you given any thought to the "cross font" kerning problem.
Eg, if the parens comes from one file, and the 2 from another and the
\pi from a 3rd, how to we kern "( 2 \pi )?

I haven't looked at how TeX does this yet, though I suspect the job is easier there, being that it has its own universe of math fonts that are built to work together. The current implementation does kern consecutive characters of the same font and fontsize, but that's all. I have noticed, though, that TeX kerns a lot less in math mode than one might expect. For instance, $AVA$ isn't kerned at all (presumably to keep variables looking like separate entities), though $\rm AVA$ is.

* have you succeeded in get any non bakoma fonts to work, eg with the
UnicodeFont code?

No, I haven't tried that yet. I've certainly broken the UnicodeFont classes by changing the API a little bit. Is there a font you would suggest that was known to work with the old mathtext that I should test with?

Anyway, this is really great stuff. Thanks!

It's been fun working through this. You may want to check out mathtext_examples.py in my branch for examples that exercise other new features.

Cheers,
Mike

···

On 7/25/07, Michael Droettboom <mdroe@...31...> wrote:

Maybe you haven't committed it?

johnh@...539...:examples> pwd
/home/titan/johnh/python/svn/matplotlib/mathtext_mgd/examples
johnh@...539...:examples> svn up
At revision 3613.
johnh@...539...:examples> ls mathtext*.py
mathtext2_demo.py mathtext_demo.py

JDH

···

On 7/25/07, Michael Droettboom <mdroe@...31...> wrote:

It's been fun working through this. You may want to check out
mathtext_examples.py in my branch for examples that exercise other new
features.

John Hunter wrote:

···

On 7/25/07, Michael Droettboom <mdroe@...31...> wrote:

It's been fun working through this. You may want to check out
mathtext_examples.py in my branch for examples that exercise other new
features.

Maybe you haven't committed it?

Doh! Try now.

Cheers,
Mike