Mathtext questions, continued...

Hi all, Please John, take some time before SciPy conf to

    > answer at least some of this questions, because the SoC
    > deadline (21st August) is *very* near.

Alas, I am already here and have been a little out of email contact
while traveling. Sorry for the delay.

    > 1) I'm having some problems regarding FT2Font. The problem
    > is when I instantiate FT2Font like: font = FT2Font(filename)
    > and when I call it's method font.set_text("Some text"), and
    > afterwards, font.draw_glyphs_to_bitmap(), the latter simply
    > deletes every glyph that was drawn before it, and just
    > paints in the internal image buffer the text that was passed
    > on last invocation of set_text (or load_char).

This is a feature, not a bug :slight_smile: You can clear the image buffer if
you want with the clear method, as we do in the mathtexy code

        for fontface in self.fonts.values():
            fontface.clear()

    > This is a pain, because draw_glyphs_to_bitmap implements the
    > layout (with kerning etc.), but if one wants to paint
    > several words in different x,y positions in the same image
    > buffer, he has to do the layout for every character in every
    > word manually via draw_glyph_to_bitmap(x, y, glyph) (like
    > you did with the BaKoMa fonts in mathtext).

    > Why hasn't draw_glyphs_to_bitmap been implemented so that it
    > takes x, y as arguments (draw_glyphs_to_bitmap(x, y)) and
    > leaves the image buffer intact (as does
    > draw_glyph_to_bitmap)?

You can add optional x and y args to draw_glyphs_to_bitmap if you need
them. Just make sure any changes you make pass
examples/backend_driver.py and unit/memleak_hawaii3.py

    > 2) As I have said before, I have started the complete
    > rewrite of mathtext (the parsing stuff etc.). I have
    > completely removed the dependency on pyparsing (please don't
    > yell at me :), and I was wondering about how much of TeX

OK, I won't yell. Quietly scold maybe :slight_smile:

I am skeptical of your -- or anyone's except Robert's -- ability to
parse TeX mathematical expressions w/o a true recursive descent
parser. I took a look at your code but w/o any running examples I
could not test it. From reading it, I do not think it will be able to
handle the deeply recursive structures that are currently supported by
the existing, working parser. Can you handle recursively nested
super/sub scripts? Can it parse this

  tex = r'\\cal\{R\}\\prod\_\{i=\\alpha\_\{i\+1\}\}^\\infty a\_i\\rm\{sin\}\(2 \\pi f x\_i\)'

If so, I apologize for my skepticism, but my working assumption is you
will need a proper parser to parse tex and string methods aren't going
to get you there. What I really need to see before even considering a
system that replaces the working system is an argument about why it
needs to be replaced, and more importantly, code that can do
everything the old code can do, such as render the mathtext_demo.py
example?

    > should mathtext support. I'm not talking about support for
    > \frac, \above, \choose (which I plan to add one by one)
    > etc., but about more general things - macros (\def etc.). I

\above, \frac and \sqrt yes, \def no. Others I'm not sure about.

    > was thinking of just simulating them, at least to a
    > tolerable extent, via notion of an enviroment. Example: \rm
    > in plain TeX sets the current font to roman (until the end
    > of the current scope - 'till it hits "}"). Implementation:
    > At render time, when the parser hits "\rm", it does the
    > folowing: env["facetype"] = "rm", where env is the
    > environment in the current scope.

Can we use classes here rather than dictionaries? Syntactically, I
prefer env.facetype = "rm" to the dictionary syntax.

    > Also, I am planing to create a separate class for every new
    > layout item that gets implemented. Example:
    > sub/superscripted item (nucleus_sub^sup) gets translated to
    > an instance of class Scripted that has the attributes
    > nucleus, superscript and subscript.

    > 3) I was thinking of focusing on just the Agg backend for
    > now (that is till' the deadline). Is this OK? 4) I think
    > that we should move the job of math_parse_s_ft2font,
    > math_parse_s_ft2font_svg, and math_parse_s_ps etc. to the
    > corresponding backends, and that some general function like:
    > math_parse_s(x, y, s, prop, angle) should be implemented
    > directly in mathtext.py (perhaps even without the "angle"
    > parameter) so that it returns a list of the following type:
    > [(x1, y1, s1, prop1, angle1), ... , (xn, yn, sn, propn,
    > anglen)]

I can't address these questions until I understand why you are trying
to rewrite, rather than extend or fix, the existing code. The agg and
the postscript backends work with the existing frameowork. As far as
I can see, you could fix the layout engine in the existing framework
and not have to think too much about backends, with the exception that
you need a basic backend line drawing API for things like the
horizontal line in \frac.

    > 5) What would be the consequences of distributing a GPL font
    > (FreeFont) with matplotlib. I mean, it's not that using a
    > GPL font in some non-GPL app could be a breach of the
    > GPL. Is there any interest in this?

Don't underestimate the zealousness of GPL advocates. I have been
informed by the FSF that merely providing a backend that does

import qt

in a backend that is optionally included at runtime by a user who may
be using a GPL version of QT or may be using a commercially licensed
version of qt (we can't know which) makes the entire code base of mpl
GPL'd. This claim was provided w/o argument and I frankly find it
ludicrous. Robert can probably tell you the IANAL response to
including GPL'd fonts but I'm not too interested in distributing them.
We can always point the user to a web site and they can download them
from.

    > The new mathtext.py is attached. Please do not try it at
    > home :wink: because nothing visible yet works.

OK.

JDH

Here are the reasons for rewriting mathtext2 that I can come up with:

* One of the reasons I started the complete rewrite of the parser is that
I'm a newbie at parsing, and I wanted to try it out a bit. I didn't understand
why was it so difficult to parse TeX (or anything a little bit complicated
for that matter). Well, now I know :wink:

* The other reason was that I didn't understand fully the current parsing
code, or more precisely, the part when what's interpreted get's rendered.
And I'm not talking about glyphs, but about the complex constructs (scripts,
frac's etc.)

* The third reason was that I can now try out in pararel the new and the old
code. Also, because I'm not touching the PS and SVG backends for now we can
have the following code in the current mathtext:

if rcParams[some_parameter]:
   from matplotlib.mathtext2 import math_parse_s_ft2font
else:
   math_parse_s_ft2font = math_parse_s_ft2font_common('BMP')
math_parse_s_ft2font_svg = math_parse_s_ft2font_common('SVG')
math_parse_s_ps = math_parse_s_ft2font_common('PS')
math_parse_s_pdf = math_parse_s_ft2font_common('PDF')

Also, I thought that the author of the current code base did some design
mistakes at the begining. And, being a developer newbie, it's a lot easier
to start things from scratch, than make fixes to old stuff you don't
understand well.

As for the mathtext_demo.py part, even real TeX can't handle it :). The point
is that, i.e. \cal sets the current fontface to "cal", and the change is
propagated till the end of the current scope (or untill it hits \rm, for
example). Old mathtext applies it only to the first item after the command.

I promise that I'll post more about the implementation in a day or two.
I'll also post screenshots (TeX/mathtext/mathtext2 shootout).

Cheers,
Edin

···

On 8/18/06, John Hunter <jdhunter@...5...> wrote:

    > Hi all, Please John, take some time before SciPy conf to
    > answer at least some of this questions, because the SoC
    > deadline (21st August) is *very* near.

Alas, I am already here and have been a little out of email contact
while traveling. Sorry for the delay.

    > 1) I'm having some problems regarding FT2Font. The problem
    > is when I instantiate FT2Font like: font = FT2Font(filename)
    > and when I call it's method font.set_text("Some text"), and
    > afterwards, font.draw_glyphs_to_bitmap(), the latter simply
    > deletes every glyph that was drawn before it, and just
    > paints in the internal image buffer the text that was passed
    > on last invocation of set_text (or load_char).

This is a feature, not a bug :slight_smile: You can clear the image buffer if
you want with the clear method, as we do in the mathtexy code

        for fontface in self.fonts.values():
            fontface.clear()

    > This is a pain, because draw_glyphs_to_bitmap implements the
    > layout (with kerning etc.), but if one wants to paint
    > several words in different x,y positions in the same image
    > buffer, he has to do the layout for every character in every
    > word manually via draw_glyph_to_bitmap(x, y, glyph) (like
    > you did with the BaKoMa fonts in mathtext).

    > Why hasn't draw_glyphs_to_bitmap been implemented so that it
    > takes x, y as arguments (draw_glyphs_to_bitmap(x, y)) and
    > leaves the image buffer intact (as does
    > draw_glyph_to_bitmap)?

You can add optional x and y args to draw_glyphs_to_bitmap if you need
them. Just make sure any changes you make pass
examples/backend_driver.py and unit/memleak_hawaii3.py

    > 2) As I have said before, I have started the complete
    > rewrite of mathtext (the parsing stuff etc.). I have
    > completely removed the dependency on pyparsing (please don't
    > yell at me :), and I was wondering about how much of TeX

OK, I won't yell. Quietly scold maybe :slight_smile:

I am skeptical of your -- or anyone's except Robert's -- ability to
parse TeX mathematical expressions w/o a true recursive descent
parser. I took a look at your code but w/o any running examples I
could not test it. From reading it, I do not think it will be able to
handle the deeply recursive structures that are currently supported by
the existing, working parser. Can you handle recursively nested
super/sub scripts? Can it parse this

  tex = r'\\cal\{R\}\\prod\_\{i=\\alpha\_\{i\+1\}\}^\\infty a\_i\\rm\{sin\}\(2 \\pi f x\_i\)'

If so, I apologize for my skepticism, but my working assumption is you
will need a proper parser to parse tex and string methods aren't going
to get you there. What I really need to see before even considering a
system that replaces the working system is an argument about why it
needs to be replaced, and more importantly, code that can do
everything the old code can do, such as render the mathtext_demo.py
example?

    > should mathtext support. I'm not talking about support for
    > \frac, \above, \choose (which I plan to add one by one)
    > etc., but about more general things - macros (\def etc.). I

\above, \frac and \sqrt yes, \def no. Others I'm not sure about.

    > was thinking of just simulating them, at least to a
    > tolerable extent, via notion of an enviroment. Example: \rm
    > in plain TeX sets the current font to roman (until the end
    > of the current scope - 'till it hits "}"). Implementation:
    > At render time, when the parser hits "\rm", it does the
    > folowing: env["facetype"] = "rm", where env is the
    > environment in the current scope.

Can we use classes here rather than dictionaries? Syntactically, I
prefer env.facetype = "rm" to the dictionary syntax.

    > Also, I am planing to create a separate class for every new
    > layout item that gets implemented. Example:
    > sub/superscripted item (nucleus_sub^sup) gets translated to
    > an instance of class Scripted that has the attributes
    > nucleus, superscript and subscript.

    > 3) I was thinking of focusing on just the Agg backend for
    > now (that is till' the deadline). Is this OK? 4) I think
    > that we should move the job of math_parse_s_ft2font,
    > math_parse_s_ft2font_svg, and math_parse_s_ps etc. to the
    > corresponding backends, and that some general function like:
    > math_parse_s(x, y, s, prop, angle) should be implemented
    > directly in mathtext.py (perhaps even without the "angle"
    > parameter) so that it returns a list of the following type:
    > [(x1, y1, s1, prop1, angle1), ... , (xn, yn, sn, propn,
    > anglen)]

I can't address these questions until I understand why you are trying
to rewrite, rather than extend or fix, the existing code. The agg and
the postscript backends work with the existing frameowork. As far as
I can see, you could fix the layout engine in the existing framework
and not have to think too much about backends, with the exception that
you need a basic backend line drawing API for things like the
horizontal line in \frac.

    > 5) What would be the consequences of distributing a GPL font
    > (FreeFont) with matplotlib. I mean, it's not that using a
    > GPL font in some non-GPL app could be a breach of the
    > GPL. Is there any interest in this?

Don't underestimate the zealousness of GPL advocates. I have been
informed by the FSF that merely providing a backend that does

import qt

in a backend that is optionally included at runtime by a user who may
be using a GPL version of QT or may be using a commercially licensed
version of qt (we can't know which) makes the entire code base of mpl
GPL'd. This claim was provided w/o argument and I frankly find it
ludicrous. Robert can probably tell you the IANAL response to
including GPL'd fonts but I'm not too interested in distributing them.
We can always point the user to a web site and they can download them
from.

    > The new mathtext.py is attached. Please do not try it at
    > home :wink: because nothing visible yet works.

OK.

JDH

Here are the reasons for rewriting mathtext2 that I can come up with:

* One of the reasons I started the complete rewrite of the parser is that
I'm a newbie at parsing, and I wanted to try it out a bit. I didn't
understand why was it so difficult to parse TeX (or anything a little bit
complicated for that matter). Well, now I know :wink:

* The other reason was that I didn't understand fully the current parsing
code, or more precisely, the part when what's interpreted get's rendered.
And I'm not talking about glyphs, but about the complex constructs
(scripts, frac's etc.)

* The third reason was that I can now try out in pararel the new and the
old code. Also, because I'm not touching the PS and SVG backends for now we
can have the following code in the current mathtext:

if rcParams[some_parameter]:
   from matplotlib.mathtext2 import math_parse_s_ft2font
else:
   math_parse_s_ft2font = math_parse_s_ft2font_common('BMP')
math_parse_s_ft2font_svg = math_parse_s_ft2font_common('SVG')
math_parse_s_ps = math_parse_s_ft2font_common('PS')
math_parse_s_pdf = math_parse_s_ft2font_common('PDF')

Also, I thought that the author of the current code base did some design
mistakes at the begining. And, being a developer newbie, it's a lot easier
to start things from scratch, than make fixes to old stuff you don't
understand well.

Just a general comment. Eric Raymond observed in The Cathedral and the Bazaar
that "Good programmers know what to write. Great ones know what to rewrite
(and reuse)." In the future, I suggest you consider that it may be in your
own long term interest to try to understand and extend existing code instead
of rewriting it from scratch. You'll learn quickly by being exposed to other
people's ideas and techniques, and you will get faster results since you
won't be reinventing the wheel.

As for the mathtext_demo.py part, even real TeX can't handle it :).

If TeX isn't yielding the result you expect, then you were expecting the wrong
result.

The point is that, i.e. \cal sets the current fontface to "cal", and the
change is propagated till the end of the current scope (or untill it hits
\rm, for example). Old mathtext applies it only to the first item after the
command.

What does this have to do with real TeX? Maybe you could post an example. It
is possibly just an mpl bug that needs to be addressed.

Darren

···

On Sunday 20 August 2006 10:25 am, Edin Salković wrote: