saving images using pure matplotlib in Sage cuts off the bottom part (and produces corrupt file?)

Hi everyone,

We're having a problem in Sage where if we specify the dpi of a figure, the bottom of the figure is cut off, but only the first time we save it. If we save the figure again, with the same arguments, the resulting image looks fine. I'm puzzled whether this is a Sage problem or a problem with interfacing with matplotlib, since it only happens when we are launch Sage (as opposed to just launching the ipython that Sage uses).

More details:

Running the following code in the Sage notebook or from the Sage command line gives a figure in which the bottom is cut off and the file is 12K instead of 13K. While firefox displays an image in which the bottom is cut off, eog (the gnome image viewer) doesn't even display the image, so I think the image might be corrupted. When I open the image with Konqueror, it seems like the cut off part of the image (about the lower quarter of the image) is completely transparent. I've attached the cut off image.

import matplotlib.pyplot as plt
import numpy
plt.figure()
plt.plot(numpy.arange(0,1.1,0.01))
plt.savefig('foo.png',dpi=72)

However, if we immediately save the figure again:

plt.savefig('foo.png',dpi=72)

the resulting image looks fine (i.e., the bottom is not cut off, the file is 13K, and eog displays it).

If we run the same code again using "sage -ipython" (which just launches Sage's ipython, without any Sage initialization), the same code:

import matplotlib.pyplot as plt
import numpy
plt.figure()
plt.plot(numpy.arange(0,1.1,0.01))
plt.savefig('foo.png',dpi=72)

works just fine (i.e., the file is correct and complete).

I'm puzzled since we are just using matplotlib commands, not any Sage commands. If we remove the "dpi=72" argument from savefig, everything works perfectly in all cases. My guess is that there is some sort of configuration option that Sage is setting when it starts up, but that is overridden or something after the first time matplotlib saves a figure. But I have no clue what sort of thing to look for. Does anyone have an idea of what could be causing this issue?

Thanks,

Jason

foo.png

···

--
Jason Grout

As one of who never used Sage, I don't think I'll be any help here.
Anyhow, can you tell us what kind of backed is used by default in the
two environment? I mean the type of the canvas that is initially
created.

It seems to be some dpi issue, but MPL supposed to handle this correctly.
I guess it also works fine if the figure is initially created with the
matching dpi as the savefig (72),?

Regards,

-JJ

···

On Thu, Oct 1, 2009 at 12:26 PM, <jason-sage@...2130...> wrote:

Hi everyone,

We're having a problem in Sage where if we specify the dpi of a figure, the
bottom of the figure is cut off, but only the first time we save it. If we
save the figure again, with the same arguments, the resulting image looks
fine. I'm puzzled whether this is a Sage problem or a problem with
interfacing with matplotlib, since it only happens when we are launch Sage
(as opposed to just launching the ipython that Sage uses).

More details:

Running the following code in the Sage notebook or from the Sage command
line gives a figure in which the bottom is cut off and the file is 12K
instead of 13K. While firefox displays an image in which the bottom is cut
off, eog (the gnome image viewer) doesn't even display the image, so I think
the image might be corrupted. When I open the image with Konqueror, it
seems like the cut off part of the image (about the lower quarter of the
image) is completely transparent. I've attached the cut off image.

import matplotlib.pyplot as plt
import numpy
plt.figure()
plt.plot(numpy.arange(0,1.1,0.01))
plt.savefig('foo.png',dpi=72)

However, if we immediately save the figure again:

plt.savefig('foo.png',dpi=72)

the resulting image looks fine (i.e., the bottom is not cut off, the file is
13K, and eog displays it).

If we run the same code again using "sage -ipython" (which just launches
Sage's ipython, without any Sage initialization), the same code:

import matplotlib.pyplot as plt
import numpy
plt.figure()
plt.plot(numpy.arange(0,1.1,0.01))
plt.savefig('foo.png',dpi=72)
works just fine (i.e., the file is correct and complete).

I'm puzzled since we are just using matplotlib commands, not any Sage
commands. If we remove the "dpi=72" argument from savefig, everything works
perfectly in all cases. My guess is that there is some sort of
configuration option that Sage is setting when it starts up, but that is
overridden or something after the first time matplotlib saves a figure. But
I have no clue what sort of thing to look for. Does anyone have an idea of
what could be causing this issue?

Thanks,

Jason

--
Jason Grout

------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users

Jae-Joon Lee wrote:

As one of who never used Sage, I don't think I'll be any help here.
Anyhow, can you tell us what kind of backed is used by default in the
two environment? I mean the type of the canvas that is initially
created.
  
Thanks for following up on this. Is there an easy way to ask matplotlib what its default canvas is (i.e., what canvas pyplot will use for savefig)? I believe it's probably using the FigureCanvasAgg canvas, but I'm not sure. Since I've reproduced the problem with pure matplotlib code, I'm not sure where the canvas is set up and initialized.

It seems to be some dpi issue, but MPL supposed to handle this correctly.
I guess it also works fine if the figure is initially created with the
matching dpi as the savefig (72),?
  
I'm not sure what you're asking here or how to test your assumption. I posted the code that gave the problem in a Sage session, but worked fine in a normal python session. I also saw the problem (cut-off bottom in firefox, which looked like a transparent bottom portion of the figure in gimp) with varying amounts when I specified other dpi values. Is there something I can insert into my script to test what you're asking?

Again, I'm really puzzled by this. We are using just pure matplotlib, and yet, there seems to be something in the Sage configuration or environment or something that is causing pure matplotlib code to generate these erroneous figures, but the exact same installation of matplotlib, when run under python, generates perfectly fine figures.

Thanks,

Jason

Jae-Joon Lee wrote:

As one of who never used Sage, I don't think I'll be any help here.
Anyhow, can you tell us what kind of backed is used by default in the
two environment? I mean the type of the canvas that is initially
created.

Thanks for following up on this. Is there an easy way to ask matplotlib
what its default canvas is (i.e., what canvas pyplot will use for savefig)?
I believe it's probably using the FigureCanvasAgg canvas, but I'm not sure.
Since I've reproduced the problem with pure matplotlib code, I'm not sure
where the canvas is set up and initialized.

In [17]: fig = plt.figure()

In [18]: print type(fig.canvas)
<class 'matplotlib.backends.backend_gtkagg.FigureCanvasGTKAgg'>

See if matplotlib session and sage session uses different backend.

It seems to be some dpi issue, but MPL supposed to handle this correctly.
I guess it also works fine if the figure is initially created with the
matching dpi as the savefig (72),?

I'm not sure what you're asking here or how to test your assumption. I
posted the code that gave the problem in a Sage session, but worked fine in
a normal python session. I also saw the problem (cut-off bottom in firefox,
which looked like a transparent bottom portion of the figure in gimp) with
varying amounts when I specified other dpi values. Is there something I can
insert into my script to test what you're asking?

plt.figure takes dpi parameter. This is the dpi of the figure instance
(not the output). Set this to same value as one you use for savefig.

Again, I'm really puzzled by this. We are using just pure matplotlib, and
yet, there seems to be something in the Sage configuration or environment or
something that is causing pure matplotlib code to generate these erroneous
figures, but the exact same installation of matplotlib, when run under
python, generates perfectly fine figures.

I do not have any clue either. However, one possibility I can think of
is that it is a bug in certain backend whose savefig command does not
take care of changing dpi correctly. If it turns out that the sage
session and ipython session uses a same backend, then, I have no idea.
I hope some other developer who uses sage take a look at this.

Regards,

-JJ

···

On Mon, Oct 5, 2009 at 10:25 PM, <jason-sage@...2130...> wrote:

Thanks,

Jason

Jae-Joon Lee wrote:

  

Jae-Joon Lee wrote:
    

As one of who never used Sage, I don't think I'll be any help here.
Anyhow, can you tell us what kind of backed is used by default in the
two environment? I mean the type of the canvas that is initially
created.

Thanks for following up on this. Is there an easy way to ask matplotlib
what its default canvas is (i.e., what canvas pyplot will use for savefig)?
I believe it's probably using the FigureCanvasAgg canvas, but I'm not sure.
Since I've reproduced the problem with pure matplotlib code, I'm not sure
where the canvas is set up and initialized.

In [17]: fig = plt.figure()

In [18]: print type(fig.canvas)
<class 'matplotlib.backends.backend_gtkagg.FigureCanvasGTKAgg'>

See if matplotlib session and sage session uses different backend.

It seems to be some dpi issue, but MPL supposed to handle this correctly.
I guess it also works fine if the figure is initially created with the
matching dpi as the savefig (72),?

I'm not sure what you're asking here or how to test your assumption. I
posted the code that gave the problem in a Sage session, but worked fine in
a normal python session. I also saw the problem (cut-off bottom in firefox,
which looked like a transparent bottom portion of the figure in gimp) with
varying amounts when I specified other dpi values. Is there something I can
insert into my script to test what you're asking?

plt.figure takes dpi parameter. This is the dpi of the figure instance
(not the output). Set this to same value as one you use for savefig.

Again, I'm really puzzled by this. We are using just pure matplotlib, and
yet, there seems to be something in the Sage configuration or environment or
something that is causing pure matplotlib code to generate these erroneous
figures, but the exact same installation of matplotlib, when run under
python, generates perfectly fine figures.
    
I do not have any clue either. However, one possibility I can think of
is that it is a bug in certain backend whose savefig command does not
take care of changing dpi correctly. If it turns out that the sage
session and ipython session uses a same backend, then, I have no idea.
I hope some other developer who uses sage take a look at this.

I ran some more tests, this time on sage.math.washington.edu, which is a 64-bit Ubuntu system. This time, instead of errors, all I saw were zero-length files! (i.e., the file was there, but it was completely empty, i.e., zero bytes). I consistently produced a zero-length file the first time I used the dpi keyword argument to savefig. If I called savefig again, the correct figure was saved to the file. Any ideas about what in the world could be going on here? The logs of my trials are below.

Using Sage's copy of python (2.6.2), the code printed out the exact same values for fig.canvas and fig.dpi each time, but there were no erroneous figures (i.e., all of the figures were saved to images correctly the first time savefig was called and the dpi was specified in savefig).

Thanks,

Jason

<new Sage session>
sage: import matplotlib.pyplot as plt;import numpy
sage: fig=plt.figure()
sage: print fig.canvas
<matplotlib.backends.backend_agg.FigureCanvasAgg instance at 0x4ad8440>
sage: print fig.dpi
80
sage: plt.plot(numpy.arange(0,1.1,0.01))
[<matplotlib.lines.Line2D object at 0x51275d0>]
sage: plt.savefig('foo.png',dpi=72) # zero-length file generated
sage: plt.savefig('foo.png',dpi=72) # correct figure generated

<new Sage session>
sage: import matplotlib.pyplot as plt;import numpy
sage: fig=plt.figure(dpi=72)
sage: print fig.canvas
<matplotlib.backends.backend_agg.FigureCanvasAgg instance at 0x4ad94d0>
sage: print fig.dpi
72
sage: plt.plot(numpy.arange(0,1.1,0.01))
[<matplotlib.lines.Line2D object at 0x51285d0>]
sage: plt.savefig('foo.png',dpi=72) # zero-length file generated
sage: plt.savefig('foo.png',dpi=72) # correct figure generated

<new Sage Session>
sage: import matplotlib.pyplot as plt;import numpy
sage: fig=plt.figure()
sage: print fig.canvas
<matplotlib.backends.backend_agg.FigureCanvasAgg instance at 0x4ad7440>
sage: print fig.dpi
80
sage: plt.plot(numpy.arange(0,1.1,0.01))
[<matplotlib.lines.Line2D object at 0x51305d0>]
sage: plt.savefig('foo.png',dpi=80) # zero-length file generated
sage: plt.savefig('foo.png',dpi=80) # correct figure generated

<new Sage Session>
sage: import matplotlib.pyplot as plt;import numpy
sage: fig=plt.figure(dpi=72)
sage: print fig.canvas
<matplotlib.backends.backend_agg.FigureCanvasAgg instance at 0x4ad94d0>
sage: print fig.dpi
72
sage: plt.plot(numpy.arange(0,1.1,0.01))
[<matplotlib.lines.Line2D object at 0x51325d0>]
sage: plt.savefig('foo.png') # correct figure generated

<new Sage session>
sage: import matplotlib.pyplot as plt;import numpy
sage: fig=plt.figure()
sage: print fig.canvas
<matplotlib.backends.backend_agg.FigureCanvasAgg instance at 0x4ad8440>
sage: print fig.dpi
80
sage: plt.plot(numpy.arange(0,1.1,0.01))
[<matplotlib.lines.Line2D object at 0x51275d0>]
sage: plt.savefig('foo.png') # correct figure generated

···

On Mon, Oct 5, 2009 at 10:25 PM, <jason-sage@...2130...> wrote:

I think this has nothing to do with dpi, but a file flushing issue.
Try something like below with sage (note that the code does not work
with svn version of matplotlib, i'll commit the fix soon).

import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
fig=plt.figure()
plt.plot([1,2,3])
f = open("test.png", "wb")
plt.savefig(f, format="png", dpi=50)
f.flush()

I'll report some more details later.

Regards,

-JJ

Jae-Joon Lee wrote:

I think this has nothing to do with dpi, but a file flushing issue.
Try something like below with sage (note that the code does not work
with svn version of matplotlib, i'll commit the fix soon).

import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
fig=plt.figure()
plt.plot([1,2,3])
f = open("test.png", "wb")
plt.savefig(f, format="png", dpi=50)
f.flush()

I'll report some more details later.

This seems to have solved the problem on the two systems on which I've run experiments. Now it makes perfect sense why I was always seeing a corrupt file size that was a multiple of 4K (e.g., 12K instead of 13K) :slight_smile: .

So just to confirm, is this a bug in matplotlib when using the plt.savefig(filename,dpi=72) form of calling savefig?

Thanks,

Jason

I'm not sure.

In short, what savefig does is (when filename ending with "png" is
provided) is following.

def savefig(filename):
   f = open(filename,"wb")
   write_png(f)

I think there are two issue here.

1) write_png does not flush the file : this might be a bug or a
feature, I'm not sure.
2) f is somehow not flushed when savefig returns (which should happen
if file gets deleted). This only seems to happen in sage and this
seems to be the reason why savefig does not work.

As far as the second issues is concerned, I currently don't see
anything wrong in mpl's side. We may try to explicitly flush (or
close) the file after write_png, though.

I hope someone who is more knowledgeable than me take a look at this.
Meanwhile, I'll change the code to explicitly close the opened file.

Regards,

-JJ

···

On Tue, Oct 6, 2009 at 1:44 PM, <jason-sage@...2130...> wrote:

Jae-Joon Lee wrote:

I think this has nothing to do with dpi, but a file flushing issue.
Try something like below with sage (note that the code does not work
with svn version of matplotlib, i'll commit the fix soon).

import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
fig=plt.figure()
plt.plot([1,2,3])
f = open("test.png", "wb")
plt.savefig(f, format="png", dpi=50)
f.flush()

I'll report some more details later.

This seems to have solved the problem on the two systems on which I've
run experiments. Now it makes perfect sense why I was always seeing a
corrupt file size that was a multiple of 4K (e.g., 12K instead of 13K) :slight_smile: .

So just to confirm, is this a bug in matplotlib when using the
plt.savefig(filename,dpi=72) form of calling savefig?

Thanks,

Jason

------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users

I meant if the variable "f" gets dereferenced.

-JJ

···

On Tue, Oct 6, 2009 at 2:24 PM, Jae-Joon Lee <lee.j.joon@...287...> wrote:

2) f is somehow not flushed when savefig returns (which should happen
if file gets deleted).