unicode filenames and MPL.

Hi all,

I'm writing a little MPL embedded in wxPython app, and just ran into this:

Traceback (most recent call last):
...
"..../matplotlib-0.91.2-py2.5-macosx-10.3-fat.egg/matplotlib/backends/backend_agg.py",

line 397, in print_png
     self.get_renderer()._renderer.write_png(filename, self.figure.dpi.get())
TypeError: cannot return std::string from Unicode object

As soon as I saw it, I knew what happened: I'm running the unicode version of wxPython, so the filename I got back from a SaveFile Dialog is in unicode. Now, it's easy enough for me to turn that into a string for now, but as more and more file systems are implemented in unicode, it might be nice if we could use unicode file names with MPL.

Are there any plans along these lines? Or are we just going to have to wait for py3k?

-CHB

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...

I think the real reason this wasn't done is that its tricky to do at the C level in a cross-platform way. At present it uses the regular POSIX fopen in C, which isn't really "Unicode aware". See the "Unicode filenames" section of the link below for some of the complications. Linux is particularly hard to get right:

http://www.amk.ca/python/howto/unicode

However, the Python runtime does take care of many of these details in a (mostly) reasonable way. I think the easiest solution is to open the file on the Python side (rather than using POSIX fopen in C as we do now), and pass the file descriptor itself over to C. There was a bug in extracting the file descriptor from a Python file object in earlier versions of mpl, so it was removed a while ago, but I think I have it working again.

This should now be fixed on r5025 (branch) and r5026 (trunk)

Note that if you are running 0.91.2 (and not SVN, where this was subsequently broken in r4874), you can do the following as a workaround (with a performance hit from making many Python function calls):

   savefig(open(u"CrazyUnicodeFilename.png", "w"))

On a related note -- there is still an issue where fonts with Unicode paths can not be loaded. That can hopefully be addressed in a similar manner, but there are a few more code paths to fix up there.

Mike

Christopher Barker wrote:

···

Hi all,

I'm writing a little MPL embedded in wxPython app, and just ran into this:

Traceback (most recent call last):
...
"..../matplotlib-0.91.2-py2.5-macosx-10.3-fat.egg/matplotlib/backends/backend_agg.py",

line 397, in print_png
     self.get_renderer()._renderer.write_png(filename, self.figure.dpi.get())
TypeError: cannot return std::string from Unicode object

As soon as I saw it, I knew what happened: I'm running the unicode version of wxPython, so the filename I got back from a SaveFile Dialog is in unicode. Now, it's easy enough for me to turn that into a string for now, but as more and more file systems are implemented in unicode, it might be nice if we could use unicode file names with MPL.

Are there any plans along these lines? Or are we just going to have to wait for py3k?

-CHB

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Michael Droettboom wrote:

I think the real reason this wasn't done is that its tricky to do at the C level in a cross-platform way. At present it uses the regular POSIX fopen in C, which isn't really "Unicode aware".

The actual error is from trying to put the filename in a std::string, but yes, I'm sure the fopen issue is the driver. Does C++ offer anything better?

   See the "Unicode

filenames" section of the link below for some of the complications. Linux is particularly hard to get right:

http://www.amk.ca/python/howto/unicode

Thanks, that's a good one.

> you can do the following as a workaround

(with a performance hit from making many Python function calls):

  savefig(open(u"CrazyUnicodeFilename.png", "w"))

thanks, I'll give that a try. I'm confused, though, why the many Python function calls? the C++ code doesn't just grab the file pointer?

Thanks,
-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@...259...

Christopher Barker wrote:

Michael Droettboom wrote:
  

I think the real reason this wasn't done is that its tricky to do at the C level in a cross-platform way. At present it uses the regular POSIX fopen in C, which isn't really "Unicode aware".
    
The actual error is from trying to put the filename in a std::string,
  

Presumably because it doesn't decode into ASCII.

but yes, I'm sure the fopen issue is the driver. Does C++ offer anything better?
  

Not in the C++ standard library in a cross platform way, that I'm aware of. All three platforms (if you include Gtk, Qt etc.) have special Unicode-aware APIs, and each deals with things differently. The approach I took basically leverages all the trouble the Python core guys put into to make this work cross-platform.

   See the "Unicode
  

filenames" section of the link below for some of the complications. Linux is particularly hard to get right:

http://www.amk.ca/python/howto/unicode
    
Thanks, that's a good one.

> you can do the following as a workaround
  

(with a performance hit from making many Python function calls):

  savefig(open(u"CrazyUnicodeFilename.png", "w"))
    
thanks, I'll give that a try. I'm confused, though, why the many Python function calls? the C++ code doesn't just grab the file pointer?
  

No -- a while back, it was segfaulting when trying to do that (but I've since figured out why -- it was a reference counting problem). In 0.91.2, it calls the "write" method on the Python object for each block of PNG data. SVN still supports that approach (to handle arbitrary Python file-like objects), but will extract the real C file handle if it is in fact a real file.

Mike

···

--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA