image module questions

I'm starting to think about adding image support and wanted to get
some input about what it should include and how it should be designed.
The ideas are still pretty nascent but here is where I was planning to
start.

Create an image extension built around agg (not backend_agg). This
would be a free standing extension not tied to any of the backends
with the responsibility of loading image data from a variety of
sources into a pixel buffer, and resizing it to a desired pixel size
(dependent on the axes window) with customizable interpolation.

Inputs: what file formats should be supported?

  * I can do PNG rather easily since I already had to interface agg
    with png for save capabilities in backend_agg.

  * As for raw pixel data, should we try to support
    grayscale/luminance, rgb and rgba with the platform dependent byte
    ordering problems, or leave it to the user to load these into a
    numeric/numarray and init the image with that? Should we follow
    PILs lead here and just provide a fromstring method with format
    strings?

  * What raw types should be supported: 8 bit luminance, 16 bit
    luminance, 8 bit rgb, 8bit rgba, 16 bit rgb or rgba?

Resizing: Generally the axes viewport and the image dimensions will
not agree. Several possible solutions - perhaps all need to be
supported:

  * a custom axes creation func that fits the image when you just want
    to view and draw onto single image (ie no multiple subplots).

  * resize to fit, resize constrained aspect ratio, plot in current
    axes and clip image outside axes viewlim

  * with resizing, what pixel interpolation schemes are critical? agg
    supports several: nearest neighbor, blinear, bicubic, spline,
    sinc.

Backends:

  I am thinking about using the same approach as in ft2font. Have the
  image backend provide the load/resize/interpolate methods and fill a
  pixel buffer of appropriate size and letting the backends do
  whatever they want with it. Agg can blend the image with the
  drawing buffer, gtk can draw from an rgba buffer. Not sure about PS
  yet. paint and wx can use their respective APIs to copy the pixel
  buffer.

Any other thoughts welcome...

JDH

*Any other thoughts welcome...*

Not really related to images but…

I’ve been thinking a bit about mapplotlib (no, that is not a typo).

Quite often I find myself with numbers for different parts of the world that I want to map, shading regions different colours according to the numbers.

In fact, one of my early experiments with python and wxPython was to produce such a beast, but I’m not terribly happy with what I produced.

matplotlib has lots of the goodies that mapplotlib would require: it has axes you can zoom and scroll, is great a drawing coloured polygons and can do legends.

The problem I’ve tended to run into with mapping projects has been getting shape files that aren’t distributed under a restrictive licence.

Anyway, is there any interest out there in a mapplotlib?

John

John Hunter writes:

I'm starting to think about adding image support and wanted to get
some input about what it should include and how it should be designed.
The ideas are still pretty nascent but here is where I was planning to
start.

Create an image extension built around agg (not backend_agg). This
would be a free standing extension not tied to any of the backends
with the responsibility of loading image data from a variety of
sources into a pixel buffer, and resizing it to a desired pixel size
(dependent on the axes window) with customizable interpolation.

I guess I'm confused by terminology. What do you intend "backend"
to mean for images. A common interface for reading different
image formats? Speaking of which...

Inputs: what file formats should be supported?

  * I can do PNG rather easily since I already had to interface agg
    with png for save capabilities in backend_agg.

I guess I would argue for what you refer to below, that the
functionality to read image formats should be decoupled, at least
initially, from the plotting (display?) package. In fact, initially
it may make sense just to use PIL for that functionality alone until
we understand better what really needs to be integrated into the
display package. (The main drawback of PIL is that it doesn't support
either Numeric or numarray, and Lundt isn't inclined to support
either unless either is part of the Python Standard Library. It
may turn out that we could add it to PIL, or extract from PIL
the image file support component for our purposes. I suspect that
that stuff is pretty stable). But initially, reading images into
arrays seems like the most flexible and straightforward thing to
do.

  * As for raw pixel data, should we try to support
    grayscale/luminance, rgb and rgba with the platform dependent byte
    ordering problems, or leave it to the user to load these into a
    numeric/numarray and init the image with that? Should we follow
    PILs lead here and just provide a fromstring method with format
    strings?

I haven't given this a great deal of thought, but again, arguing
for simplicity, that the array representations should be simple.
For example, nxm dim array implies luminance, nxmx3 implies
rgb, nxmx4 implies rgba. The I/O module always fills the arrays
in native byte order. I suppose that some thought should be given
to the default array type. One possibility is to use Float32 with
normalized values (1.0=max), but it is probably important to keep
integer values from some image formats (like png). Floats give
the greatest flexibility and independence from the display hardware,
if sometimes wasteful of memory. The second type to support would be
UInt8 (I admit I could be stupidly overlooking something).

These arrays are passed to matplotlib rendering methods
or functions and the dimensionality will tell the rendering engine
how to interpret it. The question is how much the engine needs to
know about the depth and representation of the display buffer
and how much of these details are handled by agg (or other backends)

  * What raw types should be supported: 8 bit luminance, 16 bit
    luminance, 8 bit rgb, 8bit rgba, 16 bit rgb or rgba?

Resizing: Generally the axes viewport and the image dimensions will
not agree. Several possible solutions - perhaps all need to be
supported:

  * a custom axes creation func that fits the image when you just want
    to view and draw onto single image (ie no multiple subplots).

  * resize to fit, resize constrained aspect ratio, plot in current
    axes and clip image outside axes viewlim

  * with resizing, what pixel interpolation schemes are critical? agg
    supports several: nearest neighbor, blinear, bicubic, spline,
    sinc.

Here again I would argue that the resizing functions could be separated
into a separate module until we understand better how they should
be integrated into the interface. So for now, require a user to
apply a resampling function to an image. Something like this might
be a good initial means of handling images.

im = readpng("mypicture.png") # returns a rgb array (nxmx3) unless alpha
                              # is part of png files (I'm that ignorant).
rebinned_im = bilinear(im, axisinfo...)

Then use rebinned_im for a pixel-to-pixel display in the plot canvas
(with appropriate offset and clipping). This isn't quite as convenient
as one step from file to display, but it should get us some flexible
functionality faster and doesn't restrict more integrated means of
displaying images. There are other approaches to decoupling that are
probably more object oriented.

I'll think more about this (and you can clarify more what you mean
as well if I'm confused about what you are saying).

Perry

<snipped lots of interesting discussion...>

I guess I would argue for what you refer to below, that the
functionality to read image formats should be decoupled, at least
initially, from the plotting (display?) package. In fact, initially
it may make sense just to use PIL for that functionality alone until
we understand better what really needs to be integrated into the
display package. (The main drawback of PIL is that it doesn't support
either Numeric or numarray, and Lundt isn't inclined to support
either unless either is part of the Python Standard Library. It
may turn out that we could add it to PIL, or extract from PIL
the image file support component for our purposes. I suspect that
that stuff is pretty stable). But initially, reading images into
arrays seems like the most flexible and straightforward thing to
do.

Disclaimer: I surely don't understand all of the requirements for matplotlib imaging.

I would also suggest PIL, sticking as much as possible to the pure-Python interface. I suggest this because some OSs (well, Mac OS X is what I'm thinking of) are supposed to have native image-handling routines which are fine-tuned to their hardware and unlikely to be beat by some multi-platform project. (For example, I've heard that Apple's PNG handling is faster and better than libpng.) If someone modified PIL to take advantage of these platform-specific features, we would reap the benefits. Then again, this argument assumes someone will someday do that.

  * As for raw pixel data, should we try to support
    grayscale/luminance, rgb and rgba with the platform dependent byte
    ordering problems, or leave it to the user to load these into a
    numeric/numarray and init the image with that? Should we follow
    PILs lead here and just provide a fromstring method with format
    strings?

I haven't given this a great deal of thought, but again, arguing
for simplicity, that the array representations should be simple.
For example, nxm dim array implies luminance, nxmx3 implies
rgb, nxmx4 implies rgba. The I/O module always fills the arrays
in native byte order. I suppose that some thought should be given
to the default array type. One possibility is to use Float32 with
normalized values (1.0=max), but it is probably important to keep
integer values from some image formats (like png). Floats give
the greatest flexibility and independence from the display hardware,
if sometimes wasteful of memory. The second type to support would be
UInt8 (I admit I could be stupidly overlooking something).

I think the format string idea is a good one. An nxm dim array could be luminance, alpha, or possibly colormapped -- it's not necessarily known in advance what data type it is, although a guess would probably be right 99% of the time.

I also vote use a floating-point representation whenever possible. It's clear that 8 bits per color aren't enough for many purposes.

<snipped more interesting discussion...>

PGP.sig (155 Bytes)

Hi Perry,
You may not be aware that, although not officially supported, Lundh
provided the following sample code (which I have used successfully to read
bmp files and do fft's on them) for converting PIL to/from Numeric arrays
via strings. It is quite fast. I think I probably found it in an email
archive,
Gary Ruben

-PilConvert.py-

···

#
# convert between numarrayal arrays and PIL image memories
#
# fredrik lundh, october 1998
#
# fredrik@...38...
# http://www.pythonware.com
#

import numarray
import Image

def image2array(im):
    if im.mode not in ("L", "F"):
        raise ValueError, "can only convert single-layer images"
    if im.mode == "L":
        a = numarray.fromstring(im.tostring(), numarray.UInt8)
    else:
        a = numarray.fromstring(im.tostring(), numarray.Float32)
    a.shape = im.size[1], im.size[0]
    return a

def array2image(a):
    if a.typecode() == numarray.UInt8:
        mode = "L"
    elif a.typecode() == numarray.Float32:
        mode = "F"
    else:
        raise ValueError, "unsupported image mode"
    return Image.fromstring(mode, (a.shape[1], a.shape[0]), a.tostring())

-ffts.py-

import PilConvert
import Image
from numarray import *
from numarray.fft import *

def doFft(im):
    nim = PilConvert.image2array(im)
    im_fft = abs(real_fft2d(nim))
    im_fft = log(im_fft + 1) # convert levels for display
    scale = 255. / max(im_fft.flat)
    im_fft = (im_fft * scale).astype(UInt8)
    imo = PilConvert.array2image(im_fft)
    return imo

im = Image.open('rm.bmp')
imo = doFft(im)
imo.save('rm_fourier.bmp', 'BMP')

*********** REPLY SEPARATOR ***********

On 9/03/2004 at 17:58 Perry Greenfield wrote:

John Hunter writes:

> I'm starting to think about adding image support and wanted to get
> some input about what it should include and how it should be designed.
> The ideas are still pretty nascent but here is where I was planning to
> start.
>
> Create an image extension built around agg (not backend_agg). This
> would be a free standing extension not tied to any of the backends
> with the responsibility of loading image data from a variety of
> sources into a pixel buffer, and resizing it to a desired pixel size
> (dependent on the axes window) with customizable interpolation.
>
I guess I'm confused by terminology. What do you intend "backend"
to mean for images. A common interface for reading different
image formats? Speaking of which...

> Inputs: what file formats should be supported?
>
> * I can do PNG rather easily since I already had to interface agg
> with png for save capabilities in backend_agg.
>
I guess I would argue for what you refer to below, that the
functionality to read image formats should be decoupled, at least
initially, from the plotting (display?) package. In fact, initially
it may make sense just to use PIL for that functionality alone until
we understand better what really needs to be integrated into the
display package. (The main drawback of PIL is that it doesn't support
either Numeric or numarray, and Lundt isn't inclined to support
either unless either is part of the Python Standard Library. It
may turn out that we could add it to PIL, or extract from PIL
the image file support component for our purposes. I suspect that
that stuff is pretty stable). But initially, reading images into
arrays seems like the most flexible and straightforward thing to
do.

> * As for raw pixel data, should we try to support
> grayscale/luminance, rgb and rgba with the platform dependent byte
> ordering problems, or leave it to the user to load these into a
> numeric/numarray and init the image with that? Should we follow
> PILs lead here and just provide a fromstring method with format
> strings?
>
I haven't given this a great deal of thought, but again, arguing
for simplicity, that the array representations should be simple.
For example, nxm dim array implies luminance, nxmx3 implies
rgb, nxmx4 implies rgba. The I/O module always fills the arrays
in native byte order. I suppose that some thought should be given
to the default array type. One possibility is to use Float32 with
normalized values (1.0=max), but it is probably important to keep
integer values from some image formats (like png). Floats give
the greatest flexibility and independence from the display hardware,
if sometimes wasteful of memory. The second type to support would be
UInt8 (I admit I could be stupidly overlooking something).

These arrays are passed to matplotlib rendering methods
or functions and the dimensionality will tell the rendering engine
how to interpret it. The question is how much the engine needs to
know about the depth and representation of the display buffer
and how much of these details are handled by agg (or other backends)

> * What raw types should be supported: 8 bit luminance, 16 bit
> luminance, 8 bit rgb, 8bit rgba, 16 bit rgb or rgba?
>
> Resizing: Generally the axes viewport and the image dimensions will
> not agree. Several possible solutions - perhaps all need to be
> supported:
>
> * a custom axes creation func that fits the image when you just want
> to view and draw onto single image (ie no multiple subplots).
>
> * resize to fit, resize constrained aspect ratio, plot in current
> axes and clip image outside axes viewlim
>
> * with resizing, what pixel interpolation schemes are critical? agg
> supports several: nearest neighbor, blinear, bicubic, spline,
> sinc.
>
Here again I would argue that the resizing functions could be separated
into a separate module until we understand better how they should
be integrated into the interface. So for now, require a user to
apply a resampling function to an image. Something like this might
be a good initial means of handling images.

im = readpng("mypicture.png") # returns a rgb array (nxmx3) unless alpha
                              # is part of png files (I'm that ignorant).
rebinned_im = bilinear(im, axisinfo...)

Then use rebinned_im for a pixel-to-pixel display in the plot canvas
(with appropriate offset and clipping). This isn't quite as convenient
as one step from file to display, but it should get us some flexible
functionality faster and doesn't restrict more integrated means of
displaying images. There are other approaches to decoupling that are
probably more object oriented.

I'll think more about this (and you can clarify more what you mean
as well if I'm confused about what you are saying).

Perry

-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

------------------------------------
Gary Ruben gruben@...1...
<http://users.bigpond.net.au/gazzar>

Hi Perry,
You may not be aware that, although not officially supported, Lundh
provided the following sample code (which I have used successfully to read
bmp files and do fft's on them) for converting PIL to/from Numeric arrays
via strings. It is quite fast. I think I probably found it in an email
archive,
Gary Ruben

Thanks for showing the code. I guess what I was referring to was
that native support by PIL would eliminate unnecessary memory copies
which occur when fromstring and tostring are used. But as I argue
in a previous message, I'm not currently worried that much about
that, but it seems that it would be nice if it weren't necessary
to go through that copying (from PIL image to string to array
rather than directly from PIL image to array)

[And I *meant* to check the spelling of Fredrik's name; I have a hard
time remembering the correct spelling :slight_smile: ]

Perry

John Gill writes:

Any other thoughts welcome...

Not really related to images but...

I've been thinking a bit about mapplotlib (no, that is not a typo).

Quite often I find myself with numbers for different parts of the world
that I want to map, shading regions different colours according to the
numbers.

In fact, one of my early experiments with python and wxPython was to
produce such a beast, but I'm not terribly happy with what I produced.

matplotlib has lots of the goodies that mapplotlib would require: it has
axes you can zoom and scroll, is great a drawing coloured polygons and can
do legends.

The problem I've tended to run into with mapping projects has been getting
shape files that aren't distributed under a restrictive licence.

Anyway, is there any interest out there in a mapplotlib?

John

Well, sort of though on our part the interest is more in plotting
things on a map of the sky than the Earth (though occasionally, we
need to do that also). For us the biggest issue is handling all
the possible map coordinate projections. I would assume that is
also something you would have to worry about. We've given some
thought about how to do that sort of thing (as well as do thing
like polar plots). This would be a generalization of the matplotlib
transform mechanism. It isn't a real high priority for us yet.
The image stuff that John is talking about is much higher priority.
But if you have any thoughts of expanding on matplotlib for this
and are planning to use something other than simple rectangular
coordinates, I'd be interested in understanding how you will handle
map projections.

Perry