Hi, folks! I’m again encountering the problem - imshow generating a MemoryError exception trying to image a very large array - discussed in this thread I started almost a year and a half ago.
Question 1) has anything changed in MPL in that time interval which would provide an “easy” solution?
Question 2) is there some way I can add pieces of the array incrementally to the image into their proper place, i.e., modify the following code:
ax.imshow(image[0:ny/2+1, 0:nx/2+1]) # upper left corner of image
ax.hold(True)
ax.imshow(argW[ny/2+1:-1, 0:nx/2+1]) # lower left corner of image
ax.imshow(argW[0:ny/2+1, nx/2+1:-1]) # upper right corner of image
ax.imshow(argW[ny/2+1:-1, nx/2+1:-1]) # lower right corner of image
so that each subsequent imshow doesn’t cover up the previous imshow and is placed in the right place relative to each of the other pieces?
Question 3) Would such incremental addition work to get around the memory limit, or does the fact (if the following statement is in fact correct) that eventually the entire too-large image needs to be handled doom this strategy to failure?
Question 4) would I have this problem if I was running 64 bit (i.e., OS, as well as 64 bit builds of Python, numpy, MPL, etc.), i.e., is it most likely a memory addressing problem?
Question 5) can anyone suggest any other work-around(s)?
Thanks!
DG
···
On Sat, Sep 6, 2008 at 4:00 PM, David Goldsmith <d_l_goldsmith@…9…> wrote:
Ah, Ich verstehe now. I’ll try RGBA-ing it; in the meantime, let me know if the colormapping conversion gets changed to 32 bit. Thanks again!
DG
— On Sat, 9/6/08, Eric Firing <efiring@…202…> wrote:
From: Eric Firing <efiring@…202…>
Subject: Re: [Matplotlib-users] imshow size limitations?
To: d_l_goldsmith@…9…
Date: Saturday, September 6, 2008, 3:13 PM
David Goldsmith wrote:
Thanks, Eric!
— On Sat, 9/6/08, Eric Firing > > > <efiring@…202…> wrote:
– snip OP –
It looks to me like you simply ran out of
memory–this is
not an imshow
problem as such. Your array is about 1e8
elements, and as
floats that
would be close to a GB–just for that array alone.
Do you
Well, I anticipated that, so I do initialize the
storage for the numpy array as numpy.uint8 and have
confirmed that the data in the array returned by the
function which creates it remains numpy.uint8, so it should
“only” be ~100MB (indeed, the .na file into which
I tofile it is 85,430 KB, just as it should be for a 10800 x
8100 array of uint8 elements). And the ax.imshow statement
doesn’t (directly) cause the crash (but I don’t know
that it isn’t making either a float copy or an in-place
conversion of the array). So, AFAIK, right up until the
statement:
canvas.print_figure(‘HiResHex’)
the data being imaged are all numpy.uint8 type.
Yes, but it looks to me like they are still getting
color-mapped, and
this requires conversion to numpy.float. This may be a bad
aspect of
the mpl design, but it is quite deeply embedded. I suspect
the best we
could do would be to use float32 instead of float64;
certainly for color
mapping one does not need 64 bits.
Using numpy.uint8 helps only if you are specifying RGBA
directly,
bypassing the colormapping.
really need
all that resolution?
Well, there’s the rub: I fancy myself a fractal
“artist” and I have
access to an HP DesignJet 500ps plotter with a maximum
resolution of
1200 x 600 dpi. For the size images I’m trying to
make (I’m hoping to go
even bigger than 36" x 27", but I figured
that as a good starting point)
even I regard that resolution as too much - I was
thinking of 300 x
300 dpi (which is its “normal” resolution)
as certainly worthy of giving
a try.
If you do, you will probably have to
get a much
more capable machine.
Possible, but I was hoping to generate at least one
“proof” first to determine how hard I’d need
to try.
Otherwise, you need to knock down
the size of
that array before trying to plot or otherwise
manipulate
it.
Forgive me, but I’d like a more detailed
explanation as to why: I
have
ample (~35 GB, just on my built-in disc, much more
than that on external
discs) harddisc space - isn’t there some way to
leverage that?
I don’t know enough about virtual memory
implementations–especially on
Win or Mac–to say. In practice, I suspect you would find
that as soon
as you are doing major swapping during a calculation, you
will thrash
the disk until you run out of patience.
With respect to imshow, probably you can get it to
handle
larger images
Again, imshow doesn’t appear to be the culprit
(contrary to my
original subject line), rather it would appear to be
canvas.print_figure. (While I’m on the subject of
canvas.print_figure,
isn’t there some way for MPL to “splash”
the image directly to the
screen, without first having to write to a file? I
didn’t ask this
before because I did eventually want to write the
image to a file, but I
would prefer to do so only after I’ve had a look
at it.)
It is imshow in the sense that most of the action in mpl
doesn’t happen
when you call imshow or plot or whatever–they just set
things up. The
real work is done in the backend when you display with
show() or write
to a file.
if you feed them in as NxMx4 numpy.uint8 RGBA
arrays–but I
doubt this
is going to be enough, or the right approach, for
your
present situation.
Right: I don’t see how that would be better than
having a single 8
bit
datum at each point w/ color being determined from a
color map (which is
how I’d prefer to do it anyway).
The way it is better is that it avoids a major operation,
including the
generation of the double-precision array. The rgba array
can go
straight to agg.
Eric
Thanks again,
DG
Eric
Platform Details: MPL 0.91.2 (sorry, I
didn’t
realize I was running such an old version, maybe I
just need
to upgrade?), Python 2.5.2, Windows XP 2002 SP3,
504MB
physical RAM, 1294MB VM Page size (1000MB init.,
5000MB max)
Thanks!
DG
