Segmentation fault using imshow on large image

Adam_Ginsburg · April 24, 2009, 12:27am

Hi, I've been getting a segmentation fault when trying to display
large images. A transcript of a sample session is below. I'm using
the TkAgg backend, and I am using numpy, but otherwise I have made no
modifications to the matplotlib setup.

milkyway /data/glimpseii $ alias pylab
alias pylab='/usr/local/adm/config/python/bin/ipython -pylab -log'
milkyway /data/glimpseii $ pylab
Activating auto-logging. Current session state plus future input saved.
Filename : ipython_log.py
Mode : rotate
Output logging : False
Raw input log : False
Timestamping : False
State : active
Python 2.5 (r25:51908, Dec 22 2006, 16:08:43)
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

Welcome to pylab, a matplotlib-based Python environment.
For more information, type 'help(pylab)'.

In [1]: import matplotlib,pyfits,numpy,scipy

In [2]: scipy.__version__
Out[2]: '0.7.0'

In [3]: numpy.__version__
Out[3]: '1.3.0'

In [4]: matplotlib.__version__
Out[4]: '0.98.5.2'

In [5]: f = pyfits.open('GLM_00600+0000_mosaic_I3.fits')

In [6]: f[0].data.shape
Out[6]: (10370, 9320)

In [7]: f[0].data.dtype
Out[7]: dtype('>f4')

In [8]: imshow(f[0].data)
Segmentation fault

Any ideas?

Thanks,
Adam

Michael_Droettboom1 · April 24, 2009, 1:52pm

On my machine (32-bit Fedora 10 with 2GB RAM), it chugs along swapping for a loooong time and then fails with a Python MemoryError exception -- which is at least reasonable.

I suspect you're running on a 64-bit machine and we're running into some sort of non-64-bit-clean issue. We try to be 64-bit clean, but it doesn't get verified on a regular basis, and not all of us (myself included) are running 64-bit OSes.

Can you try running python inside of gdb and getting a traceback? That might provide some clues.

We can estimate a little bit as to the memory requirements -- though it's hard to account for everything.

Input array is (10370, 9320) x 4 = 386MB
This array is always converted to doubles to convert to colors (this is probably a place ripe for opimtization) so you get also 786MB.
Then this gets converted to an RGBA array for another 386MB

Mike

Adam Ginsburg wrote:

···

Hi, I've been getting a segmentation fault when trying to display
large images. A transcript of a sample session is below. I'm using
the TkAgg backend, and I am using numpy, but otherwise I have made no
modifications to the matplotlib setup.

milkyway /data/glimpseii alias pylab alias pylab='/usr/local/adm/config/python/bin/ipython \-pylab \-log' milkyway /data/glimpseii pylab
Activating auto-logging. Current session state plus future input saved.
Filename : ipython_log.py
Mode : rotate
Output logging : False
Raw input log : False
Timestamping : False
State : active
Python 2.5 (r25:51908, Dec 22 2006, 16:08:43)
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

Welcome to pylab, a matplotlib-based Python environment.
For more information, type 'help(pylab)'.

In [1]: import matplotlib,pyfits,numpy,scipy

In [2]: scipy.__version__
Out[2]: '0.7.0'

In [3]: numpy.__version__
Out[3]: '1.3.0'

In [4]: matplotlib.__version__
Out[4]: '0.98.5.2'

In [5]: f = pyfits.open('GLM_00600+0000_mosaic_I3.fits')

In [6]: f[0].data.shape
Out[6]: (10370, 9320)

In [7]: f[0].data.dtype
Out[7]: dtype('>f4')

In [8]: imshow(f[0].data)
Segmentation fault

Any ideas?

Thanks,
Adam

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensign option that enables unlimited
royalty-free distribution of the report engine for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

Adam_Ginsburg · April 25, 2009, 2:53pm

Yep, I'm running on a 64 bit machine. I've been dealing with larger
than 4GB data files in IDL, but I'd rather use python/numpy/matplotlib
if possible.

Here's the gdb session. The error didn't happen in imshow, only when
I specified show(); I guess that means I must have had ioff() set
although I don't think that was my default choice last time I used
matplotlib.

milkyway /data/glimpseii $ gdb /usr/local/python/bin/python
GNU gdb Red Hat Linux (6.3.0.0-1.159.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
libthread_db library "/lib64/tls/libthread_db.so.1".

(gdb) run
Starting program: /usr/local/python/bin/python
[Thread debugging using libthread_db enabled]
[New Thread 182900715072 (LWP 18039)]
Python 2.5 (r25:51908, Dec 22 2006, 16:08:43)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import matplotlib,scipy,numpy,pyfits
from pylab import *
f = pyfits.open('GLM_00600+0000_mosaic_I3.fits')
imshow(f[0].data)

<matplotlib.image.AxesImage object at 0x2aa3f45890>

show()

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 182900715072 (LWP 18039)]
0x0000002aa3170ab5 in _image_module::fromarray (this=Variable "this"
is not available.
) at src/image.cpp:872
872 src/image.cpp: No such file or directory.
in src/image.cpp
Current language: auto; currently c++
(gdb)

I've never used gdb before, so is there anything else I should be
doing at this point?

Thanks,
Adam

···

On Fri, Apr 24, 2009 at 7:52 AM, Michael Droettboom <mdroe@...86...> wrote:

On my machine (32-bit Fedora 10 with 2GB RAM), it chugs along swapping for a loooong time and then fails with a Python MemoryError exception -- which is at least reasonable.

I suspect you're running on a 64-bit machine and we're running into some sort of non-64-bit-clean issue. We try to be 64-bit clean, but it doesn't get verified on a regular basis, and not all of us (myself included) are running 64-bit OSes.

Can you try running python inside of gdb and getting a traceback? That might provide some clues.

We can estimate a little bit as to the memory requirements -- though it's hard to account for everything.

Input array is (10370, 9320) x 4 = 386MB
This array is always converted to doubles to convert to colors (this is probably a place ripe for opimtization) so you get also 786MB.
Then this gets converted to an RGBA array for another 386MB

Mike

Adam Ginsburg wrote:

Hi, I've been getting a segmentation fault when trying to display
large images. A transcript of a sample session is below. I'm using
the TkAgg backend, and I am using numpy, but otherwise I have made no
modifications to the matplotlib setup.

milkyway /data/glimpseii alias pylab alias pylab='/usr/local/adm/config/python/bin/ipython \-pylab \-log' milkyway /data/glimpseii pylab
Activating auto-logging. Current session state plus future input saved.
Filename : ipython_log.py
Mode : rotate
Output logging : False
Raw input log : False
Timestamping : False
State : active
Python 2.5 (r25:51908, Dec 22 2006, 16:08:43)
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

Welcome to pylab, a matplotlib-based Python environment.
For more information, type 'help(pylab)'.

In [1]: import matplotlib,pyfits,numpy,scipy

In [2]: scipy.__version__
Out[2]: '0.7.0'

In [3]: numpy.__version__
Out[3]: '1.3.0'

In [4]: matplotlib.__version__
Out[4]: '0.98.5.2'

In [5]: f = pyfits.open('GLM_00600+0000_mosaic_I3.fits')

In [6]: f[0].data.shape
Out[6]: (10370, 9320)

In [7]: f[0].data.dtype
Out[7]: dtype('>f4')

In [8]: imshow(f[0].data)
Segmentation fault

Any ideas?

Thanks,
Adam

_John_Hunter1 · April 25, 2009, 3:34pm

Could you try and create an image using random data that is the same dimensions and datatype as the fits data you are using htat replicates the segfault, so we can try and reproduce the error as well as add it to our test suite?

Thanks

JDH

···

On Sat, Apr 25, 2009 at 9:53 AM, Adam Ginsburg <adam.ginsburg@…179…> wrote:

Yep, I’m running on a 64 bit machine. I’ve been dealing with larger

than 4GB data files in IDL, but I’d rather use python/numpy/matplotlib

if possible.

Here’s the gdb session. The error didn’t happen in imshow, only when

I specified show(); I guess that means I must have had ioff() set

although I don’t think that was my default choice last time I used

matplotlib.

milkyway /data/glimpseii $ gdb /usr/local/python/bin/python

GNU gdb Red Hat Linux (6.3.0.0-1.159.el4rh)

Copyright 2004 Free Software Foundation, Inc.

GDB is free software, covered by the GNU General Public License, and you are

welcome to change it and/or distribute copies of it under certain conditions.

Type “show copying” to see the conditions.

There is absolutely no warranty for GDB. Type “show warranty” for details.

This GDB was configured as “x86_64-redhat-linux-gnu”…Using host

libthread_db library “/lib64/tls/libthread_db.so.1”.

(gdb) run

Starting program: /usr/local/python/bin/python

[Thread debugging using libthread_db enabled]

[New Thread 182900715072 (LWP 18039)]
Python 2.5 (r25:51908, Dec 22 2006, 16:08:43)

[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2

Type “help”, “copyright”, “credits” or “license” for more information.

import matplotlib,scipy,numpy,pyfits

from pylab import *
f = pyfits.open(‘GLM_00600+0000_mosaic_I3.fits’)

imshow(f[0].data)

<matplotlib.image.AxesImage object at 0x2aa3f45890>

show()

Program received signal SIGSEGV, Segmentation fault.

[Switching to Thread 182900715072 (LWP 18039)]

0x0000002aa3170ab5 in _image_module::fromarray (this=Variable “this”

is not available.

) at src/image.cpp:872

872 src/image.cpp: No such file or directory.
    in src/image.cpp
Current language: auto; currently c++

(gdb)

_Troels_Kofoed_Jaco1 · April 25, 2009, 3:47pm

Hi

It could be that you just have to much data for the stack. You can see/set
your stack size with ulimit -s (on linux/solaris at least). Try to set it to
unlimited:
ulimit -s unlimited

This has solved similar problems for me in the past.

Best Regards
Troels Kofoed Jacobsen

···

On Saturday 25 April 2009 16:53:27 Adam Ginsburg wrote:

Yep, I'm running on a 64 bit machine. I've been dealing with larger
than 4GB data files in IDL, but I'd rather use python/numpy/matplotlib
if possible.

Here's the gdb session. The error didn't happen in imshow, only when
I specified show(); I guess that means I must have had ioff() set
although I don't think that was my default choice last time I used
matplotlib.

milkyway /data/glimpseii $ gdb /usr/local/python/bin/python
GNU gdb Red Hat Linux (6.3.0.0-1.159.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are welcome to change it and/or distribute copies of it under certain
conditions. Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
libthread_db library "/lib64/tls/libthread_db.so.1".

(gdb) run
Starting program: /usr/local/python/bin/python
[Thread debugging using libthread_db enabled]
[New Thread 182900715072 (LWP 18039)]
Python 2.5 (r25:51908, Dec 22 2006, 16:08:43)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import matplotlib,scipy,numpy,pyfits
>>> from pylab import *
>>> f = pyfits.open('GLM_00600+0000_mosaic_I3.fits')
>>> imshow(f[0].data)

<matplotlib.image.AxesImage object at 0x2aa3f45890>

>>> show()

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 182900715072 (LWP 18039)]
0x0000002aa3170ab5 in _image_module::fromarray (this=Variable "this"
is not available.
) at src/image.cpp:872
872 src/image.cpp: No such file or directory.
in src/image.cpp
Current language: auto; currently c++
(gdb)

I've never used gdb before, so is there anything else I should be
doing at this point?

Thanks,
Adam

On Fri, Apr 24, 2009 at 7:52 AM, Michael Droettboom <mdroe@...86...> wrote:
> On my machine (32-bit Fedora 10 with 2GB RAM), it chugs along swapping
> for a loooong time and then fails with a Python MemoryError exception --
> which is at least reasonable.
>
> I suspect you're running on a 64-bit machine and we're running into some
> sort of non-64-bit-clean issue. We try to be 64-bit clean, but it
> doesn't get verified on a regular basis, and not all of us (myself
> included) are running 64-bit OSes.
>
> Can you try running python inside of gdb and getting a traceback? That
> might provide some clues.
>
> We can estimate a little bit as to the memory requirements -- though it's
> hard to account for everything.
>
> Input array is (10370, 9320) x 4 = 386MB
> This array is always converted to doubles to convert to colors (this is
> probably a place ripe for opimtization) so you get also 786MB. Then this
> gets converted to an RGBA array for another 386MB
>
> Mike
>
> Adam Ginsburg wrote:
>> Hi, I've been getting a segmentation fault when trying to display
>> large images. A transcript of a sample session is below. I'm using
>> the TkAgg backend, and I am using numpy, but otherwise I have made no
>> modifications to the matplotlib setup.
>>
>>
>> milkyway /data/glimpseii alias pylab >> alias pylab='/usr/local/adm/config/python/bin/ipython \-pylab \-log' >> milkyway /data/glimpseii pylab
>> Activating auto-logging. Current session state plus future input saved.
>> Filename : ipython_log.py
>> Mode : rotate
>> Output logging : False
>> Raw input log : False
>> Timestamping : False
>> State : active
>> Python 2.5 (r25:51908, Dec 22 2006, 16:08:43)
>> Type "copyright", "credits" or "license" for more information.
>>
>> IPython 0.9.1 -- An enhanced Interactive Python.
>> ? -> Introduction and overview of IPython's features.
>> %quickref -> Quick reference.
>> help -> Python's own help system.
>> object? -> Details about 'object'. ?object also works, ?? prints more.
>>
>> Welcome to pylab, a matplotlib-based Python environment.
>> For more information, type 'help(pylab)'.
>>
>> In [1]: import matplotlib,pyfits,numpy,scipy
>>
>> In [2]: scipy.__version__
>> Out[2]: '0.7.0'
>>
>> In [3]: numpy.__version__
>> Out[3]: '1.3.0'
>>
>> In [4]: matplotlib.__version__
>> Out[4]: '0.98.5.2'
>>
>> In [5]: f = pyfits.open('GLM_00600+0000_mosaic_I3.fits')
>>
>> In [6]: f[0].data.shape
>> Out[6]: (10370, 9320)
>>
>> In [7]: f[0].data.dtype
>> Out[7]: dtype('>f4')
>>
>> In [8]: imshow(f[0].data)
>> Segmentation fault
>>
>>
>> Any ideas?
>>
>> Thanks,
>> Adam

---------------------------------------------------------------------------
--- Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensign option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

Adam_Ginsburg · April 25, 2009, 4:38pm

A 10000x10000 array reproduces the error:

milkyway /data/glimpseii $ gdb /usr/local/python/bin/python
GNU gdb Red Hat Linux (6.3.0.0-1.159.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
libthread_db library "/lib64/tls/libthread_db.so.1".

(gdb) run
Starting program: /usr/local/python/bin/python
[Thread debugging using libthread_db enabled]
[New Thread 182900715072 (LWP 19947)]
Python 2.5 (r25:51908, Dec 22 2006, 16:08:43)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from pylab import *
x=rand(10000,10000)
imshow(x)

<matplotlib.image.AxesImage object at 0x2a9f532390>

show()

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 182900715072 (LWP 19947)]
0x0000002a9eac7ab5 in _image_module::fromarray (this=Variable "this"
is not available.
) at src/image.cpp:872
872 src/image.cpp: No such file or directory.
in src/image.cpp
Current language: auto; currently c++
(gdb)

···

Could you try and create an image using random data that is the same
dimensions and datatype as the fits data you are using htat replicates the
segfault, so we can try and reproduce the error as well as add it to our
test suite?

Thanks
JDH

_Adam_Ginsburg · April 27, 2009, 4:50pm

Tried it, no luck. Thanks for the tip, though.

···

Bugzilla from tkjacobsen@...287... wrote:

Hi

It could be that you just have to much data for the stack. You can see/set
your stack size with ulimit -s (on linux/solaris at least). Try to set it
to
unlimited:
ulimit -s unlimited

This has solved similar problems for me in the past.

Best Regards
Troels Kofoed Jacobsen

--
View this message in context: http://www.nabble.com/Segmentation-fault-using-imshow-on-large-image-tp23207792p23260454.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

_Adam_Ginsburg · June 22, 2009, 5:37pm

Anyone else have ideas on how to display large images?

Thanks,
Adam

···

--
View this message in context: http://www.nabble.com/Segmentation-fault-using-imshow-on-large-image-tp23207792p24152022.html
Sent from the matplotlib - users mailing list archive at Nabble.com.