Hi Darren,
I have tried rerunning our code with the change you suggested in the make_dvi and make_png functions. I am still noticing failures however. I put these at the bottom of this message. Strangely enough, these errors don’t seem to occur when there are a lot of files in my tex.cache directory. For example, I ran the code (consisting of ~40 codes all making ~10-20 plots each), successfully 3 times (the OSError wasn’t raised at all, I used a print statement to check). I realised after this that a lot of temp files were in my tex.cache directory, so I emptied it and then I noticed that a lot of failures occured when I ran the code the next time (the OSError I showed previously was raised as well as the error messages shown below). It seems weird that it should run fine when a lot of files are left in my temp directory and not when it is empty?
Here are the error messages that are occuring now:
Traceback (most recent call last):
File “/home/spxiwh/ihope/852450000-852700000/nsbhinj_summary_plots/…/executables/plotinspmissed”, line 625, in ?
savePlot( opts, filename, titleText)
File “/home/spxiwh/ihope/852450000-852700000/nsbhinj_summary_plots/…/executables/plotinspmissed”, line 108, in savePlot
dpi_thumb=opts.figure_resolution)
File “/home/spxiwh/lscsoft/executables/cbc_s5_1yr_20070129/pylal//lib64/python2.4/site-packages/pylal/InspiralUtils.py”, line 54, in savefig_pylal
fig.savefig(filename, dpi=dpi)
File “/home/spxiwh/test/matplotlib/figure.py”, line 682, in savefig
self.canvas.print_figure(*args, **kwargs)
File “/home/spxiwh/test/matplotlib/backends/backend_agg.py”, line 456, in print_figure
self.draw()
File “/home/spxiwh/test/matplotlib/backends/backend_agg.py”, line 392, in draw
self.figure.draw(renderer)
File “/home/spxiwh/test/matplotlib/figure.py”, line 544, in draw
for a in self.axes: a.draw(renderer)
File “/home/spxiwh/test/matplotlib/axes.py”, line 1063, in draw
a.draw(renderer)
File “/home/spxiwh/test/matplotlib/axis.py”, line 595, in draw
self.label.draw(renderer)
File “/home/spxiwh/test/matplotlib/text.py”, line 340, in draw
bbox, info = self._get_layout(renderer)
File “/home/spxiwh/test/matplotlib/text.py”, line 187, in _get_layout
w,h = renderer.get_text_width_height(
File “/home/spxiwh/test/matplotlib/backends/backend_agg.py”, line 240, in get_text_width_height
Z = texmanager.get_rgba(s, size, self.dpi.get(), rgb)
File “/home/spxiwh/test/matplotlib/texmanager.py”, line 334, in get_rgba
pngfile = self.make_png(tex, fontsize, dpi, force=False)
File “/home/spxiwh/test/matplotlib/texmanager.py”, line 255, in make_png
fh = file(outfile)
IOError: [Errno 2] No such file or directory: ‘/home/spxiwh/.matplotlib/tex.cache/fb2014e54961855bd04020b61190867c.output’
Traceback (most recent call last):
File “/home/spxiwh/ihope/852450000-852700000/bnsinj_summary_plots/…/executables/plotinspinj”, line 569, in ?
‘end_time’, ‘days’, opts.time_axis, plot_type = ‘linear’ )
File “/home/spxiwh/ihope/852450000-852700000/bnsinj_summary_plots/…/executables/plotinspinj”, line 94, in plot_parameter_accuracy
dpi_thumb=opts.figure_resolution)
File “/home/spxiwh/lscsoft/executables/cbc_s5_1yr_20070129/pylal//lib64/python2.4/site-packages/pylal/InspiralUtils.py”, line 54, in savefig_pylal
fig.savefig(filename, dpi=dpi)
File “/home/spxiwh/test/matplotlib/figure.py”, line 682, in savefig
self.canvas.print_figure(*args, **kwargs)
File “/home/spxiwh/test/matplotlib/backends/backend_agg.py”, line 456, in print_figure
self.draw()
File “/home/spxiwh/test/matplotlib/backends/backend_agg.py”, line 392, in draw
self.figure.draw(renderer)
File “/home/spxiwh/test/matplotlib/figure.py”, line 544, in draw
for a in self.axes: a.draw(renderer)
File “/home/spxiwh/test/matplotlib/axes.py”, line 1063, in draw
a.draw(renderer)
File “/home/spxiwh/test/matplotlib/axis.py”, line 561, in draw
tick.draw(renderer)
File “/home/spxiwh/test/matplotlib/axis.py”, line 161, in draw
if self.label1On: self.label1.draw(renderer)
File “/home/spxiwh/test/matplotlib/text.py”, line 838, in draw
Text.draw(self, renderer)
File “/home/spxiwh/test/matplotlib/text.py”, line 340, in draw
bbox, info = self._get_layout(renderer)
File “/home/spxiwh/test/matplotlib/text.py”, line 187, in _get_layout
w,h = renderer.get_text_width_height(
File “/home/spxiwh/test/matplotlib/backends/backend_agg.py”, line 240, in get_text_width_height
Z = texmanager.get_rgba(s, size, self.dpi.get(), rgb)
File “/home/spxiwh/test/matplotlib/texmanager.py”, line 334, in get_rgba
pngfile = self.make_png(tex, fontsize, dpi, force=False)
File “/home/spxiwh/test/matplotlib/texmanager.py”, line 247, in make_png
dvifile = self.make_dvi(tex, fontsize)
File “/home/spxiwh/test/matplotlib/texmanager.py”, line 223, in make_dvi
fh = file(outfile)
IOError: [Errno 2] No such file or directory: ‘/home/spxiwh/.matplotlib/tex.cache/7e534aafdc12681d1ef0d36df4963de8.output’
And once I noticed:
Traceback (most recent call last):
File “/home/spxiwh/ihope/852450000-852700000/allinj_summary_plots/…/executables/plotinspmissed”, line 661, in ?
dpi_thumb=opts.figure_resolution)
File “/home/spxiwh/lscsoft/executables/cbc_s5_1yr_20070129/pylal//lib64/python2.4/site-packages/pylal/InspiralUtils.py”, line 54, in savefig_pylal
fig.savefig(filename, dpi=dpi)
File “/usr/lib64/python2.4/site-packages/matplotlib/figure.py”, line 682, in savefig
self.canvas.print_figure(*args, **kwargs)
File “/usr/lib64/python2.4/site-packages/matplotlib/backends/backend_agg.py”, line 456, in print_figure
self.draw()
File “/usr/lib64/python2.4/site-packages/matplotlib/backends/backend_agg.py”, line 392, in draw
self.figure.draw(renderer)
File “/usr/lib64/python2.4/site-packages/matplotlib/figure.py”, line 544, in draw
for a in self.axes: a.draw(renderer)
File “/usr/lib64/python2.4/site-packages/matplotlib/axes.py”, line 1063, in draw
a.draw(renderer)
File “/usr/lib64/python2.4/site-packages/matplotlib/text.py”, line 340, in draw
bbox, info = self._get_layout(renderer)
File “/usr/lib64/python2.4/site-packages/matplotlib/text.py”, line 187, in _get_layout
w,h = renderer.get_text_width_height(
File “/usr/lib64/python2.4/site-packages/matplotlib/backends/backend_agg.py”, line 240, in get_text_width_height
Z = texmanager.get_rgba(s, size, self.dpi.get(), rgb)
File “/usr/lib64/python2.4/site-packages/matplotlib/texmanager.py”, line 330, in get_rgba
X = readpng(os.path.join(self.texcache, pngfile))
RuntimeError: _image_module::readpng: file not recognized as a PNG file
Cheers
Ian
2008/7/10 Darren Dale <dsdale24@…287…>:
···
Hi Ian,
On Thursday 10 July 2008 06:03:54 am Ian Harry wrote:
Hi all,
Myself and my colleagues use, and have used, matplotlib and it’s Tex
capabilities quite extensively to create plots to assist in the
gravitational wave searches we perform. (and it has been a great tool for
us
). However recently we have been running into problems when we have
started automating our plot generation by running multiple plotting jobs
concurrently using the condor scheduler (and dagmans). Many of our plotting
jobs fail with messages such as the one below:
—snip—
Traceback (most recent call last):
File
"/home/romain/Projects/
ligovirgo/s5_2yr_lv_lowcbc_20080625/868815014-868901414/868815014-868901414
/inj001_summary_plots/…/executables/plotinjnum", line 298, in ?
‘eff_dist_h’)
File
"/home/romain/Projects/ligovirgo/s5_2yr_lv_lowcbc_20080625/868815014-868901
414/868815014-868901414/inj001_summary_plots/…/executables/plotinjnum",
line 119, in plot_found_missed
fname_thumb = InspiralUtils.savefig_pylal(filename=fname,
doThumb=True, dpi_thumb=opts.figure_resolution)
File
"/home/romain/codes/s5_2yr_lv_lowcbc_20080625/pylal/lib64/python2.4/site-pa
ckages/pylal/InspiralUtils.py", line 58, in savefig_pylal
fig.savefig(filename_thumb, dpi=dpi_thumb)
…
File “/usr/lib64/python2.4/site-packages/matplotlib/texmanager.py”, line
259, in make_png
os.remove(outfile)
OSError: [Errno 2] No such file or directory:
'/home/romain/.matplotlib/tex.cache/ae479c90ff242327b54af004a0846188.output
’
—snip—
My feeling is that when the code invokes the Tex ‘bit’ it creates a temp
file in ~/matplotlib/tex.cache and then deletes it and all other temp tex
files when it finishes the Tex ‘bit’. This would cause problems if another
job is in the middle of running Tex when the other job deletes it’s temp
files!
We are running a slightly old version of matplotlib (0.87.7), as we run on
multiple clusters our sys admins tend to only update software when there is
a need to and we have had no other problems with matplotlib, I apologize if
this has been fixed in the meantime (I did do a quick search of the mailing
list archive but found nothing). All our clusters currently run Fedora Core
4 (we’re going to move to CentOS 5).
Currently we are getting around this by forcing condor to retry the failed
jobs 2/3 times, this catches most of these errors. Another solution would
be to limit the number of jobs running to 1 BUT as we run dagmen from
within one ‘super’ dagman it would prove difficult to limit jobs from
multiple dagmen.
Anyway if anyone has any ideas of how to solve this I would appreciate
this. Also if there are any options where we can set the location of these
temp tex files and use a different directory for each job (or stop
matplotlib deleting other temp files) that would help us.
I’m really hesitant to mess around with the location of the temp files. It was
a bit painfull trying to get usetex to work across platforms.
Instead, would you try replacing:
os.remove(outfile)
with:
try: os.remove(outfile)
except OSError: pass
Let me know if that fixes it, and if you need to wrap any other file
deletions.
Thanks,
Darren
–
Ian Harry
School of Physics & Astronomy
Queens Buildings, The Parade
Cardiff, CF24 3AA
Email: Ian.Harry@…1663…
Phone: (+44) 29 208 75120
Mobile: (+44) 7890 479090