OSError on check of _is_writable_dir(p)

Community matplotlib-users
1

Hi all,

I recently run into a problem with the .matplotlib directory. I run a script as a daemon, that in its turn runs several scripts to create graphs (often the same script with different input parameters), dependent on an outside trigger. Recently, I found that these script crashed with an OSError during the 'import matplotlib' phase, at the point where it tries to check that the .matplotlib directory is writeable. The exact traceback is appended below.

Searching on the mailing list, I found someone had the same problem almost a year ago ( http://sourceforge.net/mailarchive/message.php?msg_id=44964AD1.605%40yahoo.com ), but it appears that that post doesn't have a response.
A wider search resulted in this post, http://osdir.com/ml/python.peak/2006-06/msg00019.html , where the response suggests these problems are caused by all the things mpl does in its __init__ file.

A few notes on my setup:
- the .matplotlib is a symbolic link to a .python/matplotlib directory, since I'd like to keep my python resources together (eg, .python also contains an ipython directory). I've currently changed the symlink to a proper directory, but haven't had a chance to test this new setup. Anyway, afaik, symlinks should work properly
- the scripts that are run from the daemon script, are run in the background, and quite often shortly after each other. Whether that creates a deadlock situation or something, I don't know. Obviously, I'm using a non-interactive backend, agg in this case.
- the problem only showed after I upgraded from mpl version 0.87 to 0.9.
- interactively, everything runs fine.
- everything runs under a locally installed python, which is called with the appropriate path from the script
- python version 2.4.4, numpy version 1.0

Does anyone have a suggestion what went wrong, and how to properly fix all this? As a temporary measure, I now ignore the _is_writeable(p) result in __init__.py (the directory, and therefore the symlink, were/are writeable when this occurred, as far as I can tell).

I like mpl very much, but this problem may render it much less usable for me; a temporary fix would be fine for now, but it actually feels like a bug to me.

Thanks a lot,

   Evert

full traceback (note: replaced directories with ~)

Traceback (most recent call last):
   File "~/myscript.py", line 64, in ?
     sys.exit(main())
   File "~/myscript.py", line 59, in main
     return run(parser)
   File "~/myscript.py", line 27, in run
     sigmaclip=parser['sigmaclip'], invert=True, compass=True)
   File "~/python/modules/fitsimage/FITSImage_numpy.py", line 777, in plot
     import matplotlib as mpl
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 1019, in ?
     rcParams = rc_params()
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 976, in rc_params
     fname = matplotlib_fname()
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 922, in matplotlib_fname
     fname = os.path.join(get_configdir(), 'matplotlibrc')
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 273, in wrapper
     ret = func(*args, **kwargs)
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 329, in _get_configdir
     os.mkdir(p)
OSError: [Errno 17] File exists: '<...>/.matplotlib'

2

One way this could happen is if the .matplotlib directory exists but is not writable; such a case would give the error traceback you see with little clue as to what and where the problem really is. So, __init__.py certainly could be improved. (Offhand, I don't even know why the existence of such a writable config directory should be required.)

In the meantime, to solve your particular problem: my guess is that when you run a script as a daemon, and it runs more scripts, the environment is not what you think it is (or what it is when you run interactively), so __init__.py is not looking where you think it is. Either that, or the uid/gid of the script are not the same as yours, so the script can't write to your .matplotlib directory even though you can. Instead of ignoring the _is_writable(p) result, you might want to try printing out os.path.abspath(p) to verify where p really is.

Eric

Evert Rol wrote:

···

   Hi all,

I recently run into a problem with the .matplotlib directory. I run a script as a daemon, that in its turn runs several scripts to create graphs (often the same script with different input parameters), dependent on an outside trigger. Recently, I found that these script crashed with an OSError during the 'import matplotlib' phase, at the point where it tries to check that the .matplotlib directory is writeable. The exact traceback is appended below.

Searching on the mailing list, I found someone had the same problem almost a year ago ( http://sourceforge.net/mailarchive/message.php? msg_id=44964AD1.605%40yahoo.com ), but it appears that that post doesn't have a response.
A wider search resulted in this post, OSDIR 2006-06/msg00019.html , where the response suggests these problems are caused by all the things mpl does in its __init__ file.

A few notes on my setup:
- the .matplotlib is a symbolic link to a .python/matplotlib directory, since I'd like to keep my python resources together (eg, .python also contains an ipython directory). I've currently changed the symlink to a proper directory, but haven't had a chance to test this new setup. Anyway, afaik, symlinks should work properly
- the scripts that are run from the daemon script, are run in the background, and quite often shortly after each other. Whether that creates a deadlock situation or something, I don't know. Obviously, I'm using a non-interactive backend, agg in this case.
- the problem only showed after I upgraded from mpl version 0.87 to 0.9.
- interactively, everything runs fine.
- everything runs under a locally installed python, which is called with the appropriate path from the script
- python version 2.4.4, numpy version 1.0

Does anyone have a suggestion what went wrong, and how to properly fix all this? As a temporary measure, I now ignore the _is_writeable (p) result in __init__.py (the directory, and therefore the symlink, were/are writeable when this occurred, as far as I can tell).

I like mpl very much, but this problem may render it much less usable for me; a temporary fix would be fine for now, but it actually feels like a bug to me.

Thanks a lot,

   Evert

full traceback (note: replaced directories with ~)

Traceback (most recent call last):
   File "~/myscript.py", line 64, in ?
     sys.exit(main())
   File "~/myscript.py", line 59, in main
     return run(parser)
   File "~/myscript.py", line 27, in run
     sigmaclip=parser['sigmaclip'], invert=True, compass=True)
   File "~/python/modules/fitsimage/FITSImage_numpy.py", line 777, in plot
     import matplotlib as mpl
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 1019, in ?
     rcParams = rc_params()
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 976, in rc_params
     fname = matplotlib_fname()
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 922, in matplotlib_fname
     fname = os.path.join(get_configdir(), 'matplotlibrc')
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 273, in wrapper
     ret = func(*args, **kwargs)
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 329, in _get_configdir
     os.mkdir(p)
OSError: [Errno 17] File exists: '<...>/.matplotlib'

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
matplotlib-users List Signup and Options

3

It is writeable because we use it for caching fonts and tex-related stuff.
Don't most programs require a writeable config directory? My home directory
is filled with them.

Darren

···

On Monday 30 April 2007 02:31:59 pm Eric Firing wrote:

One way this could happen is if the .matplotlib directory exists but is
not writable; such a case would give the error traceback you see with
little clue as to what and where the problem really is. So, __init__.py
certainly could be improved. (Offhand, I don't even know why the
existence of such a writable config directory should be required.)

4

Hi Eric,

Thanks for your response.

One way this could happen is if the .matplotlib directory exists but is not writable; such a case would give the error traceback you see with little clue as to what and where the problem really is. So, __init__.py certainly could be improved. (Offhand, I don't even know why the existence of such a writable config directory should be required.)

I'm pretty sure the directory is writeable, as my default settings are 755 for new files/dirs (umask=022). Unless, indeed, some process changed that. But when I looked after the crash, the permissions were writeable/executable.
And yes, I was somewhat surprised by the writeable requirement of the config dir as well.

In the meantime, to solve your particular problem: my guess is that when you run a script as a daemon, and it runs more scripts, the environment is not what you think it is (or what it is when you run interactively), so __init__.py is not looking where you think it is. Either that, or the uid/gid of the script are not the same as yours, so the script can't write to your .matplotlib directory even though you can. Instead of ignoring the _is_writable(p) result, you might want to try printing out os.path.abspath(p) to verify where p really is.

The OSError does show it to be the correct directory, unless there's a way to fool that traceback. Also, I haven't set MATPLOTLIBRC anywhere, so I guess it picks up .matplotlibrc in my HOME (which I assume is set automatically by the system, even for daemon scripts). I'll put in the os.path.abspath output anyway.
I wouldn't know 100% about the uid/gid, but everything is done under my user account (I don't have access to any other account on this system anyway), so might be hard to have that changed).

Thanks again for the response.

Cheers,

   Evert

···

   Hi all,
I recently run into a problem with the .matplotlib directory. I run a script as a daemon, that in its turn runs several scripts to create graphs (often the same script with different input parameters), dependent on an outside trigger. Recently, I found that these script crashed with an OSError during the 'import matplotlib' phase, at the point where it tries to check that the .matplotlib directory is writeable. The exact traceback is appended below.
Searching on the mailing list, I found someone had the same problem almost a year ago ( http://sourceforge.net/mailarchive/message.php? msg_id=44964AD1.605%40yahoo.com ), but it appears that that post doesn't have a response.
A wider search resulted in this post, OSDIR 2006-06/msg00019.html , where the response suggests these problems are caused by all the things mpl does in its __init__ file.
A few notes on my setup:
- the .matplotlib is a symbolic link to a .python/matplotlib directory, since I'd like to keep my python resources together (eg, .python also contains an ipython directory). I've currently changed the symlink to a proper directory, but haven't had a chance to test this new setup. Anyway, afaik, symlinks should work properly
- the scripts that are run from the daemon script, are run in the background, and quite often shortly after each other. Whether that creates a deadlock situation or something, I don't know. Obviously, I'm using a non-interactive backend, agg in this case.
- the problem only showed after I upgraded from mpl version 0.87 to 0.9.
- interactively, everything runs fine.
- everything runs under a locally installed python, which is called with the appropriate path from the script
- python version 2.4.4, numpy version 1.0
Does anyone have a suggestion what went wrong, and how to properly fix all this? As a temporary measure, I now ignore the _is_writeable (p) result in __init__.py (the directory, and therefore the symlink, were/are writeable when this occurred, as far as I can tell).
I like mpl very much, but this problem may render it much less usable for me; a temporary fix would be fine for now, but it actually feels like a bug to me.
Thanks a lot,
   Evert
full traceback (note: replaced directories with ~)
Traceback (most recent call last):
   File "~/myscript.py", line 64, in ?
     sys.exit(main())
   File "~/myscript.py", line 59, in main
     return run(parser)
   File "~/myscript.py", line 27, in run
     sigmaclip=parser['sigmaclip'], invert=True, compass=True)
   File "~/python/modules/fitsimage/FITSImage_numpy.py", line 777, in plot
     import matplotlib as mpl
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 1019, in ?
     rcParams = rc_params()
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 976, in rc_params
     fname = matplotlib_fname()
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 922, in matplotlib_fname
     fname = os.path.join(get_configdir(), 'matplotlibrc')
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 273, in wrapper
     ret = func(*args, **kwargs)
   File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 329, in _get_configdir
     os.mkdir(p)
OSError: [Errno 17] File exists: '<...>/.matplotlib'

5

Darren Dale wrote:

···

On Monday 30 April 2007 02:31:59 pm Eric Firing wrote:

One way this could happen is if the .matplotlib directory exists but is
not writable; such a case would give the error traceback you see with
little clue as to what and where the problem really is. So, __init__.py
certainly could be improved. (Offhand, I don't even know why the
existence of such a writable config directory should be required.)

It is writeable because we use it for caching fonts and tex-related stuff. Don't most programs require a writeable config directory? My home directory is filled with them.

This occurred to me a couple minutes after I hit "send". Yes, this sort of thing is common, but I wouldn't say "most programs"; and it is exactly the sort of thing that can cause trouble and confusion when running a program from something like a cron job or modpython instead of the usual command line.

Eric