Python randomly exits with Linux OS error -9 or -15

[Janis]

Hello!

I have this problem with my script exiting randomly with Linux OS
status code -9 (most often) or -15 (also sometimes, but much more
rarely). As far as I understand -9 corresponds to Bad file descriptor
and -15 Block device required.

1) Is there a way how I could find out what exactly causes Python
process to exit?
2) What could be the reason of Python exiting with these status code?

The script is a crawler that crawls several web sites to download web
pages and extract information. Most often it exits after having run
for 2 hours and having downloaded ~24 000 files. Some specific web
sites are more affected than others, i.e., there are other instances
of the script running in parallel that download more pages and
complete normally. That could be related to the speed each page is
returned etc.

I have a try-catch block in the root of the script which works very
well to catch any kind of exceptions, but in these cases either Python
does not catch the exception or fails to log it into MySQL error log
table.

I can not use debugger because the code exits after it has run for an
hour or more. In order to try to catch the exact place I have put
logging after each line in some places. The logging prints to stdout,
to a file and logs the line into MySQL DB. This has revealed that code
may exit upon a random simple lines such as: time.sleep(0.1) - most
often, f = opener.open(request) - also often, but sometimes also such
simple statements as list.add('string').

It can also be that the problem occues and then the script exits upon
attempt to do any output - stdout (i.e. regular print '1'), writing to
file and logging into MySQL. I have changed either of these to be the
first ones in the debug log function, but each time the script did not
fail in between these lines.

The main libraries that are in use: MySQLDB, urllib and urllib2, but
none of them seems to be the direct cause of the problem, there is no
direct call of any of these upon exit. I suspect that this could be
related to some garbadge collecting process.

Versions tried with: Python 2.6, Python 2.7 (and I think it happened
also in Python 2.5, but I'm not sure)

Thanks,
Janis