[issue3115] os.listdir randomly fails on occasions when it shouldn't

[philipspencer]

New submission from philipspencer <pspencer at fields.utoronto.ca>:

Python's os.listdir function has the following code in
Modules/posixmodule.c:

   errno = 0
   ...
    for (;;) {
         Py_BEGIN_ALLOW_THREADS
         ep = readdir(dirp);
         Py_END_ALLOW_THREADS
         if (ep == NULL)
              break;
         ...
         a bunch of other stuff, including PyString_FromStringAndSize
         which calls malloc
         ...
    }
    if (errno != 0 && d != NULL) {

The assumption is that errno will be nonzero only if readdir failed.
However, this is not the case. GLibc's malloc will, in some rare cases,
set errno even when it succeeds. So, during one pass through the loop
errno gets set to ENOMEM and then it is still set to ENOMEM when the
final readdir returns null at the end of the directory listing.

The fix is to move the line "errno = 0" from outside the loop to
right before the readdir inside the loop. That is the only way to
ensure that, when the loop is exited with readdir returning null,
the condition "errno != 0" is equivalent to "readdir actually failed."

The attached patch does this. Without this patch, we experience
frequent failures with the python tools "creatrepo" and "repomanage"
when run on very large directories (> 5000 packages) on Fedora 8
systems; see RedHat bugzilla entry #451494. With the patch, the commands
work as expected.

The patch is against 2.5.1 but from what I can see of the posixmodule.c
code from SVN it should apply cleanly there too.

----------
components: Extension Modules
messages: 68223
nosy: philipspencer
severity: normal
status: open
title: os.listdir randomly fails on occasions when it shouldn't
type: behavior
versions: Python 2.5

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3115>
_______________________________________