[ python-Bugs-1776160 ] Buffer overflow when listing deeply nested directory

[SourceForge.net]

Bugs item #1776160, was opened at 2007-08-17 13:24
Message generated for change (Comment added) made by sonderblade
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1776160&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Windows
Group: Platform-specific
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Björn Lindqvist (sonderblade)
Assigned to: Nobody/Anonymous (nobody)
Summary: Buffer overflow when listing deeply nested directory

Initial Comment:
This code:

import os
import os.path
TARGET='C:/code/python/foo'
base = TARGET
for x in range(200):
    subdirs = os.listdir(base)
    base = os.path.join(base, subdirs[0])
    print base

Produces a TypeError (buffer overflow) when run on a to deeply nested directory for windows to handle:

.. more output here..
C:code/python/foo\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.p
ng\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png
Traceback (most recent call last):
  File "killdir.py", line 6, in <module>
    subdirs = os.listdir(base)
TypeError: listdir() argument 1 must be (buffer overflow), not str


----------------------------------------------------------------------

>Comment By: Björn Lindqvist (sonderblade)
Date: 2007-08-22 12:26

Message:
Logged In: YES 
user_id=51702
Originator: YES

Yes, it is the error message and the exception that is the problem. First,
it shouldn't raise TypeError (which indicates a programming error), it
should raise either IOError, OSError or WindowsError. Second, the exception
message is whacky: "listdir() argument 1 must be (buffer overflow), not
str" I realize that it is probably impossible to detect this specific error
condition but I still want something more explanatory than what it
currently is.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2007-08-22 07:56

Message:
Logged In: YES 
user_id=21627
Originator: NO

Can you please explain what specifically you consider a bug here?

I can see that the error message is confusing, so it could be improved.
However, there is nothing we can do to make the error go away. The
Microsoft C library simply does not support file names longer than
MAX_PATH; you have to use Unicode file names to go beyond this limit.

----------------------------------------------------------------------

Comment By: Björn Lindqvist (sonderblade)
Date: 2007-08-21 10:49

Message:
Logged In: YES 
user_id=51702
Originator: YES

Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)]
on win32
MS Windows XP, Version 5.1, SP2


----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2007-08-21 10:18

Message:
Logged In: YES 
user_id=21627
Originator: NO

To rephrase Skip's comment: Can you please report what operating system
and Python version you are using?

----------------------------------------------------------------------

Comment By: Skip Montanaro (montanaro)
Date: 2007-08-18 13:38

Message:
Logged In: YES 
user_id=44345
Originator: NO

Worked as expected for me on Mac OS X 10.4.10 running from
the trunk (you didn't mention what version you were using).
In ~/tmp/deep I created a maximally nested directory tree from the shell
like so:

    cd /Users/skip/tmp/deep
    for i in `range 1000` ; do
        x=`printf %04d $i`
        echo $x
        mkdir $x
        cd $x
    done

where the range command is analogous to Python's range
builtin:

    % range 20
    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

The for loop barfed after making directory 0205.

In Python I then executed these statements:

    import os.path
    base = "/Users/skip/tmp/deep"
    for x in range(210):
        subdirs = os.listdir(base)
        base = os.path.join(base, subdirs[0])
        print base

This went until it got to dir 0200 where it raised an
OSError:

    [Errno 63] File name too long:
'/Users/skip/tmp/deep/0000/0001/.../0199/0200'

which stands to reason since base was 1025 characters long
at that point.  MAXPATHLEN is defined to be 1024 on my
system, so the OSError is to be expected.

Skip


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1776160&group_id=5470