[ python-Bugs-1478529 ] size limit exceeded for read() from network drive

Wed May 3 12:38:20 CEST 2006

Bugs item #1478529, was opened at 2006-04-28 17:46
Message generated for change (Comment added) made by markshep
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1478529&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Windows
Group: 3rd Party
Status: Closed
Resolution: Wont Fix
Priority: 5
Submitted By: Mark Sheppard (markshep)
Assigned to: Nobody/Anonymous (nobody)
Summary: size limit exceeded for read() from network drive

Initial Comment:
If you've got a network share mounted as a local drive
then Windows has a limit of 67,076,095 (0x03ff7fff)
bytes for a read from an open file on that drive.

Running the python read() method on an open file larger
than this size throws an "IOError: [Errno 22] Invalid
argument" exception.

A fix would be for python to internally use multiple
reads so as to not exceed this limit.

----------------------------------------------------------------------

>Comment By: Mark Sheppard (markshep)
Date: 2006-05-03 11:38

Message:
Logged In: YES 
user_id=1512331

Thanks for closing this bug without giving me a chance to
follow up!

The problem isn't caused by a limitation of my machine -
it's got 3 GiB of RAM.

I've done some more testing on this and the problem only
appears when connected to a server running certain SMB
implementations:

  The local Windows XP machine
  A remote Windows XP machine
  Samba 3.0.22 on Linux

When connected to servers running the following SMB
implementations the problem isn't present:

  Windows NT 4.0 Server
  Windows Server 2000
  Windows Server 2003 Standard Edition

As this error is being returned by the underlying fread()
call the proper place for it to be fixed is there, but the
chances of Microsoft doing so for Windows XP are negligible.

As you're trying to provide a cross-platform language then
having to put up with OS's undocumented warts is just part
of the job.  As it's entirely possible for you to implement
a work-around for this problem Python I think you should. 
One of reasons for using a high level language like Python
is to be insulated from system quirks likes this.  If you're
refusing to smooth over these quirks where possible then
you're undermining that reason.

The documentation for Python's read() method on a file
handle already says "Note that this method may call the
underlying C function fread() more than once", so this
possibility is already catered for in the documentation.

As this problem only affects remotely mounted filesystems
the workaround need only be used for such filesystems.  You
can determine whether or not a drive is a network one by
using the GetDriveType() Windows call.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-05-03 02:45

Message:
Logged In: YES 
user_id=31435

Sorry, I'm closing this as "3rd Party, Won't Fix".  It's
certainly not Python's doing that Microsoft's stdio
implementation has peculiar undocumented warts (Python just
calls the platform C's fread() here), so at best this is a
request for enhancement rather than a Python bug.

If there is a bug here, it's Microsoft's bug, and then the
proper source for a fix is also Microsoft.  This is
especially true since the two people who have tried this
here don't see the same behavior -- we don't even know what
"the bug" is.

----------------------------------------------------------------------

Comment By: Martin v. LÃ¶wis (loewis)
Date: 2006-05-02 20:00

Message:
Logged In: YES 
user_id=21627

I could reproduce the write problem on XPSP2; I get the
Win32 error ERROR_NO_SYSTEM_RESOURCES after fwrite returns
(from GetLastError).

I can't reproduce the fread problem, though: in Python,
f.read(90*2**20) just returns with a 90MiB string. So it
could be a limitation of your machine (e.g. it might not
have enough memory), or of the server machine. I'm hesitant
to add a work-around for that into Python if this isn't a
system limitation. Performing multiple reads is also bad:
what happens if the first read succeeds, and the second one
fails? It might be that the system *really* is out of resources.

----------------------------------------------------------------------

Comment By: Mark Sheppard (markshep)
Date: 2006-05-02 11:48

Message:
Logged In: YES 
user_id=1512331

I'm running Windows XP.  I've been unable to find any
documentation about this exact problem - only that fwrite
thing.  But my testing shows that it works if I do
file.read(67076095), but throws an exception with
file.read(67076096).

I'm not suggesting limiting all reads from Python.  All I'm
suggesting is that under the hood the Windows implementation
of Python's read() call actually uses multiple fread() (or
whatever) calls if more than 67076095 bytes need to be read.
 That's all.  No interface changes.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-04-30 17:23

Message:
Logged In: YES 
user_id=31435

Martin, here's an MS article seemingly related to this:

http://support.microsoft.com/default.aspx?scid=kb;en-us;899149

However, it's about writing to a file on a network drive,
not reading from it.  It says that opening the file in 'w+b'
mode, instead of 'wb' mode, is a workaround.

I couldn't find anything documenting the same kind of
problem for reading.

----------------------------------------------------------------------

Comment By: Martin v. LÃ¶wis (loewis)
Date: 2006-04-30 11:10

Message:
Logged In: YES 
user_id=21627

What version of Windows are you using? Do you know of any
documentation of this limit? (without actually testing, I
find it hard to believe that this limit exists in Windows)

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-04-29 14:23

Message:
Logged In: YES 
user_id=849994

How can it be determined whether exactly this restriction
caused the "invalid argument" error? If it can't, there's
nothing that can be done -- restricting all reads just
because of a Windows limitation doesn't seem right.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1478529&group_id=5470