[Numpy-discussion] numpy.load truncates read from network file on XP

[Dan Halbert]

On Thursday, April 28, 2011 5:22pm, "Dan Halbert" <halbert at halwitz.org> said:
> I'm having trouble loading a large remote .npy file on Windows XP. This is on
> numpy-1.3.0 on Windows XP SP3:
> 
>     numpy.load(r'\\myserver\mydir\big.npy')
> 
> will fail with this sort of error being printed:
>     "14328000 items requested but only 54 read"

I have tracked this down: it's not specific to numpy, or even Python, though
there's a Python bug report about it:
"size limit exceeded for read() from network drive"
http://bugs.python.org/issue1478529

The problem stems from a buffer size limit of not quite 64MB for the Windows API
ReadFile()function. ReadFile() is used (maybe indirectly) by the Microsoft
fread() implementation, which is in turn used by Python file.read().

Apparently most of the time the underlying Windows code is cognizant of this limit,
but under certain circumstances, a read from a large networked file is not handled
correctly (I think when there is also an entry in "My Network Places" for the path).
It's quite obscure, and there's nothing that can be fixed at the numpy level.

Dan