OverflowError while sending large file via socket

[Steven D'Aprano]

On Mon, 13 Apr 2009 00:21:34 +0200, Ryniek90 wrote:

> When i wanted to send an .iso file of 4GB length, i had traceback:
> "OverflowError: requested number of bytes is more than a Python string
> can hold"
> 
> Sockets are being used in every network app, i.e: p2p progs (like
> BitTorrent), and exchanged data is often bigger than 4GB.

But they don't transfer the entire file as ONE packet. Split your data 
into smaller packets. I don't know what a good size for each packet would 
be, but if I were doing this, I'd probably start with 4096 or 8192 
*bytes*.

http://www.amk.ca/python/howto/sockets/


> So why i've 
> had that Traceback? How many number of bytes Python string can hold?

The documentation doesn't seem to specify a maximum string length:

http://docs.python.org/library/stdtypes.html

which suggests to me that it will be implementation dependent. However, 
I'd guess that the current CPython implementation will have a hard limit 
of 2**32 bytes (4GB), and a soft limit on the amount of memory that you 
have. You're trying to create a single, continuous block of memory 4GB in 
size! Unless you've got *at least* 4GB of RAM, this is impossible even in 
principle, and in practice you need more than that to allow for the 
overhead of the operating system, Python, and any other applications you 
have running.



-- 
Steven