ftplib error- Large file

[half.italian at gmail.com]

On Jun 6, 11:21 pm, "Hendrik van Rooyen" <m... at microcorp.co.za> wrote:
>  <hal..... at gmail.com> wrote:
> > Hi all,
>
> > I'm using ftplib to transfer large files to remote sites.  The process
> > seems to work perfectly with small files, but when the file gets to
> > large ~20GB I begin getting errors that sometimes seem to be non-
> > fatal, and other times the transfer does not complete.  I've debugged
> > the hell out of the code, and can't figure out what is going on..is it
> > the remote site?  Packet loss?
>
> 20 gig is a big thing - what is your link speed, and how long does the thing
> run before falling over - Is your ISP maybe giving you or the remote site a
> new IP addy periodically?  
>
> Are there any "anti jabber" precautions taken anywhere?
>
> you may have to break it up into smaller chunks and glue them together
> remotely...
>
> hth - Hendrik

It's different sites (state to state) within the same company, so I'm
sure of the IP addresses.  Whats odd is that the transfers don't fail
most of the time.  The transfer usually completes properly but
ftplib.FTP.storbinary() throws an IOError at the end of the file
instead of returning the status of the transfer.  I've resorted to
just catching and logging the exception and then comparing remote vs.
local md5's to check for integrity, but it seems like if a connection
error occurs, storbinary should handle the error and return an error
code rather than throwing an exception.  The 20gb files take about an
hour to transfer, and I've had success with files up to 60gb so far...
once I figured out what ftplib was doing and starting cathing the
IOError.

Why would storbinary throw an exception even when the file transfer
was successful?  Why would the exception not be thrown until after the
file was sent?  Shouldn't ftplib return something like (104,
"Connection reset by peer") instead of throwing an error?  Is my
reasoning off somewhere?

~Sean