Odd/Weird errors with FTPLib

[Bakes]

On Sep 15, 1:32 am, MRAB <pyt... at mrabarnett.plus.com> wrote:
> Bakes wrote:
> > On Sep 13, 11:47 pm, MRAB <pyt... at mrabarnett.plus.com> wrote:
> >> Bakes wrote:
> >>> On 13 Sep, 22:41, Chris Rebert <c... at rebertia.com> wrote:
> >>>> On Sun, Sep 13, 2009 at 2:34 PM, Bakes <ba... at ymail.com> wrote:
> >>>>> I am using a simple python script to download my logfiles. This is on
> >>>>> a while loop, the logfile grows rapidly, so it is necessary for python
> >>>>> to start downloading the new script as soon as it has finished the
> >>>>> old.
> >>>>> It works fine (for about 20 minutes), then crashes. I have removed a
> >>>>> couple of excepts, and have narrowed the error down to a 'error_perm:
> >>>>> 550 logfile.log: The data is invalid.' error.
> >>>>> Does anyone know what the problem might be regarding this, and what I
> >>>>> might do to fix it?
> >>>> Including an actual code snippet and the full error traceback would help a lot.
> >>>> According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
> >>>> error code 550 translates to:
> >>>> "Requested action not taken. File unavailable (e.g., file not found,
> >>>> no access)."
> >>>> Does the logfile get rotated or something, thus causing it to briefly not exist?
> >>>> It might also help if you explain how your logfile system works.
> >>>> Cheers,
> >>>> Chris
> >>>> --http://blog.rebertia.com
> >>> It's a cod4 gameserver logfile, being downloaded for a python bot to
> >>> parse.
> >>> The logfile is downloaded using this try/except while loop.
> >>>     while True:
> >>>         try:
> >>>             if ftp == False:
> >>>                 self.debug('FTP connection not active, attempting to (re)connect')
> >>>                 ftp = self.ftpconnect()
> >>>             size=os.path.getsize('games_mp.log')
> >>>             ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig ['path']), handleDownload, rest=size)
> >>>             if self.console._paused:
> >>>                 self.console.unpause()
> >>>         except:
> >>>             print error
> >>>             self.debug('Lost connection to server, pausing until updated properly, Sleeping 10 seconds')
> >>>             self.console.pause()
> >>>             try:
> >>>                 ftp.close()
> >>>                 self.debug('FTP Connection Closed')
> >>>             except:
> >>>                 self.debug('FTP does not appear to be open, so not closed')
> >>>             ftp = False
> >>>             time.sleep(10)
> >>> I can only assume that occasionally, the logfile is being written to
> >>> by the gameserver at the same time that it's downloading.
> >>> If this was the case, do you think a try: download except: sleep
> >>> 900msec then download loop would work?
> >> Bare excepts are almost always a bad idea because they'll catch _all_
> >> exceptions, both those you expect could happen and those you don't.
> >> Catch only those you expect.
>
> >> For example, if the file 'games_mp.log' doesn't exist then
> >> os.path.getsize('games_mp.log') will raise an exception, and if you
> >> forgot to import the os module then that will raise a NameError
> >> exception.
>
> >> Anyway, I can't see how you leave the loop; I'd expect something like a
> >> 'break' statement.
>
> >> And as a matter of style, I'd prefer None to False to indicate when
> >> there's no FTP connection (and "if not ftp" instead of "if ftp ==
> >> False").
>
> > I removed the try/except and saw when it failed.
>
> > I'll change those things, games_mp.log is guaranteed to be there (file
> > made in another script), os is imported correctly.
>
> > So, what do you think the error could be?
>
> How does control leave the while loop?
>
> If you're running on Windows and you're opening the file 'games_mp.log'
> with mode 'w' then there'll be the issue of the line-endings. If the log
> file on the server uses '\n' for the line endings and you're using
> '\r\n' then os.path.getsize('games_mp.log') will return a larger size
> then you expect. In that case, could ftp.retrbinary() be complaining
> because it's given you the entire file and then you're trying to
> download from an offset that's beyond the end?
>
> For example, suppose the file on the server contains just "foo\n" (4
> bytes):
>
> 1. You open a file locally in text mode ('w').
>
> 2. You download the entire file, "foo\n".
>
> 3. You write out the data, but because you've opened in text mode it
> writes "foo\r\n" to the file. The local file size is now 5 bytes.
>
> 4. The loop means that you then try to download from offset 5, which is
> beyond the end of the file on the server.
>
> 5. Error?

It opens it in append mode, and if it errors with that message, the
ending is different. I have come across that problem before, and it
has been fixed (download in binary is the fix, mostly).