Odd/Weird errors with FTPLib

[MRAB]

Bakes wrote:
> On Sep 13, 11:47 pm, MRAB <pyt... at mrabarnett.plus.com> wrote:
>> Bakes wrote:
>>> On 13 Sep, 22:41, Chris Rebert <c... at rebertia.com> wrote:
>>>> On Sun, Sep 13, 2009 at 2:34 PM, Bakes <ba... at ymail.com> wrote:
>>>>> I am using a simple python script to download my logfiles. This is on
>>>>> a while loop, the logfile grows rapidly, so it is necessary for python
>>>>> to start downloading the new script as soon as it has finished the
>>>>> old.
>>>>> It works fine (for about 20 minutes), then crashes. I have removed a
>>>>> couple of excepts, and have narrowed the error down to a 'error_perm:
>>>>> 550 logfile.log: The data is invalid.' error.
>>>>> Does anyone know what the problem might be regarding this, and what I
>>>>> might do to fix it?
>>>> Including an actual code snippet and the full error traceback would help a lot.
>>>> According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
>>>> error code 550 translates to:
>>>> "Requested action not taken. File unavailable (e.g., file not found,
>>>> no access)."
>>>> Does the logfile get rotated or something, thus causing it to briefly not exist?
>>>> It might also help if you explain how your logfile system works.
>>>> Cheers,
>>>> Chris
>>>> --http://blog.rebertia.com
>>> It's a cod4 gameserver logfile, being downloaded for a python bot to
>>> parse.
>>> The logfile is downloaded using this try/except while loop.
>>>     while True:
>>>         try:
>>>             if ftp == False:
>>>                 self.debug('FTP connection not active, attempting to (re)connect')
>>>                 ftp = self.ftpconnect()
>>>             size=os.path.getsize('games_mp.log')
>>>             ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig ['path']), handleDownload, rest=size)
>>>             if self.console._paused:
>>>                 self.console.unpause()
>>>         except:
>>>             print error
>>>             self.debug('Lost connection to server, pausing until updated properly, Sleeping 10 seconds')
>>>             self.console.pause()
>>>             try:
>>>                 ftp.close()
>>>                 self.debug('FTP Connection Closed')
>>>             except:
>>>                 self.debug('FTP does not appear to be open, so not closed')
>>>             ftp = False
>>>             time.sleep(10)
>>> I can only assume that occasionally, the logfile is being written to
>>> by the gameserver at the same time that it's downloading.
>>> If this was the case, do you think a try: download except: sleep
>>> 900msec then download loop would work?
>> Bare excepts are almost always a bad idea because they'll catch _all_
>> exceptions, both those you expect could happen and those you don't.
>> Catch only those you expect.
>>
>> For example, if the file 'games_mp.log' doesn't exist then
>> os.path.getsize('games_mp.log') will raise an exception, and if you
>> forgot to import the os module then that will raise a NameError
>> exception.
>>
>> Anyway, I can't see how you leave the loop; I'd expect something like a
>> 'break' statement.
>>
>> And as a matter of style, I'd prefer None to False to indicate when
>> there's no FTP connection (and "if not ftp" instead of "if ftp ==
>> False").
> 
> I removed the try/except and saw when it failed.
> 
> I'll change those things, games_mp.log is guaranteed to be there (file
> made in another script), os is imported correctly.
> 
> So, what do you think the error could be?

How does control leave the while loop?

If you're running on Windows and you're opening the file 'games_mp.log'
with mode 'w' then there'll be the issue of the line-endings. If the log
file on the server uses '\n' for the line endings and you're using
'\r\n' then os.path.getsize('games_mp.log') will return a larger size
then you expect. In that case, could ftp.retrbinary() be complaining
because it's given you the entire file and then you're trying to
download from an offset that's beyond the end?

For example, suppose the file on the server contains just "foo\n" (4
bytes):

1. You open a file locally in text mode ('w').

2. You download the entire file, "foo\n".

3. You write out the data, but because you've opened in text mode it
writes "foo\r\n" to the file. The local file size is now 5 bytes.

4. The loop means that you then try to download from offset 5, which is
beyond the end of the file on the server.

5. Error?