[Tutor] Need help appending data to a logfile

[Matt D]

On 06/30/2013 04:00 AM, Alan Gauld wrote:
> On 30/06/13 04:41, Matt D wrote:
> 
>> So i have a program that gets pickled data from a thread and displays
>> the data (9 strings) in TextCtrl fields in the UI.
> 
> The fact that it is a UI is largely irrelevant apart from the
> fact that it forces an event driven architecture.
> 
>>  When the pickle is
>> received it is processed like this:
>>
>> def display_data(self,event):
> ...
>>      attrs = pickle.loads(pickled_dict)
>>      self.update(attrs)
>>
>> attrs is passed into the update function:
>>
>>   def update(self, field_values):
>>      #  logger code----------
>>      self.logfile.write('\n')
>>      self.logfile.write('%s,'%(str(strftime("%Y-%m-%d %H:%M:%S",
>> localtime()))))
>>      for k,v in self.fields.items():
>>          f = field_values.get(k, None)
>>          if f:
>>              self.logfile.write('%s,'%(str(f)))
>>       #end logger code ----------------
> 
> Why keep the logfile open? Why not store the logfile name
> (as selected by the user) and open the file in the update
> function and close it after writing?
> 
>> so i have the logfile.txt formatted in a was that is very useful for
>> later inspection with time series analysis.
> 
>> ...  up near the main class constructor the log file is opened like
>> this:
>>
>>      self.logfile = open('logfile.txt', 'a')
> 
> This is the bit I don't understand. Is there a reason you are
> using a hard coded filename? That is the source of much of
> your complexity. And why are you opening it so early? Why
> not wait till you need to write?
> 
>> it is stored in the home folder
> 
> That's a choice you make with the hard coded name. If you
> allow the user to select the name (and folder) it can
> be stored anywhere. The user choice can be via a GUI
> or via a config file or via a startup argument.
> 
>> and gets appended every time the program
>> is run.  Only problem is i need multiple log files
> 
> Why do you need multiple log files?
> Is there one per user and you have multiple users?
> Or do you need multiple files per user?
> Or do you only "need" multiple files because you hardcoded
> the name and then allow the user to choose their own name?
> 
>> and the user needs to
>> be able to choose the file from the UI.
> 
> Are you saying the user needs to be able to rename
> the file in the middle of processing? If so thats
> an unusual requirement but there are ways of doing
> it that are easier than what you seem to be doing.
> 
>>      def openFile(self, evt):
>>          with wx.FileDialog(self, "Choose a file", os.getcwd(), "",
>> "*.txt*", wx.OPEN) as dlg:
>>             if dlg.ShowModal() == wx.ID_OK:
>>                  path = dlg.GetPath()
>>                  mypath = os.path.basename(path)
>>                  uf = open(mypath, "a")
>>                  uf.write(self.logfile)
> 
> And as we've pointed out you can't write the logfile
> *object* you need to read the data and then write it out.
> And to read it you need to mess with the cursor using seek()
> But if you have already closed the file due to the way
> you opened it with 'with' you won;t be able to seek or write.
> Hence, I suspect, the error message.
> 
>> I have been unable to get the logfile.txt written
>> into the user opened file.
> 
> But I'd really like to understand why you believe this
> is needed. Why not just get the user to specify the file
> up front and then do all the logging into that.
> 
> 
> PS. In the update function you have this:
> 
>>      self.logfile.write('\n')
>>      self.logfile.write('%s,'%(str(strftime("%Y-%m-%d %H:%M:%S",
>> localtime()))))
> 
> It might help if you separate out the string formatting from the file
> writing
> 
> output = ('\n')
> output += ('%s,'%(str(strftime("%Y-%m-%d %H:%M:%S",localtime()))))
> self.logfile.write(output)
> 
> And hopefully this makes it obvious that the formatting is overly
> complex you don't need all the string conversion stuff. It can
> just be:
> 
> )
> output = '\n%s,' % strftime("%Y-%m-%d %H:%M:%S",localtime())
> self.logfile.write(output)
> 

> Why keep the logfile open? Why not store the logfile name
> (as selected by the user) and open the file in the update
> function and close it after writing?

if the user selected file is opened by the click event in:

    # open file dialog
    def openFile(self, evt):

then how i can i open it in :

    def update(self, field_values):

!?!?

> This is the bit I don't understand. Is there a reason you are
> using a hard coded filename? That is the source of much of
> your complexity. And why are you opening it so early? Why
> not wait till you need to write?

When i first started this project i wanted to be able to have a look at
the file so i could get the format proper. So this was the easiest way
to write the file.  since then I have tried to open it in other places
and it simply doesn't work. I suspect because it needs to be available
to the main class to write in the home folder or be available across
defs in order to copy it to the user selected file.

> Why do you need multiple log files?

There are multiple frequencies that I need to log data from. And
multiple geographic locations. and multiple time periods.

> Are you saying the user needs to be able to rename
> the file in the middle of processing? If so thats
> an unusual requirement but there are ways of doing
> it that are easier than what you seem to be doing.

no i am not saying the user needs to be able to rename in the middle of
processing.  what i am saying is that the user needs to be able to
choose the file that the 'logfile.txt' is copied to.

> But I'd really like to understand why you believe this
> is needed. Why not just get the user to specify the file
> up front and then do all the logging into that.

that sounds ideal, but i have not found a way to pass the user selected
file into the loop that writes the data without opening another file.

> And hopefully this makes it obvious that the formatting is overly
> complex you don't need all the string conversion stuff.

i will try making the code more pretty after i get it functional.

All that being said. . . i think shutil may the way to go here even
though it creams existing files.  so if open self.logfile like this:

    self.logfile = tempfile.mktemp('.txt)

how would i use shutil to copy the tempory self.logfile into the user
selected file in this open fileFile()?:

    # open file dialog
    def openFile(self, evt):
        with wx.FileDialog(self, "Choose a file", os.getcwd(), "",
"*.txt*", wx.OPEN) as dlg:
           if dlg.ShowModal() == wx.ID_OK:
                path = dlg.GetPath()
                mypath = os.path.basename(path)
                with open('mypath', 'a') as handle:

So 'handle' is the user opened file, i set the dialog to only allow
opening .txt files.  how can i use the shutil.copyfile() to copy the
temporary .txt file into the user opened .txt file?
Thanks!