File IO question.

[J Poirier]

On Thu, 15 Jul 2004 12:38:36 -0500, Larry Bates
<lbates at swamisoft.com> wrote:

> You answered your own question:
>
>> I'm guessing the hard drives would fill up
eventually
>> anyway regardless of the size of the hard drive,
the
>> crashes just wouldn't occur quite as often.
>
> If this statement is true, compressing the files
won't
> help either as that is just the same as a larger
hard
> drive.
>
> But, you could limit the size and number of trace
files
> and periodically throw away the oldest ones to keep
the
> trace files at a nearly constant size.  I just can't
tell
> from your description exactly what is getting
written to
> the trace file that makes it so large (debugging
info?).
> Maybe you could only log errors/warnings to the
trace
> file so it grows more slowly?
>
> Larry Bates
> Syscon, Inc.
>
> "J Poirier" <oo1z at yahoo.com> wrote in message
>
news:mailman.445.1089911565.5135.python-list at python.org...
>> Hi All,
>>
>> I'm hoping that someone might have some pointers or
>> examples on how to proceed with a solution to the
>> following problem:
>>
>> A test application, which produces a trace file, is
>> being run for very long periods of time. Say 72
hours
>> or more.
>>
>> The application is often running on older PCs that
>> have relatively small hard drives in comparison to
>> to how big the trace file can become.
>>
>> The trace files tend to accumulate as they're not
>> always deleted once archived to a network database,
>> and often times a machine will crash in the middle
of
>> a
>> very long test due to the hard drive filling up.
>>
>> There are dozens of PCs being used for the tests so
>> buying bigger hard drives isn't really feasible.
And
>> I'm guessing the hard drives would fill up
eventually
>> anyway regardless of the size of the hard drive,
the
>> crashes just wouldn't occur quite as often.
>>
>> The nice thing is that the trace files compress
quite
>> well.
>>
>> I messed around with the mmap and file object stuff
>> as well as the win32 extensions thinking that I
could
>> extract and compress the data that was being
written
>> to
>> the trace file, by the application, in chuncks.
>>
>> Although I was able to get it to work on a
contrived
>> setup, it didn't work when used with the real
>> application.
>>
>> Any hints on how to get something similar to the
above
>> to work or recommendations on alternate solutions
>> would
>> be *greatly* appreciated.
>>
>> Thanks,
>> Joe

The files contain lots of detailed mobile info
(RLC/MAC, etc) from GSM and WCDMA networks.

Running a test involves running through a bunch of
test
cases.

All test cases must finish without interruption
for the trace data to be considered valid. If an
interruption does occur the entire test must be 
re-started from the beginning. 

It's easy to have a trace that's several hundred meg
to
several gig for a test that's being run for several
days. 

A trace file can usually be compressed to a tenth of
its original uncompressed size. So removing the info
from a trace file that the application is writing to,
while it's still periodically being written to by the
test application, and compressing the piped out trace
data, seemed like the way to go. 

I was unable to retrieve any of the trace data from
a file that was created by the application, and the
application wouldn't write any trace info to a file
created by my script, while the script was accessing
the file.

Ultimately at some point old trace files need to be
deleted from the hard drive, but that doesn't help 
with handling the size of the trace file. 

If a file's data could be piped to a compressed file
as the trace was being taken, more comprehensive test
cases could be run at a single session.

E.g., a 7 day test would currently be out of the
question because it generates a 25 gig trace. But if
the trace file's contents was piped to a compressed 
file it would only end up taking up 2.5 gig of space
once the test completed.

Unfortunately my script was unable to process a trace
file created by the application and the application
wouldn't write to a file that my script created while
the script was accessing the file.
 

-Joe 


		
__________________________________
Do you Yahoo!?
Read only the mail you want - Yahoo! Mail SpamGuard.
http://promotions.yahoo.com/new_mail