[Tutor] reading output from a c executable.

[bob gailer]

Ravi Kondamuru wrote:
> Denis, These are 32bit, 64bit counters (essentially numbers).
> Bob, There are well over 10K counters in the log file that are updated 
> every 5 secs. If a counter1's graph was requested, log will have to be 
> parsed once to get the data points. If a user asked for a counter2, 
> now it needs to be retrieved from the log file. Which essentially 
> means having to go through the logfile again. This is because i want 
> to try to avoid using a database to store values after parsing the file.
Here is a little test program I wrote to check timing on a file of 
*100,000* 64 bit counters.

import time, struct
ctrs = 100000
s = time.time()
# create a file of 800,000 characters (100,000 64 bit numbers)
f = open('ravi.dat', 'wb')
for i in range(ctrs):
  f.write('abcdefgh') # it does not matter what we write as long as it 
is 8 bytes.
print time.time() - s

s = time.time()
l = []
# read the file
f = open('ravi.dat', 'rb').read()

# unpack each 8 characters into an integer and collect in a list
for b in range(0, ctrs, 8):
  n = struct.unpack('q', f[b:b+8])
  l.append(n)
print time.time() - s

Writing the file took 0.14 seconds on my computer
Reading and unpacking 0.04 seconds.
I think you can set performance issues aside.

There is a principle in programming that proposes:
1 - get a program running. Preferably in Python as development time is 
much faster.
2 - check its performance.
3 - refine as needed.

-- 
Bob Gailer
Chapel Hill NC 
919-636-4239