A design problem

[Paddy]

On Jan 31, 3:57 am, "Dan Upton" <up... at virginia.edu> wrote:
> Or: How to write Python like a Python programmer, not a Java
> programmer.  This will be a little long-winded...
>

> ...
>
> and so on.  The way I was going to approach it was to every time
> through the loop, read the data for one of the processes, open its
> file, write out to it, and close it, then do the same for the next
> process, and so on.  Really though I don't need to be able to look at
> the data until the processes are finished, and it would be less I/O,
> at the expense of memory, to just save all of the lists of data as I
> go along and then dump them out to disk at the end of the Python
> program's execution.  I feel like Python's lists or dictionaries
> should be useful here, but I'm not really sure how to apply them,
> particularly in a "Python-like" way.
> Thanks,
> -dan

For 'reasonable' n you might want to go with something like your
original idea except don't close the files until  the end:
  * Open a file for each process
  * loop:
    * get data for all processes
    * Write one record to each file
    * Optionally flush the file for data integrity
    * sleep for some time
  * close all files.
If you flush then you get some data back if the program fails part
way. If you don't then you may get some data. If you kept all the data
in memory then on a crash, it would all be gone.
The above is close to what you were originally thinking so should be
easier for you to code and see if its performance is adequate - You
automatically take advantage of the OS and disk caches this way too.

- Paddy.