Parallel(?) programming with python

[Louis Krupp]

On 8/8/2022 4:47 AM, Andreas Croci wrote:
> tI would like to write a program, that reads from the network a fixed 
> amount of bytes and appends them to a list. This should happen once a 
> second.
>
> Another part of the program should take the list, as it has been 
> filled so far, every 6 hours or so, and do some computations on the 
> data (a FFT).
>
> Every so often (say once a week) the list should be saved to a file, 
> shorthened in the front by so many items, and filled further with the 
> data coming fom the network. After the first saving of the whole list, 
> only the new part (the data that have come since the last saving) 
> should be appended to the file. A timestamp is in the data, so it's 
> easy to say what is new and what was already there.
>
> I'm not sure how to do this properly: can I write a part of a program 
> that keeps doing its job (appending data to the list once every 
> second) while another part computes something on the data of the same 
> list, ignoring the new data being written?
>
> Basically the question boils down to wether it is possible to have 
> parts of a program (could be functions) that keep doing their job 
> while other parts do something else on the same data, and what is the 
> best way to do this.

You might be able to do what you need by making the file system work for 
you:

Use numbered files, something like DATA/0001, DATA/0002, etc.

Start by initializing a file number variable to 1 and creating an empty 
file, DATA/0001. The current time will be your start time.

In an infinite loop, just as in Stefan's example:

Read from the network and append to the current data file. This 
shouldn't take long unless the file is on a remote system.

If six hours have gone by (compare the current time to the start time), 
close the current date file, create a thread (see Stefan's example) to 
call your FFT with the name of the current file, increment the file 
number, and open a new empty data file.

If you want to, you can consolidate files every week or so. The Python 
library has functions that will let you get a list files in a directory. 
If you're on a Linux or UNIX system, you can use shell commands to 
append, copy or rename files.

Have fun.

Louis