Parallel(?) programming with python

[avi.e.gross at gmail.com]

Stefan,

You are correct that the goal of a lock is to do something rather quickly
and atomically, so your design should not do something complex or long
before releasing the lock.

In your example, you have a producer adding data as regularly as every
second and another that wakes up rarely and processes all the data since the
last time. So you may want to augment the code you had to do something fast
like point another variable at the data gathered so far and move the
original variable to an empty list or whatever. Then you release the lock
within fractions of a second and let the regular job keep adding to the
initially empty list while the other part of the code processes without a
lock. 

A design like the above has the busy worker constantly checking the lock. An
alternative if you are sure the other process will only show up almost
exactly at 6 hours on the clock, is to have the busy one check the time
instead, but that may be more expensive. 

Still other architectures are possible, such as writing to not a single list
for six hours, but some data structure with multiple sub-lists  such as one
where you switch every minute or so. The second process can note how many
entries there are at the moment, and does all but the last and notes the
location so the next time it starts there. This would work if you did not
need every last bit of data as the two do not interfere with each other. And
no real locks would be needed as the only thing the two parts share is the
position or identity of the current last fragment which only one process
actually touches.

Just some ideas. Lots of other variations are very possible.



-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com at python.org> On
Behalf Of Stefan Ram
Sent: Monday, August 8, 2022 7:21 AM
To: python-list at python.org
Subject: Re: Parallel(?) programming with python

Andreas Croci <andrea.croci at gmx.de> writes:
>Basically the question boils down to wether it is possible to have 
>parts of a program (could be functions) that keep doing their job while 
>other parts do something else on the same data, and what is the best 
>way to do this.

  Yes, but this is difficult. If you ask this question here,
  you might not be ready for this.

  I haven't learned it yet myself, but nevertheless tried to
  write a small example program quickly, which might still
  contain errors because of my lack of education.

import threading
import time

def write_to_list( list, lock, event ):
    for i in range( 10 ):
        lock.acquire()
        try:
            list.append( i )
        finally:
            lock.release()
        event.set()
        time.sleep( 3 )

def read_from_list( list, lock, event ):
    while True:
        event.wait()
        print( "Waking up." )
        event.clear()
        if len( list ):
            print( "List contains " + str( list[ 0 ]) + "." )    
            lock.acquire()
            try:
                del list[ 0 ]
            finally:
                lock.release()
        else:
            print( "List is empty." )

list = []
lock = threading.Lock()
event = threading.Event()
threading.Thread( target=write_to_list, args=[ list, lock, event ]).start()
threading.Thread( target=read_from_list, args=[ list, lock, event ]).start()

  In basketball, first you must learn to dribble and pass,
  before you can begin to shoot.

  With certain reservations, texts that can be considered
  to learn Python are:

"Object-Oriented Programming in Python Documentation" - a PDF file,
Introduction to Programming Using Python - Y Daniel Liang (2013), How to
Think Like a Computer Scientist - Peter Wentworth (2012-08-12), The Coder's
Apprentice - Pieter Spronck (2016-09-21), and Python Programming - John
Zelle (2009).


--
https://mail.python.org/mailman/listinfo/python-list