python notifying calling script that multiprocessing tasks have finished at lower level scripts

[Daiyue Weng]

Hi, I have a master script that executes two sequences (lists) of child
scripts, i.e. script_1 to script_3, and script_4 to_script_6 (the structure
is attached as a png file).

The execution is sequential, e.g. running script_1, then 2 then 3.

After executing the 1st sequence (script_1 to 3), master will execute the
2nd sequence (script_4 to 6).

Each child script will be calling a multiprocessing function to process a
task.

Master script is like,

for seq in seqs_to_launch:

    for script in seq:

        script().execute(data)



Each child script is like,

import multi_process_update

class Script_n():

    def execute(self, data):
        # some data processing

        multi_process_task(data_task, task_name)



The multiprocessing function is like,

def multi_process_task(tasks, task_name):

    cores_to_use = how_many_core()

    handler = task_handling(task_name)  # task handling class

    task_blocks = slice_list(tasks, cores_to_use)

    for block in task_blocks:
        # spawn processes for each row block assigned to every cpu core
        p = multiprocessing.Process(target=handler.execute, args=(block,))
        p.start()


I like to know how to notify the master when the 2nd sequence is finished
that all multiprocess tasks (seq 1 and 2) are completed, and then maybe
close the multiprocessing.

thanks