Multithreading? How?

pozz pozzugno at gmail.com
Fri Apr 28 06:31:36 EDT 2023


I need to develop a Python application that is a sort of gateway between 
to networks: a "serial bus" network (think of a serial port or a USB 
connection) and a MQTT connection (in my case, it's AWS IoT service).

On the bus, a proprietary protocol is implemented. From the bus, the app 
knows the status of the system (think of a simple ON/OFF status).
The user can retrieve the status of the system through MQTT: it sends a 
message to read/status MQTT topic and receive back a message with the 
current status on read/status/reply MQTT topic. Of course, they are just 
examples.

On the contrary, when the app detects a status change reported from the 
serial bus (think of a transition from ON to OFF), it sends a MQTT message.

I'm thinking to split the application in three classes: Bus, App and 
IoT. Bus and IoT are also threads.
The thread of Bus manages the serial protocol, while the thread of IoT 
manages MQTT connection with the broker (the server).

However I don't know if it could be a good architecture. Suppone Bus 
thread receives a new status from the system. In the context of 
ThreadBus, the object Bus could call a method of App object:

   app.set_status(new_status_from_the_bus)

In the App I have:

   class App():
     ..
     def set_status(new_status):  # Could be called from ThreadBus
       if new_status != self.new_status:
         self.new_status = new_status
         # Do some actions on status change

     def get_status():            # Could be called from ThreadIoT
       return self.status

Of course, IoT object needs to know the current status of the system 
when a message is received from MQTT. So ThreadIoT could call 
app.get_status().

I think this architecture has some problems with race conditions or 
threads synchronization. What happens if IoT calls get_status() exactly 
when set_status() called by ThreadBus is executing? If status is a big 
data structure, set_status() could be interrupted by get_status() that 
could get a completely corrupted status, because it was only partly 
updated by set_status().

I know I can use locks or semaphores in get_status() and set_status(), 
but I don't know if this is a good approach. Consider that the system is 
complex, it isn't composed by a simple single status. It has many many 
parameters that are collected from the serial bus. Should I use a lock 
for every [get|set]_status(), [get|set]_dimensions(), 
[get|set]_battery_level(), [get|set]_mains_present(), and so on?


Another possibility is to use a Queue for Bus and a Queue for IoT. So 
the set_status(new_status) called from Bus object will be transformed in 
a put in the queue:

   app_queue.put({"type": "new_status", "data": ...})

However how could be transformed the get_status() from IoT? How the 
return value (the current status) is real retrieved?

   class IoT():
     ..
     def status_request_from_MQTT():
       app_queue.put({"type": "get_status"})
       # How to get the status to return?
       return current_status

Should the app put the status on the same queue and should IoT waits for 
a new message in the Queue?

     def status_request_from_MQTT():
       app_queue.put({"type": "get_status"})
       try:
         current_status = app_queue.get(timeout=10)
       except Empty:
         # What to do?
       return current_status


Again another approach is to avoid multi-threading at all and create a 
single "main loop" function that waits at the same time for incoming 
events on the serial bus and MQTT (how?). I don't know if this could be 
done in my case, because I'm using awscrt Python module and it works 
through callbacks that I think is called from another thread.


Any suggestions on this architecture?


More information about the Python-list mailing list