Data unchanged when passing data to Python in multiprocessing shared memory

[Barry Scott]

> On 1 Feb 2022, at 23:40, Jen Kris <jenkris at tutanota.com> wrote:
> 
> Barry, thanks for your reply.  
> 
> On the theory that it is not yet possible to pass data from a non-Python language to Python with multiprocessing.shared_memory, I bypassed the problem by attaching 4 bytes to my FIFO pipe message from NASM to Python:
> 
> byte_val = v[10:14]
> 
> where v is the message read from the FIFO.  Then:
> 
> breakup = int.from_bytes(byte_val, "big")
> print("this is breakup " + str(breakup))
> 
> Python prints:  this is breakup 32894
> 
> Note that I had to switch from little endian to big endian.  Python is little endian by default, but in this case it's big endian.  
> 
> However, if anyone on this list knows how to pass data from a non-Python language to Python in multiprocessing.shared_memory please let me (and the list) know.  

By using the struct module you can control for endian of the data.

Barry

> 
> Thanks.  
> 
> 
> Feb 1, 2022, 14:20 by barry at barrys-emacs.org:
> 
> On 1 Feb 2022, at 20:26, Jen Kris via Python-list <python-list at python.org> wrote:
> 
> I am using multiprocesssing.shared_memory to pass data between NASM and Python. The shared memory is created in NASM before Python is called. Python connects to the shm: shm_00 = shared_memory.SharedMemory(name='shm_object_00',create=False). 
> 
> I have used shared memory at other points in this project to pass text data from Python back to NASM with no problems. But now this time I need to pass a 32-bit integer (specifically 32,894) from NASM to Python. 
> 
> First I convert the integer to bytes in a C program linked into NASM:
> 
> unsigned char bytes[4]
> unsigned long int_to_convert = 32894;
> 
> bytes[0] = (int_to_convert >> 24) & 0xFF;
> bytes[1] = (int_to_convert >> 16) & 0xFF;
> bytes[2] = (int_to_convert >> 8) & 0xFF;
> bytes[3] = int_to_convert & 0xFF;
> memcpy(outbuf, bytes, 4);
> 
> where outbuf is a pointer to the shared memory. On return from C to NASM, I verify that the first four bytes of the shared memory contain what I want, and they are 0, 0, -128, 126 which is binary 00000000 00000000 10000000 01111110, and that's correct (32,894). 
> 
> Next I send a message to Python through a FIFO to read the data from shared memory. Python uses the following code to read the first four bytes of the shared memory:
> 
> byte_val = shm_00.buf[:4]
> print(shm_00.buf[0])
> print(shm_00.buf[1])
> print(shm_00.buf[2])
> print(shm_00.buf[3])
> 
> But the bytes show as 40 39 96 96, which is exactly what the first four bytes of this shared memory contained before I called C to overwrite them with the bytes 0, 0, -128, 126. So Python does not see the updated bytes, and naturally int.from_bytes(byte_val, "little") does not return the result I want. 
> 
> I know that Python refers to shm00.buf, using the buffer protocol. Is that the reason that Python can't see the data that has been updated by another language? 
> 
> So my question is, how can I alter the data in shared memory in a non-Python language to pass back to Python?
> 
> Maybe you need to use a memory barrier to force the data to be seen by another cpu?
> Maybe use shm lock operation to sync both sides?
> Googling I see people talking about using stdatomic.h for this.
> 
> But I am far from clear what you would need to do.
> 
> Barry
> 
> Thanks,
> 
> Jen
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
>