Data unchanged when passing data to Python in multiprocessing shared memory

[Jen Kris]

Barry, thanks for your reply.  

On the theory that it is not yet possible to pass data from a non-Python language to Python with multiprocessing.shared_memory, I bypassed the problem by attaching 4 bytes to my FIFO pipe message from NASM to Python:

byte_val = v[10:14]

where v is the message read from the FIFO.  Then:

 breakup = int.from_bytes(byte_val, "big")
print("this is breakup " + str(breakup))

Python prints:  this is breakup 32894

Note that I had to switch from little endian to big endian.  Python is little endian by default, but in this case it's big endian.  

However, if anyone on this list knows how to pass data from a non-Python language to Python in multiprocessing.shared_memory please let me (and the list) know.  

Thanks.  


Feb 1, 2022, 14:20 by barry at barrys-emacs.org:

>
>
>> On 1 Feb 2022, at 20:26, Jen Kris via Python-list <python-list at python.org> wrote:
>>
>> I am using multiprocesssing.shared_memory to pass data between NASM and Python.  The shared memory is created in NASM before Python is called.  Python connects to the shm:  shm_00 = shared_memory.SharedMemory(name='shm_object_00',create=False). 
>>
>> I have used shared memory at other points in this project to pass text data from Python back to NASM with no problems.  But now this time I need to pass a 32-bit integer (specifically 32,894) from NASM to Python. 
>>
>> First I convert the integer to bytes in a C program linked into NASM:
>>
>>  unsigned char bytes[4]
>>  unsigned long int_to_convert = 32894;
>>
>>  bytes[0] = (int_to_convert >> 24) & 0xFF;
>>  bytes[1] = (int_to_convert >> 16) & 0xFF;
>>  bytes[2] = (int_to_convert >> 8) & 0xFF;
>>  bytes[3] = int_to_convert & 0xFF;
>>  memcpy(outbuf, bytes, 4);
>>
>> where outbuf is a pointer to the shared memory.  On return from C to NASM, I verify that the first four bytes of the shared memory contain what I want, and they are 0, 0, -128, 126 which is binary 00000000 00000000 10000000 01111110, and that's correct (32,894). 
>>
>> Next I send a message to Python through a FIFO to read the data from shared memory.  Python uses the following code to read the first four bytes of the shared memory:
>>
>>  byte_val = shm_00.buf[:4]
>>  print(shm_00.buf[0])
>>  print(shm_00.buf[1])
>>  print(shm_00.buf[2])
>>  print(shm_00.buf[3])
>>
>> But the bytes show as 40 39 96 96, which is exactly what the first four bytes of this shared memory contained before I called C to overwrite them with the bytes 0, 0, -128, 126.  So Python does not see the updated bytes, and naturally int.from_bytes(byte_val, "little") does not return the result I want. 
>>
>> I know that Python refers to shm00.buf, using the buffer protocol.  Is that the reason that Python can't see the data that has been updated by another language? 
>>
>> So my question is, how can I alter the data in shared memory in a non-Python language to pass back to Python?
>>
>
> Maybe you need to use a memory barrier to force the data to be seen by another cpu?
> Maybe use shm lock operation to sync both sides?
> Googling I see people talking about using stdatomic.h for this.
>
> But I am far from clear what you would need to do.
>
> Barry
>
>>
>> Thanks,
>>
>> Jen
>>
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list
>>