Data unchanged when passing data to Python in multiprocessing shared memory

[Avi Gross]

Jen,

I would not be shocked at incompatibilities in the system described making it hard to exchange anything, including text, but am not clear if there is a limitation of four bytes in what can be shared. For me, a character string can use any number of contiguous bytes in memory that some kind of pointer or table lookup provides access to. 

Clearly you could squeeze larger number in by not writing decimals but using hexadecimal notation that adds a to f as valid entries, and of course you can make your own customized base 32 or if you use more symbols (such as upper and lower case as different) you could  add a few symbols and stretch it out to base 60 or 64.

Like I said, I may have missed something. If you KNOW that system A will interact with systems B and C and so on, and each machine knows what kind it is, and especially if it knows which kind of machine left it something in shared memory, there has to be a way to coordinate between them. If character strings in ASCII or UTF8 or EBCDIC or some kind of raw are allowed, and used with just a few characters that can spell out numbers, I would think much is possible. If needed, perhaps a fixed size could be set aside and the string use a null terminator if shorter. 

Of course if this big-endian issue also scrambles bytes used in strings, forget it.

Or, maybe shared memory is not the easy way to go, even it it might be faster.



-----Original Message-----
From: Jen Kris <jenkris at tutanota.com>
To: Avi Gross <avigross at verizon.net>
Cc: python-list at python.org <python-list at python.org>
Sent: Wed, Feb 2, 2022 1:27 pm
Subject: Re: Data unchanged when passing data to Python in multiprocessing shared memory



An ASCII string will not work.  If you convert 32894 to an ascii string you will have five bytes, but you need four.  In my original post I showed the C program I used to convert any 32-bit number to 4 bytes.  






Feb 2, 2022, 10:16 by python-list at python.org:

I applaud trying to find the right solution but wonder if a more trivial solution is even being considered. It ignores big and little endians and just converts your data into another form and back.



If all you want to do is send an integer that fit in 32 bits or 64 bits, why not convert it to a character string in a form that both machines will see the same way and when read back, convert it back to an integer? 



As long as both side see the same string, this can be done in reasonable time and portably.



Or am I missing something? Is "1234" not necessarily seen in the same order, or "1.234e3" or whatever?



Obviously, if the mechanism is heavily used and multiple sides keep reading and even writing the same memory location, this is not ideal. But having different incompatible processors looking at the same memory is also not.



-----Original Message-----

From: Dennis Lee Bieber <wlfraed at ix.netcom.com>

To: python-list at python.org

Sent: Wed, Feb 2, 2022 12:30 am

Subject: Re: Data unchanged when passing data to Python in multiprocessing shared memory





On Wed, 2 Feb 2022 00:40:22 +0100 (CET), Jen Kris <jenkris at tutanota.com>



declaimed the following:






breakup = int.from_bytes(byte_val, "big")



>print("this is breakup " + str(breakup))





>Python prints:  this is breakup 32894





>Note that I had to switch from little endian to big endian.  Python is little endian by default, but in this case it's big endian.  





    Look at the struct module. I'm pretty certain it has flags for big or



little end, or system native (that, or run your integers through the



various "network byte order" functions that I think C and Python both



support.







https://www.gta.ufrj.br/ensino/eel878/sockets/htonsman.html











>However, if anyone on this list knows how to pass data from a non-Python language to Python in multiprocessing.shared_memory please let me (and the list) know.  







    <pondering>MMU cache lines not writing through to RAM? Can't find



anything on Google to force a cache flush</pondering> Can you test on a



different OS? (Windows vs Linux)















-- 



    Wulfraed                 Dennis Lee Bieber         AF6VN



    wlfraed at ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/



-- 



https://mail.python.org/mailman/listinfo/python-list



-- 

https://mail.python.org/mailman/listinfo/python-list