shared memory pointer

[Tim Golden]

Tim wrote:
> I saw the mmap function in the shared memory example. I had some
> concern with my large memory size being written to the disk drive. I
> though it might slow down my application. The reason I am writting
> this new code is because the existing method using a file. I thought
> shared memory would be much faster.
> 
> I have tried using the memcpy and addresses, casting, etc. Is there a
> way to use memcpy with the pData returned by MapViewOfFile()?

Tim, let's try to back up on this a moment. As I understand, you're
running under Windows and you want to use shared memory, presumably
between two processes. You know about the CreateFileMapping/MapViewOfFile
API, possibly from this example on MSDN [1] and you want to use that
technique in Windows *without* an explicit file backing.

Am I right so far?

I suggested the mmap module, and you seemed to pick up on it
and be trying to use both your ctypes solution *and* the mmap
module as two halves of the same mechanism. Maybe I misunderstood,
but that's what it looked like.

Then you asked a question about getting hold of a Python object's
memory address to be able to pass it into your ctypes solution.
(Which makes sense, given the nature of that solution).

What you seem to have missed is that your ctypes code is doing
*exactly* what the mmapmodule.c code (which is the implementation
of the mmap module) is doing for you behind the scenes.

Here's your code (very slightly reformatted):

<code>
szName = c_char_p(name)
hMapObject = windll.kernel32.CreateFileMappingA(
   INVALID_HANDLE_VALUE,
   None,
   PAGE_READONLY,
   0,
   TABLE_SHMEMSIZE,
   szName
)
if (hMapObject == 0):
   print "OpenKey: Could not open name file mapping object"
   raise WinError()

self.pData = windll.kernel32.MapViewOfFile(
   hMapObject,
   FILE_MAP_ALL_ACCESS,
   0,
   0,
   TABLE_SHMEMSIZE
)

</code>

and here's the code from mmapmodule.c (also reformatted
and snipped about):

<code>
m_obj->map_handle = CreateFileMapping(
   m_obj->file_handle,
   NULL,
   flProtect,
   size_hi,
   size_lo,
   m_obj->tagname
);
if (m_obj->map_handle != NULL) {
   m_obj->data = (char *) MapViewOfFile(
     m_obj->map_handle,
     dwDesiredAccess,
     0,
     0,
     0
);

</code>

I hope you can see that they're basically doing the same
thing. (given the appropriate parameters). The only
clear difference is that the final param to MapViewOfFile
is 0 in the Python code, which the MSDN docs[2] indicate
"If this parameter is 0 (zero), the mapping extends
from the specified offset to the end of the file mapping."
It's not clear from that how it applies to a non-file-backed
FileMapping, but I *assume* that the Python devs have tested
that out.

In short, to have two Python processes talk via shared
memory (not linking to a specific file) the following
works for me:

<code-a>
import mmap

#
# The 0 special file value can be -1 in Python 2.5
#
shmem = mmap.mmap (0, 1000, "TJG", mmap.ACCESS_WRITE)
shmem.write ("blah blah")
</code-a>

<code-b>
import mmap

shmem = mmap.mmap (0, 1000, "TJG", mmap.ACCESS_WRITE)
print shmem.read (9)
</code>

Obviously, I've used dummy codes and values, but since
the .write and .read (and the other helper methods)
use strings, you can always pickle or marshal arbitrary
Python data to move it around.

I hope all that's helpful; if nothing else, it's given
me some exercise in reading the code of the stdlib, which
can't be bad!

TJG

[1] http://msdn2.microsoft.com/en-us/library/aa366551.aspx
[2] http://msdn2.microsoft.com/en-us/library/aa366761.aspx