shared memory pointer

[Tim]

Wow....You put some detail in this response. Thanks so much!

I think I want to stay away from mmap because it uses the disk to
store my memory. I am trying to stay away from that. I am building
strip charts in this python project to view my data. Currently, I am
using a file and I have to open the file when my simulation is
complete. The simulation takes too long to execute when it has to dump
all of the data to a file. The simulation is written in C++ but I want
the strip charts to be written in python so I can using the free plots
in matplotlib. I am trying to place the very large data block in
shared memory so I don't have to wait for the data to be written to a
disk.

I have about 200 shared memory segments that the simulation creates. I
did not know that my code would write to the hard drive because I use
ctypes. Could you explain that. Even if I use mmap, how could I mmap
200 shared memory segments? The first argument to mmap is a zero. How
does mmap know to map the returned object from MapViewOfFile to the
tag?

I have been looking into using a dll to do all of the mappings. Then,
I can call the dll from my python script to access my array of floats.
What do you think of this approach? I am having trouble finding good
documentation of this approach. Can you help?


Thanks


On Sep 11, 7:58 am, Tim Golden <m... at timgolden.me.uk> wrote:
> Tim wrote:
> > I saw the mmap function in the shared memory example. I had some
> > concern with my large memory size being written to the disk drive. I
> > though it might slow down my application. The reason I am writting
> > this new code is because the existing method using a file. I thought
> > shared memory would be much faster.
>
> > I have tried using the memcpy and addresses, casting, etc. Is there a
> > way to use memcpy with the pData returned by MapViewOfFile()?
>
> Tim, let's try to back up on this a moment. As I understand, you're
> running under Windows and you want to use shared memory, presumably
> between two processes. You know about the CreateFileMapping/MapViewOfFile
> API, possibly from this example on MSDN [1] and you want to use that
> technique in Windows *without* an explicit file backing.
>
> Am I right so far?
>
> I suggested the mmap module, and you seemed to pick up on it
> and be trying to use both your ctypes solution *and* the mmap
> module as two halves of the same mechanism. Maybe I misunderstood,
> but that's what it looked like.
>
> Then you asked a question about getting hold of a Python object's
> memory address to be able to pass it into your ctypes solution.
> (Which makes sense, given the nature of that solution).
>
> What you seem to have missed is that your ctypes code is doing
> *exactly* what the mmapmodule.c code (which is the implementation
> of the mmap module) is doing for you behind the scenes.
>
> Here's your code (very slightly reformatted):
>
> <code>
> szName = c_char_p(name)
> hMapObject = windll.kernel32.CreateFileMappingA(
>    INVALID_HANDLE_VALUE,
>    None,
>    PAGE_READONLY,
>    0,
>    TABLE_SHMEMSIZE,
>    szName
> )
> if (hMapObject == 0):
>    print "OpenKey: Could not open name file mapping object"
>    raise WinError()
>
> self.pData = windll.kernel32.MapViewOfFile(
>    hMapObject,
>    FILE_MAP_ALL_ACCESS,
>    0,
>    0,
>    TABLE_SHMEMSIZE
> )
>
> </code>
>
> and here's the code from mmapmodule.c (also reformatted
> and snipped about):
>
> <code>
> m_obj->map_handle = CreateFileMapping(
>    m_obj->file_handle,
>    NULL,
>    flProtect,
>    size_hi,
>    size_lo,
>    m_obj->tagname
> );
> if (m_obj->map_handle != NULL) {
>    m_obj->data = (char *) MapViewOfFile(
>      m_obj->map_handle,
>      dwDesiredAccess,
>      0,
>      0,
>      0
> );
>
> </code>
>
> I hope you can see that they're basically doing the same
> thing. (given the appropriate parameters). The only
> clear difference is that the final param to MapViewOfFile
> is 0 in the Python code, which the MSDN docs[2] indicate
> "If this parameter is 0 (zero), the mapping extends
> from the specified offset to the end of the file mapping."
> It's not clear from that how it applies to a non-file-backed
> FileMapping, but I *assume* that the Python devs have tested
> that out.
>
> In short, to have two Python processes talk via shared
> memory (not linking to a specific file) the following
> works for me:
>
> <code-a>
> import mmap
>
> #
> # The 0 special file value can be -1 in Python 2.5
> #
> shmem = mmap.mmap (0, 1000, "TJG", mmap.ACCESS_WRITE)
> shmem.write ("blah blah")
> </code-a>
>
> <code-b>
> import mmap
>
> shmem = mmap.mmap (0, 1000, "TJG", mmap.ACCESS_WRITE)
> print shmem.read (9)
> </code>
>
> Obviously, I've used dummy codes and values, but since
> the .write and .read (and the other helper methods)
> use strings, you can always pickle or marshal arbitrary
> Python data to move it around.
>
> I hope all that's helpful; if nothing else, it's given
> me some exercise in reading the code of the stdlib, which
> can't be bad!
>
> TJG
>
> [1]http://msdn2.microsoft.com/en-us/library/aa366551.aspx
> [2]http://msdn2.microsoft.com/en-us/library/aa366761.aspx