ob_type in shared memory

[Aaron Brady]

Hi Mark, nice to have your comment.

On Jan 25, 9:47 pm, Mark Wooding <m... at distorted.org.uk> wrote:
> Aaron Brady <castiro... at gmail.com> writes:
snip
> > Object 'A' is of type 'Ta'.  When process 'P' is looking at it, it
> > needs to have an 'ob_type' that is 'Ta' as process 'P' sees it.  When
> > process 'Q' is looking at it, it needs to have an 'ob_type' that is
> > 'Ta' as process 'Q' sees it.  If it referred to 'Ta' in process 'P'
> > when 'Q' was calling it, 'Q' would have to access memory that is in
> > another process.
>
> I see.  My immediate reaction was to suggest that you just put the
> PyTypeObject in shared memory -- but then I realised that the shared
> memory region will probably be in different addresses in the two
> processes.

Yes, exactly.

> I'll assume that you've got enough synchronization between
> the processes involved to stop everything from turning to mush.

Have at thee, synchronization!  Yes, it is possible theoretically.
(By the way, did you know you can detect deadlock before it strikes
with a simple breadth-first search?)  The idea is still not out of the
proof-of-concept stage-- the concept not having been proved.

> > Therefore, I need a field and an array.  The field indicates which
> > type should be loaded, and the array contains the types.  Quick
> > example:
>
> > PyTypeObject* array_of_types[]= { &SharedList, &SharedTuple };
>
> > Then, when a list is being accessed, it can set its own 'ob_type'
> > field to 'array_of_types[ 0 ]', and similarly for a tuple.
>
> > However, I'm having trouble getting 'array_of_types' in the right
> > module during compilation.  My question is: Where do 'array_of_types'
> > and the forward declarations for the types go?
>
> I'm not sure I understand the difficulty.  They'll want to go in your C
> module somewhere, but as long as SharedList and SharedTuple are either
> in the same source file or not declared `static' you just write the
> definition you've got somewhere after declaring or defining the actual
> types in question.

The problem I ran into was that different modules (C files) were
seeing different versions of the structure.  '&SharedList' showed up
as different addresses in the debugger!  I might be missing something,
but I think the forward declaration got interpreted as an actual
declaration in the different files that included the header.
Mysteriously, it disappeared when I tried it again now.  It's still a
possibility.

The solution I looked at today was declare the array like this:

PyTypeObject* keep_types[];

It's even ok in a shared header (multiply included).  Then the actual
definition just gives it NULLs and a size, which are filled in as the
needed modules are loaded.  That solves the (additional) problem of
not having called the owner module's initialization code.

The catch is that if you have an object in a SharedList, you need to
have imported its module before you load it.  However 'pickle' has
that problem too, so I don't feel bad.

> There's a comment in Extending and Embedding (2.1) about initializing
> PyTypeObjects:
>
> :          PyObject_HEAD_INIT(NULL)
> :
> : This line is a bit of a wart; what we'd like to write is:
> :
> :          PyObject_HEAD_INIT(&PyType_Type)
> :
> : as the type of a type object is "type", but this isn't strictly
> : conforming C and some compilers complain.
>
> The comment here is wrong: &PyType_Type is a perfectly good constant
> expression as far as C is concerned, but alas Microsoft's dynamic
> linking system isn't clever enough to cope with it even so (because it's
> in a separate link unit).  But this isn't a problem in our case:
> presumably SharedList and SharedTuple are in the same module, so this
> should all just work.

Here is another quote from the docs:

"Portability therefore requires not to make any assumptions about
symbol visibility. This means that all symbols in extension modules
should be declared static, except for the module’s initialization
function, in order to avoid name clashes with other extension modules
(as discussed in section The Module’s Method Table and Initialization
Function)."

All declared static?  Dear.  So I don't know what to make of it,
especially given the need to call 'PyType_Ready'.