[Python-Dev] advice needed: best approach to enabling "metamodules"?

[Nathaniel Smith]

On Sat, Nov 29, 2014 at 11:32 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 29 Nov 2014 01:59:06 +0000
> Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Option 1: Make it possible to change the type of a module object
>> in-place, so that we can write something like
>>
>>    sys.modules[__name__].__class__ = MyModuleSubclass
>>
>> Option 1 downside: The invariants required to make __class__
>> assignment safe are complicated, and only implemented for
>> heap-allocated type objects. PyModule_Type is not heap-allocated, so
>> making this work would require lots of delicate surgery to
>> typeobject.c. I'd rather not go down that rabbit-hole.
>
> Option 1b: have __class__ assignment delegate to a tp_classassign slot
> on the old class, so that typeobject.c doesn't have to be cluttered with
> many special cases.

I'm intrigued -- how would this help?

I have a vague impression that one could add another branch to
object_set_class that went something like

if at least one of the types is a subtype of the other type, and the
subtype is a heap type with tp_dealloc == subtype_dealloc, and the
subtype doesn't add any important slots, and ... then the __class__
assignment is legal.

(This is taking advantage of the fact that if you don't have any extra
slots added, then subtype_dealloc just basically defers to the base
type's tp_dealloc, so it doesn't really matter which one you end up
calling.)

And my vague impression is that there isn't really anything special
about the module type that would allow a tp_classassign function to
simplify this logic.

But these are just vague impressions :-)

>> Option 3: Make it legal to assign to the __dict__ attribute of a
>> module object, so that we can write something like
>>
>>    new_module = MyModuleSubclass(...)
>>    new_module.__dict__ = sys.modules[__name__].__dict__
>>    sys.modules[__name__].__dict__ = {}     # ***
>>    sys.modules[__name__] = new_module
>>
> [...]
>>
>> Option 4: Add a new function sys.swap_module_internals, which takes
>> two module objects and swaps their __dict__ and other attributes. By
>> making the operation a swap instead of an assignment, we avoid the
>> lifecycle pitfalls from Option 3. By making it a builtin, we can make
>> sure it always handles all the module fields that matter, not just
>> __dict__. Usage:
>
> How do these two options interact with the fact that module functions
> store their globals dict, not the module itself?

I think that's totally fine? The whole point of all these proposals is
to make sure that the final module object does in fact have the
correct globals dict.

~$ git clone git at github.com:njsmith/metamodule.git
~$ cd metamodule
~/metamodule$ python3.4
>>> import examplepkg
>>> examplepkg
<FancyModule 'examplepkg' from '/home/njs/metamodule/examplepkg/__init__.py'>
>>> examplepkg.f.__globals__ is examplepkg.__dict__
True

If anything this is another argument for why we NEED something like this :-).

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org