importing pyc from memory?

[Bengt Richter]

On Mon, 4 Jul 2005 13:01:06 -0500, Jeff Epler <jepler at unpythonic.net> wrote:

>
>--s2ZSL+KKDSLx8OML
>Content-Type: text/plain; charset=us-ascii
>Content-Disposition: inline
>
>This stupid code works for modules, but not for packages.  It probably has bugs.
>
>
>import marshal, types
>
>class StringImporter:
>    def __init__(self, old_import, modules):
>        self._import = old_import
>        self._modules = modules
>
>    def __call__(self, name, *args):
>        module = self._modules.get(name, None)
>        if module is None:
>            return self._import(name, *args)
>        code = marshal.loads(module)
>        mod = types.ModuleType(name)
>        exec code in mod.__dict__
>        return mod
>
>def test():
>    import __builtin__
>    __builtin__.__import__ = StringImporter(__builtin__.__import__,
>        { 'test_importer': open("/usr/lib/python2.3/os.pyc").read()[8:] })
>
>    import test_importer
>    print test_importer.path.join("a", "b")
>    print test_importer.__doc__
>
>if __name__ == '__main__': test()

I have a feeling that the OP might be looking for a way to convert
source -> source, e.g., (sketch only, not even remotely tested ;-)

    # main.py
    import some_module
    import another

into

    # main.py
    # prelim
    import marshal, sys        #  builtin
    ModuleType = type(__builtins__) # avoid import types
    def unscramble_proprietary_format(proprietary_string):
        # ... whatever
    def extract_code(proprietary_string):
        return marshal.loads(unscramble_proprietary_format(proprietary_string))

    # replacing each import with
    try: some_module = sys.modules['some_module']  # don't re-"import"
    except KeyError:
        some_module = ModuleType('some_module')
        exec extract_code(
             "< ... string literal for proprietarily scrambled marshalled module body code"
             " generated by source -> source rewrite utility ... >"
             ) in some_module.__dict__
        sys.modules['some_module'] = some_module

    try: another = sys.modules['another']
    except NameError:
        another = ModuleType('another')
        exec extract_code('<proprietary marshalled code string for module source another.py>')
        sys.modules['another'] = another

    # etc

so that his some_module and another scripts can be pre-compiled by the rewrite utility and embedded in
a single source (e.g. main.py) so that he doesn't have to have external .pyc file accesses.

His startup will then still cost the execs, but not the compiles, and if he doesn't "import" stuff
he doesn't need, until he needs it, it should spread the import time load, if so desired.

All the proprietary string literal sources should become efficiently represented as part of the
single main.pyc IWT, assuming at least that one is used. Chasing various forms of import of
non-builtins recursively to eliminate imports in imported modules before they are converted to
marshalled form etc., all to avoid real imports, and statically determining that some imports
don't need to be converted to marshalled string import form because a prior import can be proved,
should be an interesting exercise, which I can't pursue at this point... ;-)

But I may be reading too much between the lines ;-)

Regards,
Bengt Richter