Technique for safely reloading dynamically generated module

[Chris Angelico]

On Fri, Jul 22, 2016 at 12:42 AM, Malcolm Greene <python at bdurham.com> wrote:
> We're designing a server application that parses a custom DSL (domain
> specific language) source file, generates a Python module with the
> associated logic, and runs the associated code. Since this is a server
> application, we need to reload the module after each regeneration. Is
> this process is simple as the following pseudo code or are there other
> issues we need to be aware of? Are there better techniques for this
> workflow (eval, compile, etc)?
>
> We're working in Python 3.5.1.
>
> import importlib
>
> # custom_code is the module our code will generate - a version of this
> # file will always be present
> # if custom_code.py is missing, a blank version of this file is created
> # before this step
> import custom_code
>
> while True:
>     # (re)generates custom_code.py visible in sys.path
>    generate_custom_code( source_file )
>
>    # reload the module whose source we just generated
>    importlib.reload( custom_code )
>
>    # run the main code in generated module
>    custom_code.go()
>

I would avoid importlib.reload() if I can. Using compile/eval is
probably more effective; you might need to change how the module gets
organized, but if you already require a go() function, that's probably
enough. Something like this:

while True:
    python_code = generate_custom_code(source_file)
    ns = {}
    exec(python_code, ns)
    ns["go"]()

(Presumably your go() function doesn't return until you want to
regenerate the code?)

Personally, what I'd do is have an initializer function, and have that
register the module in whatever way is appropriate (eg some sort of
"handle_request" function). Then the main program runs an event loop,
everything's handled asynchronously, and you can have either a file
system trigger "this file has changed, go and regenerate it" or an
explicit action by the admin to regenerate. But definitely I'd craft
the source code as a string, not writing anything to disk, and exec it
directly; the import machinery is unnecessary here.

ChrisA