Technique for safely reloading dynamically generated module

[Peter Otten]

Malcolm Greene wrote:

> We're designing a server application that parses a custom DSL (domain
> specific language) source file, generates a Python module with the
> associated logic, and runs the associated code. Since this is a server
> application, we need to reload the module after each regeneration. Is
> this process is simple as the following pseudo code or are there other
> issues we need to be aware of? Are there better techniques for this
> workflow (eval, compile, etc)?
>  
> We're working in Python 3.5.1.
> 
> import importlib
>  
> # custom_code is the module our code will generate - a version of this
> # file will always be present
> # if custom_code.py is missing, a blank version of this file is created
> # before this step
> import custom_code
>  
> while True:
>     # (re)generates custom_code.py visible in sys.path
>    generate_custom_code( source_file )
>  
>    # reload the module whose source we just generated
>    importlib.reload( custom_code )
>  
>    # run the main code in generated module
>    custom_code.go()

If the go() function in that loop is the only place where the generated 
module is used you can avoid writing code to the file system altogether.

Here's a really simple alternative suggestion based on exec():

SOURCEFILE = "whatever.dsl"


def make_func(sourcefile):
    ns = {}
    exec(generated_code(sourcefile), ns)
    return ns["go"]


def generated_code(sourcefile):
    # placeholder for your actual source generation
    return """
def go():
    print("Generated from", {!r})
""".format(sourcefile)


while True:
    # XXX should we wait for updates of sourcefile here?
    go = make_func(SOURCEFILE)
    go()