[Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version

[Steve Dower]

Thanks for adding your input, Gregory! It's much appreciated.

I'll shuffle your comments around a bit, as I'd rather address the 
themes than each individual point.

On 15May2019 2134, Gregory Szorc wrote:
> PyPreConfig_INIT and PyConfig_INIT as macros that return a struct feel
> weird to me. Specifically, the `PyPreConfig preconfig =
> PyPreConfig_INIT;` pattern doesn't feel right.

I see Victor agreed here, but I think this is the right pattern for 
PreConfig. The "_INIT" macro pattern is common throughout as a way to 
initialize a stack-allocated struct - we really can't change it to be 
anything than "{ .member = static value }" without breaking users, but 
if you have another way to initialize it correctly then that is fine. 
The important factor here is that this struct has to be allocated 
_without_ any memory management provided by Python.

That said, I don't particularly like this approach for PyConfig. As you 
said:

> Also, one thing that tripped me up a few times when writing PyOxidizer
> was managing the lifetimes of memory that various global variables point
> to.

My preference here is for PreConfig to get far enough that we can 
construct the full configuration as regular Python objects (e.g. using 
PyDict_New, PyUnicode_FromString, etc.)[1] rather than a brand new C 
struct with a new set of functions. That will cost copies/allocations at 
startup, but it will also ensure that the lifetime of the configuration 
info is managed by the runtime.

I assume you already have code/helpers for constructing Python strings 
and basic data structures, so I wonder whether it would be helpful to be 
able to use them to create the configuration info?

([1]: Yes, this requires implementation changes so they work 
pre-interpreter and cross-interpreter. This work probably has to happen 
anyway, so I don't see any harm in assuming it will happen.)

> I'm a little confused about the pre-initialization functions that take
> command arguments. Is this intended to only be used for parsing the
> arguments that `python` recognizes? Presumably a custom application
> embedding Python would never use these APIs unless it wants to emulate
> the behavior of `python`? (I suppose this can be clarified in the API
> docs once this is implemented.)

> One feature that I think is missing from the proposal (and this is
> related to the previous paragraph) is the ability to prevent config
> fallback to things that aren't PyConfig and PyPreConfig. 

This is certainly my intent, and I *think* Victor is coming around to it 
too ;)

My preference is that embedding by default does not use any information 
outside of the configuration provided by the host application. Then our 
"python" host application can read the environment/argv/etc. and convert 
it into configuration. Since some embedders also want to do this, we can 
provide helper functions to replicate the behaviour.

Does this sound like a balance that would suit your needs? Would you 
expect an embedded Python to be isolated by default? Or would you assume 
that it's going to pick up configuration from various places and that 
you (as an embedder) need to explicitly suppress that?

(Some parts of the stdlib and some 3rd-party libraries use their own 
environment variables at runtime. We may not be able to convert all of 
those to configuration, but at least they're read lazily, and so Python 
code can override them by setting the variables.)

> What about PyImport_FrozenModules? 

> FWIW I /might/ be interested in a mechanism to better control importlib
> initialization because PyOxidizer is currently doing dirty things at
> run-time to register the custom 0-copy meta path importer.

Controlling imports early in initialization is one of our broader goals 
here, and one that I would particularly like to figure out before we 
commit to a new public API. Registering new importers should not have to 
be "dirty".

Cheers,
Steve