[Python-Dev] Best practice for new namespace (from C/API)

[Campbell Barton]

Hi, I'm writing because I'm working on a project where the user can
run scripts that don't reference a file but use internal application
text data.
Otherwise we are not doing anything tricky, only that the scripts
should each run independently (no cruft left from the previous scripts
namespace, sharing sys.modules etc is fine).

Something which is unclear to me even after looking over pythonrun.c
is the correct way to setup a namespace.

For years we have been doing this and it seemed to work fine...
  PyObject *d = PyDict_New(); // new namespace
  PyDict_SetItemString(d, "__builtins__", PyEval_GetBuiltins());
  PyDict_SetItemString(d, "__file__", PyString_FromString(filename));
// fake, avoids sys.argv[0] being used for warnings.
/* --- snip ---*/
  PyEval_EvalCode(compiled_text, d, d);


Recently a developer reported a bug where pickle wasn't working, it
turns out that in a few places python expects the __main__ modules
namespace to match that if the running script: _pickle.c's
save_global() in this case

eg:
  >>> spam = 10
  >>> print(__import__("__main__").__dict__["spam"])
  ... 10

Once I found this was the problem it was simple to use __main__'s
namespace however there are still things that are not clear about
exactly how this should be done.

Simplified code...
  PyObject *item, *dict= PyModule_GetDict(PyImport_AddModule("__main__"));
  PyDict_Clear(dict);
  PyDict_SetItemString(dict, "__builtins__", PyImport_AddModule("builtins"));
  item = PyUnicode_FromString( "__main__" );
  PyDict_SetItemString( dict, "__name__", item );
  Py_DECREF(item);
  PyDict_SetItemString(d, "__file__", PyString_FromString(filename));
// fake, avoids sys.argv[0] being used for warnings.
/* --- snip ---*/
  PyEval_EvalCode(compiled_text, dict, dict);


Still this leaves me with the following questions...
- Whats the best way to manage the namespace for running multiple
scripts one after another? clear and initialize __main__'s dict each
time?, keep a copy and restore it after each run?
- should the original __main__ namespace be restored after running a script?
- pickle expects: __import__("__main__").__dict__ == ***the current
namespace***, is this apart of the python spec or just something
specific to pickle?
- PyDict_SetItemString(d, "__builtins__", PyEval_GetBuiltins()) acts
differently to PyDict_SetItemString(dict, "__builtins__",
PyImport_AddModule("builtins")), using the PyEval_GetBuiltins() dict
adds every member of __builtins__ when running:
print(__import__("__main__").__dict__.keys()), rather then just
showing __builtins__.

-- 
- Campbell