Being a bit confused after hacking Python

[Wolfgang Draxinger]

Within my 3D engine project I decided to choose python for all scripting
purposes. It's fast, flexible, you know what I mean. So I embedded it
the normal way. It's clear, that I've to redirect a few parts of the
builtin module stuff, e.g. sys.stdin/out/err.

So I did e.g. PySys_SetObject("stdout", engine_py_stdout); etc. just
after PyInitialize(); It works all well until a reload(sys) which is of
course the defined behaviour, and restores sys.

Since this is not what I want, I hacked a bit the Python sources and
found out the following, beginning at

void
Py_Initialize(void)
{
/*snipped interpreter initialisation*/

/*Ok here are the Builtin ...*/

	bimod = _PyBuiltin_Init();
	if (bimod == NULL)
		Py_FatalError("Py_Initialize: can't initialize __builtin__");
	interp->builtins = PyModule_GetDict(bimod);
	Py_INCREF(interp->builtins);

/*... and Sys module initialized, all well so long*/
	sysmod = _PySys_Init();
	if (sysmod == NULL)
		Py_FatalError("Py_Initialize: can't initialize sys");
	interp->sysdict = PyModule_GetDict(sysmod);
	Py_INCREF(interp->sysdict);
	_PyImport_FixupExtension("sys", "sys");
	PySys_SetPath(Py_GetPath());
	PyDict_SetItemString(interp->sysdict, "modules",
			     interp->modules);

	_PyImport_Init();
/*and the rest snipped*/
}

It seems at first to me, that PySys and PyBuiltin only get initialized
once. Now to the reload process, no wait, the _PyImport_Inittab first:

struct _inittab _PyImport_Inittab[] = {
/*snipped the obvious stuff*/

          /* These entries are here for sys.builtin_module_names */
          {"__main__", NULL},
          {"__builtin__", NULL},
          {"sys", NULL},
	{"exceptions", NULL},

          /* Sentinel */
          {0, 0}
};

As we can see, initfunc of the "always there" builtin functions are
NULL, hmmm....

now what does reload? It's the function that screws up my config:

PyObject *
PyImport_ReloadModule(PyObject *m)
{
/*stuff to find out, if module is loaded already, to reload parent
modules and so on, right, snipped */

	buf[0] = '\0';
	fdp = find_module(subname, path, buf, MAXPATHLEN+1, &fp);
	Py_XDECREF(path);
	if (fdp == NULL)
		return NULL;
	m = load_module(name, fp, buf, fdp->type);
	if (fp)
		fclose(fp);
	return m;
}

Ok, it just calls load_module, for a builtin with type C_BUILTIN, what
does load_module:

static PyObject *
load_module(char *name, FILE *fp, char *buf, int type)
{
	PyObject *modules;
	PyObject *m;
	int err;

/*for the moment just the builtin stuff matters*/

	case C_BUILTIN:
	case PY_FROZEN:
		if (buf != NULL && buf[0] != '\0')
			name = buf;
		if (type == C_BUILTIN)
			err = init_builtin(name);

/*here comes error checking not of interesst now*/

	}

	return m;
}

Ahm, for a builtin it just calls init_builtin(), more hmmm...
Have a look at init_builtin():

static int
init_builtin(char *name)
{
	struct _inittab *p;

	if (_PyImport_FindExtension(name, name) != NULL)
		return 1;

	for (p = PyImport_Inittab; p->name != NULL; p++) {
		if (strcmp(name, p->name) == 0) {
			if (p->initfunc == NULL) {
				PyErr_Format(PyExc_ImportError,
				    "Cannot re-init internal module %.200s",
				    name);
				return -1;
			}
			if (Py_VerboseFlag)
				PySys_WriteStderr("import %s # builtin\n", name);
			(*p->initfunc)();
			if (PyErr_Occurred())
				return -1;
			if (_PyImport_FixupExtension(name, name) == NULL)
				return -1;
			return 1;
		}
	}
	return 0;
}

Alright, now is the confusion perfect: If I look at _PyImport_Inittab I
see, that sys doesn't define a init function. Other builtins do, and all
is clear to me, no problem at all, great.

*BUT* what/where the hell is the function that reload sys and invalids
my changes I made juat after Py_Initialize, when there is no initfunc
that can be called ???
I must have missed something.

And now a few other questions:

1st:
It'd be interessting to know, how to prevent scripts to delete
imported modules. My engines load some standard modules into the
interpreter by default. It's not bad when the modules are unloaded and
reloaded, but this means for some modules a huge performance hit. Thats
a bit disturbing in a realtime engine.

2nd:
Is there anybody who knows, how to *import* a source module, that may be 
compiled already, from contents in memory? My problem is that my engine 
has a virtual file system that packs data compresed into packages, to 
minimize bandwidth when loading from removeable media. The modules are 
contained within such a package and thus cannot be accesed by a normal 
operating system file.

3rd:
There are a few modules, like os or posix, which allow low level access 
to the system. Since my engine has network support and the server 
provides a remote console that works in python interactive mode, it 
would be good to remove some of the builtin modules after Py_Initialize, 
that were initialized via _PyImport_Inittab. It would be even possible 
to write a virus that distributes within a multiplayer game, if that 
access isn't restricted and a new type of virus is definitely what the 
targeted users not want.
There must be a official way to remove specific module entries, I think 
of the things that rexec can do, but I didn't get out how it does, what 
it does. I could of course create a own _PyImport_Inittab and thus just 
not initialize the dangerous stuff. Is that a good solution?

Ok, that was now a huge amount of questions, but I hope, that you will 
find a bit time for them.

Thanks in advance
-- 
+------------------------------------------------+
| +----------------+ WOLFGANG DRAXINGER          |
| | ,-.   DARKSTAR | lead programmer             |
| |(   ) +---------+ wdraxinger at darkstargames.de |
| | `-' / GAMES /                                |
| +----+''''''''     http://www.darkstargames.de |
+------------------------------------------------+