Python does not play well with others

[sjdevnull at yahoo.com]

On Feb 6, 4:27 pm, Paul Rubin <http://phr...@NOSPAM.invalid> wrote:
> "sjdevn... at yahoo.com" <sjdevn... at yahoo.com> writes:
> > In our case, the issue is this: we load a ton of info at server
> > restart, from the database.  Some of it gets processed a bit based on
> > configuration files and so forth.  If this were done in my own C
> > server, I'd do all of that and set up the (read-only) runtime data
> > structures prior to forking.  That would mean that:
> > a) The processing time would be lower since you're just doing the pre-
> > processing once; and
> > b) The memory footprint could be lower if large data structures were
> > created prior to fork; they'd be in shared copy-on-write pages.
>
> If you completely control the server, write an apache module that
> dumps this data into a file on startup, then mmap it into your Python app.

The final data after loading is in the form of a bunch of python
objects in a number of complex data structures, so that's not really a
good solution as far as I can tell.  We read in a bunch of data from
the database and build a data layer describing all the various classes
(and some kinds of global configuration data, etc) used by the various
applications in the system.

It's possible that we could build it all in a startup module and then
pickle everything we've built into a file that each child would
unpickle, but I'm a bit leery about that approach.