[Python-Dev] shared data

[Greg Stein]

On Tue, 16 Nov 1999, Gordon McMillan wrote:
> Barry A. Warsaw writes:
> > One approach might be to support loading modules out of jar files
> > (or whatever) using Greg imputils.  We could put the bootstrap
> > .pyc files in this jar and teach Python to import from it first. 
> > Python installations could even craft their own modules.jar file
> > to include whatever modules they are willing to "hard code". 
> > This, with -S might make Python start up much faster, at the
> > small cost of some flexibility (which could be regained with a
> > c.l. switch or other mechanism to bypass modules.jar).
> 
> Couple hundred Windows users have been doing this for 
> months (http://starship.python.net/crew/gmcm/install.html). 
> The .pyz files are cross-platform, although the "embedding" 
> app would have to be redone for *nix, (and all the embedding 
> really does is keep Python from hunting all over your disk). 
> Yeah, it's faster. And I can put Python+Tcl/Tk+IDLE on a 
> diskette with a little room left over.

I've got a patch from Jim Ahlstrom to provide a "standardized" library
file. I've got to review and fold that thing in (I'll post here when that
is done).

As Gordon states: yes, the startup time is considerably improved.

The DBM approach is interesting. That could definitely be used thru an
imputils Importer; it would be quite interesting to try that out.

(Note that the library style approach would be even harder to deal with
updates, relative to what Sjoerd saw with the DBM approach; I would guess 
that the "right" approach is to rebuild the library from scratch and
atomically replace the thing (but that would bust people with open
references...))

Certainly something to look at.

Cheers,
-g

p.s. I also want to try mmap'ing a library and creating code objects that
use PyBufferObjects (rather than PyStringObjects) that refer to portions
of the mmap. Presuming the mmap is shared, there "should" be a large
reduction in heap usage. Question is that I don't know the proportion of
code bytes to other heap usage caused by loading a .pyc.

p.p.s. I also want to try the buffer approach for frozen code.

--
Greg Stein, http://www.lyra.org/