Shared memory python between two separate shell-launched processes

[Charles Fox (Sheffield)]

On Feb 10, 6:22 pm, Jean-Paul Calderone <calderone.jeanp... at gmail.com>
wrote:
> On Feb 10, 12:21 pm, "Charles Fox (Sheffield)" <charles.... at gmail.com>
> wrote:
>
>
>
> > On Feb 10, 3:43 pm, Jean-Paul Calderone <calderone.jeanp... at gmail.com>
> > wrote:
>
> > > On Feb 10, 9:30 am, "Charles Fox (Sheffield)" <charles.... at gmail.com>
> > > wrote:
>
> > > > Hi guys,
> > > > I'm working on debugging a large python simulation which begins by
> > > > preloading a huge cache of data.  I want to step through code on many
> > > > runs to do the debugging.   Problem is that it takes 20 seconds to
> > > > load the cache at each launch.  (Cache is a dict in a 200Mb cPickle
> > > > binary file).
>
> > > > So speed up the compile-test cycle I'm thinking about running a
> > > > completely separate process (not a fork, but a processed launched form
> > > > a different terminal)
>
> > > Why _not_ fork?  Load up your data, then go into a loop forking and
> > > loading/
> > > running the rest of your code in the child.  This should be really
> > > easy to
> > > implement compared to doing something with shared memory, and solves
> > > the
> > > problem you're trying to solve of long startup time just as well.  It
> > > also
> > > protects you from possible bugs where the data gets corrupted by the
> > > code
> > > that operates on it, since there's only one copy shared amongst all
> > > your
> > > tests.  Is there some other benefit that the shared memory approach
> > > gives
> > > you?
>
> > > Of course, adding unit tests that exercise your code on a smaller data
> > > set
> > > might also be a way to speed up development.
>
> > > Jean-Paul
>
> > Thanks Jean-Paul, I'll have a think about this.  I'm not sure if it
> > will get me exactly what I want though, as I would need to keep
> > unloading my development module and reloading it, all within the
> > forked process, and I don't see how my debugger (and emacs pdb
> > tracking) will keep up with that to let me step though the code.
>
> Not really.  Don't load your code at all in the parent.  Then there's
> nothing to unload in each child process, just some code to load for
> the very first time ever (as far as that process is concerned).
>
> Jean-Paul


Jean, sorry I'm still not sure what you mean, could you give a couple
of lines of pseudocode to illustrate it?   And explain how my emacs
pdbtrack would still be able to pick it up?
thanks,
charles