Shared memory python between two separate shell-launched processes

[Jean-Paul Calderone]

On Feb 10, 9:30 am, "Charles Fox (Sheffield)" <charles.... at gmail.com>
wrote:
> Hi guys,
> I'm working on debugging a large python simulation which begins by
> preloading a huge cache of data.  I want to step through code on many
> runs to do the debugging.   Problem is that it takes 20 seconds to
> load the cache at each launch.  (Cache is a dict in a 200Mb cPickle
> binary file).
>
> So speed up the compile-test cycle I'm thinking about running a
> completely separate process (not a fork, but a processed launched form
> a different terminal)

Why _not_ fork?  Load up your data, then go into a loop forking and
loading/
running the rest of your code in the child.  This should be really
easy to
implement compared to doing something with shared memory, and solves
the
problem you're trying to solve of long startup time just as well.  It
also
protects you from possible bugs where the data gets corrupted by the
code
that operates on it, since there's only one copy shared amongst all
your
tests.  Is there some other benefit that the shared memory approach
gives
you?

Of course, adding unit tests that exercise your code on a smaller data
set
might also be a way to speed up development.

Jean-Paul