Pickling recursive user defined types

[MADDEN Richard]

Hi!

   This is beginning to drive me nuts, so any help appreciated. I have a
python application I am using to track bacterial cell motion in growing
microcolonies. It works great. But now I want to combine all of the
colonies in a single database. Since I am storing the cell data as a
generic python class, looking at memory usage I'm finding a single cell
instance is taking several kilobytes. Since I may have millions of cell
instances this is a prohibitive memory load. Figured I could do a lot
better by writing C code to implement a specialized type for the cell
data. Did so. Works fine. Except I can no longer pickle and unpickle the
databases. This seems to be because the cells contain references to other
cells, so the database is highly recursive. When I pickle and unpickle I
find a cell class can come back with the wrong type, a tuple or worse.

   My overall strategy was to write a C type using T_OBJECT references for
the recursive links. I then use copy_reg to register a customized pickling
routine along the following lines:

# ccell object pickling registration
import copy_reg
from newcell import *

def ccellrebuild(num,anc,des):
  c=ccell()
  c.num=num
  c.anc=anc
  c.des=des
  return(c)

def ccellextract(c):
  p=(c.num,c.anc,c.des)
  return((ccellrebuild,p))

copy_reg.constructor(ccellrebuild)
copy_reg.pickle(type(ccell()),ccellextract)

  ccell is the object constructor (defined in 'newcell') with a trivial
__init___ and anc and des are lists of ancestor and descendent ccell type
objects. In my attempts to solve this on my own I actually dug in to the
unpickling modules and found stuff like:

atp19
ag19

this appears to me that it is making a tuple out of a ccell object (how?)
and storing it instead of the ccell object. I feel I'm missing something
really obvious here. un/picking works fine if they are native python
classes.

Ciao and thanks in advance, Dick

Dick Madden                         Home: Studio 5
IHES                                Residence de l'Ormaille
Le Bois-Marie                       Route de la Faculte
35, Route de Chartres               F-91440 Bures-sur-Yvette, France
F-91440 Bures-sur-Yvette, France
Phone: 01 60 92 66 61               Phone: 01 60 92 67 25