[C++-sig] [pygccxml] caching fails due to pickling recursion limit

[Kirill Lapshin]

Hi Roman, everybody,

First of all my apologies if it is a wrong list for pygccxml discussion.

We are using py++/pygccxml on moderately sized project, and recently 
py++ started to fail silently on us. It runs gccxml and then silently quits.

It turned out the offending piece of code was pickling cache. Something 
fails in cPickle and terminates the whole process. Most likely it is a 
stack overflow, because if cPickle replaced with pickle, then we get 
recursion limit runtime error. One can raise recursion limit and pickle 
will save cache just fine, but really slowly (not surprising at all).

At the moment we've worked around this problem by disabling cache, 
however it would be nice to fix it properly.

After reading a bit about pickling, it looks like (c)pickle don't work 
very well with recursive structures, which declarations obviously are.

I just want to list some ideas I have on how to tackle this problem, and 
I would love to get some comments on their feasibility, or better yet 
alternative approaches.

1. Simple one, just set all parents to None before pickling and rebuild 
parents upon loading. Should be relatively simple, I'll try it first 
thing. Hopefully that would kill fair amount of recursion, but not all 
of it (e.g. class A refers to class B and B to A).

2. Add unique id to each declaration, have a global store that saves 
declarations by id, and define for each declaration __getstate__, 
__setstate__ that would serialize ids, rather then declaration 
instances. Then we just pickle the global store. In reality somewhat 
more complex approach needed to tackle recursive references.

References:
Reversible flatting out: 
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/302535

Some details: we are running:
Python 2.4.3 on Windows XP
Py++/pygccxml 0.9
gccxml -- somewhat old CVS snapshot

Kirill