[Python-Dev] ast-objects branch created

[Neal Norwitz]

On 12/5/05, Jeremy Hylton <jeremy at alum.mit.edu> wrote:
> On 12/5/05, James Y Knight <foom at fuhm.net> wrote:
> >
> > ISTM that having to remember which pointers are arena-allocated and
> > which are normally-refcounted-allocated removes the major gain that
> > an arena method is supposed to bring: resistance to mistakes. I'd
> > find having a single way to allocate and track memory easier than
> > multiple. Then you just have to follow the single set of best
> > practices for memory management, and you're all set. (and with
> > PyObjects, the same practices the rest of python uses, another win.)
>
> It's a question of degree, right?  If you can find a small number of
> rules that are easy to understand then you are still likely to avoid
> mistakes.

This is my understanding of the two approaches from what I've seen so
far (Jeremy or Martin should correct me if I'm wrong).

With current arena impl:
 * need to call PyArena_AddPyObject() for any allocated PyObject
 * need to call PyArena_AddMallocPointer() for any malloc()ed memory
(there are current no manual calls like this, all the calls are in
generated code?)

With the PyObject imp:
 * need to init all PyObjects to NULL
 * need to Py_XDECREF() on exit
 * need to goto error if there is any failure

Both impls have a bit more details, but those are the highlights. 
>From what I've seen of both, the arena is easier to deal with even
though it is different from the rest of python.  There is only one
thing to remember.

I didn't look at the changes much, but from what I saw I think it may
be better to move the arenas off the branch and onto the head now.  It
appears to be much easier to get right since there is virtually no
error handling code in line.  It's all taken care of in a few central
places.

We can then decide between the arenas in the head vs PyObjects.

> > I'd also like to parrot the concern others have had that if the AST
> > nodes are not made of PyObjects, then a mirror hierarchy of PyObject-
> > ified AST nodes will have to be created, which seems like quite a
> > wasteful duplication. If it is required that there be a collection of
> > AST python objects (which I think it is), is there really any good
> > reason to make the "real" AST objects not be the _only_ AST objects?
> > I've not heard one.
>
> The PyObject-ified AST nodes are only needed if user code requests an
> AST from the compiler.  That is, if we add a new feature that exposes
> AST, we would need AST objects represented in Python code.  I think
> this feature would be great to add, but it doesn't seem like a primary
> concern for the internal compiler implementation.

FWIW, I agree with this approach.  I don't care that much about the
internal AST for its own sake.  I want to consume the AST and I only
care about the internals insofar as the result is correct and
maintainable.

So my view of the best approach is one that is easy to get right and
maintain.  That's why I think the arena should be moved to the head
now.  From what I saw it was much easier to get right, it removed a
bunch of code and should be more maintainable.

I will also probably work on the PyObject approach, since if that's
more maintainable I'd prefer that in the end.  I don't know which
approach is best.

I also really like Martin's idea about generating a lot more (all?) of
the manually written Python/ast.c code.  I'd prefer much less C code
to maintain.

n