[Python-Dev] RE: cloning iterators again

[Guido van Rossum]

> Armin Rigo posted an URL to a C extension module he has recently
> developed, and used.  That module is able to copy running instances
> of a generator.  Currently, it does so by just "knowing", and type by
> type dealing with, several types whose instances could be values
> referred to by the generator's frame.  I was suggesting extending this
> so that, when values of other types are met, instead of automatically
> failing (or, as I believe I recall Armin's extension does today, copying
> the reference rather than the value), the copy process would...:

I haven't seen Armin's code, but I don't believe that the type alone
gives enough information about whether they should be copied.

Consider a generator that uses a dict as a cache or memo for values it
computes.  Multiple instances of the generator share the dict, but for
efficiency the generator references it in a local variable.  This dict
should not be copied when copying the generator's stack frame.

But consider another generator that uses a dict to hold some of its
state.  This dict should be copied.

> > > > Cool!  Why don't you try copy.copy on types you don't automatically
> > > > recognize and know how to deal with, BTW?  That might make this
> 
> and Armin said that __copy__ seems weak to him but __deepcopy__
> might not be:
> 
> > > I will try.  Note that only __deepcopy__ makes sense, as far as I can
> > > tell, because there is too much state that really needs to be copied and
> > > not shared in a generator (in particular, the sequence iterators from
> > > 'for' loops).
> 
> 
> So, now he went on to ask about __deepcopy__ and you answered:
> 
> > > I'm not sure about how deep-copying should be defined for built-in
> > > types.  Should a built-in __deepcopy__ method try to import and call
> > > copy.deepcopy() on the sub-items? This doesn't seem to be right.
> >
> > Almost -- you have to pass the memo argument that your __deepcopy__
> > received as the second argument to the recursive deepcopy() calls, to
> > avoid looping on cycles.
> 
> 
> Now, if Armin's code can only provide __deepcopy__ and not __copy__,
> then it's probably of little use wrt the __copy__ functionality I
> talk about in PEP 323 (which I still must revise to take into
> account your feedback and Raymond's -- plan to get to that as soon
> as I've cleared my mbox) -- the memory and time expenditure is
> likely to be too high for that.

Right.

> It's still going to be a cool hack, well worth "publishing" as such,

As a third-party module?  Sure.

> and probably able to be "user-installed" as the way deepcopy deals
> with generators even though generators themselves may not sprout a
> __deepcopy__ method themselves (fortunately, copy.copy does a lot of
> "ad-hoc protocol adaptation" -- it's occasionally a bit rambling or
> hard to follow, but often allows plugging in "third party copiers"
> for types which their authors hadn't imagined would be copied or
> deep copied, so that other innocent client code which just calls
> copy.copy(x) will work... essentially how "real" adaptation would
> work, except that registering a third-party protocol adapter would
> be easier:-).

--Guido van Rossum (home page: http://www.python.org/~guido/)