[Python-ideas] Copy (and/or pickle) generators

[Micheál Keane]

I wanted to sound out a couple things.

First, I couldn't find any real discussion about it after 2011 so I had no
idea if the reasons it was ruled unfeasible with Python 2 still held nearly
10 years later with Python 3. I was mainly wondering if all the recent
asynchronous work had changed things significantly. Apparently not?

Secondly, one SO comment had included the suggestion that it be posted to
this list - my searching couldn't find it ever having been done so here it
is.

Finally, another comment made the point that there wasn't a strong use case
given for it. With the data science libraries that have sprung up around
Python in the intervening years, I believe there now is one.

Washington, DC  USA
ffaristocrat at gmail.com

On Wed, Jun 20, 2018 at 12:25 AM, Guido van Rossum <guido at python.org> wrote:

> The state of a generator is not much more that a single Python stack frame
> plus an integer indicating where in the bytecode the resume point is. But
> copying/pickling a stack frame is complicated -- it's not just all the
> locals but also the try/except stack and the expression evaluation stack.
> Have a look here: https://github.com/python/cpython/blob/master/Include/
> frameobject.h. I'm not sure that I want to sign up for making all that
> stuff copyable (pickling will be an even harder challenge). But perhaps you
> (and/or another fearless hacker) are interested in trying?
>
> Or were you just trying to see if the core dev team has spare cycles to
> implement this for you?
>
> --Guido
>
> On Tue, Jun 19, 2018 at 3:56 PM Micheál Keane <ffaristocrat at gmail.com>
> wrote:
>
>>
>> Add a function to generator objects to copy the entire state of it:
>>
>> Proposed example code:
>>
>> game1 = complicated_game_type_thing()
>>
>> # Progress the game to the first decision point
>> choices = game1.send(None)
>>
>> # Choose something
>> response = get_a_response(choices)
>>
>> # Copy the game generator
>> game2 = game1.copy()
>>
>> # send the same response to each game
>> x = game1.send(response)
>> y = game2.send(response)
>>
>> # verify the new set of choices is the same
>> assert x == y
>>
>>
>> History:
>>
>> I found this stackoverflow Q&A
>> <https://stackoverflow.com/questions/7180212/why-cant-generators-be-pickled> which
>> among other things linked to an in-depth explanation of why generators
>> could not be pickled
>> <http://peadrop.com/blog/2009/12/29/why-you-cannot-pickle-generators/> and
>> this enhancement request for 2.6 <https://bugs.python.org/issue1092962> on
>> the bugtracker. All the reasons given there are perfectly valid.... but
>> they were also given nearly 10 years ago. It may be time to revisit the
>> issue.
>>
>> I couldn't turn up any previous threads here related to this so I'm
>> throwing it out for discussion.
>>
>>
>> Use case:
>>
>> My work involves Monte Carlo Tree Searches of games, eventually in
>> combination with tensorflow. MCTS involves repeatedly copying the state of
>> a simulation to explore the potential outcomes of various choices in depth.
>>
>> If you're doing a game like Chess or Go, a game state is dead simple to
>> summarize - you have a list of board positions with which pieces they have
>> and whose turn it is.
>>
>> If you're doing complex games that don't have an easily summarized state
>> at any given moment, you start running into problems. Think something
>> along the lines of Magic the Gathering with complex turn sequences between
>> players and effect resolutions being done in certain orders that are
>> dependent on choices made by players, etc.
>>
>> Generators are an ideal way to run these types of simulations but the
>> inability to copy the state of a generator makes it impossible to do this
>> in MCTS.
>>
>> As Python is being increasingly used for data science, this use case will
>> be increasingly common. Being able to copy generators will save a lot of
>> work.
>>
>> Keep in mind, I don't necessarily propose that generators should be fully
>> picklable; there are obviously a number of concerns and problems there.
>> Just being able to duplicate the generator's state within the interpreter
>> would be enough for my use case.
>>
>>
>> Workarounds:
>>
>> The obvious choice is to refactor the simulation as an iterator that
>> stores each state as something that's easily copied/pickled. It's probably
>> possible but it'll require a lot of thought and code for each type of
>> simulation.
>>
>> There's a Python2 package from 2009 called generator_tools
>> <https://pypi.org/project/generator_tools/> that purports to do this. I
>> haven't tried it yet to see if it still works in 2.x and it appears beyond
>> my skill level to port to 3.x.
>>
>> PyPy & Stackless Python apparently support this within certain limits?
>>
>>
>> Thoughts?
>>
>>
>> Washington, DC  USA
>> ffaristocrat at gmail.com
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180620/5d0e24ab/attachment.html>