[Python-ideas] Copy (and/or pickle) generators

[Nathaniel Smith]

You might find this useful, either to use directly or as a source of
inspiration:

https://github.com/llllllllll/cloudpickle-generators

-n

On Tue, Jun 19, 2018, 15:55 Micheál Keane <ffaristocrat at gmail.com> wrote:

>
> Add a function to generator objects to copy the entire state of it:
>
> Proposed example code:
>
> game1 = complicated_game_type_thing()
>
> # Progress the game to the first decision point
> choices = game1.send(None)
>
> # Choose something
> response = get_a_response(choices)
>
> # Copy the game generator
> game2 = game1.copy()
>
> # send the same response to each game
> x = game1.send(response)
> y = game2.send(response)
>
> # verify the new set of choices is the same
> assert x == y
>
>
> History:
>
> I found this stackoverflow Q&A
> <https://stackoverflow.com/questions/7180212/why-cant-generators-be-pickled> which
> among other things linked to an in-depth explanation of why generators
> could not be pickled
> <http://peadrop.com/blog/2009/12/29/why-you-cannot-pickle-generators/> and
> this enhancement request for 2.6 <https://bugs.python.org/issue1092962> on
> the bugtracker. All the reasons given there are perfectly valid.... but
> they were also given nearly 10 years ago. It may be time to revisit the
> issue.
>
> I couldn't turn up any previous threads here related to this so I'm
> throwing it out for discussion.
>
>
> Use case:
>
> My work involves Monte Carlo Tree Searches of games, eventually in
> combination with tensorflow. MCTS involves repeatedly copying the state of
> a simulation to explore the potential outcomes of various choices in depth.
>
> If you're doing a game like Chess or Go, a game state is dead simple to
> summarize - you have a list of board positions with which pieces they have
> and whose turn it is.
>
> If you're doing complex games that don't have an easily summarized state
> at any given moment, you start running into problems. Think something
> along the lines of Magic the Gathering with complex turn sequences between
> players and effect resolutions being done in certain orders that are
> dependent on choices made by players, etc.
>
> Generators are an ideal way to run these types of simulations but the
> inability to copy the state of a generator makes it impossible to do this
> in MCTS.
>
> As Python is being increasingly used for data science, this use case will
> be increasingly common. Being able to copy generators will save a lot of
> work.
>
> Keep in mind, I don't necessarily propose that generators should be fully
> picklable; there are obviously a number of concerns and problems there.
> Just being able to duplicate the generator's state within the interpreter
> would be enough for my use case.
>
>
> Workarounds:
>
> The obvious choice is to refactor the simulation as an iterator that
> stores each state as something that's easily copied/pickled. It's probably
> possible but it'll require a lot of thought and code for each type of
> simulation.
>
> There's a Python2 package from 2009 called generator_tools
> <https://pypi.org/project/generator_tools/> that purports to do this. I
> haven't tried it yet to see if it still works in 2.x and it appears beyond
> my skill level to port to 3.x.
>
> PyPy & Stackless Python apparently support this within certain limits?
>
>
> Thoughts?
>
>
> Washington, DC  USA
> ffaristocrat at gmail.com
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180619/15f80f8e/attachment.html>