[Python-Dev] PEP 550 v3

[Guido van Rossum]

>
> On Sun, Aug 20, 2017, 03:08 Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>> On Sat, 19 Aug 2017 17:21:03 -0700
>> Guido van Rossum <gvanrossum at gmail.com> wrote:
>> > The way we came to "logical context" was via "logical thread (of
>> control)",
>> > which is distinct from OS thread. But I think we might need to search
>> for
>> > another term...
>>
>> Perhaps "task context"?  A "task" might be a logical thread, OS thread,
>> or anything else that deserves a distinct set of implicit parameters.
>>
>
I think you're on to something here, though I hesitate to use "task"
because asyncio.Task is a specific implementation of it.

On Sun, Aug 20, 2017 at 7:04 PM, Brett Cannon <brett at python.org> wrote:

> Maybe this is skirting too loose to the dynamic scoping, but maybe
> ContextFrame? This does start to line up with frames of execution which I
> know is a bit low-level, but then again most people will never need to know
> about this corner of Python.
>

I've been thinking that the missing link here may be the execution stack. A
logical thread (LT) has a "logical stack" (LS). While a thread of control
is a fairly fuzzy concept (once you include things that aren't OS threads),
an execution stack is a concrete object, even if not all logical threads
represent their execution stack in the same way. For example, a suspended
asyncio Task has a stack that is represented by a series of stack frames
linked together by await (or yield from), and greenlet apparently uses a
different representation again (their term is micro-thread -- maybe we
could also do something with that?).

Here's the result of some more thinking about this PEP that I've been doing
while writing and rewriting this message (with a surprising ending).

Let's say that the details of scheduling an LT and managing its mapping
onto an LS is defined by a "framework". In this terminology, OS threads are
a framework, as are generators, asyncio, and greenlet. There are
potentially many different such frameworks. (Some others include Twisted,
Tornado and concurrent.futures.ThreadPoolExecutor.)

The PEP's big idea is to recognize that there are also many different,
non-framework, libraries (e.g. Decimal or Flask) that need to associate
some data with an LT. The PEP therefore proposes APIs that allow libraries
to do this without caring about what framework is managing the LT, and vice
versa (the framework doesn't have to care about how libraries use the
per-LT data).

The proposed APIs uses two sets of concepts: one set for the framework and
one for the library.

The library-facing API is simple: create a ContextKey (CK) instance as a
global variable in the library, and use its get() and set() methods to
access and manipulate the data for that library associated with the current
logical thread (LT). Its role is similar to threading.local(), although the
API and implementation are completely different, and threading.local() is
tied to a specific framework (OS threads).

For frameworks the API is more complicated. There are two new classes,
LogicalContext (LC) and ExecutionContext (EC). The PEP gives pseudo code
suggesting that LC is/contains a dict (whose items are (CK, value) pairs)
and an EC is/contains a list of LCs. But in actuality that's only one
possible implementation (and not the one proposed for CPython). The key
idea is rather that a framework needs to be able to take the data
associated with one LT and clone it as the starting point for the data
associated for a new LT. This cloning is done by
sys.get_execution_context(), and the PEP proposes to use a Hash Array
Mapped Trie (HAMT) as the basis for the implementation of LC and EC, to
make this cloning fast. IIUC it needs to be fast to match the speed with
which many frameworks create and destroy their LTs.

The PEP proposes a bunch of new functions in sys for frameworks to
manipulate LCs and ECs and their association with the current OS-level
thread. Note that OS threads are important here because in the end all
frameworks build on top of them.

Honestly I'm not sure we need the distinction between LC and EC. If you
read carefully some of the given example code seems to confuse them. If we
could get away with only a single framework-facing concept, I would be
happy calling it ExecutionContext.

(Another critique of the proposal I have is that it adds too many
similarly-named functions to sys. But this email is already too long and I
need to go to bed.)

-- 
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170820/24d6cf82/attachment.html>