[Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Brett Cannon]

On Tue, 30 Aug 2016 at 09:08 Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Mon, 29 Aug 2016 21:38:19 +0000
> Brett Cannon <brett at python.org> wrote:
> > For quick background for those that don't remember, part of PEP 523
> > proposed adding a co_extra field to code objects along with making the
> > frame evaluation function pluggable (
> > https://www.python.org/dev/peps/pep-0523/#expanding-pycodeobject). The
> idea
> > was that things like JITs and debuggers could use the field as a scratch
> > space of sorts to store data related to the code object. People who
> > objected to the new field did either for memory ("it adds another pointer
> > to the struct that won't be typically used"), or for conceptual reasons
> > ("the code object is immutable and you're proposing a mutable field").
> The
> > latter is addressed by not exposing the field in Python and clearly
> stating
> > that code should never expect the field to be filled.
>
> Just a question: Maciej mentioned the field may be useful for vmprof.
> That's already two potential users (vmprof and Pyjion) for a single
> field.


PTVS has also said they would find it useful for debugging.


>   OTOH, the PEP says:
>
> """It is not recommended that multiple users attempt to use the co_extra
> simultaneously. While a dictionary could theoretically be set to the
> field and various users could use a key specific to the project, there
> is still the issue of key collisions as well as performance degradation
> from using a dictionary lookup on every frame evaluation. Users are
> expected to do a type check to make sure that the field has not been
> previously set by someone else."""
>
> Does it mean that, in the event vmprof comes in and changes the field,
> Pyjion will have to recompile the function the next time it comes to
> execute it?
>

As of right now Pyjion simply doesn't JIT the function.


>
> And, conversely, if Pyjion compiles a function while vmprof is enabled,
> vmprof will lose timing information (or whatever else, because I'm not
> sure what vmprof plans to store there) for that code object?
>

Depends on what vmprof chooses to do. Since the data is designed to be
disposable it could decide it should always take precedence and overwrite
the data if someone beat it to using the field. Basically I don't think we
want co_extra1, co_extra2, etc. But we don't want to require a dict either
as that kills performance. Using a list where users could push on objects
might work, but I have no clue what that would do to perf as you would have
to still needlessly search the list when only one piece of code uses the
field.

Basically I don't see a good way to make a general solution for people who
want to use the field simultaneously, so tools that use the field will need
to be clear on how they choose to handle the situation, such "as we use it
if it isn't set" or "we always use it no matter what". This isn't a perfect
solution in all cases and I think that's just going to have to be the way
it is.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/377aa98d/attachment.html>