[Python-ideas] Move optional data out of pyc files

[Chris Angelico]

On Wed, Apr 11, 2018 at 2:14 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> Currently pyc files contain data that is useful mostly for developing and is
> not needed in most normal cases in stable program. There is even an option
> that allows to exclude a part of this information from pyc files. It is
> expected that this saves memory, startup time, and disk space (or the time
> of loading from network). I propose to move this data from pyc files into
> separate file or files. pyc files should contain only external references to
> external files. If the corresponding external file is absent or specific
> option suppresses them, references are replaced with None or NULL at import
> time, otherwise they are loaded from external files.
>
> 1. Docstrings. They are needed mainly for developing.
>
> 2. Line numbers (lnotab). They are helpful for formatting tracebacks, for
> tracing, and debugging with the debugger. Sources are helpful in such cases
> too. If the program doesn't contain errors ;-) and is sipped without
> sources, they could be removed.
>
> 3. Annotations. They are used mainly by third party tools that statically
> analyze sources. They are rarely used at runtime.
>
> Docstrings will be read from the corresponding docstring file unless -OO is
> supplied. This will allow also to localize docstrings. Depending on locale
> or other settings different docstring file can be used.
>
> For suppressing line numbers and annotations new options can be added.

A deployed Python distribution generally has .pyc files for all of the
standard library. I don't think people want to lose the ability to
call help(), and unless I'm misunderstanding, that requires
docstrings. So this will mean twice as many files and twice as many
file-open calls to import from the standard library. What will be the
impact on startup time?

ChrisA