[issue39221] Cross compiled python installed wrong version of lib2to3/Grammar pickle

[Andrew Aladjev]

Andrew Aladjev <aladjev.andrew at gmail.com> added the comment:

Hello. I've received more problems with pickle generation, reviewed source code, invented better bike for solving it. So I can provide full description of this issue and reduce other developers time on debugging.

Problem:

1. Martin v. Löwis <martin at v.loewis.de> introduced pickle generation Wed Mar 19 04:43:46 2008 commit https://github.com/python/cpython/commit/5e37baea8007cb64b65a180e4d6c80de292a8a4a#diff-c9bd4064884726c21716df7118dcaeb72a63cc8ddd49484c7f4676934b37d9bb

2. Martin provided code for generating of pickle file: head + tail + ".".join(map(str, sys.version_info)) + ".pickle". This code has never been changed and comes as is in 2021 year.

Code means that current python interpreter uses its own version for creating a new pickle file for new cpython interpreter. It will work only when current python interpreter equals to new cpython interpreter. This code is broken by design.

3. We can see file "Lib/lib2to3/patcomp.py" in the commit above, today (2021) it is a part of "Lib/lib2to3/pygram.py". It comes with the following code:

_GRAMMAR_FILE = ... "Grammar.txt"
_PATTERN_GRAMMAR_FILE = ... "PatternGrammar.txt"

4. We can review step-by-step "load_grammar" real world usage from "Lib/lib2to3/pgen2/driver.py":

driver.load_grammar(_GRAMMAR_FILE)
driver.load_grammar(_PATTERN_GRAMMAR_FILE)

def load_grammar(gt="Grammar.txt", gp=None
gp = head + tail + ".".join(map(str, sys.version_info)) + ".pickle"

if force or not _newer(gp, gt):
  logger.info("Generating grammar tables from %s", gt)

def _newer(a, b):
  if not os.path.exists(a):
    return False

This "code" comes as is today as a part of latest stable python 3.9.5. This workaround means that when pickle files doesn't exist - python will recreate it in runtime inside "lib/lib2to3" folder.

This workaround will ruin your production if "lib/lib2to3" is readonly and pickle files are not inside. Everybody will try to use this workaround as a security hole. But it looks like nobody from cpython cares.

Solution:

1. Apply provided remove_python_version_from_pickle.patch.
2. Cross compile new cpython.
3. Cross compile portage.
4. Chroot.
5. Run python -c "import lib2to3.pygram", it will create pickles without version postfix, stop python from creating junk in runtime, fix sandbox permission issues temporaly.
6. Reinstall portage.
7. Reinstall cpython without patch, it will recreate pickles during installation.
8. Reinstall portage.

This is just bike on the top of Martin's workaround. The best variant is to ignore cpython (and maybe python) completely, ignore software written in python where possible.

----------
Added file: https://bugs.python.org/file50153/remove_python_version_from_pickle.patch

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39221>
_______________________________________