[New-bugs-announce] [issue32054] Creating RPM on Python 2 works, but Python 3 fails because of sys.implementation.cache_tag

[Pedro]

New submission from Pedro <pgacv2+pythonbugs at gmail.com>:

I am trying to create an RPM on SLES12 SP2. (I think that corresponds to OpenSUSE 42.2.) This is my setup.py file, nothing special:

    import setuptools
    setuptools.setup(name='MyApp',
                     version='1.2.3',
                     options={'bdist_rpm': {'post_install': 'post_install.sh',
                                            'post_uninstall': 'post_uninstall.sh'}})

Running `python setup.py bdist_rpm` with Python 2 works. However, running `python3 setup.py bdist_rpm` outputs the following:

    running bdist_rpm
    running egg_info
    writing top-level names to MyApp.egg-info/top_level.txt

    ...

    byte-compiling /home/pedro/MyApp/build/bdist.linux-x86_64/rpm/BUILDROOT/MyApp-1.2.3-1.x86_64/usr/lib/python3.4/site-packages/MyApp/my_file.py to my_file.cpython-34.pyc

    ...

    Processing files: MyApp-1.2.3-1.noarch
    error: File not found: /home/pedro/MyApp/build/bdist.linux-x86_64/rpm/BUILDROOT/MyApp-1.2.3-1.x86_64/usr/lib/python3.4/site-packages/MyApp/my_file.py
    
    RPM build errors:
        File not found: /home/pedro/MyApp/build/bdist.linux-x86_64/rpm/BUILDROOT/MyApp-1.2.3-1.x86_64/usr/lib/python3.4/site-packages/MyApp/my_file.py
    error: command 'rpmbuild' failed with exit status 1

The problem appears to be that setuptools is generating a file that ends in `.cpython-34.pyc`, and later looks for a file without the `.cpython-34` but can't find it.

The RPM generation process on Python 3 goes through `distutils.util.byte_compile()`, which in turn calls `importlib.util.cache_from_source(path)`, where `path` is the file to be byte-compiled. `cache_from_source()` injects the value of `sys.implementation.cache_tag` (which is equal to 'cpython-34' on SLES12 SP2) into the filename of the compiled file. This attribute of `sys` does not exist in Python 2. So it looks like `setuptools` alters the filename with that tag during byte compilation, but later forgets that it modified the filename and fails because it's looking for the original name.

I ended up working around this by patching `distutils.file_util.write_file` to eliminate the .pyc entries from INSTALLED_FILES:

    orig_bytecode_var = os.environ.get('PYTHONDONTWRITEBYTECODE', '')
    os.environ['PYTHONDONTWRITEBYTECODE'] = '1'
    orig_write_file = distutils.file_util.write_file

    def my_patch(*args, **kwargs):
        new_args = list(args)
        if args[0] == 'INSTALLED_FILES':
            new_args[1] = [fname for fname in args[1] if fname[-4:] not in ('.pyc', '.pyo')]
        orig_write_file(*new_args, **kwargs)
    distutils.file_util.write_file = my_patch

    setuptools.setup(name='MyApp',
                     version='1.2.3',
                     options={'bdist_rpm': {'post_install': 'post_install.sh',
                                            'post_uninstall': 'post_uninstall.sh'}})

    os.environ['PYTHONDONTWRITEBYTECODE'] = orig_bytecode_var
    distutils.file_util.write_file = orig_write_file


But I don't think I should have to do anything special to build an RPM on Python 3 vs. Python 2. Is this expected behavior?

----------
components: Distutils
messages: 306400
nosy: dstufft, eric.araujo, pgacv2
priority: normal
severity: normal
status: open
title: Creating RPM on Python 2 works, but Python 3 fails because of sys.implementation.cache_tag
type: behavior
versions: Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue32054>
_______________________________________