[New-bugs-announce] [issue7187] importlib/_bootstrap.py write_bytecode fails if it can't open the .pyc file for writing

[Dave Malcolm]

New submission from Dave Malcolm <dmalcolm at redhat.com>:

I'm working on an RPM package of Python 3.1.1 for Fedora [1].  Within my RPMs are 
precompiled .pyc and .pyo files that are not writable by users, and are hardlinked 
together if they have equal content.  For example:
$ ls -al /usr/lib/python3.1/tkinter/__init__.*
-rw-r--r--. 1 root root 157859 2009-10-22 15:43 
/usr/lib/python3.1/tkinter/__init__.py
-rw-r--r--. 2 root root 226817 2009-10-22 15:42 
/usr/lib/python3.1/tkinter/__init__.pyc
-rw-r--r--. 2 root root 226817 2009-10-22 15:42 
/usr/lib/python3.1/tkinter/__init__.pyo

I see numerous IOError permission errors running the regression test suite, with 
importlib/_bootstrap.py trying and failing to open the .pyc files for writing.  
The IOError bubbles up each time and causes the test to fail.

Here's a minimal reproducer (assuming such an install as above)
>>> from tkinter import Tcl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 151, in decorated
    return fxn(self, module)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 399, in load_module
    return self._load_module(module)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 324, in _load_module
    code_object = self.get_code(module.__name__)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 443, in get_code
    self.write_bytecode(fullname, data)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 171, in inner
    return method(self, name, *args, **kwargs)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 525, in write_bytecode
    file = _io.FileIO(bytecode_path, 'w')  # Assuming bytes.
IOError: [Errno 13] Permission denied: '/usr/lib/python3.1/tkinter/__init__.pyc'

Upon investigating Lib/importlib/_bootstrap.py, the logic for handling permission 
errors in "write_bytecode" appears to be wrong:
  - the initializer for _io.FileIO can raise an "IOError" if it can't open the 
.pyc file for writing
    - see Modules/_io/fileio.c:fileio_init, the exception is indeed raised if the 
call to:
        self->fd = open(name, flags, 0666);
    fails.

So it looks like the try/except clause needs to be extended to also cover opening 
the file for writing, as well as writing the bytes to it.

I'm attaching a patch that makes this change.

With this change, the above failing import succeeds, and an invocation of 
/usr/lib/python3.1/test/regrtest.py improves from:
276 tests OK.
39 tests failed:
    test_cmd_line test_codeccallbacks test_codecencodings_cn
    test_codecencodings_hk test_codecencodings_jp
    test_codecencodings_kr test_codecencodings_tw test_cprofile
    test_distutils test_docxmlrpc test_email test_heapq
    test_htmlparser test_httpservers test_imp test_lib2to3
    test_linecache test_modulefinder test_multiprocessing test_osx_env
    test_plistlib test_pyclbr test_pydoc test_runpy test_socket
    test_sqlite test_sundry test_tcl test_threading_local test_tk
    test_ttk_guionly test_ttk_textonly test_uuid test_warnings
    test_wsgiref test_xml_etree_c test_xmlrpc test_xmlrpc_net
    test_zipfile
to:
304 tests OK.
10 tests failed:
    test_email test_httpservers test_imp test_lib2to3 test_linecache
    test_socket test_tk test_ttk_guionly test_ttk_textonly
    test_zipfile

(I'm working on these other failures; some of them are due to errors in my 
packaging)

FWIW the initial checkin of py3k/Lib/importlib/_bootstrap.py appears to have the 
initialization of the FileIO outside the try/except block:
http://svn.python.org/view/python/branches/py3k/Lib/importlib/_bootstrap.py?
view=markup&pathrev=68698

though at that time it was _fileio._FileIO, rather than _io.FileIO.  (did FileIO 
only test for perms upon writing, and change behavior to test upon opening?  or 
has this case always led to this failure?  or am I misreading this?)

Thanks
Dave

[1] https://bugzilla.redhat.com/show_bug.cgi?id=526126

----------
components: Library (Lib)
files: python-3.1.1-importlib-fix-handling-of-readonly-pyc-files.patch
keywords: patch
messages: 94371
nosy: dmalcolm
severity: normal
status: open
title: importlib/_bootstrap.py write_bytecode fails if it can't open the .pyc file for writing
type: behavior
versions: Python 3.1
Added file: http://bugs.python.org/file15185/python-3.1.1-importlib-fix-handling-of-readonly-pyc-files.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7187>
_______________________________________