[issue13146] Writing a pyc file is not atomic

[Antoine Pitrou]

New submission from Antoine Pitrou <pitrou at free.fr>:

One of the buildbots recently showed the following failure:

======================================================================
ERROR: test_rapid_restart (test.test_multiprocessing.WithProcessesTestManagerRestart)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildslave/3.x.murray-gentoo/build/Lib/test/test_multiprocessing.py", line 1442, in test_rapid_restart
    queue = manager.get_queue()
  File "/var/lib/buildslave/3.x.murray-gentoo/build/Lib/multiprocessing/managers.py", line 670, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "/var/lib/buildslave/3.x.murray-gentoo/build/Lib/multiprocessing/managers.py", line 568, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "/var/lib/buildslave/3.x.murray-gentoo/build/Lib/multiprocessing/connection.py", line 778, in XmlClient
    import xmlrpc.client as xmlrpclib
  File "/var/lib/buildslave/3.x.murray-gentoo/build/Lib/xmlrpc/client.py", line 137, in <module>
    import http.client
  File "/var/lib/buildslave/3.x.murray-gentoo/build/Lib/http/client.py", line 69, in <module>
    import email.parser
  File "/var/lib/buildslave/3.x.murray-gentoo/build/Lib/email/parser.py", line 12, in <module>
    from email.feedparser import FeedParser
  File "/var/lib/buildslave/3.x.murray-gentoo/build/Lib/email/feedparser.py", line 28, in <module>
    from email import policy
EOFError: EOF read where not expected

(http://www.python.org/dev/buildbot/all/builders/x86%20Gentoo%203.x/builds/942/steps/test/logs/stdio)

"EOF read where not expected" comes from reading a pyc file in marshal.c. It is raised when the pyc file is somehow truncated or incomplete. Writing and reading the same pyc file is protected by the import lock when in a single interpreter, but not when running several Python processes at the same time (which test_multiprocessing obviously does).

Under POSIX, import.c could do the traditional write-then-rename dance which guarantees that the file contents appear atomically.
And so could importlib.

----------
components: Interpreter Core
messages: 145313
nosy: brett.cannon, ncoghlan, pitrou, r.david.murray
priority: normal
severity: normal
status: open
title: Writing a pyc file is not atomic
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13146>
_______________________________________