[issue23838] linecache and MemoryError

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

When format a traceback of MemoryError, linecache tries to read all lines of source file and likely raises a MemoryError. For example:

http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/9712/steps/test/logs/stdio

Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1280, in runtest_inner
    test_runner()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 5340, in test_main
    support.run_unittest(*tests)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/support/__init__.py", line 1808, in run_unittest
    _run_suite(suite)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/support/__init__.py", line 1774, in _run_suite
    result = runner.run(suite)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/runner.py", line 176, in run
    test(result)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 122, in run
    test(result)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 122, in run
    test(result)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 625, in __call__
    return self.run(*args, **kwds)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 585, in run
    self._feedErrorsToResult(result, outcome.errors)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 518, in _feedErrorsToResult
    result.addError(test, exc_info)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/runner.py", line 67, in addError
    super(TextTestResult, self).addError(test, err)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 17, in inner
    return method(self, *args, **kw)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 115, in addError
    self.errors.append((test, self._exc_info_to_string(err, test)))
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 186, in _exc_info_to_string
    exctype, value, tb, limit=length, capture_locals=self.tb_locals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 456, in __init__
    capture_locals=capture_locals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 341, in extract
    f.line
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 270, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 44, in getlines
    return updatecache(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 134, in updatecache
    lines = fp.readlines()
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 532, in main
    result = runtest(*args, **kwargs)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 967, in runtest
    display_failure=False)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1304, in runtest_inner
    msg = traceback.format_exc()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 161, in format_exc
    return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 115, in format_exception
    type(value), value, tb, limit=limit).format(chain=chain))
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 456, in __init__
    capture_locals=capture_locals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 341, in extract
    f.line
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 270, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 44, in getlines
    return updatecache(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 134, in updatecache
    lines = fp.readlines()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
MemoryError
Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/__main__.py", line 3, in <module>
    regrtest.main_in_temp_cwd()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1564, in main_in_temp_cwd
    main()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 738, in main
    raise Exception("Child error on {}: {}".format(test, result[1]))
Exception: Child error on test_socket: 
make: *** [buildbottest] Error 1
program finished with exit code 2

Proposed patch clears the cache and repeats an attempt to read source file, and returns empty result if the second attempt fails (as when the source file can't be read).

I hesitate how to classify this issue. On one hand, it looks as new feature, on other hand it is related to testing on all versions.

----------
components: Library (Lib)
files: linecache_memoryerror.patch
keywords: patch
messages: 239781
nosy: ezio.melotti, haypo, michael.foord, pitrou, serhiy.storchaka
priority: normal
severity: normal
stage: patch review
status: open
title: linecache and MemoryError
Added file: http://bugs.python.org/file38779/linecache_memoryerror.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23838>
_______________________________________