[New-bugs-announce] [issue16743] mmap accepts files > 1 GB, but processes only 1 GB

[Marc Schlaich]

New submission from Marc Schlaich:

Platform: Windows 7 64 bit
Interpreter: Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit Intel)] on win32

Here are the steps to reproduce:

1. Create a big file (5 GB):

with open('big', 'wb') as fobj:
    for _ in xrange(1024 * 1024 * 5):
        fobj.write('1' + '0' * 1023)

2. Open and process it with `mmap`:

import mmap
import re
import sys

with open('big', 'rb') as fobj:
    data = mmap.mmap(fobj.fileno(), 0, access=mmap.ACCESS_READ)
    print data.size()
    try:
        counter = 0
        for match in re.finditer('1' + '0' * 1023, data):
            counter += 1
        print len(data[1073740800:1073741824]) # (1 GB - 1024, 1 GB)
        print len(data[1073741824:1073742848]) # (1 GB, 1 GB + 1024)
    finally:
        data.close()

    print counter

This returns the following lines:

    5368709120
    1024
    0
    1048576

So this is a behavioral issue. `mmap` accepts a file which cannot fit in the interpreter memory but fits in the system memory. On processing the data, it only reads data until the maximum interpreter memory is reached (1 GB).

----------
components: None
messages: 177879
nosy: schlamar
priority: normal
severity: normal
status: open
title: mmap accepts files > 1 GB, but processes only 1 GB
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue16743>
_______________________________________