[ python-Bugs-1451466 ] reading very large files

[SourceForge.net]

Bugs item #1451466, was opened at 2006-03-16 18:21
Message generated for change (Comment added) made by richardchristen
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1451466&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: christen (richardchristen)
Assigned to: Nobody/Anonymous (nobody)
Summary: reading very large files

Initial Comment:
I work on the human genome
I extracted words from chromosomes using a suffix tree
(C compiled for 64 done on a SUN with 300 Go RAM, since
my suffix tree requires 150 Go RAM for chromosome 1,
the largest one)

this gave some >5 Go files, for example with 163763326
lines for chr 4, the one presently analyzed.

Using python 2.4.2 on a windows 32-computer (1.5 Go
RAM), reading this file line by line either

for li in file:
    do something

or

while li!='':
    li=file.readline()

I got problems seemingly around the 4 Go boundary
(after reading the problematic first line), for some
lines (not all), the li returned the correct content
but with the first word of the next line also within li
(see below)

As a result a simple
file1=open('1')
file2=open('2','w')
li=file1.readline()
while li!='':
    file2.write(li) 
    li=file1.readline()

produced a second file of only
163754385 lines
problem lines were "seemingly random", i.e. not in a
row, with the last line being OK.


The same code on the same file but on my OSX
64-dualcore machine went fine, despite the use of
default Python 2.2.3 and "file Python" showing it is a
Mach-0 executable ppc, i.e. a 32 bit app.

Everything was run from the command line.


the first file looks like that
...
TCAGCCACAGCAGAAAGTGA:\t33240 551212 751185
TCAGCCACAGCAGAAAGTGC:\t131324047
TCAGCCACAGCACTGTGTTA:\t61641912
....

the second file contains lines like these :
TCAGCCACAGCAGAAAGTGC:\t131324047TCAGCCACAGCAGAAGAAGA:  

which is 'first line'+'1rst word of next line'

PS1 : no problem to read the big file with UEdit on the
windows machine. Therefore the OS itself is not the
problem (also I transfered the bigfile from the Windows
to the Mac, if the file had had problems, it would have
been corrupted on the Mac)
PS2 : I tried python 2.3.5 on windows with the same
problem.
PS3: If needed, I can run the same test on a similar
file but for chromosome 8 which is slightly below the 4
Go limit (3.99).
PS4: I think I remember having done a similar parsing
on a Linux Athlon 64 monoCPU a month ago, with no trouble.

----------------------------------------------------------------------

>Comment By: christen (richardchristen)
Date: 2006-03-18 08:29

Message:
Logged In: YES 
user_id=1477618

In reply to previous comment

Are you sure this is a text file?
Yes I made it myself.
Besides I transfered it from the UX machine to the windows
one by ftp with change of the end of line character to the
window's kind. I checked with type myfile, that the control
character was indeed changed. Also, I mentioned that I
manually checked with Uedit, both in ASCII and HEX modes for
the akward lines.

"windows 32-computer" is too vague."
I agree, I should have been more specific:
System: Microsoft Windows 2000 Professionnel
Version 5.0.2195 Service Pack 4 version 2195
Mother card : ASUSTek 
System Model A7N8X-E
BIOS Phoenix AwardBIOS v6-00PG
Memory 1.5Go
Swap 2.4 Go

File System NTFS

Best Regards

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-03-18 03:33

Message:
Logged In: YES 
user_id=31435

"windows 32-computer" is too vague.  Which operating system
(Win95, Win98, WinME, NT, Win2K, WinXP), and which
filesystem (FAT, FAT32, NTFS)?

Are you sure this is a text file?  If it's a binary file,
then  all sorts of bad things can happen opening it in text
mode (which your sample code does).

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2006-03-18 01:35

Message:
Logged In: YES 
user_id=341410

Sounds like an issue with file objects on certain platforms
not being able to handle offsets of 2**32 or larger.  I
personally have read and written files > 4gb on the windows
platform, but I seem to recall having issues on 32 bit linux
some time in the past.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1451466&group_id=5470