[ python-Bugs-1636950 ] Newline skipped in "for line in file"

[SourceForge.net]

Bugs item #1636950, was opened at 2007-01-16 10:56
Message generated for change (Comment added) made by mark-roberts
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1636950&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Andy Monthei (amonthei)
Assigned to: Nobody/Anonymous (nobody)
Summary: Newline skipped in "for line in file"

Initial Comment:
When processing huge fixed block files of about 7000 bytes wide and several hundred thousand lines long some pairs of lines get read as one long line with no line break when using "for line in file:".  The problem is even worse when using the fileinput module and reading in five or six huge files consisting of 4.8 million records causes several hundred pairs of lines to be read as single lines. When a newline is skipped it is usually followed by several more in the next few hundred lines. I have not noticed any other characters being skipped, only the line break.

O.S. Windows (5, 1, 2600, 2, 'Service Pack 2')
Python 2.5

----------------------------------------------------------------------

Comment By: Mark Roberts (mark-roberts)
Date: 2007-01-17 23:24

Message:
Logged In: YES 
user_id=1591633
Originator: NO

How wide are the min and max widths of the lines?  This problem is of
particular interest to me.

----------------------------------------------------------------------

Comment By: Andy Monthei (amonthei)
Date: 2007-01-17 15:58

Message:
Logged In: YES 
user_id=1693612
Originator: YES

I can not upload the files that trigger this because of the data that is
in them but I am working on getting around that.

In my data line 617391 in a fixed block file of 6990 bytes wide gets read
in with the next line after it.  The line break is 0d0a (same as the
others) where the bug happens so I am wondering if it is a buffer issue
where the linebreak falls at the edge, however no other characters are ever
missed. The total file is 888420 lines and this happens in four spots.

I will hopefully have a file to send soon.

----------------------------------------------------------------------

Comment By: Brett Cannon (bcannon)
Date: 2007-01-16 16:33

Message:
Logged In: YES 
user_id=357491
Originator: NO

Do you happen to have a sample you could upload that triggers the bug?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1636950&group_id=5470