[Python-ideas] Iterating non-newline-separated files should be easier

[Paul Moore]

On 19 July 2014 10:01, Steven D'Aprano <steve at pearwood.info> wrote:
> I open a file with some record-separator
>
>   fp = open(filename, newline="\0")
>
> then pass it to a function:
>
>   spam(fp)
>
> which assumes that each chunk ends with a linefeed:
>
>    assert next(fp).endswith('\n')

I will often do

for line in fp:
    line = line.strip()

to remove the line ending ("record separator"). This fails if you have
an arbitrary separator. And for that matter, how would you remove an
arbitrary separator? Maybe line = line[:-1] works, but what if at some
point people ask for multi-character separators ("\n\n" for "paragraph
separated", for example - ignoring the universal newline complexities
in that).

A splitrecord method still needs a means for code to to remove the
record separator, of course, but the above demonstrates how reusing
line separation could break the assumptions of *current* code.

Paul