itertools.groupby

[Oscar Benjamin]

On 22 April 2013 15:24, Neil Cerutti <neilc at norwich.edu> wrote:
>
> Hrmmm, hoomm. Nobody cares for slicing any more.
>
> def headered_groups(lst, header):
>     b = lst.index(header) + 1
>     while True:
>         try:
>             e = lst.index(header, b)
>         except ValueError:
>             yield lst[b:]
>             break
>         yield lst[b:e]
>         b = e+1

This requires the whole file to be read into memory. Iterators are
typically preferred over list slicing for sequential text file access
since you can avoid loading the whole file at once. This means that
you can process a large file while only using a constant amount of
memory.

>
> for group in headered_groups([line.strip() for line in open('data.txt')],
>         "Starting a new group"):
>     print(group)

The list comprehension above loads the entire file into memory.
Assuming that .strip() is just being used to remove the newline at the
end it would be better to just use the readlines() method since that
loads everything into memory and removes the newlines. To remove them
without reading everything you can use map (or imap in Python 2):

with open('data.txt') as inputfile:
    for group in headered_groups(map(str.strip, inputfile)):
        print(group)


Oscar