looping through a file

[Andrew Bennetts]

On Wed, Apr 30, 2003 at 05:12:46PM +0000, Alex Martelli wrote:
> Andrew Bennetts wrote:
>     ...
> > 
> > I thought there might be something in 2.3's itertools module to help with
> > this -- I was hoping you'd be able to write something like:
> > 
> >     InFile = open('users.dat', 'r')
> >     
> >     for username, password in multistep(InFile, 2):
> 
> Careful!!!  You'd still need to strip them before you compare them,
> else the trailing \n or any other whitespace will make the comparisons
> fail.  imap(str.strip, multistep(InFile, 2)) might work here (if
> multistep existed).

Oops!  Yes, I forgot about that.

> > Where "multistep" would probably have a better name, and be defined like:
> > 
> >     def multistep(iterable, count):
> >         # XXX: What should happen when len(iterable) % count != 0 ?
> 
> I think the last trailing part X with 0<=len(X)<count should be dropped.
> Otherwise, unpacking assigments such as the above would fail -- and,
> "padding with None's" isn't in fashion any more;-).

That's my inclination too, because it's the easiest thing to do :)

> >         iterator = iter(iterable)
> >         while 1:
> >             values = []
> >             for i in range(count):
> >                 values.append(iterator.next())
> >             yield values
> 
> I think I'd prefer expressing this loop (naming issues apart) as:
> 
>    while True: return [iterator.next() for i in range(count)]

I think you meant to use "yield", not "return".

But yes, that occurred to me after I posted :)   It's much nicer that way.

> > 
> > I couldn't see anything in itertools to save me writing that out.  But I
> > just figured out how to do it with itertools -- izip can do it:
> > 
> >     def multistep(iterable, count):
> >         return itertools.izip(*((iter(iterable),)*count))
> > 
> > I guess you could write this directly:
> > 
> >     iterator = iter(InFile)
> 
> Unneeded, btw: iter(inFile) is inFile (I also exploit that in my
> less-elegant snippet above where I call inFile.next(), secure that
> this will consume a line).

Hmm.  I couldn't remember if that was safe or not; I tested this stuff
interactively with xrange, and 

    x = xrange(10)
    print list(izip(x, x))

didn't do what I expected until I remembered to change the first line to

    x = iter(xrange(10))

So I played it safe.

> >     for username, password in izip(iterator, iterator):
> >         ...
> > 
> > So it's probably not worth defining a seperate function...
> 
> I think it is (if anything can still be added to new module itertools
> in 2.3 -- I'm not sure ANY functionality can be added after beta-1), as
> it's frequently needed.  But, if it isn't, the "trick" with the
> iterator explicitly repeated is good and IMHO worth keeping in mind.
> 
> But one does still need the .strip call...;-)

Yes :)

-Andrew.