Strange behavior with iterables - is this a bug?

[akameswaran at gmail.com]

My original concern and reason for goint the iterator/generator route
was exactly for large large lists :)  Unnecessary in this example, but
exactly what I was exploring.  I wouldn't be using list comprehension
for generating the permutiations.  Where all this came from was
creating a generator/iterator to handle very large permutations.



Gary Herron wrote:
> akameswaran at gmail.com wrote:
>
> >Gary Herron wrote:
> >
> >
> >>List comprehension is a great shortcut, but when the shortcut starts
> >>causing trouble, better to go with the old ways. You need to reopen each
> >>file each time you want to iterate through it. You should be able to
> >>understand the difference between these two bits of code.
> >>
> >>The first bit opens each file but uses (two of them) multiple times.
> >>Reading from a file at EOF returns an empty sequence.
> >>
> >>The second bit opened the file each time you want to reuse it. That
> >>works correctly.
> >>
> >>And that suggest the third bit of correctly working code which uses list
> >>comprehension.
> >>
> >># Fails because files are opened once but reused
> >>f1 = open('word1.txt')
> >>f2 = open('word2.txt')
> >>f3 = open('word3.txt')
> >>for i1 in f1:
> >>  for i2 in f2:
> >>    for i3 in f3:
> >>      print (i1.strip(),i2.strip(),i3.strip())
> >>
> >>and
> >>
> >># Works because files are reopened for each reuse:
> >>f1 = open('word1.txt')
> >>for i1 in f1:
> >>f2 = open('word2.txt')
> >>for i2 in f2:
> >>f3 = open('word3.txt')
> >>for i3 in f3:
> >>print (i1.strip(),i2.strip(),i3.strip())
> >>
> >>and
> >>
> >># Also works because files are reopened for each use:
> >>print [(i1.strip(),i2.strip(),i3.strip())
> >>          for i1 in open('word1.txt')
> >>            for i2 in open('word2.txt')
> >>              for i3 in open('word3.txt')]
> >>
> >>Hope that's clear!
> >>
> >>Gary Herron
> >>
> >>
> >
> >
> >My original problem was with recursion.  I explicitly nested it out to
> >try and understand the behavior - and foolishly looked in the wrong
> >spot for the problem, namely that file is not reitreable.  In truth I
> >was never concerned about file objects, the problem was failing with my
> >own custom iterators (wich also were not reiterable) and I switched to
> >file, to eliminate possible code deficiencies on my own part.  I was
> >simply chasing down the wrong problem.   As was pointed out to me in a
> >nother thread - the cleanest implementation which would allow me to use
> >one copy of the file (in my example the files are identical) would be
> >to use a trivial iterator class that opens the file, uses tell to track
> >position and seek to set position, and returns the appropriate line for
> >that instance - thus eliminating unnecessary file opens and closes.
> >
> >
> >
> I see.
>
> I wouldn't call "tell" and "seek" clean.  Here's another suggestion.  Use
>   l1 = open(...).readlines()
> to read the whole file into a (nicely reiterable) list residing in
> memory, and then iterate through the list as you wish.  Only if your
> files are MANY megabytes long would this be a problem with memory
> consumption.  (But if they were that big, you wouldn't be trying to find
> all permutations would you!)
> 
> Gary Herron