[Python-ideas] itertools.chunks()

[Oscar Benjamin]

On 7 April 2013 10:37, Wolfgang Maier
<wolfgang.maier at biologie.uni-freiburg.de> wrote:
>>Also I find myself often writing helper functions like these:
>>
>>def chunked(sequence,size):
>>       i = 0
>>       while True:
>>               j = i
>>               i += size
>>               chunk = sequence[j:i]
>>               if not chunk:
>>                       return
>>               yield chunk
>
> This is just an alternate version of the grouper recipe from the itertools
> documentation, just that grouper should be way faster and will also work
> with iterators.

It's not quite the same as grouper as it doesn't use fill values; I've
never found that I wanted fill values in this situation.

Also I'm not sure why you think that grouper would be "way faster". If
sequence is a concrete sequence with efficient random access (e.g. a
list or tuple) then grouper will just be extracting slices from it. If
it can do that faster than the sequence.__getslice__ method then
there's probably something wrong with the implementation of sequence.

I've written a generator function like the above before and it was
intended for numpy ndarrays. Since ndarray slices are views into the
original array, using a slice is a zero copy operation. This means
that using slices has time complexity of O(number of chunks) rather
than O(number of elements) for grouper. It also has a constant memory
requirement rather than O(chunk size) for grouper.


Oscar