Fast pythonic way to process a huge integer list

[Steven D'Aprano]

On Thu, 7 Jan 2016 01:36 pm, high5storage at gmail.com wrote:

> 
> I have a list of 163.840 integers. What is a fast & pythonic way to
> process this list in 1,280 chunks of 128 integers?


py> from itertools import izip_longest
py> def grouper(iterable, n, fillvalue=None):
...     "Collect data into fixed-length chunks or blocks"
...     # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx
...     args = [iter(iterable)] * n
...     return izip_longest(fillvalue=fillvalue, *args)
...
py> alist = range(163840)
py> count = 0
py> for block in grouper(alist, 128):
...     assert len(list(block)) == 128
...     count += 1
...
py> count
1280


This was almost instantaneous on my computer. 163840 isn't a very large
number of ints.




-- 
Steven