Pool Module: iterator does not yield consistently with different chunksizes

syockit syockit at gmail.com
Fri Jul 2 04:44:02 EDT 2010


I've been playing around with custom iterators to map into Pool. When
I run the code below:

def arif(arr):
    return arr

def permutate(n):
    k = 0
    a = list(range(6))
    while k<n:
        for i in range(6):
            a.insert(0, a.pop(5)+6)
        #yield a[:]   <-- produces correct results
        yield a
        k += 1
    return

def main():
    from multiprocessing import Pool
    pool = Pool()
    chksize = 15
    for x in pool.imap_unordered(arif, permutate(100), chksize):
        print(x)

if __name__=="__main__":
    main()

.... will output something like this:


[36, 37, 38, 39, 40, 41]
[36, 37, 38, 39, 40, 41]
[36, 37, 38, 39, 40, 41]
[36, 37, 38, 39, 40, 41]
[36, 37, 38, 39, 40, 41]
[36, 37, 38, 39, 40, 41]
[72, 73, 74, 75, 76, 77]
[72, 73, 74, 75, 76, 77]
[72, 73, 74, 75, 76, 77]
[72, 73, 74, 75, 76, 77]
[72, 73, 74, 75, 76, 77]
[72, 73, 74, 75, 76, 77]
[108, 109, 110, 111, 112, 113]
[108, 109, 110, 111, 112, 113]
[108, 109, 110, 111, 112, 113]
[108, 109, 110, 111, 112, 113]
[108, 109, 110, 111, 112, 113]
[108, 109, 110, 111, 112, 113]
[144, 145, 146, 147, 148, 149]

... where results are duplicated number of times equal to chunk size,
and the results between the gap are lost. Using a[:] instead, i get:

[6, 7, 8, 9, 10, 11]
[12, 13, 14, 15, 16, 17]
[18, 19, 20, 21, 22, 23]
[24, 25, 26, 27, 28, 29]
[30, 31, 32, 33, 34, 35]
[36, 37, 38, 39, 40, 41]
[42, 43, 44, 45, 46, 47]
[48, 49, 50, 51, 52, 53]

.... it comes out okay. Any explanation for such behavior?

Ahmad Syukri



More information about the Python-list mailing list