[issue19395] lzma hangs for a very long time when run in parallel using python's muptiprocessing module?
cantor
report at bugs.python.org
Fri Oct 25 21:20:43 CEST 2013
New submission from cantor:
import lzma
from functools import partial
import multiprocessing
def run_lzma(data,c):
return c.compress(data)
def split_len(seq, length):
return [str.encode(seq[i:i+length]) for i in range(0, len(seq), length)]
def lzma_mp(sequence,threads=3):
lzc = lzma.LZMACompressor()
blocksize = int(round(len(sequence)/threads))
strings = split_len(sequence, blocksize)
lzc_partial = partial(run_lzma,c=lzc)
pool=multiprocessing.Pool()
lzc_pool = list(pool.map(lzc_partial,strings))
pool.close()
pool.join()
out_flush = lzc.flush()
return b"".join(lzc_pool + [out_flush])
sequence = 'AAAAAJKDDDDDDDDDDDDDDDDDDDDDDDDDDDDGJFKSHFKLHALWEHAIHWEOIAH IOAHIOWEHIOHEIOFEAFEASFEAFWEWWWWWWWWWWWWWWWWWWWWWWWWWWWWWEWFQWEWQWQGEWQFEWFDWEWEGEFGWEG'
lzma_mp(sequence,threads=3)
----------
components: ctypes
messages: 201278
nosy: cantor
priority: normal
severity: normal
status: open
title: lzma hangs for a very long time when run in parallel using python's muptiprocessing module?
type: behavior
versions: Python 3.3
_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue19395>
_______________________________________
More information about the Python-bugs-list
mailing list