problem with multiprocessing and defaultdict

[Wolodja Wentland]

On Tue, Jan 12, 2010 at 11:48 +0100, wiso wrote:
> They sent back the object filled with data. The problem is very simple: I 
> have a container, the container has a method read(file_name) that read a 
> huge file and fill the container with datas. I have more then 1 file to read 
> so I want to parallelize this process. The reading method is quite slow 
> because it involves regex.

Take a look at multiprocessing.Manager and use one to proxy access to a
*shared* container to your container from all processes.

If your container is a dict it is as easy as:

manager = multiprocessing.Manager()
managed_dict = manager.dict()
...

-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20100112/e64dd688/attachment-0001.sig>