asyncio question

Joseph L. Casale jcasale at activenetwerx.com
Thu Mar 13 17:24:15 EDT 2014


I have a portion of code I need to speed up, there are 3 api calls to an external system
where the first enumerates a large collection of objects I then loop through and perform
two additional api calls each. The first call is instant, the second and third per object are
very slow. Currently after accumulating all the data I write the relevant data into a database.

I have the ability to hold all this in memory and dump it once fully accumulated, so performing
the second and third call in parallel with fixed batches would be great,

I took a look at coroutines and some skeleton code worked fine, but I am not sure how to
perform the acquisition in fixed groups like I might for example with multiprocessing and
a pool of workers.

Anyone done something like this or have an opinion?

Thanks,
jlc


More information about the Python-list mailing list