python - handling HTTP requests asynchronously

[lordluke80 at gmail.com]

Il giorno sabato 7 maggio 2016 21:04:47 UTC+2, Michael Selik ha scritto:
> On Fri, May 6, 2016 at 3:01 AM <lordluke80 at gmail.com> wrote:
> 
> > The PDF is generated through an external API. Since currently is generated
> > on demand, this is handled synchronously via an HTTP request/response.
> 
> 
> Are you sending the request or are you receiving the request?
> If you are sending, you can just use threads as you are only doing IO.
> If you are receiving the requests and generating PDFs, you may want to use
> subprocesses if the PDF-generation is compute-intensive.
> 
> 
> > 3) multiprocessing module, with e.g. 10 as workers limit.
> >
> 
> multiprocessing.Pool is an easy way to use subprocesses
> multiprocessing.pool.ThreadPool is an easy way to use threads. It's not
> well documented, but has the exact same interface as Pool.
> 
> the goal is to use the API concurrently (e.g. send 10 or 100 http requests
> > simultaneously, depending on how many concurrent requests the API can
> > handle).
> >
> 
> Sounds like you want to use threads. How does the API let you know you're
> hitting it too frequently? Perhaps you want to code an exponential backoff
> and retry wrapper for your API requests.

Thanks for the reply.
Currently the django view that does the job does three http request:
- the first does a POST and send the payload used during the PDF rendering, the response contains a url to check the pdf generation progress;
- the second loop over that url, checking the progress of pdf generation. Once the response contains the keyword 'status': 'complete', then it give also a url for the file retrieval;
- the third one is a GET to retrieve the file, the reponse contains the binary content of the file, then this content is read and wrapped as attachment of a django http response, and then returned to the user.

the goal is to reuse the existing code as much as possible, possibly doing concurrently, and saving the file instead on a folder.