What is the fastest way to do 400 HTTP requests using requests library?

[Ian Kelly]

On Mon, Jan 4, 2016 at 4:38 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Tue, 5 Jan 2016 07:50 am, livemsn22 at gmail.com wrote:
>
>> So what is the fastest way to make 400 HTTP requests using "requests"
>> library and also using tor proxy?
>
>
> Since this will be I/O bound, not CPU bound, probably use separate threads.
>
> Push the 400 requests into a queue, then create N threads, where N will need
> to be determined by experiment, but will probably be something like 4 or 8,
> and let each thread pop a request from the queue as needed.
>
> Are you experienced with threads? Do you need further information about
> using threads and queues?

Also see the concurrent.futures module in the standard library, which
makes this sort of setup very simple to implement.