How to benchmark a HTTP connection with requests module?

[Etienne Robillard]

Hi Dennis,

Here's my code so far:

import os
import requests
import threading
import time

from Queue import Queue
from test_support import unittest

test_queue = Queue()

def worker(ident, url, queue):

     while True:
         start = time.clock()
         r = requests.get(url)
         end = time.clock()
         queue.put((ident, r.status_code, end - start))
         queue.task_done()
     return queue

class HTTPBenchmarkTestCase(unittest.TestCase):

     url = 'http://localhost/benchmark/'
     threads = 5

     def setUp(self):
         self.hold = True


     def test_benchmark_concurrency(self):
         for i in range(self.threads):
             t = threading.Thread(target=worker, args=(i, self.url, 
test_queue))
             t.daemon = True
             t.start()
             obj = test_queue.get()
             print obj
         test_queue.join()


erob at marina:~/src/django-hotsauce-0.9/tests$ pypy ./run.py -C benchmarks
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
(0, 200, 0.12999503499999943)
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
(1, 200, 0.053242928999999606)
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
(0, 200, 0.11221248100000025)
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
(1, 200, 0.12075822500000033)
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
(0, 200, 0.12590276600000028)
.
----------------------------------------------------------------------
Ran 1 test in 0.468s

OK


What do you think? :-)

Etienne


Le 2018-02-16 à 10:58, Dennis Lee Bieber a écrit :
> On Fri, 16 Feb 2018 06:22:04 -0500, Etienne Robillard <tkadm30 at yandex.com>
> declaimed the following:
>
>> Hi Dennis,
>>
>> Nice pseudo code! :-)
>>
>> Is it possible benchmark/measure the latency of a HTTP connection for
>> each threads by timing the duration of the requests.get method to complete?
>>
> 	Wrap each .get with calls for start time and end time? (Need to check
> if time.time or time.clock is more precise on your hardware, docs recommend
> time.clock for benchmarking)
>
>>> def worker(ID):
>>> 	while hold: pass
> 	st = time.clock()
>>> 	r = requests.get(...)
> 	en = time.clock()
> 	resultQ.put( (ID, r.status_code, en - st) )
> 	#return duration as third parameter
>
>
>>> 	for _ in range(NUMBEROFREQUESTS):
> 		(ID, code, duration) = resultQ.get()
>>> 		requestTasks[ID].join()

-- 
Etienne Robillard
tkadm30 at yandex.com
https://www.isotopesoftware.ca/