requests.{get,post} timeout

[Chris Angelico]

On Wed, Aug 23, 2017 at 10:52 PM, Jon Ribbens <jon+usenet at unequivocal.eu> wrote:
> On 2017-08-23, Chris Angelico <rosuav at gmail.com> wrote:
>> On Wed, Aug 23, 2017 at 9:10 PM, Jon Ribbens <jon+usenet at unequivocal.eu> wrote:
>>> I am interested to learn what you mean by "with a thread". How would
>>> one execute a requests, er, request in a thread with a proper timeout?
>>
>> Assuming that by "proper timeout" you mean "limit the entire
>> download's wall time": Use one thread to do the request, and another
>> thread to monitor it. Generally, the monitoring thread is your UI
>> thread (most commonly, the main thread of the program), though it
>> doesn't have to be. If the monitoring thread decide that the
>> requesting thread has taken too long, it can cut it off and report
>> failure to the user.
>
> Yes, what I was interested to learn was how the monitoring thread can
> "cut off" the requesting thread.

Ah, I see. That partly depends on your definition of "cut off", and
how it needs to interact with other things. I'm not sure about the
requests module specifically, but one very effective method of
terminating a network query is to close the socket (since resources
are owned by processes, not threads, any thread can close the
underlying socket connection); it won't instantly terminate the
thread, but it will result in an end-of-connection read shortly
afterwards. You'd have to do some sort of high-level "hey, I'm
cancelling this request" for the user's benefit, too - or maybe the
user initiated it in the first place. For example, in a web browser,
you can hit Esc to cancel the current page download; that can
immediately close the socket, and it probably has to tell the cache
subsystem not to hold that data, and maybe some logging and stuff, but
in terms of aborting the download, closing the socket is usually
sufficient.

How this interacts with the actual specific details of the requests
module I'm not sure, especially since a lot of the work usually
happens inside requests.get() or one of its friends; but if you
explicitly construct a Request object before sending it [1], it would
be conceivable to have a "req.close()" or "req.abort()" method that
closes the underlying socket. (Or possibly that goes on the
PreparedRequest. Or maybe the Session. I don't know; normally, when I
use requests, I use the higher level interfaces.) It would need to be
a feature provided by requests ("abort this request"), as it would
potentially interact with connection pooling and such. But at the
simplest level, closing the socket WILL abort the connection.

[1] http://docs.python-requests.org/en/master/user/advanced/#prepared-requests

ChrisA