requests.{get,post} timeout

[Chris Angelico]

On Fri, Aug 25, 2017 at 3:40 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
> Chris Angelico <rosuav at gmail.com>:
>
>> On Fri, Aug 25, 2017 at 12:17 AM, Jon Ribbens
>> <jon+usenet at unequivocal.eu> wrote:
>>> By that, do you mean "kill the process"? That's obviously not a
>>> sensible answer in general, especially given we were including
>>> processes which have no terminal or user sitting there watching them.
>>
>> Only in the sense that "kill" is the Unix term for "send signal".
>> Python catches the signal, the system call terminates with EINTR, and
>> the exception is raised. Give it a try.
>
> Signals are an arcane Unix communication method. I strongly recommend
> against using signals for anything but terminating a process, and even
> then you have to be extra careful.
>
> I have seen code that uses signals for runtime communication, but none
> of it was free from race conditions.

Strongly disagree. Signals exist so that they can be used. Sending
SIGHUP to a daemon to tell it to reload its configs is well-supported
by the ecosystem; use of SIGCHLD and SIGWINCH for non-termination
conditions is also vital. How else should an operating system or
desktop environment inform a process of something important?

Ctrl-C sends SIGINT meaning "interrupt". There are many, MANY
situations in which "interrupting" a process doesn't terminate it.
Here's one very simple example:

$ python3
Python 3.7.0a0 (heads/master:3913bad495, Jul 21 2017, 20:53:52)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> (
...
KeyboardInterrupt
>>>

You hit Ctrl-C in the middle of typing a multi-line command (maybe you
didn't expect it to be multi-line?), and the current input is
cancelled. Easy. No termination needed. And no race condition. I don't
know why you'd get those, but they're not hard to eliminate.

>>>>> Basically the only way to use requests safely is to fork an
>>>>> individual process for every request, which is of course
>>>>> spectacularly inefficient.
>>>>
>>>> Yes. Spectacularly inefficient and almost certainly unnecessary.
>
> Processes are a nice way to exercise multiple CPUs and also a way to
> deal with obnoxious synchronous function calls.
>
> However, you don't want to fork a process (or a thread, for that matter)
> for each context. Rather, you should have a pool of processes in the
> order of, say, 2 * CPU count, and the processes should fetch work from a
> queue.
>
> And if a process should get stuck, killing it is trivial.

Yeah, if you DO need to split them out, a pool is good. I didn't
bother mentioning it as a single process was enough for the scenario
in question (which definitely sounded like an I/O-bound app), but even
if you do fork, there's not a lot of point forking exactly one per
request.

Interestingly, the 2*CPUs figure isn't always optimal. I've messed
around with -j options on repeatable tasks, and sometimes the best
figure is lower than that, and other times it's insanely higher. On my
four-core-hyperthreading CPU, I've had times when 12 is right, I've
had times when 4 is right, and I even had one situation when I was
debating between -j64 and -j128 on a task that was theoretically
CPU-bound (ray-tracing). There are a lot of weird things happening
with caching and such, and the only way to truly know what's best is
to try it!

ChrisA