[Python-ideas] solving multi-core Python

[Devin Jeanpierre]

On Mon, Jun 22, 2015 at 4:29 PM, Gregory P. Smith <greg at krypto.org> wrote:
>
> On Mon, Jun 22, 2015 at 3:51 PM Sturla Molden <sturla.molden at gmail.com>
> wrote:
>>
>> "Gregory P. Smith" <greg at krypto.org> wrote:
>>
>> > What this means for subinterpreters in this case is not much different
>> > from
>> > starting up multiple worker processes: You need to start them up and
>> > wait
>> > for them to be ready to serve, then reuse them as long as feasible
>> > before
>> > recycling them to start up a new one. The startup cost is high.
>>
>> The statup cost for worker processes is high on Windows. It is very small
>> on nearly any other OS.
>
> While I understand that Windows adds some overhead there, startup time for
> Python worker processes is high on all OSes.
>
> Python startup is slow in general. It slows down further based on the
> modules you must import before you can begin work.

Python does *very* little work on fork, which is what Sturla is
alluding to. (Fork doesn't exist on Windows.)

The only part I've found forking to be slow with is if you need to
delay initialization of a thread pool and everything that depends on a
thread pool until after the fork. This could hypothetically be made
faster with subinterpreters if the thread pool was shared among all
subinterpreters (e.g. if it was written in C.), but I would *expect*
fork to be faster overall.

That said, worker startup time is not actually very interesting
anyway, since workers should restart rarely. I think its biggest
impact is probably the time it takes to start your entire task from
scratch.

-- Devin