[Python-Dev] Non-blocking (asynchronous) timer without thread?

[Josiah Carlson]

Evgeniy Khramtsov <xramtsov at gmail.com> wrote:
> Mike Klaas пишет:
> 
> > I'm not sure how having python execute code at an arbitrary time would
> > _reduce_ race conditions and/or deadlocks. And if you want to make it
> > safe by executing code that shares no variables or resources, then it
> > is no less safe to use threads, due to the GIL.
> >
> Ok. And what about a huge thread overhead? Just try to start 10-50k 
> threading timers :)

That is a foolish task in any language, and is why heap-based schedulers
exist.  You can use a Queue as synchronization between timer producers
and the scheduler, and another Queue as synchronization between the
scheduler and your threadpool (5-10 should be fine and have minimal
overhead).

Also 10-50k threads, in just about any language or runtime (except
using C in solaris on SPARC, which can handle that pretty well), is way
too much to handle.  With thread stack sizes as they are, I wouldn't be
surprised if you were running into the limit of your main memory size,
and you would certianly be destroying the effectiveness of most caches.

One thing that came to mind is that perhaps the "overhead" you are
experiencing when using threads isn't context switch time, but is the
fact that you have thousands of threads waiting on signals from the
OS to continue, or even that Python switches threads after a certain
number of bytecodes have been interpreted (if they are not waiting on
wait syscalls). Python 2.5 sets the interval at 100 bytecodes (see
sys.[get|set]checkinterval()), which is either 1 million or 5 million
bytecodes with your load of 10k-50k.  For most operations, this would be
a palpable delay of up to a couple seconds before a thread gets back to
doing what it was doing.

Seriously, threadpool with a scheduler.


 - Josiah