logging from time critical tasks -- QueueListener().stop() takes the whole CPU

[Thomas Jollans]

On 16/07/18 08:24, Gerlando Falauto wrote:
> On Monday, July 16, 2018 at 8:13:46 AM UTC+2, Thomas Jollans wrote:
>> On 16/07/18 07:39, Gerlando Falauto wrote:
>>> On Monday, July 16, 2018 at 6:56:19 AM UTC+2, dieter wrote:
>>>>> ...
>>>>> Why is the main thread taking up so much CPU?
>>>>> I believe at this point listener.stop() should only be waiting for the helper thread to terminate, which I reckon would be implemented by waiting on a semaphore or something (i.e. iowait i.e. 0% CPU).
>>>>
>>>> Maybe, you look at the source code of "listener.stop"?
>>>
>>> I did, forgot to mention that. Culprit is self._thread.join().
>>> Which is where it waits for the internal thread to terminate,
>>> which I would've expected to wait on a lock or semaphore (pthread_join()?)
>>> So there's something I'm totally missing here, which has more to do
>>> with queues and threads in general than it has with logging.
>>> Any ideas?
>>>
>>
>> I have no idea what's really going on there, but a quick look through
>> the source reveals that the actual waiting in Thread.join() is done by
>> sem_wait().
>>
>> via
>> https://github.com/python/cpython/blob/13ff24582c99dfb439b1af7295b401415e7eb05b/Python/thread_pthread.h#L304
>> via
>> https://github.com/python/cpython/blob/master/Modules/_threadmodule.c#L45
> 
> Hmm... do you think it's possible it's really getting interrupted the whole time, effectively turning the lock into a sort of spinlock?
> Any idea how to confirm that without recompiling the whole code?
> 

Profiling, maybe? Or you might be able to find out if it's being
interrupted by listening for signals yourself (though I'm not sure which
one(s))?