Counting Python threads vs C/C++ threads

[Barry Scott]

I'm going to assume you are on linux.

On 16 Jul 2019, at 18:35, Dan Stromberg <drsalists at gmail.com> wrote:
> 
> I'm looking at a performance problem in a large CPython 2.x/3.x codebase
> with quite a few dependencies.
> 
> I'm not sure what's causing the slowness yet.  The CPU isn't getting hit
> hard, and I/O on the system appears to be low - but throughput is poor.
> I'm wondering if it could be CPU-bound Python threads causing the problem
> (because of the threading+GIL thing).

Does top show the process using 100% CPU?

> 
> The non-dependency Python portions don't Appear to have much in the way of
> threading going on based on a quick grep, but csysdig says a process
> running the code has around 32 threads running - the actual thread count
> varies, but that's the ballpark.
> 
> I'm wondering if there's a good way to find two counts of those threads -
> how many are from CPython code that could run afoul of the GIL, and how
> many of them are from C/C++ extension modules that wouldn't be responsible
> for a GIL issue.

From the docs on threading:

threading.active_count()
 <file:///Library/Frameworks/Python.framework/Versions/3.7/Resources/English.lproj/Documentation/library/threading.html?highlight=threading#threading.active_count>
Return the number of Thread <file:///Library/Frameworks/Python.framework/Versions/3.7/Resources/English.lproj/Documentation/library/threading.html?highlight=threading#threading.Thread> objects currently alive. The returned count is equal to the length of the list returned by enumerate() <file:///Library/Frameworks/Python.framework/Versions/3.7/Resources/English.lproj/Documentation/library/threading.html?highlight=threading#threading.enumerate>.


Try running strace on the process to see what system calls its making.

You could also connect gdb to the process and find out what code the threads are running.

Barry

> 
> Does anyone know how to find out these different thread counts, without
> mounting a huge effort to scrutinize all the code and its dependencies?
> I'd prefer a tool that can scrutinize the process(es) from the outside, but
> modifying the code is not out of the question.
> 
> Please see
> https://stackoverflow.com/questions/56958009/how-can-i-tell-how-many-python-threads-a-process-has-from-the-outside
> for more information.  Or perhaps
> https://python-forum.io/Thread-How-to-get-a-count-of-Python-threads-from-the-outside-or-via-code-instrumentation
> 
> Thanks!
> -- 
> https://mail.python.org/mailman/listinfo/python-list
>