Counting Python threads vs C/C++ threads

[Barry Scott]

> On 16 Jul 2019, at 20:48, Dan Stromberg <drsalists at gmail.com> wrote:
> 
> 
> 
> On Tue, Jul 16, 2019 at 11:13 AM Barry Scott <barry at barrys-emacs.org <mailto:barry at barrys-emacs.org>> wrote:
> I'm going to assume you are on linux.
> Yes, I am.  Ubuntu 16.04.6 LTS sometimes, Mint 19.1 other times.
> 
> On 16 Jul 2019, at 18:35, Dan Stromberg <drsalists at gmail.com <mailto:drsalists at gmail.com>> wrote:
> > 
> > I'm looking at a performance problem in a large CPython 2.x/3.x codebase
> > with quite a few dependencies.
> > 
> > I'm not sure what's causing the slowness yet.  The CPU isn't getting hit
> > hard, and I/O on the system appears to be low - but throughput is poor.
> > I'm wondering if it could be CPU-bound Python threads causing the problem
> > (because of the threading+GIL thing).
> 
> Does top show the process using 100% CPU?
> Nope.  CPU utilization and disk use are both low.

Then your problem is latency. You need to find the slow operation.

> We've been going into top, and then hitting '1' to see things broken down by CPU core (there are 32 of them, probably counting hyperthreads as different cores), but the CPU use is in the teens or so.
> 
> I've also tried dstat and csysdig.  The hardware isn't breaking a sweat, but throughput is poor.

> > The non-dependency Python portions don't Appear to have much in the way of
> > threading going on based on a quick grep, but csysdig says a process
> > running the code has around 32 threads running - the actual thread count
> > varies, but that's the ballpark.
> > 
> > I'm wondering if there's a good way to find two counts of those threads -
> > how many are from CPython code that could run afoul of the GIL, and how
> > many of them are from C/C++ extension modules that wouldn't be responsible
> > for a GIL issue.
> 
> >From the docs on threading:
> 
> threading.active_count()
>  <file:///Library/Frameworks/Python.framework/Versions/3.7/Resources/English.lproj/Documentation/library/threading.html?highlight=threading#threading.active_count>
> Return the number of Thread <file:///Library/Frameworks/Python.framework/Versions/3.7/Resources/English.lproj/Documentation/library/threading.html?highlight=threading#threading.Thread> objects currently alive. The returned count is equal to the length of the list returned by enumerate() <file:///Library/Frameworks/Python.framework/Versions/3.7/Resources/English.lproj/Documentation/library/threading.html?highlight=threading#threading.enumerate>.
> 
> Are you on a Mac?

Opss a file: link sorry should have search the online docs.

I use many operating systems: Fedora, macOS, Windows, NetBSD, CentOS and others in the past.

> 
> https://docs.python.org/2/library/threading.html <https://docs.python.org/2/library/threading.html> appears to have some good info. I'll probably try logging threading.active_count()
>  
> A question arises though: Does threading.active_count() only show Python threads created with the threading module?  What about threads created with the thread module?

Only pythons threads, if you think about it why would python care about threads it does not control?


> 
> Try running strace on the process to see what system calls its making.
> I've tried it, but thank you.  It's a good suggestion.
> 
> I often find that when strace'ing a program, there's a bunch of mostly-irrelevant stuff at Initial Program Load (IPL), but then the main loop fails into a small cycle of system calls.

And what are thoses sys calls and what is the timing of them?
If you are use select/poll how long before the call returns.
If you in a read how long before it returns.

> 
> Not with this program.  Its main loop is busy and large.

Does the code log any metrics or telemetry to help you?
I work on a product that produces time-series data to show key information about the service.
TPS, cache hit rates etc.

Should have mention before you can run the code under python's cprofile.

Do a test run against the process and then run analysis on the data that cprofile
produces to find out elapse times and cpu times of the code.

> 
> You could also connect gdb to the process and find out what code the threads are running.
> 
> I used to use gdb, and wrappers for gdb, when I was doing C code, but I don't have much experience using it on a CPython interrpreter.
> 
> Would I be doing a "thread apply all bt" or what?  I'm guessing those backtraces could facilitate identifying the origin of a thread.

Yes  thread apply all bt works great on a python process. recent gdb releases knows how to format the stack and show you the python stack,
forgot the command, but its easy to google for.

Barry


> 
> Thanks a bunch.
>