[Python-Dev] DTRACE support
Jesus Cea
jcea at jcea.es
Sat Sep 7 05:40:04 CEST 2013
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 06/09/13 20:33, Antoine Pitrou wrote:
> On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote:
>>
>> It is intrusive. Yes. I think it must be, by its own nature.
>> Probably room for improvement and code transparency. But... are
>> Python-DEVs interested in the project?. That is the point :)
>
> As a concrete data point: - here are Dave's modifications to
> ceval.c for systemtap:
> http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here
> are your modifications to ceval.c for dtrace:
> http://bugs.python.org/review/13405/diff/6151/Python/ceval.c
Unfair, because that code is not doing the same thing.
Most of the extra complexity is there to deal with DTRACE ability to
provide meaningful stackframes, with Python code instead of CPython
evaluation loop. This is kind of magical.
Look at this:
""
[root at stargate-host /]# ps -lAf|grep -i correo
0 S correo 1371 1 0 40 20 ? 277063 ? Jul 29
? 90:43 /usr/local/bin/python /export/home/
0 S root 20397 20373 0 40 20 ? 1294 ? 05:18:16
pts/12 0:00 grep -i correo
[root at stargate-host /]# dtrace -n 'python1371:::function-entry
{jstack();}'
5 3164 PyEval_EvalFrameEx:function-entry
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x89a
[ /usr/local/lib/python2.7/logging/__init__.py:1332
(isEnabledFor) ]
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
[ /usr/local/lib/python2.7/logging/__init__.py:1202
(log) ]
libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae
[ /home/correo/durus/connection.py:483 (shrink) ]
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
[ /home/correo/durus/connection.py:225 (shrink_cache) ]
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
[ /home/correo/durus/connection.py:303 (commit2) ]
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
[ /home/correo/durus/connection.py:261 (commit) ]
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
[ /export/home/correo/lmtp.py:191 (_monitor) ]
libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae
[ /export/home/correo/lmtp.py:125 (process_vacation2) ]
libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae
[ /export/home/correo/lmtp.py:138 (process_vacation) ]
libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
libpython2.7.so.1.0`function_call+0x192
libpython2.7.so.1.0`PyObject_Call+0x5c
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94
[ /usr/local/lib/python2.7/threading.py: ]
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
libpython2.7.so.1.0`function_call+0xa4
libpython2.7.so.1.0`PyObject_Call+0x5c
libpython2.7.so.1.0`instancemethod_call+0xa1
libpython2.7.so.1.0`PyObject_Call+0x5c
libpython2.7.so.1.0`PyEval_CallObjectWithKeywords+0x58
libpython2.7.so.1.0`t_bootstrap+0x52
libc.so.1`_thr_setup+0x4e
libc.so.1`_lwp_start
^C
"""
Lets say I want to know about what codepaths are hitting the OS "sync"
features, so I do this:
"""
[root at stargate-host /]# dtrace -n
'syscall::fdsync:entry/pid==1371/{jstack();}'
dtrace: description 'syscall::fdsync:entry' matched 1 probe
CPU ID FUNCTION:NAME
3 58344 fdsync:entry
libc.so.1`__fdsync+0x15
libdb-6.0.so`__os_fsync+0x91
libdb-6.0.so`__log_flush_int+0x685
libdb-6.0.so`__log_flush+0x6a
libdb-6.0.so`__log_flush_pp+0x121
_pybsddb.so`DBEnv_log_flush+0x39
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6992
[
/export/home/correo/durus-berkeleydbstorage/berkeleydb_storage.py:918
(flush) ]
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
[ /export/home/correo/lmtp.py:1005 (lmtp2) ]
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
[ /export/home/correo/lmtp.py:763 (lmtp) ]
libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
libpython2.7.so.1.0`function_call+0x192
libpython2.7.so.1.0`PyObject_Call+0x5c
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94
[ /export/home/correo/lmtp.py:214
(_ignore_socket_errors) ]
libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
libpython2.7.so.1.0`function_call+0x192
libpython2.7.so.1.0`PyObject_Call+0x5c
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94
[ /usr/local/lib/python2.7/threading.py:504 (run) ]
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
[ /usr/local/lib/python2.7/threading.py:551
(__bootstrap_inner) ]
libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
[ /usr/local/lib/python2.7/threading.py:524
(__bootstrap) ]
libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
libpython2.7.so.1.0`function_call+0xa4
libpython2.7.so.1.0`PyObject_Call+0x5c
libpython2.7.so.1.0`instancemethod_call+0xa1
libpython2.7.so.1.0`PyObject_Call+0x5c
libpython2.7.so.1.0`PyEval_CallObjectWithKeywords+0x58
libpython2.7.so.1.0`t_bootstrap+0x52
libc.so.1`_thr_setup+0x4e
libc.so.1`_lwp_start
"""
I could even aggregate stacktraces and print them by frequency, cost,
whatever.
Be aware that to provide this information I am not even using the
probes I have integrated in Python. The program is in production and I
don't need to stop it to make those measurements. They are 100%
"live", not interrupting the service and without any kind of support
in that Python program.
I can even plug to TCP Dtrace providers in kernel and know what exact
line of Python code is sending those bytes that are being delayed
because the Nagle algorithm
<https://en.wikipedia.org/wiki/Nagle%27s_algorithm>. I actually digged
this five years ago. what I hunt it was! :-)
PS: And it is supporting Unicode :-), another big complexity in the code.
- --
Jesús Cea Avión _/_/ _/_/_/ _/_/_/
jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
Twitter: @jcea _/_/ _/_/ _/_/_/_/_/
jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQCUAwUBUiqgFJlgi5GaxT1NAQLezQP2PRkvn6+9Zev6GlGu14hfYTpwAMie9gcn
tEieelaQniisLy22oj/MO2d2j78qlH4RoU3U/vzOTPru6uzt3OKy0UBivnBFqTcc
fD/dPUCcfkigZO4nBzcqjAP4lz93wcDkx1Y8RcL3ygxeI6fSe8JKdyZKyXvyRf/9
f+3MfWJVDA==
=29tQ
-----END PGP SIGNATURE-----
More information about the Python-Dev
mailing list