PEP idea: Instrumented Python

[Erik Jones]

On Oct 12, 2007, at 10:09 AM, Christopher Nelson wrote:

> I was looking at adding dtrace-like dynamic tracing to Python.   
> Note that this isn't dtrace itself.  The basic rationale:
>
> 1. A lot of enterprise-level software is written in Python.  It is  
> difficult to impossible to reproduce the customer environment in a  
> test lab.  Sometimes applications hang mysteriously, and we are  
> unable to duplicate the problems in our environment.
> 2. dtrace is a good design.  It has little to no overhead for  
> collection points that are not collecting data.  Emulating that  
> behavior allows you to test what you fly and fly what you test.
> 3. Having the ability to only get trace data you want (or need)  
> instead of trace data for every instruction executed is very  
> useful.  It also reduces bandwidth requirements when archiving the  
> traces, or shipping them out over a network connection.
> 4. The ability to dynamically expand and contract the tracing scope  
> during a trace is vital to getting the right data.
>
> The general plan would be to provide collectors in all of the  
> public python API functions.  The filtering code would also be in  
> the local system, for speed and bandwidth-conserving purposes.  The  
> filters should generally be native code.  The data should be  
> writeable to a file or to a network socket.  In addition, there  
> should be a network socket listening for commands from a tracing  
> client.  The client would be able to upload additional filter  
> scripts and start or stop tracing transparently to the running source.
>
> For security there should be an authentication mechanism that  
> allows the application to decide whether or not a connection to the  
> trace facility should be allowed, and a way to disable the trace  
> listening port on startup.  (Or perhaps it should be disabled on  
> startup by default and enabled if so desired.)
>
> Finally, there should, of course, be Pythonic access to the trace  
> system.
>
> The essential differences between the current tracing system and  
> the proposal involve the selectable granularity, the transparent  
> remote access, and automatic triggering of collection points.
>
> Comments? Please make sure to reply to all, I am not subscribed to  
> the list.

I've recently starting thinking of something along these lines  
although I don't the need to alter existing libraries or even mimic  
Dtrace given Python's extremely dynamic nature and reflection  
capabilities.  Ian Bicking wrote something along these lines for  
Paste that is used in Pylons wherein exceptions raised by the  
application during debug mode drop you into an interactive debugger  
with a web interface -- very handy for developing webapps.

This got me thinking about building a module that could be included  
by projects that creates a socket and responds to messages on that  
socket in a separate thread from the main app so that you can connect  
to the app and explore its current state.  Take a look at the  
evalexception module in Paste to see what he does.

Erik Jones

Software Developer | Emma®
erik at myemma.com
800.595.4401 or 615.292.5888
615.292.0777 (fax)

Emma helps organizations everywhere communicate & market in style.
Visit us online at http://www.myemma.com