[Python-Dev] s/hotshot/lsprof

[Brett Cannon]

Just  for everyone's FYI while we are talking about profilers, Floris
Bruynooghe (who I am cc'ing on this so he can contribute to the
conversation), for Google's Summer of Code, wrote a replacement for
'profile' that uses Hotshot directly.  Thanks to his direct use of
Hotshot and rewrite of pstats it loads Hotshot data 30% faster and
also alleviates keeping 'profile' around and its slightly questionable
license.

You can find his project at
http://savannah.nongnu.org/projects/pyprof/ .  I believe he also
tweaked Hotshot to accept custom timing functions.  I have not had a
chance to go over his code to clean it up for putting it up on SF, but
I thought people should be aware of it.

-Brett

On 11/19/05, Armin Rigo <arigo at tunes.org> wrote:
> Hi!
>
> The current Python profilers situation is a mess.
>
> 'profile.Profile' is the ages-old pure Python profiler.  At the end of a
> run, it builds a dict that is inspected by 'pstats.Stats'.  It has some
> recent support for profiling C calls, which however make it crash in
> some cases [1].  And of course it's slow (makes a run take about 10x
> longer).
>
> 'hotshot', new from 2.2, is quite faster (reportedly, only 30% added
> overhead).  The log file is then loaded and turned into an instance of
> the same 'pstats.Stats'.  This loading takes ages.  The reason is that
> the log file only records events, and loading is done by instantiating a
> 'profile.Profile' and sending it all the events.  In other words, it
> takes exactly as long as the time it spared in the first place!
> Moreover, for some reasons, the results given by hotshot seem sometimes
> quite wrong.  (I don't understand why, but I've seen it myself, and it's
> been reported by various people, e.g. [2].)  'hotshot' doesn't know
> about C calls, but it can log line events, although this information is
> lost(!) in the final conversion to a 'pstats.Stats'.
>
> 'lsprof' is a third profiler by Brett Rosen and Ted Czotter, posted on
> SF in June [2].  Michael Hudson and me did some minor clean-ups and
> improvements on it, and it seems to be quite useful.  It is, for
> example, the only of the three profilers that managed to give sensible
> information about the PyPy translation process without crashing,
> allowing us to accelerate it from over 30 to under 20 minutes.  The SF
> patch contains a more detailed account on the reasons for writing
> 'lsprof'.  The current version [3] does not support C calls nor line
> events.  It has its own simple interface, which is not compatible with
> any of the other two profilers.  However, unlike the other two
> profilers, it can record detailed stats about children, which I found
> quite useful (e.g. how much take is spent in a function when it is
> called by another specific function).
>
> Therefore, I think it would be a great idea to add 'lsprof' to the
> standard library.  Unless there are objections, it seems that the best
> plan is to keep 'profile.py' as a pure Python implementation and replace
> 'hotshot' with 'lsprof'.  Indeed, I don't see any obvious advantage that
> 'hotshot' has over 'lsprof', and I certainly see more than one downside.
> Maybe someone has a use for (and undocumented ways to fish for) line
> events generated by hotshot.  Well, there is a script [4] to convert
> hotshot log files to some format that a KDE tool [5] can display.  (It
> even looks like hotshot files were designed with this in mind.)  Given
> that the people doing that can still compile 'hotshot' as a separate
> extension module, it doesn't strike me as a particularly good reason to
> keep Yet Another Profiler in the standard library.
>
> So here is my plan:
>
> Unify a bit more the interfaces of the pure Python and the C profilers.
> This also means that 'lsprof' should be made to use a pstats-compatible
> log format.  The 'pstats' documentation specifically says that the file
> format can change: that would give 'lsprof' a place to store its
> detailed children stats.
>
> Then we can provide a dummy 'hotshot.py' for compatibility, remove its
> documentation, and provide documentation for 'lsprof'.
>
> If anyone feels like this is a bad idea, please speak up.
>
>
> A bientot,
>
> Armin
>
>
> [1] https://sourceforge.net/tracker/?group_id=5470&atid=105470&func=detail&aid=1117670
>
> [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470&func=detail&aid=1212837
>
> [3] http://codespeak.net/svn/user/arigo/hack/misc/lsprof (Subversion)
>
> [4] http://mail.python.org/pipermail/python-list/2003-September/183887.html
>
> [5] http://kcachegrind.sourceforge.net/cgi-bin/show.cgi
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
>