Profiler for long-running application

[Ryan Stuart]

Hi Asad,

Is there any reason why you can't just use profile/cProfile? In particular,
you could use the api of that module to save out the profile stats to an
external file with a unique name and then inspect them later with a tool
like snakeviz <https://jiffyclub.github.io/snakeviz/>. The code to save
profile stats might look like the following:

    pr = cProfile.Profile()
    pr.runcall(your_celery_task_without_async)
    ps = pstats.Stats(pr)
    ps.dump_stats("my_task.profile")

Obviously you need to call your celery task function directly, not via
Celery using delay() or any derivative. Alternatively, you could try
something like line_profiler <https://github.com/rkern/line_profiler> by
again, calling the task directly.

If using this method it turn out your actual task code is running quite
fast, then I'd suggest that the majority of the time is being lost in
transferring the task to the Celery node.

Cheers

On Mon Feb 09 2015 at 5:20:43 AM Asad Dhamani <dhamaniasad at gmail.com> wrote:

> I have a Flask application where I run a specific task asynchronously
> using Celery. Its basically parsing some HTML and inserting data into a
> Postgres database(using SQLAlchemy). However, the task seems to be running
> very slowly, at 1 insert per second.
> I'd like to find out where the bottleneck is, and I've been looking for a
> good profiler that'd let me do this, however, I couldn't find anything. Any
> recommendations would be great.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20150208/2f3cb5d7/attachment.html>