Profiler for long-running application

[Asad Dhamani]

On Monday, February 9, 2015 at 1:50:58 PM UTC+5:30, Ryan Stuart wrote:
> Hi Asad,
> 
> Is there any reason why you can't just use profile/cProfile? In particular, you could use the api of that module to save out the profile stats to an external file with a unique name and then inspect them later with a tool like snakeviz. The code to save profile stats might look like the following:
> 
> 
>     pr = cProfile.Profile()
>     pr.runcall(your_celery_task_without_async)
>     ps = pstats.Stats(pr)
>     ps.dump_stats("my_task.profile")
> Obviously you need to call your celery task function directly, not via Celery using delay() or any derivative. Alternatively, you could try something like line_profiler by again, calling the task directly.
> 
> 
> If using this method it turn out your actual task code is running quite fast, then I'd suggest that the majority of the time is being lost in transferring the task to the Celery node.
> 
> Cheers
> 
> 
> On Mon Feb 09 2015 at 5:20:43 AM Asad Dhamani <dhama... at gmail.com> wrote:
> I have a Flask application where I run a specific task asynchronously using Celery. Its basically parsing some HTML and inserting data into a Postgres database(using SQLAlchemy). However, the task seems to be running very slowly, at 1 insert per second.
> 
> I'd like to find out where the bottleneck is, and I've been looking for a good profiler that'd let me do this, however, I couldn't find anything. Any recommendations would be great.
> 
> --
> 
> https://mail.python.org/mailman/listinfo/python-list

Hi  Ryan,

I was looking for something that didn't make me modify my code. It seems like I'll just use cProfile with snakeviz. 

Thanks for the help