[Tutor] Modules to work with curl.

[Martin A. Brown]

Hi,

 : I am currently working on a project , which has lots of curl 
 : commands. I am using subprocess.Popen to run these curl commands. 
 : But somehow i see its bit slow. Is there a way/module by which i 
 : can improve the performance of the program.

Wow.  Is curl a requirement or do you simply need to speak HTTP?

Some suggestions and questions come to my mind:

  * Is your problem / cost the forking (i.e. how many times are you 
    calling curl)?

  * Is the problem unresponsive or sluggish HTTP servers?

  * Have you localized the slowness?  (I use logging data and timing
    information to try to figure that out.)

  * How do you define 'slow'?  How many curl commands is 'lots'?  
    Like 100 or 100,000?

Suggestions:

  * Profile the code and see where the slowness is.
 
  * Consider python-requests [0] to handle the HTTP fetches, instead 
    of calling subprocess.

  * If after that, you discover that your bottleneck is mostly 
    network, consider a task parallelizing solution.  One option 
    could be multiprocessing [1].  There are doubtless others.

-Martin

 [0] http://docs.python-requests.org/en/latest/
 [1] https://docs.python.org/2/library/multiprocessing.html

--
Martin A. Brown
http://linux-ip.net/