[Tutor] Modules to work with curl.

[Danny Yoo]

On Fri, Jul 18, 2014 at 1:05 PM, Santosh Kumar <rhce.san at gmail.com> wrote:
> its a curl command caliing a http function.
>
> for example:
>
> curl <username>:<password> <xml> GET http://<somelink>/app/somefunction

Hi Santosh,

Ok, good.  This is some of the information that we wanted to know.

So you are performing HTTP requests to get content, and you also need
to be able to provide username and password for authentication.
Sounds good so far.



> so i have to use this to create a wrapper which can scale better.

This actually doesn't follow from the requirement.  Let me explain.

You're fixated on using the "curl" shell command to do this task.  But
that's not the only way you can get the content of a URL from Python.
For example, Python's standard library includes a URL-downloading
library in urllib (https://docs.python.org/2/library/urllib.html)
which might work just as well as curl.


You're saying that you need to scale better.  In that case, you must
identify what exactly is causing that slowness.

To that end, please respond to Martin Brown's questions in this
thread.  You should try to address them, because I share those
questions too.


If you're just talking about a single HTTP request, for example, then
I don't see an obvious solution to speed up a single download besides
throwing more hardware and networking equipment at the problem.

But if we're talking about many HTTP requests at once, where the
current solution is getting each URL's content in a serial manner,
then perhaps some kind of concurrent solution will be appropriate.
(e.g. http://www.ibm.com/developerworks/aix/library/au-threadingpython/)
 But such a concurrent approach is only effective if you aren't
already saturating your network, and you're connecting to slow
servers, and... etc.  Those are the details that matter, the ones you
need to talk about and describe.

"Slowness" isn't sufficiently descriptive to let us give you technical
advice.  Please say more about the problem by answering Martin's
questions.


Good luck!