Batching HTTP requests with httplib (Python 2.7)

Sat Sep 15 05:30:07 EDT 2012

Leading disclaimer: I'm no web service expert.

Well, I presume you're using a web service library like ZSI or suds?
I'm using suds; started with ZSI but it had apparent maintenance
stagnancy and also some other issues. I moved to suds and it's been
fine.

I make a class wrapping a suds Client object and some convenience
functions. A suds Client object is made from the WSDL file or URL
and fills out all the services with callable methods accepting the web
service parameters. So the class has a setup method like this:

  def set_client(self, wsdlfile, baseurl):
    ''' Common code to set the .client attribute to a suds SOAP Client.
    ''' 
    from suds.client import Client
    self.client = Client('file://'+wsdlfile)
    self.client.set_options(location = baseurl)
    self.client.set_options(proxy = {'https': 'proxy:3128'})

wsdlfile points at a local WSDL file, removing dependence on some server
to provide the WSDL. Makes testing easier too.

Then there's a general call wrapper. The wscall() method below is a
slightly scoured version of something I made for work. You hand it the
service name and any parameters needed for that service call and it looks
it up in the Client, calls it, sanity checks the result in a generic
sense. If things are good it returns the suds reply object. But if
anything goes wrong it logs it and returns None. "Goes wrong" includes
a successful call whose internal result contains an error response -
obviously that's application dependent.

The upshot is that the wrapper for a specific web service call then becomes:

  def thing1(self, blah, blah, ...):
    reply = self.wscall('thing1', blah, blah, ...)
    if reply is None:
      # badness happened; it has been logged, just return in whatever
      # fashion you desire - return None, raise exception, etc
      ...
    # otherwise we pick apart the reply object to get the bits that
    # matter and return them
    return stuff from the reply object

and your larger logic looks like:

  def doit(self, ...):
    self.thing1(1,2,3)
    self.thing2(...)

and so forth.

The general web service wrapper wscall() below has some stuff ripped out,
but I've left in some application specific stuff for illustration:

  - optional _client and _cred keyword parameters to override the
    default Client and credentials from the main class instance

  - the Pfx context manager stuff arranges to prefix log messages and
    exception strings with context, which I find very handy in debugging and
    logging

  - likewise the LogTime context manager logs things that exceed a time
    threshold

  - the wsQueue.submit(...) stuff punts the actual web service call
    through a capacity queue (a bit like the futures module) because
    this is a capacity limited multithreaded app.

    For your purposes you could write:

      websvc = getattr(_client.service, servicename)
      with LogTime(servicename, threshold=3):
        try:
          reply = websvc(userId=_cred.username, password=_cred.password,
                         *args, **kwargs)
        except:
          exc_info = sys.exc_info
          reply = None
        else:
          exc_info = None

    which is much easier to follow.

  - the 'callStatus' thing is application specific, left as an example
    of a post-call sanity check you would add for calls that complete
    but return with an error indication

Anyway, all that said, the general wrapper looks like this:

  def doWScall(self, servicename, _client=None, _cred=None, *args, **kwargs): 
    ''' General wrapper for a web service call.
        Returns the native SUDS reply object if OK, otherwise None.
        `_client` can be used to override use of self.client.
        `_cred` can be used to override use of self.cred.
    '''
    if _client is None:
      _client = self.client
    if _cred is None:
      _cred = self.cred
    with self.pfx:
      with Pfx("doWScall.%s(...)", servicename):
        websvc = getattr(_client.service, servicename)
        with LogTime(servicename, threshold=3):
          reply, exc_info = self.wsQueue.submit( partial(websvc,
                                                         userId=_cred.username,
                                                         password=_cred.password,
                                                         *args,
**kwargs),
                                                 name=servicename).wait()
          if exc_info:
            error("FAIL, exception in web service call",
exc_info=exc_info)
            return None
          if reply is None:
            arglist = listargs(args, kwargs)
            error("web service called but got None anyway:
arglist=(%s)",
                  ", ".join(arglist))
            return None
        if hasattr(reply, 'callStatus'):
          if reply.callStatus == "OK":
            info("OK")
            return reply
        ... log an error message including relevant stuff from the reply ...
        return None

Anyway, I hope that shows a way to get the main logic clear and the
overhead for wrapping specific calls quite lightweight.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

Well, it's one louder, isn't it?  It's not ten.  You see, most blokes are
gonna be playing at ten, you're on ten here, all the way up, all the way up,
all the way up, you're on ten on your guitar, where can you go from there?
Where?  Nowhere, exactly.  What we do is, if we need that extra push over the
cliff, you know what we do?  Eleven.  Exactly.  One louder.
        - Nigel Tufnel, _This Is Spinal Tap_