how to handle response data that is streaming and chunked?

[MRAB]

On 2019-03-21 20:12, Artie Ziff wrote:
> Hello,
> 
> I am trying to learn how to do use response library in the least confusing
> way possible. My data source is streaming.
> 
> The sample I share below looks more complicated that it needs to be. I do
> not have enough experience with this to know better. Hence why I came here
> for guidance and direction. Comments are very welcome as well as pointers
> to reading material. I want to learn this.
> 
> I expected that I was going to be able to use the JSON methods such as load
> or loads. After all, this is JSON and should be able to slurp into builtin
> python objects and data structures, I'd imagine.
> 
> Does this mailing list use a paste site or is better to paste code in-line
> for future?
> 
> Thanks for reading!
> --Art
> 
> 
> import os, sys, json, requests, signal
> from requests_oauthlib import OAuth1
> 
> api_key = 'HIDDEN'
> api_secret = 'HIDDEN'
> user_token_key = 'HIDDEN'
> user_token_secret = 'HIDDEN'
> 
> authorization = OAuth1(api_key, api_secret, user_token_key,
> user_token_secret)
> session = requests.Session()
> session.auth = authorization
> finis = 0
> 
> 
> def handler(signum, frame):
>      global finis
>      finis = 1
> 
> 
> signal.signal(signal.SIGINT, handler)
> 
> url = "https://stream.tradeking.com/v1/market/quotes.json?symbols=AAPL,DJT"
> resp = session.get(url, stream=True)
> lines = ''
> for chunk in resp.iter_content(None, decode_unicode=True):
>      if finis:
>          break
>      lines += chunk.decode('utf-8')
>      while lines.find('}}') > 0:
>          line, lines = lines.split('}}', 1)
>          print(line)
> 
As far I can tell from the docs, calling '.iter_content' with 
'decode_unicode=True' will make it decode, so you shouldn't be decoding 
it again.

Also, string indexes start at 0, and '.find' will return -1 if the 
string is not found.

As you don't need the index itself, you might as well use 'in' instead:

     while '}}' in lines:

On the other hand, you're splitting the string, so maybe it would be 
better to use '.partition':

     before, sep, after = lines.partition('}}'):
     while sep:
         print(before + sep)
         lines = after
         before, sep, after = lines.partition('}}'):

Note that when you split or partition on '}}', you'll lose the '}}' 
itself if you're not careful, hence the 'before + sep' above.