[Tutor] feedparser in python

[Mats Wichmann]

On 4/28/19 6:26 PM, nathan tech wrote:
> Hello everyone,
> 
> Most recently, I have started work using feedparser.
> 
> I noticed, almost straight away, it's a  bit slow.
> 
> For instance:
> 
>      url="http://feeds.bbci.co.uk/news/rss.xml"
> 
>      f1=feedparser.parse(url)
> 
> 
> On some feeds, this can take a few seconds, on the talk python to me 
> feed, it takes almost 10!
> 
> This, obviously, is not ideal when running a program which checks for 
> updates every once in a while. Talk about slooooow!
> 
> 
> I tried using etag, and modified, but none of the feeds seem to ever 
> have them!
> 
> Similarly, this doesn't seem to work:
> 
>      f2=feedparser.parse(url, f.headers["date"])
> 
> What am I doing wrong?
> 
> Any help appreciated.
> 
> A greatly frustrated Nate

This is just an aside... programs which depend on fetching things from
the Web are candidates for various advanced programming techniques.

One is to not write synchronously, where you ask for some data, process
the data, and present results, as if everything happened instantly.
Instead various techniques like using callbacks, or multiple threads, or
multiprocessing, or Python's asynchronous facilities can be employed.
Documentation for several of those techniques use web communications as
their examples :)  In other words, think of writing your code so other
work can happen while waiting for a particular response (for example
firing off requests to other feeds), and thing of how your update
checking can happen in the background so the data is there when you want
to look at it.

Another is when you write your unit tests, mock the responses from the
internet servers so your tests don't suffer the same delays as
interactive use of the program will see.