Using nntplib

[Steve Holden]

"jb" <jblazi at hotmail.com> ...
> Let us assume I send a LIST command to the server. Then I get a short list
> of the newsgroups, the server offers. Getting for example 100,000 lines
> from 'goliath.newsfeeds.com' takes more than a "few minutes", as Outlook,
> that I used to use in my MS-life, puts it.
>
> Now the point is, that nntplib works synchronously(!), that means that the
> thread is waiting on the LIST command until the server sends it's
> terminating '.' and all the time the temporary file the socket has created
> is polled.
>
> (1)
> This is a terrible method, but the handling of the sommands should be done
> in another thread, so maybe it is o.k.
> (I thought, I should do all this myself and use QSockets, but alas! they
are
> not supported by PyQt, however wonderful PyQt is.)
>
> (2)
> But I cannot monitor the whole procedure, that is I cannot see, how many
> lines have arrived already.
>
> (3) QUESTION:
> How can I do the monitoring I mentioned in (2)? A possibility is to tune
the
> file nntplib.py (that is, CHANGING A SYSTEM FILE!). Here is a part of
> nntplib.py:
>
>    def getlongresp(self, file=None):
>         """Internal: get a response plus following text from the server.
>         Raise various errors if the response indicates an error."""
>
>         openedFile = None
>         try:
>             # If a string was passed then open a file with that name
>             if isinstance(file, types.StringType):
>                 openedFile = file = open(file, "w")
>
>             resp = self.getresp()
>             if resp[:3] not in LONGRESP:
>                 raise NNTPReplyError(resp)
>             list = []
>             while 1:
>                 line = self.getline()
>                 if line == '.':
>                     break
>                 if line[:2] == '..':
>                     line = line[1:]
>                 if file:
>                 ############ inserted by J.B. #######
>                 ### HERE SOMETHING SHOULD BE DONE ###
>                 #####################################
>                     file.write(line + "\n")
>                 else:
>                     list.append(line)
>         finally:
>             # If this method created the file, then it must close it
>             if openedFile:
>                 openedFile.close()
>
>         return resp, list
>
> So what should I do? Rasing an excepton? And should not nntplib di this?
>

You might want to consider using the XOVER NNTP protocol feature if it's
available.  I came across it after I finished "Python Web Programming", so
it's mentioned in the support site at:

    http://pydish.holdenweb.com/pwp/chp5notes.htm

As far as informing the GUI thread about what's been received so far,
whether you use LIST or XOVER perhaps you could just put the headers onto a
Queue (see that Queue module) and letting the GUI thread take them off. That
should ease up any critical-section issues.

regards
 Steve
--
-----------------------------------------------------------------------
Steve Holden                                 http://www.holdenweb.com/
Python Web Programming                http://pydish.holdenweb.com/pwp/
-----------------------------------------------------------------------