Performance Issues with Threaded Python Network Server

[Steve Holden]

----- Original Message -----
From: "Joao Prado Maia" <JMaia at lexgen.com>
To: "'Steve Holden'" <sholden at holdenweb.com>
Cc: <python-list at python.org>
Sent: Tuesday, January 15, 2002 3:52 PM
Subject: RE: Performance Issues with Threaded Python Network Server


[...]
> >
>
> Well, I think you misunderstood the issue here. Outlook Express does a
XOVER
> 1-1500 (or whatever the range is) to get the header information about
all of
> the articles, and then does a ARTICLE 'number' on every article on the
group
> to get the actual body of the article. This only happens if the user
selects
> to have all messages downloaded to his hard disk.
>
Aah, I don't habitually download all articles, but pull them down as I
read.
>
[ xover stuff ]
> >
>
> Indeed, I understand perfectly the NNTP protocol on this issue.
However,
> even  'fairly' abbreviated means 1500 lines of header information in
just
> one message board, which grows by the day.
>
> > > The problem here is that whenever this happens, the CPU
> > usage of the NNTP
> > > server goes to about 35% and continues increasing slowly while the
> > > newsreader is receiving all the message headers and bodies.
> > >
> > Too much data! I'm guessing you aren't using XOVER?
> >
>
> It's not my fault. Outlook Express is the one asking for 1500 articles
one
> at a time because the user wants to download the information of all
articles
> to his hard disk.
Although if you set OE to download only new messages it should track
what it already has and only pull down the new ones, right? Not much
point dragging stuff down multiple times.
>
[ ... ]
>
> > > The NNTP server gets its information from a MySQL database
> > (and no, MySQL
> > is
> > > not the bottleneck as far as I know, since 'top' shows the
> > NNTP server
> > > consuming 35% of CPU, not MySQL), formats the output by using
string
> > > replacement (aka "%s %s <%s@%s>" % (v,x,z,y)) and writes to
> > the 'wfile'
> > file
> > > descriptor.
> > >
> > Your do_XOVER code doesn't look outrageously bad.
> >
>
Closer inspection of the code suggests it's quite possibly the
format_body() function in the back end that's gobbling up your CPU. Try
returning the argument, using only articles that don't need
dot-escaping, and see if that's your problem.

regards
 Steve
--
http://www.holdenweb.com/