Performance Issues with Threaded Python Network Server

[Joao Prado Maia]

> > The problem here is that whenever this happens, the CPU usage of the
> > NNTP server goes to about 35% and continues increasing 
> slowly while the
> > newsreader is receiving all the message headers and bodies.
> 
> in some of your SQL expressions you write "WHERE... A.id >= 
> %s" doesn't 
> this mean that the number of messages that have to be 
> filtered increases 
> with the id to get? (snipet above from get_XOVER) (long a go my SQL 
> experiences....)
> 

No, on this case XOVER works with a range parameter to the command that
tells the news servers which messages the newsreader is interested in. So
the 'id bigger then' above means that they are using the range way of
calling XOVER.

Since people can do 'XOVER 2-' to mean 'give me the headers for all messages
that have article number bigger than 2', I need to do that on the code. All
for NNTP compliance :)


> > My guess right now is that I'm doing something bad on the 
> routines that
> > spit the information to the newsreader somehow and because 
> of this the
> > server is consuming a lot of CPU. Since I'm not an expert in the
> > performance tweaks or even about possible bottlenecks, I 
> would love if
> > someone could take a look and maybe get some insight into 
> what could be
> > wrong. 
> 
> have you looked at the memory consuption too? maybe you 
> allacte more and 
> more space each request. (what's your log class doing?)
> 

Actually I was simply using 'top' to see the load information :)


> to find out where the CPU spends it cycles the python 
> profiler is the right 
> tool. its easy to use and helps to find bottlenecks.
> look at the docs in the stdlib for info about using it.
> i sugest running one session and then identify the function 
> that receives
> most of the execution time and look also at functions that are called 
> often.
> 

Yeah, I will have to read a little bit more on the documentation tonight.


> 
> other comments:
> i saw you compare often with 'ggg'. if you use this for not 
> yet defined 
> etc. i sugest to use "if article is not None: ..." etc. (of course 
> initializing undefined values as "article=None" at the beginning)
> 
> you don't need to compare "if somestring != '': ..." just write "if 
> somestring:..." as every non empty string will evaluate as TRUE.
> 

Hmm, indeed. The 'ggg' values are actually default values for the
'selected_article' and 'selected_group' member variables only if I remember
correctly. But yes, I guess I could make that a little more Python-friendly.


> note that most people don't care about such disclaimers, well 
> you posting 
> to a open newsgroup!
> 

Yeah, that comes from the MS Exchange mail server that handles the email
traffic at my work. Not my fault, it adds that silly banner automatically
when you send a message to outside the company.

Thanks for the pointers :)

Joao


*************************************************************************** 
 The contents of this communication are intended only for the addressee and
may contain confidential and/or privileged material. If you are not the
intended recipient, please do not read, copy, use or disclose this
communication and notify the sender.  Opinions, conclusions and other
information in this communication that do not relate to the official
business of my company shall be understood as neither given nor endorsed by
it.  
***************************************************************************