Performance Issues with Threaded Python Network Server

[Chris Liechti]

Joao Prado Maia <JMaia at lexgen.com> wrote in
news:mailman.1011124161.26356.python-list at python.org: 
> 
> Hi,
> 
> Before even starting with the description of my problem, let me explain
> my situation. I'm trying to create a custom Python based NNTP server to
> serve as a gateway of sorts to some web message boards. This way people
> can connect to the News server and get messages / reply and such.

nice

...
> The problem here is that whenever this happens, the CPU usage of the
> NNTP server goes to about 35% and continues increasing slowly while the
> newsreader is receiving all the message headers and bodies.

in some of your SQL expressions you write "WHERE... A.id >= %s" doesn't 
this mean that the number of messages that have to be filtered increases 
with the id to get? (snipet above from get_XOVER) (long a go my SQL 
experiences....)

> My guess right now is that I'm doing something bad on the routines that
> spit the information to the newsreader somehow and because of this the
> server is consuming a lot of CPU. Since I'm not an expert in the
> performance tweaks or even about possible bottlenecks, I would love if
> someone could take a look and maybe get some insight into what could be
> wrong. 

have you looked at the memory consuption too? maybe you allacte more and 
more space each request. (what's your log class doing?)

to find out where the CPU spends it cycles the python profiler is the right 
tool. its easy to use and helps to find bottlenecks.
look at the docs in the stdlib for info about using it.
i sugest running one session and then identify the function that receives
most of the execution time and look also at functions that are called 
often.


other comments:
i saw you compare often with 'ggg'. if you use this for not yet defined 
etc. i sugest to use "if article is not None: ..." etc. (of course 
initializing undefined values as "article=None" at the beginning)

you don't need to compare "if somestring != '': ..." just write "if 
somestring:..." as every non empty string will evaluate as TRUE.

have fun
chris


> ************************************************************************
> *** 
>  The contents of this communication are intended only for the addressee
>  and 
> may contain confidential and/or privileged material. If you are not the
> intended recipient, please do not read, copy, use or disclose this
> communication and notify the sender.  Opinions, conclusions and other
> information in this communication that do not relate to the official
> business of my company shall be understood as neither given nor
> endorsed by it.  
> ************************************************************************
> *** 

note that most people don't care about such disclaimers, well you posting 
to a open newsgroup!


-- 
Chris <cliechti at gmx.net>