control CPU usage

[Dave Angel]

kakarukeys wrote:
> On Sep 20, 10:57 pm, Dave Angel <da... at ieee.org> wrote:
>   
>> kakarukeys wrote:
>>     
>>> On Sep 20, 6:24 pm, Dave Angel <da... at ieee.org> wrote:
>>>       
>>>> Jiang Fung Wong wrote:
>>>>         
>>>>> Dear All,
>>>>>           
>>>>> Thank you for the information. I think I've some idea what the problem is
>>>>> about after seeing the replies.
>>>>>           
>>>>> More information about my system and my script
>>>>>           
>>>>> PIII 1Ghz, 512MB RAM, Windows XP SP3
>>>>>           
>>>>> The script monitors global input using PyHook,
>>>>> and calculates on the information collected from the events to output some
>>>>> numbers. Based on the numbers, the script then performs some automation
>>>>> using SendKeys module.
>>>>>           
>>>>> here is the memory usage:
>>>>> firefox.exe, 69MB, 109MB
>>>>> svchost.exe, 26MB, 17MB
>>>>> pythonw.exe, 22MB, 17MB
>>>>> searchindexer.exe, 16MB, 19MB
>>>>>           
>>>>> My first guess is that the script calculated for too long time after
>>>>> receiving an event before propagating it to the default handler, resulting
>>>>> the system to be non-responsive. I will try to implement the calculation
>>>>> part in another thread.
>>>>> Then the separate will have 100% CPU usage, hope the task scheduling of
>>>>> Windows works in my favour.
>>>>>           
>>>> (You top-posted this message, putting the whole stream out of order.  So
>>>> I deleted the history.)
>>>>         
>>>> All my assumptions about your environment are now invalid.  You don't
>>>> have a CPU-bound application, you have a Windows application with event
>>>> loop.  Further, you're using SendKeys to generate a keystroke to the
>>>> other process.  So there are many things that could be affecting your
>>>> latency, and all my previous guesses are useless.
>>>>         
>>>> Adding threads to your application will probably slow the system down
>>>> much more.  You need to find out what your present problem is before
>>>> complicating it.
>>>>         
>>>> You haven't really described the problem.  You say the system is
>>>> unresponsive, but you made it that way by creating a global hook;  a
>>>> notoriously inefficient mechanism.  That global hook inserts code into
>>>> every process in the system, and you've got a pretty low-end environment
>>>> to begin with.  So what's the real problem, and how severe is it?  And
>>>> how will you measure improvement?  The Task manager numbers are probably
>>>> irrelevant.
>>>>         
>>>> My first question is whether the pyHook event is calling the SendKeys
>>>> function directly (after your "lengthy" calculation) or whether there
>>>> are other events firing off  in between.  If it's all being done in the
>>>> one event, then measure its time, and gather some statistics (min time,
>>>> max time, average...).  The task manager has far too simplistic
>>>> visibility to be useful for this purpose.
>>>>         
>>>> What else is this application doing when it's waiting for a pyHook
>>>> call?  Whose event loop implementation are you using?  And the program
>>>> you're trying to control -- is there perhaps another way in?
>>>>         
>>>> DaveA
>>>>         
>>> Hi,
>>>       
>>> Sorry I wasn't sure how to use Google groups to post a msg to the
>>> newsgroup, I used Gmail to write my previous reply. What you and the
>>> other guy have provided me isn't useless. Now I understand the non-
>>> responsiveness may not be caused by high CPU usage, as the OS, be it
>>> Windows or Linux, has a way to prioritize the tasks. This is a vital
>>> clue to me.
>>>       
>>> By "not responsive", I mean, for some time, the mouse pointer is not
>>> moving smoothly, to such extent that I can't do anything with the
>>> mouse. It's like playing a multi-player game on a connection with a
>>> lot of lag. It's not caused by global hook, because it happens under
>>> certain condition, i.e. when fpa.ProcessEvent(word) is computing.
>>>       
>>> I included my main script for your reference. Comments:
>>> (1) The automation method tc.Auto() is slow, but it doesn't cause any
>>> problem, because the user would wait for the automation to finish,
>>> before he continues to do something.
>>>       
>>> (2) all other methods invoked are fast, except fpa.ProcessEvent(word)
>>> (this information is obtained from profiling). It is this method that
>>> causes 100% CPU usage. I'm planning to move this method to a separate
>>> thread, so that OnEvent(event) can finish executing, while the
>>> separate thread goes on to finish its calculation. Is this a good
>>> idea?
>>>       
>>> import pyHook
>>> import TypingAnalyzer
>>> import GUI
>>>       
>>> def OnEvent(event):
>>>    if hasattr(event, "Key") and event.Ascii =9 and event.Key == "Tab"
>>> and event.Injected =0 and event.Alt == 0:
>>>            tc.Auto()
>>>            return False
>>>    else:
>>>            recognized =k.ProcessEvent(event)
>>>            if recognized:
>>>                    tc.MatchChar(recognized)
>>>                    paragraph =c.ProcessEvent(recognized)
>>>                    if paragraph:
>>>                            for word in paragraph:
>>>                                    fpa.ProcessEvent(word)
>>>       
>>>            return True
>>>       
>>> hm =yHook.HookManager()
>>> hm.MouseAllButtonsDown =nEvent
>>> hm.KeyDown =nEvent
>>> hm.HookMouse()
>>> hm.HookKeyboard()
>>>       
>>> rk =ypingAnalyzer.ReadKey()
>>> rc =ypingAnalyzer.ReadChar()
>>> fpa =TypingAnalyzer.Analysis()
>>> tc =ypingAnalyzer.Automation(fpa)
>>>       
>>> if __name__ ='__main__':
>>>    app =UI.AWAApp()
>>>    app.MainLoop()
>>>       
>>> Thank you for your attention.
>>>       
>> I can't readily comment on your code, since it's entirely written to
>> three imports that I have never seen.  I have to assume that
>> TypingAnalyzer contains logic to automate your external program.  I have
>> no idea where GUI comes from, but I have to hope it knows how to
>> efficiently use the CPU, like most gui event loops.  If  you don't do
>> the pyHook, is the system quite responsive?
>>
>> I actually doubt if moving some of the logic to another thread is going
>> to make any difference.  In Python, threading helps to overlap CPU bound
>> stuff and I/O bound stuff.  But theoretically, as soon as you return to
>> the event loop, your main thread is going to block, so it'll schedule
>> the other thread.  The real question is whether there's communication
>> going between the two processes in those various rk, rc, etc pieces.  
>> It's easy for such protocols to become very inefficient.
>>
>> Since I can't directly help, at least I could suggest looking at XP's  
>> ControlPanel->System->Advanced.  That gets you to a dialog entitled
>> Performance Options.
>>
>> On the Advanced tab of that dialog, the first section is "processor
>> scheduling".   Mine is set to "Programs" which means give extra priority
>> to the GUI program that has the focus.  Presumably in your environment
>> that'll be the typing code, not your python app.  Setting it to
>> "Background services" would give more priority to the CPU bound of your
>> python app.
>>
>> But once these two apps are aware of each other, they could be doing all
>> sorts of things to confuse the scheduler.
>>
>> DaveA
>>     
>
> Hi DaveA,
>
>   
>> If you don't do the pyHook, is the system quite responsive
>>     
>
> I'm not sure how to answer this question. The program needs pyHook, so
> I can't leave it out. Is there other alternative that could monitor
> global input? Running other programs without pyHook creates no
> problem. Running the pyHook example demo creates no problem either.
>
>   
>> But theoretically, as soon as you return to the event loop, your main thread is going to block, so it'll schedule the other thread.
>>     
>
> Could you explain more on this part? Your statement is mind boggling
> to me.
>
> Cheers,
> W.J.F.
>
>   
An event loop should be blocking, waiting for the OS to produce another 
event.  If so, then other threads would get control as needed.

But this afternoon I came up with another plausible explanation for the 
jerky behavior on your system.  You're doing a system hook, and I 
suspect there's some form of release needed before the system is a able 
to process any more events of the same type.  As I said before, I don't 
have pyHook, so I haven't read its documentation.  But I'm guessing the 
hook gets released when you return to the event loop.  So if you have 
more processing to do, rather than moving it to another thread, move it 
to an extra event.

Read the docs for GUI.  It will undoubtedly have some way to create new 
event types.  So register a new event type, attach a handler to it, and 
post it from your current event handler using something like:   "postafter"

That new event handler is where you do the time-consuming stuff that 
you're having trouble with.

DaveA