Out-of-process Execution (was:Re: constructive critisism on pythonwin IDE)

[Mike C. Fletcher]

More verbosely, the "stale modules" problem is an issue related to the 
"easy way" to implement IDEs for Python, that is, providing an 
interpreter window.  The problem is that when a Python modules changes 
you need to reload each and every loaded module and figure out all sorts 
of tricky corner cases with inter-module dependencies and imports. 
PythonWin uses the interpreter-based approach, and punts on the reload 
requirement (likely because it's a messy problem).

Reload Requirement:
	In an interpreter window, you generally _want_ to retain previous results 
of calculations.  Some issues:

	You want all objects to update their class pointers to the newly loaded 
classes in the new modules (requires finding every object in the 
system).  You similarly need each import in the system to be re-done, in 
the same order they originally occured (and watching out for weird 
corner cases where the same names in different packages confuse the 
import mechanisms).

	There is code lying around in various usenet posts to address this issue 
by walking the object hierarchy and updating each encountered object. 
It's pretty ugly, but it's often possible to get it working right 80% 
(or whatever) of the time.  The rest of the time, something or other 
hasn't been updated (worst being exceptions and dynamically-generated 
constants).  You spend a lot of time tracking down subtle errors (the 
exception is thrown, you catch it, but it doesn't get caught, 
etceteras).  This approach needs a considerable amount of work to make 
it standard, stable and reliable.

	One approach to simplifying the problems above would be to re-run the 
interactive session on reload/change of a module.  But interpreters are 
often used for one-off calculations.  You don't want to have to re-run a 
30 minute calculation just to reload a particular module, but maybe you 
want all the other objects to be updated save your particular objects. 
If you're going to do this, you need to provide some way of marking a 
particular item static across a reload boundary.  Doable (store a 
reference in some dictionary for later usage), ugly though.  Arguably, 
you could even store all the objects, load the modules "clean" and if 
the user asks for a variable from the pre-reload session, scan it with 
algo above before giving it to them.


But, to my mind, though the approaches above might work, they miss the 
best solution, and the one you see in most (all?) of the newer IDEs for 
Python, namely out-of-process execution IN ADDITION to an interpreter 
(which is still used for "playing around" with what-ifs and interactive 
calculation sessions).  What we could really use is a robust 
(BSD-licensed) library for the following:

	running a python script in a seperate process (cross-platform, with 
ability to specify particular executable, and command-line flags 
(including -i for dropping into gui mode)) in either "standard" or 
"debug" mode

	support stdout, stderr stdin suitable for using as hooks with GUI 
libraries (i.e. ability to drop into an "interpreter" window when the 
script completes, type responses to console-mode programmes, and 
generally interact with the script as if it were being run from the 
console) GUI will be able to keep this window around after execution if 
it just wants to show user results from completed script.

	allow debugging a client script (with inspection support (i.e. be able to 
run a GUI inspector across the pipe/process barrier)
		[ you could do this with a wrapped set of stdout, stderr, stdin pipes 
(differentiating between system and "data" messages) or an out-of-band 
client-server comm mechanism (COM, CORBA, RPC) ]

	kill child process

	stop-at-breakpoint


And, add the following while you're at it ;) :

	provide for "interupt-to-debug" support (press a key, move the client 
process into debug mode) for debugging situations where code falls into 
weird states unpredictably

	"slow execution" modes for use with watch variables

	watch-variables (same interface as inspector above), basically the value 
is passed back on every single step of the programme being debugged.
	

I can't see adding this to PythonWin myself (though I use PythonWin as 
my IDE and find it the most comfortable by far, I can't see spending the 
time figuring out the raw win32api calls to create the various dialogs 
and support you'd need in the application when I just use the console 
for 98% of this kind of work), but if sameer (the original poster) feels 
like putting together a robust, reliable and complete module for 
out-of-process execution, he'd probably find lots of people willing to 
work on it with him, and one of them might be willing to do the Win32API 
coding.  Such a library would also make writing new Python IDEs 
considerably easier.

Just a thought,
Mike


F. GEIGER wrote:
> This a Python issue, not a PythonWin IDE issue.
> 
> Cheers
> Franz
> 
> "sameer" <sameer_ at email.com> schrieb im Newsbeitrag
> news:6bd9f01b.0206050736.f469082 at posting.google.com...
> 
>>First of all, let me start of by saying that Pythonwin is a great IDE.
>> Now for the constructive critisism.  The thing I hate about PythonWin
>>is, that despite the fact that it's a development environment, there
>>is a lot of caching done of modules, and the cached copy is not
>>checked against a potentialy modified copy.  Whenever I change the
>>contents of a file and try to run a module that depends on that file,
>>I always end up with the old copy.  I have to then import and then
>>reload the changed module in the interactive window for it to refresh.
>> It would greatly increase my productivity, if I can specify which
>>modules can be cached.  I understand that there is a speed and
>>development time issue, but I would like a little more fine grained
>>control on how caching works on PythonWin, as the way it's currently
>>implemented now is not acceptable.
> 
> 
> 


-- 
_______________________________________
   Mike C. Fletcher
   http://members.rogers.com/mcfletch/