Python + IIS/ASP questions (modules, namespaces, etc)

[Preston Landers]

Pythoneers:

I have some technical questions about the Python ASP server side
scripting available to IIS users.  I've seen someone post some
questions along the same lines as the questions I have (one is quoted
below), but unfortunately I have not seen any relevant responses.
Perhaps I should post this on the Active Python mailing list, as that
seems to be more win32 oriented.

Basically I have my web site, which is 100% dynamically generated in
Python.  On Unix we have Apache and mod_python, which works
beautifully.  Each Apache child process has a Python interpreter which
is re-used for each request.  Each interpreter has a namespace which
persists between requests, which allows us to stash a database
connection that can be reused.  Yet, each interpreter remains in a
separate process, so there are no issues with stepping on each others
modules, or threading issues with the global lock.  This works well
and we are happy with it, and we want to achieve a similar effect on
the Windows platform.

Currently, our Windows code works as a traditional CGI (spawning a
python.exe for each request from IIS.)  To avoid the significant
overhead of acquiring a new database connection for each request, the
CGI script ends up talking (over a socket) to a limited number of
pre-initialized "application server" processes that do the actual
work.  This works OK for limited numbers of users, but as you might
imagine it does not scale very well as you pile on the users.

The obvious solution would be to just use Apache and mod_python on
Windows, since we know our solution works there and we are happy with
it.  And this may indeed be the best way to go -- I am going to look
into the feasibility of building and packaging Win32 Apache+mod_python
for our platform.  But, for one reason or another, the powers that be
have indicated that we are to support IIS on Windows.

So, I have been examining the Python active server pages (ASP)
solution.  It seems that ASP is the only way to get state (such as a
DB connection) to persist across connections.  However, there appear
to be issues with modules being shared (as in the quoted message
below.)  All of our application modules were basically written under
the assumption that they would not be shared, and we use a number of
module-level variables, etc.  I am also worried about bugs in our code
crashing the whole site on a regular basis.  I've got a copy of
O'Reilly's "Python Programming on Win32" but the section on ASP is
woefully lacking in details like these.

The only real state I need to persist across connections is the actual
database connection object, since these can be so expensive to obtain
(about 3 or 4 seconds in my tests.) 

The idea I had for that was to register a RemoteServer COM object that
would basically act as a database connection pool.  Each web request
would be spawned as a traditional CGI (I'm not terribly worried about
the overhead there) -- and it would then talk to the COM object and
send all of its database methods as packaged requests to it, and
receive database rows (etc) in return.  All of the DB connections
would be kept inside the RemoteServer COM object.

The only drawback there (besides the work required to code up the
database pool COM object) is that we now have to worry about correctly
translating parameter (and exception) types across COM, which also
uses Unicode for all strings.  Sounds like a lot of headache just to
be able to keep a reusable database connection.  Hopefully calling
Python COM objects from Python clients should take care of most of the
gory details, but I haven't tried it yet.  The most complicated
objects we will be passing will likely be lists of tuples with varying
simple types inside (ie, database record sets.)

Basically I'm writing this post to find out what other people may have
done in this area and to ask for general guidance, etc.  Has anyone
tried using ASP or a separate COM server for the sole purpose of
database connection pooling?

I'm also going to try to find out more about the Apache + mod_python
on win32 route, and whether that is "politically" feasible for us.
Has anyone else out there had much experience with mod_python on
win32?

Any suggestions or comments will be greatly appreciated and hopefully
beneficial for other people out there who want to deploy high volume
dynamic web sites on Windows with Our Favorite Language.

yours,

Preston Landers
Senior Software Developer
Journyx, Inc.

pibble at yahoo dot com  (or)
planders at journyx dot com


------------------------------------------------------------------

Quoted posting from "chris" below (no email address given, posted Nov
20th 1998 in c.l.python.)

Follow my thought process for a moment:

As I understand the Python ASP implementation, the Python ActiveX scripting
engine is loaded by IIS on the first ASP invocation.  That engine continues
to run and stay loaded from that point forward to handle each Python ASP
request.  (IIS may unload it at some point, but it won't affect this
discussion.)

The engine (which occupies high-level and separate namespaces including
__main__) receives each ASP file from IIS, sets it up in it's own module
(and namespace) __ax_main__ (below __main__) and puts the ASP objects
(Request, Response, etc..) in the namespace for that script.  Each ASP
script can then execute and act upon the ASP objects in its namespace.  Each
ASP script/file that executes, get's its own namespace and its own separate
ASP objects to act upon.  The Python engine runs continuously above all the
ASP file invocations and manages all the requests, executions, and ASP
objects.

Furthermore:

I have verified that a single ASP file, while excuting, is run in a module
called __ax_main__ (as mentioned above).  If my previous understandings are
correct, then it would seem that subsequent ASP executions while the first
is still running CAN'T receive the same module name.  Perhaps the namespaces
for each subsequent concurrent module are __ax_main1__, __ax_main2__ ????  I
don't see how they could have the same name if all the ASP executions are
occuring in the same Python engine.  Comments?

--------------

On a related topic, it would follow that each IMPORT statement in all the
ASP files (that may be running concurrently) are all importing into the same
Python engine.  Therefore, two ASP files (running concurrently by
IIS/Python) that both perform "import foo.py" are using the same module AT
THE SAME TIME (classes generally don't count in this discussion - I'm
assuming they are safe).

In a "regular" Python program (if I don't use ASP - just standard
command-line) the multiple import issue is not a problem because the Python
command-line configuration is single-threaded.  A non-ASP Python program can
have multiple modules all importing and using each other because there is
only one thread.  The multiple uses of a module can't collide.

Now, my next assumption is that the Python ASP engine is multi-threaded.  It
would seem to be a poor implementation otherwise.  With this assumption
(multi-threaded, not a poor-implementation :-), is the multiple use of
imported modules a problem and does the engine perform any actions to
guarantee thread-safe behavior?  In the example above, what do

An example:

foo.py
=============
count=0

def getCurrentCount():
    count=count+1
    return count
=============

Assume many different ASP files with code such as:

import foo
Response.write(getCurrentCount())

What behavior do the ASP files show?  Is is deterministic?  Does each ASP
file always return 1.  Do they increment until all ASP files are finished
running and then reset back to 1?


chris