[Web-SIG] A Python Web Application Package and Format

[Graham Dumpleton]

On 14 April 2011 18:22, Alice Bevan–McGregor <alice at gothcandy.com> wrote:
> Howdy!
>
> I suspect you're thinking a little too low-level.

Exactly, I am trying to walk before running. Things always fall down
here because people try and take too large a leap rather than an
incremental approach, solving one small problem at a time.

Thus please don't think that because I am replying to your message
that I am specifically commenting about your plans. See this as a side
comment and don't try and evaluate it only in the context of your
ideas.

> On 2011-04-14 00:53:09 -0700, Graham Dumpleton said:
>
>> On 14 April 2011 16:57, Alice Bevan–McGregor <alice at gothcandy.com> wrote:
>>>>
>>>> 3. Define how to get the WSGI app.  This is WSGI specific, but (1) is
>>>> *not* WSGI specific (it's only Python specific, and would apply well to
>>>> other platforms)
>>>
>>> I could imagine there would be multiple "application types":
>>>
>>> :: WSGI application.  Define a package dot-notation entry point to a WSGI
>>> application factory.
>>
>> Why can't it be a path to a WSGI script file?
>
> No reason it couldn't be.
>
> app.type = wsgi
> app.target = /myapp.wsgi:application
>
> (Paths relative to the folder the application is installed into, and dots
> after a slash are filename parts, not module separators.)
>
> But then, how do you configure it?  Using a factory (which is passed the
> from-appserver configuration) makes a lot of sense.
>
>> This actually works more universally as it works for servers which map
>> URLs to file based
>> resources as well.
>
> First, .wsgi files (after a few quick Google searches) are only used by
> mod_wsgi.  I wouldn't call that "universal", unless you can point out the
> other major web servers that support that format.

The WGSI module for nginx used them, as does uWSGI and either one of
Phusion Passenger or new Mongrel WSGI support rely on a script file.

You also have CGI, FASTCGI, SCGI and AJP also using script files.

Don't get hung up on the extension of .wsgi, it is the concept of a
script file which is stored in the file system in an arbitrary
location to which a URL maps.

> You'll have to describe the "map URLs to file based resources" issue, since
> every web server I've ever encountered (Apache, Nginx, Lighttpd, etc.) works
> that way.

Which supports what I am saying, but you for some reason decided to
focus on '.wsgi' as an extension which wasn't the point.

> Only if someone is willing to get really hokey with the system
> described thus far would any application-scope web servers be running.

Forget for a moment trying to tie this to your larger designs and see
it as more of a basic underlying concept. Ie., the baby step before
you try and run.

>> Also allows alternate extensions than .py and also allows basename of file
>> name to be arbitrarily named, both of which help with those same servers
>> which map URLs to file base resources.
>
> Again, you'll have to elaborate or at least point to some existing
> documentation on this.
>
> I've never encountered a problem with that, nor do any of my scripts end in
> .py.

Lack of an extension is fine if you have configured Apache with a
dedicated cgi-bin or fastcgi-bin directory where an extension is
irrelevant because you have:

  SetHandler cgi-script

But many Apache server configurations use:

  AddHandler cgi-script .py

Ie., handler dispatch is based off extension, the .py extension quite
often being associated with CGI script execution.

You often see:

  AddHandler fcgid-script .fcgid

Which says certain resource is to be started up as FASTCGI process.

For both these it expects those scripts to be self contained programs
which fire up the mechanics of interfacing with CGI or FASTCGI
protocols.

This means that you usually have to stick that boilerplate at the end
of the script.

This is where though FASTCGI deployment usually sucks bad. This is
because it is put on the user to get the boilerplate and remainder of
WSGI script perfect from the outset. If you don't, because FASTCGI
technically doesn't allow for stdout/stderr at point of startup, if
there is an error on import it is lost and user has no idea. So many
times you see people winging about setting up stuff on the likes of
DreamHost because of FASTCGI being a pain like this.

In the PHP world they don't have to deal with this boilerplate
nonsense. Instead there is a PHP launcher script associated with
FASTCGI module. So you have:

  AddHandler fcgid-script .php

but also a mapping in FASTCGI module configuration that says rather
than execute .php script if runs the launcher script instead. That way
the launcher script can get everything setup properly to then call
into the script.

Nothing exists for Python like that, but if you did then it makes no
sense to use .py because of the mapping that extension often already
has in Apache. In that case you would have .wsgi script file mapped to
FASTCGI but FASTCGI configured to run a WSGI launcher. That launcher
script would setup stdout/stderr, ensure flup is loaded properly and
only then load the WSGI script file and execute. This way the system
administrators could ensure the launcher is working and users only
have to worry about dumping a WSGI script file with right extension in
a directory and it will work without all the pain. Also allows the
system admins to properly control number of processes/threads whereas
at present users can override what system admins would like to
restrict them to.

So, a concept of a script file simply works better with Apache and to
some degree other servers. This is because of how such servers
determine what handler to use from the extension

As to the file name, you can't stop people using arbitrary stuff in
file names, ie., dashes as a prime example. So when using servers
which map URLs to file system resources you have to deal with it.

>> It also allows same name WSGI script file to exist in multiple locations
>> managed by same server without having to create an overarching package
>> structure with __init__.py files everywhere.
>
> Packages aren't a bad thing.  In fact, as described so far, a top level
> package is required.

You are thinking ahead to your bigger ideas. That isn't what I am
talking about. You can't when using a web server which can map URLs to
resources within a hierarchical directory structure have that
structure be a package with __init__.py files in directories, it just
doesn't work as all the scripts could be totally unrelated and not
part of one application.

Graham