[Web-SIG] A Python Web Application Package and Format

[Ian Bicking]

On Sun, Apr 10, 2011 at 10:29 PM, Alice Bevan–McGregor
<alice at gothcandy.com>wrote:

> Howdy!
>
>
> On 2011-04-10 19:06:52 -0700, Ian Bicking said:
>
>  There's a significant danger that you'll be creating a configuration
>> management tool at that point, not simply a web application description.
>>
>
> Unless you have the tooling to manage the applications, there's no point
> having a "standard" for them.  Part of that tooling will be some form of
> configuration management allowing you to determine the requirements and
> configuration of an application /prior/ to installation.  Better to have an
> application rejected up-front ("Hey, this needs my social insurance number?
> Hells no!") then after it's already been extracted and potentially littered
> the landscape with its children.


I... think we are misunderstanding each other or something.

A nice tool that could use this format, for instance, would be a tool that
takes an app and creates a puppet recipe to setup a sever to host the
application.  A different tool (maybe better, maybe not?) would be a puppet
plugin (if that's the terminology) that uses this format to tell puppet
about all the requirements an application has, perhaps translating some
notions to puppet-native concepts, or adding high-level recipes that setup
an appropriate container (which can be as simple as a properly configured
Nginx or Apache server).

What I mean when I say there's a danger of becoming a configuration
management tool, is that if you include hooks for the application to
configure its environment you are probably stepping on the toes of whatever
other tool you might use.  And once you start down that path things tend to
cascade.



>
>  The escape valve in Silver Lining for these sort of things is services,
>> which can kind of implement anything, and presumably ad hoc services could
>> be allowed for.
>>
>
> Generic services are useful, but not useful enough.
>
>
>  You create a build process as part of the deployment (and development and
>> everything else), which I think is a bad idea.
>>
>
> Please elaborate.  There is no requirement for you to use the "application
> packaging format" and associated tools (such as an application server)
> during development.  In fact, like 2to3, that type of process would only
> slow things down to the point of uselessness.  That's not what I'm
> suggesting at all.


If you include something in the packaging format that indicates the
libraries to be installed, then you are encouraging and perhaps requiring
that the server install libraries during a deployment.

Realistically this can't be entirely avoided, but I think it is a pretty
workable separation to declare only those dependencies that can't reasonably
be included directly in the application itself (e.g., lxml, MySQLdb, git,
and so on).  In Silver Lining those dependencies were expressed as Debian
package names, installed via dpkg, but for a more general system it would
need to be somewhat more abstract.  But several configuration management
tools have managed that abstraction already, so it seems feasible to handle
this declaratively.


>  My model does not use setup.py as the basis for the process (you could
>> build a tool that uses setup.py, but it would be more a development
>> methodology than a part of the packaging).
>>
>
> I know.  And the end result is you may have to massage .pth files yourself.
>  If a tool requires you to, at any point during "normal operation", hand
> modify internal files… that tool has failed at its job.  One does not go
> mucking about in your Git repo's .git/ folder, as an example.
>

.pth files aren't exactly an "internal file" -- they are documented feature
of Python.  And .git/config is also a human-readable/editable file!

But I did note that the setup in Silver Lining was a bit too primitive.  Not
*quite* as primitive as App Engine, but close.  I think it would be better
to have a convention like adding lib/python/ to the path automatically.  If
you want, for example, src/myapp to also be added to the path then I don't
think there's anything wrong with using a .pth file to do that; that's what
they were created to do!


> How do you build a release and upload it to PyPi?  Upload docs to
> packages.python.org?  setup.py commands.  It's a convienent hook with
> access to metadata in a convienent way that would make an excellent "let's
> make a release!" type of command.
>
>
>  Also lots of libraries don't work when zipped, and an application is
>> typically an aggregate of many libraries, so zipping everything just adds a
>> step that probably has to be undone later.
>>
>
> Of course it has to be un-done later.  I had thought I had made that quite
> clear in the gist.  (Core Operation, point 1, possibly others.)
>
>
>  If a deploy process uses zip file that's fine, but adding zipping to
>> deployment processes that don't care for zip files is needless overhead.  A
>> directory of files is the most general case.  It's also something a
>> developer can manipulate, so you don't get a mismatch between developers of
>> applications and people deploying applications -- they can use the exact
>> same system and format.
>>
>
> So, how do you push the updated application around?  Using a full directory
> tree leaves you with Rsync and SFTP, possibly various SCM methods, but then
> you'd need a distinct repo (or rootless branch) just for releasing and
> you've already mentioned your dislike for SCM-based deployment models.
>
> Zip files are universal -- to the point that most modern operating systems
> treat zip files /as folders/.  If you have to, consider it a transport
> encoding.


I guess I envision tools that specifically understand this format, not using
ad hoc tools to move stuff around.  A tool that "understands" this format
could be as simple as:

#!/bin/sh
T=$(tempfile).zip
NAME=$(python -c "import webpkg, sys
print webpkg.parse(sys.argv[1]).name" "$1")
zip -r $T "$1"
scp $T "$2":/var/server/apps && rm $T


Now there's a lot more features that I'd want than that script could handle,
but it might be nice for some people.  But I would not be opposed to asking
tools to understand zip files, so long as they understand directories too.
 That would introduce a few open issues, like whether symlinks are
supported, or perhaps other details where zip is less expressive than files.


>
>  The pattern that it implements is fairly simple, and in several models you
>> have to lay things out somewhat manually.  I think some more convention and
>> tool support (e.g., in pip) would be helpful.
>>
>
> +1
>
>
>  Though there are quite a few details, the result is more reliable, stable,
>> and easier to audit than anything based on a build process (which any use of
>> "dependencies" would require -- there are *no* dependencies in a Silver
>> Lining package, only the files that are *part* of the package).
>>
>
> It might be just me (and the other people who seem to enjoy WebCore and
> Marrow) but it is fully possible to do install-time dependencies in such a
> way as things won't break accidentally.  Also, you missed Application Spec
> #4.


OK; then #4 is is the only thing I would choose to support, as it is the
most general and easiest for tools to support, and least likely to lead to
different behavior with different tools.  And not to just defer to
authority, but having written a half dozen tools in this area, not all of
them successful, I feel strongly that including dependencies is best --
simplest for both producer and consumer, and most reliable.


>
>  Some notes from your link:
>>
>> - There seems to be both the description of a format, and a program based
>> on that format, but it's not entirely clear where the boundary is.  I think
>> it's useful to think in terms of a format and a reference implementation of
>> particular tools that use that format (development management tools, like
>> installing into the format; deployment tools; testing tools; local serving
>> tools; etc).
>>
>
> Indeed; this gist was some really quickly hacked together ideas.
>
>
>  - In Silver Lining I felt no need at all for shared libraries.  Some disk
>> space can be saved with clever management (hard links), but only when it's
>> entirely clear that it's just an optimization.  Adding a concept like
>> "server-packages" adds a lot of operational complexity and room for bugs
>> without any real advantages.
>>
>
> ±0
>
>
>  - I try to avoid error conditions in the deployment, which is a big part
>> of not having any build process involved, as build processes are a source of
>> constant errors -- you can do a stage deployment, then five minutes later do
>> a production deployment, and if you have a build process there is a
>> significant chance that the two won't match.
>>
>
> I have never, in my life, encountered that particular problem.  I may be
> more careful than most in defining dependencies with version number
> boundaries, I may be more careful in utilizing my own package repository
> (vs. the public PyPi), but I don't think I'm unique in having few to no
> issues in development/sandbox/production deployment processes.
>

Well, lots (and lots and lots) of other people have ;)  Also lots of these
other techniques require consistency between development and deployment (for
instance, using the same private package repository).  This is fine when you
are careful and consider any failures to be of your own making, but if you
are deploying someone else's application you won't feel the same way, and
may make different kinds of mistakes.

A perhaps implicit goal in my mind is allowing people to deploy applications
that they did not write, nor where they care about the implementation of the
app itself.  A lot of things are different when you separate out the
developer's knowledge from the deployers.


Hell, I'm still able to successfully deploy a TurboGears 0.9 application
> without dependency issues.
>
> However, the package format I describe in that gist does include the source
> for the dependencies as "snapshotted" during bundling.  If your application
> is working in development, after snapshotting it /will/ work on sandbox or
> production deployments.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110411/e2f3b9c4/attachment.html>