Need advice: serving data using http/web-server to clients ( not only browsers )

[Thomas Weholt]

Hi,

I want to serve data from a database using the HTTP-protocol and a
web-server. The data can be in several different formats, html, xml, cvs,
plain text or binary, compressed etc. The client is mostly browsers but a
specific client is written in python that processes the data from the
web-server further and gives the user the ability to manipulate it in
several ways ( generate other views etc. ) is also in production.

So I need to be able to serve html/xml to browsers and other formats too if
the client requests it. I've defined a request-protocol using the HTTP
post-command and params in the body-part of the request are used to query
the database and athenticate the user etc.

Sometimes the client will upload a binary file, using the client or browser,
to the server too. Also using the post-command.

So far I've used Medusa/Zope to extend the default web-server to fit my
needs, but I'm having a hard time getting it stable for the most simple
tasks and I cannot get the uploading of binary files to the server to work.
The performance is also not as good as Apache either. I'm a really huge fan
of Zope and use it for all my other web-tasks, but it seems slow. For this
specific task I need very low response-time and I don't need much of the
fancier stuff in Zope.

My project is very simple ( the complexity lies outside the web-server
"scope" ) ; I have a method that takes parts of the data in the
request-header as parameter, also any uploaded files. It returns the data to
the browser. I need to be able to set the content-type ( text/plain,
text/html, application/octet-stream etc.) of the response data, use
compression ( gzip, zlib ), authenticate users and receive data from the
browser/client. So all I need is to call my method with the required params,
which should be available in the header, or collected then sent to the
method, and return data. I prefer to the user authentication in my method
manually, not use anything in Apache etc.

I need to serve different platforms and environments ( linux, windows,
netscape, IE, lynx etc. ).

My main requirements are :
1. simplicity ( in the web-server, only a few lines of code should be
needed, possibly some to decode the header-data, then just a call to my
method )
2. performance ( the web-server/python-solution must have as little as
possible overhead, ie. I can suffer some code complexity to gain
performance, but keep number req. 1 in mind )
3. stability ( the actual server must be able to cope with high workloads
without failing. My implementation in Medusa/Zope crashes terribly under
very low workloads, *but* that's probably my fault, not Medusa/Zope. Still I
cannot get it to work right. )
4. compatibility ( all normal browsers and the version 1.1 of HTTP must be
supported, I need compression of data to the browser and uploading of large
binary files ( 1kb -10mb ???) )

Can anybody point me in the right direction? Apache with : FastCGI,
mod_snake, mod_python, pros and cons of different solutions, or just ideas
about my project, how I could do things better.

Thanks,
Thomas Weholt