Beta testers needed for a high performance Python application server

[Graham Dumpleton]

On Mar 26, 7:31 am, Minor Gordon <Minor.Gor... at cl.cam.ac.uk> wrote:
> Hello all,
>
> I'm looking for beta testers for a high performance, event-driven Python
> application server I've developed.
>
> About the server: the front end and other speed-critical parts of the
> server are written in portable, multithreaded C++. The back end is an
> embedded CPython interpreter. The server is much faster than anything in
> pure Python, and it can compete with C servers (including e.g. lighttpd
> for file workloads) or outdo them (e.g. anything behind Apache) until
> CPython consumes a single processor. On the Python side it supports WSGI
> (the server can handle the static and dynamic requests of MoinMoin with
> a handful of lines), the DB API with blocking calls offloaded to a
> connection in a separate thread (MySQL, SQLite supported), Google's
> ctemplate, gzipping responses, file caching, reading and writing to URIs
> as a client, AJAX integration, debugging as a Python extension, and a
> lot of other features. The core Python API is event-driven, using
> continuations like Twisted but much cleaner (continuations are any
> callables, there are no special objects anywhere). The Python back end
> also supports Stackless Python so all of the continuation machinery can
> be hidden behind tasklet switching.
>
> Background: I'm in this to help write a "story" for Python and web
> applications. Everyone likes to go on about Ruby on Rails, and as far as
>   I can tell there's nothing that approaches Rails in Python. I want to
> code quickly in Python like I can with Rails, but without sacrificing
> single node performance on many cores.
>
> Beta testers: should be intermediate to advanced Python programmers with
> demanding applications, particularly web applications with databases and
> AJAX. The C++ is portable to Win32, Linux, and OS X, with no mandatory
> libraries beyond python-dev.
>
> Please contact me if you're interested: firstname.lastname at cl.cam.ac.uk.

Why not just put it on the net somewhere and tell us where it is?
People aren't generally going to want to help or even look at it if
you treat it like a proprietary application. So, put the documentation
and code up somewhere for all to see.

BTW, multiprocess web servers such as Apache can quite happily make
use of multiple cores. Even within a single Apache multithread process
it can still use multiple cores quite happily because all the
underlying network code and static file handling code is in C and not
subject to the GIL. So, as much as people like to bash up on the GIL,
within Apache it is not necessarily as big a deal as people make out.

Also, an event driven model, or even a large dependence on
multithreading, can actually be worse for Python web applications
where using it means one can handle a much larger number of concurrent
requests. This is because you greatly increase the risk of code having
large requirements for transient memory being hit at the same time.
The result can be large unpredictable blowouts in memory requirements
for the process, with that memory then being held by the process and
not necessarily able to be released back to operating system. Thus,
for large Python web applications, use of a multiprocess web server,
where each worker process is single threaded, is in various way still
best as it provides the most predictable memory profile.

Finally, the speed of the underlying web server (except for CGI)
generally has minimal bearing on the performance of a large Python web
application. This is because that isn't where the bottlenecks are. The
real bottlenecks are generally in the application code itself and in
any access to a back end database. Thus, pursuing absolute speed is a
bit of a fools errand when you consider that any performance gain you
may have over a competing solution may only end up resulting in
somewhat less than 1% difference when one looks at overall request
time.

Graham