Jeremy Hylton : weblog : 2003-04-28

Finding the Zope3 HTTP Server

Monday, April 28, 2003

My goal for today was to write a simple benchmark for the HTTP server that comes with Zope3. I wanted to test the performance implications of socket changes in Python 2.3. I'd also been meaning to see how Zope3 does on Joe Armstrong's challenge.

The Python 2.3 issue is that a small change to the socket module to support timeouts seems to have a big performance implication for servers. Guido explained the problem on python-dev:

I'm guessing that the slowdown comes from the fact that calling a method like recv() on the wrapper object is now a Python method which calls the C method on the wrapped object.

Joe Amstrong's challenge had to do with Erlang's user-level threads. An Erlang server can handles thousands of threads at once, and, thus, supports a very efficient multi-threaded web server. At LL2, one of Joe's challenges was to sustain high throughput as number of concurrent connections increased.

Anway, the task at hand is to find the HTTP server code in Zope3. We really need an IDE for Zope3 that integrates Python, TAL, and ZCML. It would be fun to see if Eclipse would work.

I know there's a zope.publisher.http module, but I'm not quite sure what it does. (This ended up being a false start.)

zserver.zcml has a directive for the HTTP server.

     <startup:addServer type="HTTP" port="8080" verbose="true" />

I know a little bit out zcml directives and I recognize the name startup. That's zope.app.startup -- although I'm not sure how I would have found startup otherwise. It's got a meta.zcml that says

    <directive name="defineSite"
               attributes="name threads"
               handler="zope.app.startup.metaconfigure.defineSite">
...
      <subdirective name="addServer"
                    attributes="type port verbose logClass" />

</directive>

That's a great clue, because I now have some source code to look at. There's one slight annoyance: the defineSite() function is actually an alias for zope.app.startup.sitedefinition.SiteDefinition. Is this just in need of a little refactoring? Or is there a reason for the alias?

The addServer() method isn't very helpful, though. It just adds some configuration strings to a list of servers. The start() method gets a server object using getServerType() and calls its create() method, so that must be where the HTTP server is actually created. The getServerType() function is a little odd; it uses a dictionary for registration but adds two layers of classes, modules, and interfaces to it. In the end, I see that a server must call registerServerType() to add make the server configuration work, but I can't find any code that calls registerServerType().

I see that registerServerType is called from ZCML code. (Earlier today, Guido reminded me that with Zope3 code you need to grep Python + ZCML because the only call to a function often occurs in some ZCML.) A grep revealed that the registration was still here in zope.app.startup. The directive of interest points me at zope.server instead of zope.publisher. That makes sense! (The little bit of initial knowledge I had actually hurt, because HTTP shows up as both a server and a publisher.)

  <startup:registerServerType 
    name = "HTTP"
    factory = "zope.server.http.publisherhttpserver.PublisherHTTPServer"
    requestFactory="HTTPRequestFactory"
    logFactory = "zope.server.http.commonhitlogger.CommonHitLogger"
    defaultPort="8080"
    defaultVerbose="true" />

There is some code in zope.server.http.test.test_httpserver has the example code I was looking for. I'll need to study this code a little, because it appears to demonstrate the Zope3 approach for combining threads and the asyncore mainloop. There is also a simple example at the bottom of zope.server.http.httpserver.