Why so fast a web framework?

[Benjamin Schollnick]

>> Sometimes this group reminds me of a certain large company I worked for.
>> If they didn't have a solution addressing a problem, they'd pretend it
>> didn't matter and belittle anyone who questioned that version of reality.
>> 
> 
> That's not strictly true; what's happening here is that someone's
> published a cool-looking bar graph but nobody knows what it really
> means. I don't feel the need to delve into everyone's benchmarks to
> explain why Python is purportedly worse. If someone uses those kinds
> of numbers to decide which programming language to use, they have
> bigger problems.

If you dig a bit, the benchmark is scary…
As in stupid-scary.

It consists of, 7 tests, and then a composite score is generated:

JSON Serialization - In this test, each response is a JSON serialization of a freshly-instantiated object that maps the key message to the value Hello, World!
Single Query - In this test, each request is processed by fetching a single row from a simple database table. That row is then serialized as a JSON response.
Multiple Queries - In this test, each request is processed by fetching multiple rows from a simple database table and serializing these rows as a JSON response. The test is run multiple times: testing 1, 5, 10, 15, and 20 queries per request. All tests are run at 512 concurrency.
Cached Queries - In this test, each request is processed by fetching multiple cached objects from an in-memory database (the cache having been populated from a database table either as needed or prior to testing) and serializing these objects as a JSON response. The test is run multiple times: testing 1, 5, 10, 15, and 20 cached object fetches per request. All tests are run at 512 concurrency. Conceptually, this is similar to the multiple-queries test except that it uses a caching layer.
Fortunes - In this test, the framework's ORM is used to fetch all rows from a database table containing an unknown number of Unix fortune cookie messages (the table has 12 rows, but the code cannot have foreknowledge of the table's size). An additional fortune cookie message is inserted into the list at runtime and then the list is sorted by the message text. Finally, the list is delivered to the client using a server-side HTML template. The message text must be considered untrusted and properly escaped and the UTF-8 fortune messages must be rendered properly.  Whitespace is optional and may comply with the framework's best practices.
data Updates - This test exercises database writes. Each request is processed by fetching multiple rows from a simple database table, converting the rows to in-memory objects, modifying one attribute of each object in memory, updating each associated row in the database individually, and then serializing the list of objects as a JSON response. The test is run multiple times: testing 1, 5, 10, 15, and 20 updates per request. Note that the number of statements per request is twice the number of updates since each update is paired with one query to fetch the object. All tests are run at 512 concurrency.  The response is analogous to the multiple-query test. 
plain text - In this test, the framework responds with the simplest of responses: a "Hello, World" message rendered as plain text. The size of the response is kept small so that gigabit Ethernet is not the limiting factor for all implementations. HTTP pipelining is enabled and higher client-side concurrency levels are used for this test (see the "Data table" view).

Here, I instead benchmark my django gallery app, using Apache Bench, and so forth.  I guess I’ve been over-achieving…

I have to admit, though, that these benchmarks certainly allow everyone to play.  

431	cherrypy	587	0.0%(0.0%)

Even cherrypy with it’s 587 per second replies with plain-text.  

The tasks seem deceptively (?) simple?  

But looking closer, the data table for each task, gives more details.  For example the plain text is run 4 different times, at 4 different client-side concurrency levels are used…  But the levels are: 256, 1024, 4096, and 16384.   That can’t be the concurrency/thread count??!?!?!??  I can believe 1,000 - 3,000, outrageously high, but believable.  But 16K worth of concurrency/threads?  I doubt that Wikipedia even has to dial it that high?

I have to give them points for providing API latency, and framework overhead….  

	- Benjamin