OT: why do web BBS's and blogs get so slow?

[Paul Rubin]

llothar at web.de (Lothar Scholz) writes:
> The problem is that most of them don't use an application server and
> that most of them uses databases very unoptimized, for example PhpBB
> runs about 50 SQL queries per page for medium size thread views.

Ugh!  You mean 50 separate queries, rather than getting a cursor that
you read from 50 times?  I think that could be improved a lot!

> The www.asp-shareware.org has a nice framework that let you view the
> messages in an nntp server. It's amazing fast, because everything is
> hold in memory. Optimizing this by a factor 10 is not difficult.

Is source code available?  I took a quick look there and didn't see
anything about it.  Is the idea of NNTP that you leverage off the
existing NNTP servers that are written in C and already designed to
handle very heavy traffic?

> The problem may come with slashdot or for example the german website
> "www.heise.de" which gets upto 100 requests per second (it's among the
> top ten in germany). They also use a NNTP backend. For sites like this
> you need multiple server and then its not so easy to maintain cache
> coherence. But there are only very few websites that have this
> traffic.

Are multiple servers really needed?  Is a multi-CPU box (4-way Athlon
MP or whatever) with shared memory not enough?  100 requests per second
doesn't sound like all THAT much.

For multiple servers, I've had the idea for a while of using a
Beowulf-type architecture, i.e. using fast LAN connections and special
kernel modules (I think that means MP directly over ethernet or
Myrinet, bypassing the TCP/IP stack) between the servers to get the
next best thing to shared memory.  Another way is just process all
updates (new posts) through a single server.  Every few seconds the
other servers would poll the update server (through a fast LAN) for
new posts to cache.  I think it would be hard to have new posts coming
in fast enough to swamp a server.  Most requests will be reads, not
new posts.

> At the moment there seems to be absolute no evolution in the area of
> BBS's. PHPBB has set a standart an it seems that nobody is able to
> think about more/other functionality. Maybe if more and more people
> get virtual servers and have more control over what they can install
> then it may become better. We shall see.

I think there's new development happening in blogging software, which
is really pretty similar to BBS software, in terms of supporting
comment threads, user diaries, etc.  I'm thinking of Scoop and so forth.