[Mailman-Developers] HyperKitty MongoDB vs PostgreSQL.

[Barry Warsaw]

On May 25, 2012, at 08:59 AM, Toshio Kuratomi wrote:

>We're using SQLAlchemy i(a different ORM) to abstract the database layer.

At one point the core used SA, but this was many years ago and I had lots of
problems with it.  IIRC it was mostly, when things went south, it was quite
difficult to debug.  I'm sure things have improved since then (and SA is
Python 3 compatible, whereas Storm is not, so... :).

>That means the big three open source solutions (sqlite, postgres, and mysql)
>and proprietary ones like oracle should all be supported.  However, we're
>thinking about using fulltext search indexes which would mean tying it to
>specific backends.  It may be possible to run in a degraded mode (searches
>being slower... but for small to medium lists this is likely fine) so we
>could keep the backend abstraction while running faster on certain
>databases..

That's totally reasonable, and mirrors our policy on JavaScript.

>MongoDB is not a relational database so using an ORM seems like a bit of
>a hack.  People have adapted SQLAlchemy for use with MongoDB but I'm not
>sure what sacrifices they had to make to make that work.

Agreed.

>Running comparisons using postgres was actually a choice dictated by our
>experience with the Core.  In our use, using the sqlite backend quickly
>resulted in database locks that we couldn't figure our way out of.  So that
>effectively left us with postgres as the only db for the core.  If admins
>have to run postgres for the Core then the thinking is that it wouldn't be
>hard for them to also run it for the archiver. I've never had the sqlite
>issue with SQLAlchemy,though,  so we could probably support
>a non-fulltext-optimized search in sqlite without doing much of anything.

This is a serious question, and I know that multiproc sqlite applications are
prone to database lockups.  They're also notoriously difficult to debug, or
even figure out enough to file a reasonable bug.  I think in general it's
because there some path through the code of one of the processes that doesn't
end in a commit or abort, and I know I've fixed some of those.

It would be sad if a smallish Mailman system couldn't be effectively run on
SQLite, just because it's so much easier to set up.  Certainly it's much
better for day-to-day development.  I guess we'll find out as more real
deployments happen.  In any case, I'd still like to officially support
PostgreSQL, and for that matter, MySQL (and derivatives), but I admit to not
testing very often on the former, and no one has contributed support for the
latter yet.  We really need some buildbots. :)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120601/e0140d48/attachment.pgp>