[Mailman-Developers] archiving size issue

[Barry Warsaw]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 7 Jun 2006 21:53:41 +0200
Fil <fil at rezo.net> wrote:

> my server's drive is saturated now and my problem #1 is with the
> mailing-list archives. I have a total of 5GB of archives in .mbox
> format... which make a whopping 12GB when you add the html-generated
> archives.
> 
> This makes me think that we might have room for improvement here...
> either to "archive on another computer" (using ssh maybe?), or to
> generate the archive HTML pages on the fly (with proper caching of
> the thread structure of course). It would also help a lot if the
> Message-Id were part of the url, I think. I'm not looking for a
> solution, but hints would already be great.

Ultimately, I think generating the archive html on the fly is the right
way to go.  That approach provides so many possible benefits at the
mere <wink> cost of cpu time.  When the original archive system was
designed, that cost was too high, and thus we have the static page
rendering approach.  But today I think the tradeoff tips in the other
direction.

Also, the mbox file is really only needed these days in order to regen
your archive, but that's fraught with pain anyway because there's no
guarantee (in fact a pretty good anti-guarantee) that your message urls
will be the same after regeneration.  So yes, a predictable and
reproducible message url is part of the solution.

As I always say when this topic comes up, I'd love to have some
motivated developers come forward to help with an archiver redesign, so
if you're interested in doing work here, let me know!

- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iQCVAwUBRIcxw3EjvBPtnXfVAQLdzAQAuLrOQwkiAo0SN3YwmzZ7K/8cocvxIf05
iZNJ5zq7koIjjcF4cSR/cxbVey543NIz+SBR7HeG4gZ6C7ZrZ7SSgXo+c7o+d2BL
RA5B347arj3IOmFlW64STotGE4dKzBPdDRMTK/Px4Z8g22Hs6ntntGv78uQ2wDm3
IerVcP7TMuY=
=DoDP
-----END PGP SIGNATURE-----