[Mailman-Users] problem in archiving the large mbox file

[Brad Knowles]

On 7/18/07, alpesh gajbe wrote:

>  I  am not sure whether this mail belongs to the users or developers list so
>  I am sending to both. Apologies.

This sounds to me more like a question for -users.

>  I have a large 1.2 GB mbox file which i need to archive, the messages in
>  mbox typically have jpeg image attachments
>  whose approximate image sizes is 4 KB.

4KB JPEGs?  That's pretty small for a JPEG.  You have over 300,000 of 
these messages per mailbox file?

>  when i run arch script in /usr/lib/mailman/bin/ on my ubuntu 7.04 I get the
>  following error
>
>  *File "/usr/lib/python2.5/os.py", line 172, in makedirs mkdir(name, mode)
>  OSError: [Errno 31] Too many links:

That sounds like a directory problem, and not a file problem.  With 
as large a mailbox file as you're talking about, and as many messages 
as you're talking about, you probably need to break your archives 
more frequently than once a month.  Try breaking them weekly instead. 
Alternatively, try changing your underlying filesystem to one that 
supports large numbers of files in a single directory, and preferably 
does so with an internal hashed directory/inode structure (e.g., XFS).

Also keep in mind that you'll want to make sure that your OS is built 
to support large files (files over 2GB).  Many Linux distributions 
are not built out-of-the-box to support large files.

Finally, the version of Python that is recommended for use with the 
latest release version of Mailman, is Python 2.4.x for Mailman 2.1.9, 
for whatever the most recent version of Python 2.4.x (currently 
2.4.3, I believe).  More recent versions of Python may or may not 
work with Mailman 2.1.9, and almost certainly will not work correctly 
with earlier versions of Mailman.

>  *My basic objective is to archive 250GB of mails every month for two years .
>  The size of which would exceed 7 Tera Bytes approx. Would this be a feasible
>  option using mailman.(out of curiosity !!)

That should be possible.  We have 4GB worth of archives for 
python-list at python.org going back to 1999, and I'm pretty sure those 
messages are text-only or text+code fragments.

>  What has been the largest mail archive volume anyone has ever deployed into
>  mailman ? (out of curiosity !!)

We've got some information in the FAQ Wizard about large mailing 
lists servers with regards to numbers of subscribers or numbers of 
messages, but I don't know that anyone has tried to gather any 
specific information with regards to large sizes of archives.

The largest archives I am personally aware of are the ones we have 
for python-list, but I'd love to hear any information that anyone 
else has about any others.

-- 
Brad Knowles <brad at shub-internet.org>, Consultant & Author
LinkedIn Profile: <http://tinyurl.com/y8kpxu>
Slides from Invited Talks: <http://tinyurl.com/tj6q4>

09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0