[Mailman-Users] Help getting error Too many open files in system when sending

[Greg Ward]

On 20 September 2001, Rick Backer said:
> Too many open files in system
> 
> I get this error after sending out a email to are list. The size of the list is 149271. Does someone know what is happening to create this error
> 
> I am running mailman ver 2.0.5 on a Ghz amd with 512 mem, using OS linux,GNU  kernel 2.2.19 ver  and MTA of Exim ver 3.12-10

We had a similar problem on a similar system (only difference is that
we're running a 2.4 kernel).  From Documentation/filesystems/proc.txt in
the Linux kernel source tree:

"""
The kernel  allocates file handles dynamically, but doesn't free them again at
this time.

The value  in  file-max  denotes  the  maximum number of file handles that the
Linux kernel will allocate. When you get a lot of error messages about running
out of  file handles, you might want to raise this limit. The default value is
4096. To change it, just write the new number into the file:

  # cat /proc/sys/fs/file-max 
  4096 
  # echo 8192 > /proc/sys/fs/file-max 
  # cat /proc/sys/fs/file-max 
  8192 


This method  of  revision  is  useful  for  all customizable parameters of the
kernel - simply echo the new value to the corresponding file.

The three  values  in file-nr denote the number of allocated file handles, the
number of  used file handles, and the maximum number of file handles. When the
allocated file  handles  come close to the maximum, but the number of actually
used ones  is  far  behind,  you've  encountered  a peak in your usage of file
handles and you don't need to increase the maximum.
"""

On a machine as powerful as yours, you can probably have many tens of
thousands of open files.  On our server (also with 512 MB of RAM), we
do
  echo 24576 > /proc/sys/fs/file-max
at boot time, and our "Too many open files" problem disappeared.
However, our largest list is only ~1600 subscribers.  You're two orders
of magntitude ahead of that -- wow.

Another thing you forgot to mention: how Mailman feeds messages to Exim.
I'll assume you do the standard thing and set DELIVERY_MODULE to
"SMTPDirect".

What is your setting of SMTP_MAX_RCPTS, BTW?  This is a very important
parameter that controls how many recipients Mailman asks Exim (or
whatever MTA you're using) to handle at a time.  A higher number means
less parallelism, but fewer messages in the queue.  We turned it down
from 500 (Mailman's default) to 100 to increase parallelism.  For a list
as large as yours, I'm not sure what the optimal setting is.  I hope you
have a big, fast disk for your Exim queue and logs.

        Greg
-- 
Greg Ward - software developer                gward at mems-exchange.org
MEMS Exchange                            http://www.mems-exchange.org