[Mailman-Developers] Problems if shunting fails

[Barry Warsaw]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Mark,

I'm sorry I haven't been able to respond to this thread before now,  
but I've been traveling and PyCon'ing and either haven't had time or  
have had spotty access to email.

On Feb 22, 2007, at 5:24 PM, Mark Sapiro wrote:

> The question I would like to discuss, is what is the best way to
> preserve the queue entry for analysis. The patch just renames the .bak
> file to .psv and leaves it in the original queue. This could
> potentially over time accumulate a lot of .psv files in the 'in' or
> other queues and impact processing.
>
> We can't shunt the entry in the normal way because in some cases at
> least, shunting has already thrown an exception. I can think of three
> things to do.
>
> 1) Just rename the entry .psv and leave it in the original queue.
> 2) Rename it .psv and attempt to move it to the shunt queue.
> 3) Rename it .psv and attempt to move it to the bad queue.

I'm pretty sure we don't want to leave the .psv file in the original  
queue.  I agree that over time, we'll just end up filling up the  
queue directory with files we'll never process, wasting time listing  
them, or worse.  A major reason for having the shunt queue in the  
first place was to not clutter up our processing queues with messages  
we couldn't do anything about.  Of course, culling our shunt queue is  
another issues. :/

OTOH, if we're going to move the offending message to the shunt  
queue, then there's not much point in keeping the .psv extension.  We  
pretty much always know that if the file is in shunt, it's bad, so  
maybe we just rename the offender to shunt/blah.txt (assuming it's an  
unparsed text file).

The other thing to consider is adding a configuration variable that  
let's us limit the size of the files we'll handle.  I can't imagine  
any scenario under which we'd want to (let alone be able to) handle a  
message of a half terabyte.  Heck, you have to have a pretty  
misconfigured mail system to even allow such a message to get to  
Mailman, IMO.  I think Postfix for example has a 10MB default size  
limit, and even cranking that up by a factor of 10 should allow most  
legitimate mail to go through.

So, if we added a size limit configuration variable, we'd have to  
stat the file (os.path.getsize()) and just os.rename() the file to  
shunt (with a log message) when we see a file over that size limit.

What do you think?
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBReJo43EjvBPtnXfVAQKw8QP/Qm2twCuIkyQV4zXt1+Ei9PKoBpdBSiK9
G/LgwlUz8OCBRkonb+m4NTyMmZJTQUjqcLICvgFzZ9d+dxf5ekVA8KoXM1iJjjnE
TQqS88CAgUYcWixkIfS5H1l0lVSKSGsUn8eUAXwmoVZVn3NaqCzhTo1wGV56KATc
VesBGYhpGkQ=
=juvu
-----END PGP SIGNATURE-----