[Mailman-Users] Using robots.txt

[Steff Watkins]

> -----Original Message-----
> From: mailman-users-bounces+s.watkins=nhm.ac.uk at python.org 
> [mailto:mailman-users-bounces+s.watkins=nhm.ac.uk at python.org] 
> On Behalf Of Max Pyziur
> Sent: 09 November 2009 01:09
> To: mailman-users at python.org
> Subject: [Mailman-Users] Using robots.txt
> 
> Greetings
> 
> Mailman's email lists are visible search engine spiders from 
> http://www.somedomain.com/pipermail/emaillistname/etc
> 
> However, /pipermail is an alias of /var/mailman/archives/public/
> 
> per mailman.conf
> 
> I've tried placing a basic robots.txt file at 
> /var/mailman/archives/public/ and set permissions to 644. 
> However, my lists still get spidered.
> 
> Any suggestions on where to place the robots.txt file to 
> prevent spidering?
> 
> Thanks!
> 
> Max Pyziur
> pyz at brama.com

Hello Max,

 AFAIK your robots.txt file should be in the TOP level directory of your
website, so that it is browseable via
http://www.somedomain.com/robots.txt . This is the default location for
it and 'good' spiders will look for it there.

It should contain the allow/deny details for the whole of your website
and in your case would look something like this:

----
User-agent: *
Disallow: /pipermail/
----

.. Which informs all browsers to disallow any URL starting (containing?)
the phrase "/pipermail/".

Give that a whirl and see how it does.

Regards,
Steff Watkins
=======