[Mailman-Users] How to block spiders?

[Skip Montanaro]

    Paul> Normally with an apache web server you can just place a robots.txt
    Paul> file in the root of the documents directory.  Not sure how one
    Paul> might be able to do this with mailman.

    Paul> When you take a look at the archives for a list, one sees

    Paul>    http://server-name/pipermail/list-name/

    Paul> Since there isn't a pipermail directory, one can't just place a
    Paul> robots.txt file in that directory.  

The robots.txt file only belongs in the server root.  Placing it lower in
the hierarchy isn't supported, and I doubt any spiders/crawlers would look
for it lower down.  Instead, add a Disallow: rule which references
/pipermail:

    User-agent: *
    Disallow: /pipermail/

-- 
Skip Montanaro
Got gigs? http://www.musi-cal.com/submit.html
Got spam? http://spambayes.sf.net/
skip at pobox.com