[Mailman-Developers] Proposed: remove address-obfuscation codefrom Mailman 3

[Mark Sapiro]

Barry Warsaw wrote:
>
>On Aug 31, 2009, at 1:15 PM, C Nulk wrote:
>
>> As for using robots.txt, hmm, it is not the legitimate search  
>> engines I
>> care about, it is the search engines/crawlers that do not respect my
>> robots.txt file that I care about.  If I had an effective way to
>> consistently identify those non-legitimate crawlers, I would add  
>> what I
>> needed to drop them into my firewall as I recognized them.
>
>Agreed.


The point in the original post about robots.txt was that if you think
obfuscation is undesirable and don't do it, but you get complaints
from people who find their unobfuscated addresses on your pages via
legitimate search engines, you can use robots.txt to keep the search
engines out.

However, robots.txt is not completely effective in this. You can use it
to prevent Google from crawling your site or portions thereof, but it
won't prevent Google from indexing your pages that it finds via
external links. To prevent this, you need a <meta name="robots"
content="noindex"> tag on the pages themselves.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan