msn, say hello to robots.txt
Posted by phil on 22 Nov 2004 at 10:43 am | Tagged as: Uncategorized
Hi, msnbot. Nice to see you here. Let me introduce you to a little friend of mine that I like to call the ‘Robots Exclusion Protocol’, or ‘robots.txt’ for short:
User-agent: *<br />
Disallow: *.cgi<br />
<br />
User-agent: *msn*<br />
Disallow: *<br />
The idea is that you do as I say, but hey, if you’re not cool with that, screw you.