+
−-*- mode: outline; coding: utf-8 -*-
+
−
+
−* Speeding up web site loading.
+
−
+
−  http://developer.yahoo.com/performance/rules.html
+
−
+
−* robots.txt.
+
−
+
−To exclude all robots from the entire server
+
−
+
−  User-agent: *
+
−  Disallow: /
+
−
+
−To exclude all robots from part of the server:
+
−
+
−  User-agent: *
+
−  Disallow: /cgi-bin/
+
−  Disallow: /tmp/
+
−  Disallow: /junk/
+
−
+
−To allow a single robot:
+
−
+
−  User-agent: Google
+
−  Disallow:
+
−
+
−  User-agent: *
+
−  Disallow: /
+
−
+
−To allow all robots complete access:
+
−
+
−  User-agent: *
+
−  Disallow:
+
−
+
−  http://www.robotstxt.org/
+
−                home page
+
−  http://www.robotstxt.org/robotstxt.html
+
−                About /robots.txt
+
−  http://www.robotstxt.org/faq.html
+
−                Frequently Asked Questions
+
−  http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
+
−                Improving on Robots Exclusion Protocol
+
−