# HG changeset patch # User Oleksandr Gavenko # Date 1485720292 -7200 # Node ID 1abbd5a7db80f943d5b73b56b5580c0cff6634bf # Parent 72921b56230bd866feb237f3e224718eba7f78f6 Add Wikipedia article. diff -r 72921b56230b -r 1abbd5a7db80 web-site.rst --- a/web-site.rst Sun Jan 29 21:56:25 2017 +0200 +++ b/web-site.rst Sun Jan 29 22:04:52 2017 +0200 @@ -41,14 +41,17 @@ See: - http://www.robotstxt.org/ - home page - http://www.robotstxt.org/robotstxt.html - About /robots.txt - http://www.robotstxt.org/faq.html - Frequently Asked Questions - http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html - Improving on Robots Exclusion Protocol +http://www.robotstxt.org/ + Page provides description for robots.txt usual practice and discussion about + possible standardization efforts. +http://www.robotstxt.org/robotstxt.html + About /robots.txt +http://www.robotstxt.org/faq.html + Frequently Asked Questions. +https://en.wikipedia.org/wiki/Robots_exclusion_standard + Wikipedia article on robots.txt. +http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html + Improving on Robots Exclusion Protocol. Sitemap =======