Add Wikipedia article.
--- a/web-site.rst Sun Jan 29 21:56:25 2017 +0200
+++ b/web-site.rst Sun Jan 29 22:04:52 2017 +0200
@@ -41,14 +41,17 @@
See:
- http://www.robotstxt.org/
- home page
- http://www.robotstxt.org/robotstxt.html
- About /robots.txt
- http://www.robotstxt.org/faq.html
- Frequently Asked Questions
- http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
- Improving on Robots Exclusion Protocol
+http://www.robotstxt.org/
+ Page provides description for robots.txt usual practice and discussion about
+ possible standardization efforts.
+http://www.robotstxt.org/robotstxt.html
+ About /robots.txt
+http://www.robotstxt.org/faq.html
+ Frequently Asked Questions.
+https://en.wikipedia.org/wiki/Robots_exclusion_standard
+ Wikipedia article on robots.txt.
+http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
+ Improving on Robots Exclusion Protocol.
Sitemap
=======