Check websites for broken links.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/web-site.rst Sun Jul 08 12:29:56 2012 +0300
@@ -0,0 +1,77 @@
+.. -*- coding: utf-8 -*-
+
+===========
+ Web site.
+===========
+.. contents::
+
+Speeding up web site loading.
+=============================
+
+ http://developer.yahoo.com/performance/rules.html
+
+robots.txt.
+===========
+
+To exclude all robots from the entire server::
+
+ User-agent: *
+ Disallow: /
+
+To exclude all robots from part of the server::
+
+ User-agent: *
+ Disallow: /cgi-bin/
+ Disallow: /tmp/
+ Disallow: /junk/
+
+To allow a single robot::
+
+ User-agent: Google
+ Disallow:
+
+ User-agent: *
+ Disallow: /
+
+To allow all robots complete access::
+
+ User-agent: *
+ Disallow:
+
+See:
+
+ http://www.robotstxt.org/
+ home page
+ http://www.robotstxt.org/robotstxt.html
+ About /robots.txt
+ http://www.robotstxt.org/faq.html
+ Frequently Asked Questions
+ http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
+ Improving on Robots Exclusion Protocol
+
+Web document structure useage.
+==============================
+
+ http://dev.opera.com/articles/view/mama/
+ Metadata Analysis and Mining Application
+
+Validation.
+===========
+
+ http://validator.w3.org/
+
+Add search to your site.
+========================
+
+ http://www.google.com/support/customsearch/
+ Custom Search Help
+ http://help.yahoo.com/l/uk/yahoo/search/basics/basics-13.html
+ Can I add a Yahoo! Search box to my site?
+
+Check websites for broken links.
+================================
+
+ http://linkchecker.sourceforge.net/
+ linkchecker home page.
+ http://arthurdejong.org/webcheck/
+ webcheck home page.