# HG changeset patch
# User Oleksandr Gavenko <gavenkoa@gmail.com>
# Date 1341739796 -10800
# Node ID ea51f96a6a473ba8cd1b4ece75879aa1437f2edb
# Parent  05861a2b18d8977ac1efa86571bae7ceb2772a47
Check websites for broken links.

diff -r 05861a2b18d8 -r ea51f96a6a47 web-site.rst
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/web-site.rst	Sun Jul 08 12:29:56 2012 +0300
@@ -0,0 +1,77 @@
+.. -*- coding: utf-8 -*-
+
+===========
+ Web site.
+===========
+.. contents::
+
+Speeding up web site loading.
+=============================
+
+  http://developer.yahoo.com/performance/rules.html
+
+robots.txt.
+===========
+
+To exclude all robots from the entire server::
+
+  User-agent: *
+  Disallow: /
+
+To exclude all robots from part of the server::
+
+  User-agent: *
+  Disallow: /cgi-bin/
+  Disallow: /tmp/
+  Disallow: /junk/
+
+To allow a single robot::
+
+  User-agent: Google
+  Disallow:
+
+  User-agent: *
+  Disallow: /
+
+To allow all robots complete access::
+
+  User-agent: *
+  Disallow:
+
+See:
+
+  http://www.robotstxt.org/
+                home page
+  http://www.robotstxt.org/robotstxt.html
+                About /robots.txt
+  http://www.robotstxt.org/faq.html
+                Frequently Asked Questions
+  http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
+                Improving on Robots Exclusion Protocol
+
+Web document structure useage.
+==============================
+
+  http://dev.opera.com/articles/view/mama/
+                Metadata Analysis and Mining Application
+
+Validation.
+===========
+
+  http://validator.w3.org/
+
+Add search to your site.
+========================
+
+  http://www.google.com/support/customsearch/
+                Custom Search Help
+  http://help.yahoo.com/l/uk/yahoo/search/basics/basics-13.html
+                Can I add a Yahoo! Search box to my site?
+
+Check websites for broken links.
+================================
+
+  http://linkchecker.sourceforge.net/
+                linkchecker home page.
+  http://arthurdejong.org/webcheck/
+                webcheck home page.