+
−.. -*- coding: utf-8; -*-
+
−.. include:: HEADER.rst
+
−
+
−===========
+
− Web site.
+
−===========
+
−.. contents::
+
−
+
−Speeding up web site loading.
+
−=============================
+
−
+
−  http://developer.yahoo.com/performance/rules.html
+
−
+
−robots.txt.
+
−===========
+
−
+
−To exclude all robots from the entire server::
+
−
+
−  User-agent: *
+
−  Disallow: /
+
−
+
−To exclude all robots from part of the server::
+
−
+
−  User-agent: *
+
−  Disallow: /cgi-bin/
+
−  Disallow: /tmp/
+
−  Disallow: /junk/
+
−
+
−To allow a single robot::
+
−
+
−  User-agent: Google
+
−  Disallow:
+
−
+
−  User-agent: *
+
−  Disallow: /
+
−
+
−To allow all robots complete access::
+
−
+
−  User-agent: *
+
−  Disallow:
+
−
+
−See:
+
−
+
−  http://www.robotstxt.org/
+
−                home page
+
−  http://www.robotstxt.org/robotstxt.html
+
−                About /robots.txt
+
−  http://www.robotstxt.org/faq.html
+
−                Frequently Asked Questions
+
−  http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
+
−                Improving on Robots Exclusion Protocol
+
−
+
−Sitemap.
+
−========
+
−
+
−Sitemaps protocol allows a webmaster to inform search engines about URLs on a
+
−website that are available for crawling.
+
−
+
−  http://www.sitemaps.org/protocol.html
+
−                Sitemap protocol.
+
−  http://en.wikipedia.org/wiki/Sitemaps
+
−                Wikipedia article.
+
−
+
−Web document structure useage.
+
−==============================
+
−
+
−  http://dev.opera.com/articles/view/mama/
+
−                Metadata Analysis and Mining Application
+
−
+
−Validation.
+
−===========
+
−
+
−  http://validator.w3.org/
+
−
+
−Add search to your site.
+
−========================
+
−
+
−  http://www.google.com/support/customsearch/
+
−                Custom Search Help
+
−  http://help.yahoo.com/l/uk/yahoo/search/basics/basics-13.html
+
−                Can I add a Yahoo! Search box to my site?
+
−
+
−Check websites for broken links.
+
−================================
+
−
+
−  http://linkchecker.sourceforge.net/
+
−                linkchecker home page.
+
−  http://arthurdejong.org/webcheck/
+
−                webcheck home page.