Include only local subsections into TOC. This prevent duplication of
TOC when build single page HTML document. Also this make unnecessary CSS hack
to hide document title as top level section.
.. -*- coding: utf-8; -*-
.. include:: HEADER.rst
===========
Web site.
===========
.. contents::
:local:
Speeding up web site loading.
=============================
http://developer.yahoo.com/performance/rules.html
robots.txt.
===========
To exclude all robots from the entire server::
User-agent: *
Disallow: /
To exclude all robots from part of the server::
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
To allow a single robot::
User-agent: Google
Disallow:
User-agent: *
Disallow: /
To allow all robots complete access::
User-agent: *
Disallow:
See:
http://www.robotstxt.org/
home page
http://www.robotstxt.org/robotstxt.html
About /robots.txt
http://www.robotstxt.org/faq.html
Frequently Asked Questions
http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
Improving on Robots Exclusion Protocol
Sitemap.
========
Sitemaps protocol allows a webmaster to inform search engines about URLs on a
website that are available for crawling.
http://www.sitemaps.org/protocol.html
Sitemap protocol.
http://en.wikipedia.org/wiki/Sitemaps
Wikipedia article.
Web document structure useage.
==============================
http://dev.opera.com/articles/view/mama/
Metadata Analysis and Mining Application
Validation.
===========
http://validator.w3.org/
Add search to your site.
========================
http://www.google.com/support/customsearch/
Custom Search Help
http://help.yahoo.com/l/uk/yahoo/search/basics/basics-13.html
Can I add a Yahoo! Search box to my site?
Check websites for broken links.
================================
http://linkchecker.sourceforge.net/
linkchecker home page.
http://arthurdejong.org/webcheck/
webcheck home page.