.. -*- coding: utf-8 -*-

===========
 Web site.
===========
.. contents::

Speeding up web site loading.
=============================

  http://developer.yahoo.com/performance/rules.html

robots.txt.
===========

To exclude all robots from the entire server::

  User-agent: *
  Disallow: /

To exclude all robots from part of the server::

  User-agent: *
  Disallow: /cgi-bin/
  Disallow: /tmp/
  Disallow: /junk/

To allow a single robot::

  User-agent: Google
  Disallow:

  User-agent: *
  Disallow: /

To allow all robots complete access::

  User-agent: *
  Disallow:

See:

  http://www.robotstxt.org/
                home page
  http://www.robotstxt.org/robotstxt.html
                About /robots.txt
  http://www.robotstxt.org/faq.html
                Frequently Asked Questions
  http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
                Improving on Robots Exclusion Protocol

Web document structure useage.
==============================

  http://dev.opera.com/articles/view/mama/
                Metadata Analysis and Mining Application

Validation.
===========

  http://validator.w3.org/

Add search to your site.
========================

  http://www.google.com/support/customsearch/
                Custom Search Help
  http://help.yahoo.com/l/uk/yahoo/search/basics/basics-13.html
                Can I add a Yahoo! Search box to my site?

Check websites for broken links.
================================

  http://linkchecker.sourceforge.net/
                linkchecker home page.
  http://arthurdejong.org/webcheck/
                webcheck home page.