-*- mode: outline; coding: utf-8 -*-

* Speeding up web site loading.

  http://developer.yahoo.com/performance/rules.html

* robots.txt.

To exclude all robots from the entire server

  User-agent: *
  Disallow: /

To exclude all robots from part of the server:

  User-agent: *
  Disallow: /cgi-bin/
  Disallow: /tmp/
  Disallow: /junk/

To allow a single robot:

  User-agent: Google
  Disallow:

  User-agent: *
  Disallow: /

To allow all robots complete access:

  User-agent: *
  Disallow:

  http://www.robotstxt.org/
                home page
  http://www.robotstxt.org/robotstxt.html
                About /robots.txt