1 -*- mode: outline; coding: utf-8 -*-

     2 

     3 * Speeding up web site loading.

     4 

     5   http://developer.yahoo.com/performance/rules.html

     6 

     7 * robots.txt.

     8 

     9 To exclude all robots from the entire server

    10 

    11   User-agent: *

    12   Disallow: /

    13 

    14 To exclude all robots from part of the server:

    15 

    16   User-agent: *

    17   Disallow: /cgi-bin/

    18   Disallow: /tmp/

    19   Disallow: /junk/

    20 

    21 To allow a single robot:

    22 

    23   User-agent: Google

    24   Disallow:

    25 

    26   User-agent: *

    27   Disallow: /

    28 

    29 To allow all robots complete access:

    30 

    31   User-agent: *

    32   Disallow:

    33 

    34   http://www.robotstxt.org/

    35                 home page

    36   http://www.robotstxt.org/robotstxt.html

    37                 About /robots.txt

    38   http://www.robotstxt.org/faq.html

    39                 Frequently Asked Questions

    40   http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html

    41                 Improving on Robots Exclusion Protocol

    42 

    43 * Web document structure useage.

    44 

    45   http://dev.opera.com/articles/view/mama/

    46                 Metadata Analysis and Mining Application

    47 

    48 * Validation.

    49 

    50   http://validator.w3.org/

    51 

    52 * Add search to your site.

    53 

    54   http://www.google.com/support/customsearch/

    55                 Custom Search Help

    56   http://help.yahoo.com/l/uk/yahoo/search/basics/basics-13.html

    57                 Can I add a Yahoo! Search box to my site?

    58