7 * robots.txt.

     8 

     9 To exclude all robots from the entire server

    10 

    11   User-agent: *

    12   Disallow: /

    13 

    14 To exclude all robots from part of the server:

    15 

    16   User-agent: *

    17   Disallow: /cgi-bin/

    18   Disallow: /tmp/

    19   Disallow: /junk/

    20 

    21 To allow a single robot:

    22 

    23   User-agent: Google

    24   Disallow:

    25 

    26   User-agent: *

    27   Disallow: /

    28 

    29 To allow all robots complete access:

    30 

    31   User-agent: *

    32   Disallow:

    33 

    34   http://www.robotstxt.org/

    35                 home page

    36   http://www.robotstxt.org/robotstxt.html

    37                 About /robots.txt

    38 

    39