| changeset 708 | 83ded9492a61 |
| parent 707 | fae2370e94fd |
| child 740 | 8189b7ad02d9 |
| 707:fae2370e94fd | 708:83ded9492a61 |
|---|---|
33 |
33 |
34 http://www.robotstxt.org/ |
34 http://www.robotstxt.org/ |
35 home page |
35 home page |
36 http://www.robotstxt.org/robotstxt.html |
36 http://www.robotstxt.org/robotstxt.html |
37 About /robots.txt |
37 About /robots.txt |
38 http://www.robotstxt.org/faq.html |
|
39 Frequently Asked Questions |
|
40 http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html |
|
41 Improving on Robots Exclusion Protocol |
|
38 |
42 |
39 |