|
1 .. -*- coding: utf-8 -*- |
|
2 |
|
3 =========== |
|
4 Web site. |
|
5 =========== |
|
6 .. contents:: |
|
7 |
|
8 Speeding up web site loading. |
|
9 ============================= |
|
10 |
|
11 http://developer.yahoo.com/performance/rules.html |
|
12 |
|
13 robots.txt. |
|
14 =========== |
|
15 |
|
16 To exclude all robots from the entire server:: |
|
17 |
|
18 User-agent: * |
|
19 Disallow: / |
|
20 |
|
21 To exclude all robots from part of the server:: |
|
22 |
|
23 User-agent: * |
|
24 Disallow: /cgi-bin/ |
|
25 Disallow: /tmp/ |
|
26 Disallow: /junk/ |
|
27 |
|
28 To allow a single robot:: |
|
29 |
|
30 User-agent: Google |
|
31 Disallow: |
|
32 |
|
33 User-agent: * |
|
34 Disallow: / |
|
35 |
|
36 To allow all robots complete access:: |
|
37 |
|
38 User-agent: * |
|
39 Disallow: |
|
40 |
|
41 See: |
|
42 |
|
43 http://www.robotstxt.org/ |
|
44 home page |
|
45 http://www.robotstxt.org/robotstxt.html |
|
46 About /robots.txt |
|
47 http://www.robotstxt.org/faq.html |
|
48 Frequently Asked Questions |
|
49 http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html |
|
50 Improving on Robots Exclusion Protocol |
|
51 |
|
52 Web document structure useage. |
|
53 ============================== |
|
54 |
|
55 http://dev.opera.com/articles/view/mama/ |
|
56 Metadata Analysis and Mining Application |
|
57 |
|
58 Validation. |
|
59 =========== |
|
60 |
|
61 http://validator.w3.org/ |
|
62 |
|
63 Add search to your site. |
|
64 ======================== |
|
65 |
|
66 http://www.google.com/support/customsearch/ |
|
67 Custom Search Help |
|
68 http://help.yahoo.com/l/uk/yahoo/search/basics/basics-13.html |
|
69 Can I add a Yahoo! Search box to my site? |
|
70 |
|
71 Check websites for broken links. |
|
72 ================================ |
|
73 |
|
74 http://linkchecker.sourceforge.net/ |
|
75 linkchecker home page. |
|
76 http://arthurdejong.org/webcheck/ |
|
77 webcheck home page. |