| author | Oleksandr Gavenko <gavenkoa@gmail.com> |
| Mon, 22 Feb 2016 13:52:03 +0200 | |
| changeset 1912 | 8b81a8f0f692 |
| parent 1905 | fba288d59662 |
| child 2072 | 72921b56230b |
| permissions | -rw-r--r-- |
|
1334
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
1331
diff
changeset
|
1 |
.. -*- coding: utf-8; -*- |
|
1325
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
2 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
3 |
=========== |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
4 |
Web site. |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
5 |
=========== |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
6 |
.. contents:: |
|
1905
fba288d59662
Include only local subsections into TOC. This prevent duplication of
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
1334
diff
changeset
|
7 |
:local: |
|
1325
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
8 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
9 |
Speeding up web site loading. |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
10 |
============================= |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
11 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
12 |
http://developer.yahoo.com/performance/rules.html |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
13 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
14 |
robots.txt. |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
15 |
=========== |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
16 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
17 |
To exclude all robots from the entire server:: |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
18 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
19 |
User-agent: * |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
20 |
Disallow: / |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
21 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
22 |
To exclude all robots from part of the server:: |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
23 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
24 |
User-agent: * |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
25 |
Disallow: /cgi-bin/ |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
26 |
Disallow: /tmp/ |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
27 |
Disallow: /junk/ |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
28 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
29 |
To allow a single robot:: |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
30 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
31 |
User-agent: Google |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
32 |
Disallow: |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
33 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
34 |
User-agent: * |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
35 |
Disallow: / |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
36 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
37 |
To allow all robots complete access:: |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
38 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
39 |
User-agent: * |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
40 |
Disallow: |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
41 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
42 |
See: |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
43 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
44 |
http://www.robotstxt.org/ |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
45 |
home page |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
46 |
http://www.robotstxt.org/robotstxt.html |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
47 |
About /robots.txt |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
48 |
http://www.robotstxt.org/faq.html |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
49 |
Frequently Asked Questions |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
50 |
http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
51 |
Improving on Robots Exclusion Protocol |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
52 |
|
| 1331 | 53 |
Sitemap. |
54 |
======== |
|
55 |
||
56 |
Sitemaps protocol allows a webmaster to inform search engines about URLs on a |
|
57 |
website that are available for crawling. |
|
58 |
||
59 |
http://www.sitemaps.org/protocol.html |
|
60 |
Sitemap protocol. |
|
61 |
http://en.wikipedia.org/wiki/Sitemaps |
|
62 |
Wikipedia article. |
|
63 |
||
|
1325
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
64 |
Web document structure useage. |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
65 |
============================== |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
66 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
67 |
http://dev.opera.com/articles/view/mama/ |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
68 |
Metadata Analysis and Mining Application |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
69 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
70 |
Validation. |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
71 |
=========== |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
72 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
73 |
http://validator.w3.org/ |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
74 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
75 |
Add search to your site. |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
76 |
======================== |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
77 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
78 |
http://www.google.com/support/customsearch/ |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
79 |
Custom Search Help |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
80 |
http://help.yahoo.com/l/uk/yahoo/search/basics/basics-13.html |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
81 |
Can I add a Yahoo! Search box to my site? |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
82 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
83 |
Check websites for broken links. |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
84 |
================================ |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
85 |
|
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
86 |
http://linkchecker.sourceforge.net/ |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
87 |
linkchecker home page. |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
88 |
http://arthurdejong.org/webcheck/ |
|
ea51f96a6a47
Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
89 |
webcheck home page. |