web-site.rst
author Oleksandr Gavenko <gavenkoa@gmail.com>
Mon, 22 Feb 2016 12:41:52 +0200
changeset 1903 901e7394849f
parent 1334 9bf0d5a1f0cf
child 1905 fba288d59662
permissions -rw-r--r--
Decrease intent to increase space usage on mobile.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
1334
9bf0d5a1f0cf Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1331
diff changeset
     1
.. -*- coding: utf-8; -*-
9bf0d5a1f0cf Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1331
diff changeset
     2
.. include:: HEADER.rst
1325
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     3
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     4
===========
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     5
 Web site.
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     6
===========
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     7
.. contents::
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     8
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     9
Speeding up web site loading.
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    10
=============================
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    11
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    12
  http://developer.yahoo.com/performance/rules.html
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    13
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    14
robots.txt.
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    15
===========
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    16
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    17
To exclude all robots from the entire server::
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    18
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    19
  User-agent: *
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    20
  Disallow: /
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    21
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    22
To exclude all robots from part of the server::
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    23
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    24
  User-agent: *
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    25
  Disallow: /cgi-bin/
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    26
  Disallow: /tmp/
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    27
  Disallow: /junk/
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    28
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    29
To allow a single robot::
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    30
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    31
  User-agent: Google
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    32
  Disallow:
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    33
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    34
  User-agent: *
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    35
  Disallow: /
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    36
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    37
To allow all robots complete access::
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    38
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    39
  User-agent: *
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    40
  Disallow:
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    41
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    42
See:
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    43
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    44
  http://www.robotstxt.org/
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    45
                home page
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    46
  http://www.robotstxt.org/robotstxt.html
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    47
                About /robots.txt
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    48
  http://www.robotstxt.org/faq.html
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    49
                Frequently Asked Questions
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    50
  http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    51
                Improving on Robots Exclusion Protocol
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    52
1331
7d93a4940822 Sitemap.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1325
diff changeset
    53
Sitemap.
7d93a4940822 Sitemap.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1325
diff changeset
    54
========
7d93a4940822 Sitemap.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1325
diff changeset
    55
7d93a4940822 Sitemap.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1325
diff changeset
    56
Sitemaps protocol allows a webmaster to inform search engines about URLs on a
7d93a4940822 Sitemap.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1325
diff changeset
    57
website that are available for crawling.
7d93a4940822 Sitemap.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1325
diff changeset
    58
7d93a4940822 Sitemap.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1325
diff changeset
    59
  http://www.sitemaps.org/protocol.html
7d93a4940822 Sitemap.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1325
diff changeset
    60
                Sitemap protocol.
7d93a4940822 Sitemap.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1325
diff changeset
    61
  http://en.wikipedia.org/wiki/Sitemaps
7d93a4940822 Sitemap.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1325
diff changeset
    62
                Wikipedia article.
7d93a4940822 Sitemap.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1325
diff changeset
    63
1325
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    64
Web document structure useage.
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    65
==============================
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    66
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    67
  http://dev.opera.com/articles/view/mama/
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    68
                Metadata Analysis and Mining Application
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    69
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    70
Validation.
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    71
===========
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    72
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    73
  http://validator.w3.org/
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    74
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    75
Add search to your site.
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    76
========================
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    77
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    78
  http://www.google.com/support/customsearch/
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    79
                Custom Search Help
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    80
  http://help.yahoo.com/l/uk/yahoo/search/basics/basics-13.html
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    81
                Can I add a Yahoo! Search box to my site?
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    82
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    83
Check websites for broken links.
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    84
================================
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    85
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    86
  http://linkchecker.sourceforge.net/
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    87
                linkchecker home page.
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    88
  http://arthurdejong.org/webcheck/
ea51f96a6a47 Check websites for broken links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    89
                webcheck home page.