tips: web-site.rst@7d93a4940822 (annotated)

1325 ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	1	.. -- coding: utf-8 --
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	2
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	3	===========
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	4	Web site.
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	5	===========
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	6	.. contents::
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	7
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	8	Speeding up web site loading.
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	9	=============================
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	10
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	11	http://developer.yahoo.com/performance/rules.html
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	12
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	13	robots.txt.
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	14	===========
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	15
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	16	To exclude all robots from the entire server::
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	17
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	18	User-agent: *
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	19	Disallow: /
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	20
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	21	To exclude all robots from part of the server::
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	22
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	23	User-agent: *
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	24	Disallow: /cgi-bin/
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	25	Disallow: /tmp/
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	26	Disallow: /junk/
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	27
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	28	To allow a single robot::
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	29
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	30	User-agent: Google
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	31	Disallow:
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	32
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	33	User-agent: *
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	34	Disallow: /
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	35
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	36	To allow all robots complete access::
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	37
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	38	User-agent: *
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	39	Disallow:
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	40
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	41	See:
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	42
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	43	http://www.robotstxt.org/
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	44	home page
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	45	http://www.robotstxt.org/robotstxt.html
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	46	About /robots.txt
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	47	http://www.robotstxt.org/faq.html
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	48	Frequently Asked Questions
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	49	http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	50	Improving on Robots Exclusion Protocol
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	51
1331 7d93a4940822 Sitemap. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 1325 diff changeset	52	Sitemap.
7d93a4940822 Sitemap. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 1325 diff changeset	53	========
7d93a4940822 Sitemap. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 1325 diff changeset	54
7d93a4940822 Sitemap. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 1325 diff changeset	55	Sitemaps protocol allows a webmaster to inform search engines about URLs on a
7d93a4940822 Sitemap. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 1325 diff changeset	56	website that are available for crawling.
7d93a4940822 Sitemap. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 1325 diff changeset	57
7d93a4940822 Sitemap. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 1325 diff changeset	58	http://www.sitemaps.org/protocol.html
7d93a4940822 Sitemap. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 1325 diff changeset	59	Sitemap protocol.
7d93a4940822 Sitemap. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 1325 diff changeset	60	http://en.wikipedia.org/wiki/Sitemaps
7d93a4940822 Sitemap. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 1325 diff changeset	61	Wikipedia article.
7d93a4940822 Sitemap. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 1325 diff changeset	62
1325 ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	63	Web document structure useage.
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	64	==============================
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	65
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	66	http://dev.opera.com/articles/view/mama/
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	67	Metadata Analysis and Mining Application
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	68
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	69	Validation.
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	70	===========
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	71
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	72	http://validator.w3.org/
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	73
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	74	Add search to your site.
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	75	========================
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	76
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	77	http://www.google.com/support/customsearch/
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	78	Custom Search Help
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	79	http://help.yahoo.com/l/uk/yahoo/search/basics/basics-13.html
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	80	Can I add a Yahoo! Search box to my site?
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	81
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	82	Check websites for broken links.
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	83	================================
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	84
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	85	http://linkchecker.sourceforge.net/
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	86	linkchecker home page.
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	87	http://arthurdejong.org/webcheck/
ea51f96a6a47 Check websites for broken links. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	88	webcheck home page.

author	Oleksandr Gavenko <gavenkoa@gmail.com>
	Thu, 12 Jul 2012 22:16:10 +0300
changeset 1331	7d93a4940822
parent 1325	ea51f96a6a47
child 1334	9bf0d5a1f0cf
permissions	-rw-r--r--