tips: web-site.rst@83ded9492a61 (annotated)

691 00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	1	-- mode: outline; coding: utf-8 --
00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	2
00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	3	* Speeding up web site loading.
00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	4
00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	5	http://developer.yahoo.com/performance/rules.html
00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	6
707 fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	7	* robots.txt.
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	8
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	9	To exclude all robots from the entire server
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	10
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	11	User-agent: *
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	12	Disallow: /
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	13
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	14	To exclude all robots from part of the server:
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	15
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	16	User-agent: *
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	17	Disallow: /cgi-bin/
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	18	Disallow: /tmp/
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	19	Disallow: /junk/
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	20
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	21	To allow a single robot:
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	22
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	23	User-agent: Google
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	24	Disallow:
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	25
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	26	User-agent: *
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	27	Disallow: /
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	28
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	29	To allow all robots complete access:
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	30
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	31	User-agent: *
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	32	Disallow:
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	33
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	34	http://www.robotstxt.org/
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	35	home page
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	36	http://www.robotstxt.org/robotstxt.html
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	37	About /robots.txt
708 83ded9492a61 Added url. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 707 diff changeset	38	http://www.robotstxt.org/faq.html
83ded9492a61 Added url. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 707 diff changeset	39	Frequently Asked Questions
83ded9492a61 Added url. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 707 diff changeset	40	http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
83ded9492a61 Added url. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 707 diff changeset	41	Improving on Robots Exclusion Protocol
707 fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	42

author	Oleksandr Gavenko <gavenkoa@gmail.com>
	Thu, 18 Nov 2010 20:11:32 +0200
changeset 708	83ded9492a61
parent 707	fae2370e94fd
child 740	8189b7ad02d9
permissions	-rwxr-xr-x