tips: web-site.rst@3ce5d6d2c057 (annotated)

691 00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	1	-- mode: outline; coding: utf-8 --
00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	2
00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	3	* Speeding up web site loading.
00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	4
00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	5	http://developer.yahoo.com/performance/rules.html
00f3ba84ebd9 Speeding up web site loading. Oleksandr Gavenko <gavenkoa@gmail.com> parents: diff changeset	6
707 fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	7	* robots.txt.
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	8
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	9	To exclude all robots from the entire server
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	10
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	11	User-agent: *
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	12	Disallow: /
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	13
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	14	To exclude all robots from part of the server:
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	15
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	16	User-agent: *
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	17	Disallow: /cgi-bin/
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	18	Disallow: /tmp/
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	19	Disallow: /junk/
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	20
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	21	To allow a single robot:
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	22
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	23	User-agent: Google
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	24	Disallow:
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	25
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	26	User-agent: *
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	27	Disallow: /
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	28
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	29	To allow all robots complete access:
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	30
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	31	User-agent: *
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	32	Disallow:
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	33
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	34	http://www.robotstxt.org/
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	35	home page
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	36	http://www.robotstxt.org/robotstxt.html
fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	37	About /robots.txt
708 83ded9492a61 Added url. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 707 diff changeset	38	http://www.robotstxt.org/faq.html
83ded9492a61 Added url. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 707 diff changeset	39	Frequently Asked Questions
83ded9492a61 Added url. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 707 diff changeset	40	http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
83ded9492a61 Added url. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 707 diff changeset	41	Improving on Robots Exclusion Protocol
707 fae2370e94fd robots.txt Oleksandr Gavenko <gavenkoa@gmail.com> parents: 691 diff changeset	42
740 8189b7ad02d9 Web document structure useage. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 708 diff changeset	43	* Web document structure useage.
8189b7ad02d9 Web document structure useage. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 708 diff changeset	44
8189b7ad02d9 Web document structure useage. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 708 diff changeset	45	http://dev.opera.com/articles/view/mama/
8189b7ad02d9 Web document structure useage. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 708 diff changeset	46	Metadata Analysis and Mining Application
743 3ce5d6d2c057 Validation. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 740 diff changeset	47
3ce5d6d2c057 Validation. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 740 diff changeset	48	* Validation.
3ce5d6d2c057 Validation. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 740 diff changeset	49
3ce5d6d2c057 Validation. Oleksandr Gavenko <gavenkoa@gmail.com> parents: 740 diff changeset	50	http://validator.w3.org/

author	Oleksandr Gavenko <gavenkoa@gmail.com>
	Tue, 07 Dec 2010 23:40:14 +0200
changeset 743	3ce5d6d2c057
parent 740	8189b7ad02d9
child 769	fc2e493b53ce
permissions	-rwxr-xr-x