Mercurial Mercurial > tips / file revision
summary | shortlog | changelog | graph | tags | bookmarks | branches | files | changeset | file | latest | revisions | annotate | diff | comparison | raw | help
Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
web-site.rst
author Oleksandr Gavenko <gavenkoa@gmail.com>
Mon, 09 Oct 2017 10:49:36 +0300
changeset 2188 e95731eef030
parent 2077 94a39ed90fca
child 2228 837f1337c59b
permissions -rw-r--r--
Fixed: NameError: name 'locale_encoding' is not defined File /bin/rst2html.py, line 17, in <module> from docutils.core import publish_cmdline, default_description File /usr/lib/python2.7/site-packages/docutils/core.py, line 20, in <module> from docutils import frontend, io, utils, readers, writers File /usr/lib/python2.7/site-packages/docutils/frontend.py, line 41, in <module> import docutils.utils File /usr/lib/python2.7/site-packages/docutils/utils/__init__.py, line 20, in <module> import docutils.io File /usr/lib/python2.7/site-packages/docutils/io.py, line 18, in <module> from docutils.utils.error_reporting import locale_encoding, ErrorString, ErrorOutput File /usr/lib/python2.7/site-packages/docutils/utils/error_reporting.py, line 60, in <module> codecs.lookup(locale_encoding or '') # None -> '' NameError: name 'locale_encoding' is not defined

.. -*- coding: utf-8; -*-

==========
 Web site
==========
.. contents::
   :local:

Speeding up web site loading
============================

  http://developer.yahoo.com/performance/rules.html

robots.txt
==========

To exclude all robots from the entire server::

  User-agent: *
  Disallow: /

To exclude all robots from part of the server::

  User-agent: *
  Disallow: /cgi-bin/
  Disallow: /tmp/
  Disallow: /junk/

To allow a single robot::

  User-agent: Google
  Disallow:

  User-agent: *
  Disallow: /

To allow all robots complete access::

  User-agent: *
  Disallow:

See:

http://www.robotstxt.org/
  Page provides description for robots.txt usual practice and discussion about
  possible standardization efforts.
http://www.robotstxt.org/robotstxt.html
  About /robots.txt
http://www.robotstxt.org/faq.html
  Frequently Asked Questions.
https://en.wikipedia.org/wiki/Robots_exclusion_standard
  Wikipedia article on robots.txt.
http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
  Improving on Robots Exclusion Protocol.

Sitemap
=======

Sitemaps protocol allows a webmaster to inform search engines about URLs on a
website that are available for crawling.

http://www.sitemaps.org/protocol.html
  Sitemap protocol.
http://en.wikipedia.org/wiki/Sitemaps
  Wikipedia article.

Web document structure useage
=============================

  http://dev.opera.com/articles/view/mama/
                Metadata Analysis and Mining Application

Validation
==========

  http://validator.w3.org/

Add search to your site
=======================

  http://www.google.com/support/customsearch/
                Custom Search Help
  http://help.yahoo.com/l/uk/yahoo/search/basics/basics-13.html
                Can I add a Yahoo! Search box to my site?

Check websites for broken links
===============================

  http://linkchecker.sourceforge.net/
                linkchecker home page.
  http://arthurdejong.org/webcheck/
                webcheck home page.
tips