===============

 elasticsearch

===============

.. contents::

   :local:

Elasticsearch documentation

===========================

https://amsterdam.luminis.eu/2016/10/18/elasticsearch-5-is-coming-what-is-new-and-improved/

  New features of ES 5.

Releases

========

https://github.com/elastic/elasticsearch/releases

  Git releases & tags.

https://www.elastic.co/support/eol

  Elastic Product End of Life Dates.

Installing & configuring

========================

https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

  Install Elasticsearch with Debian Package.

Basic config

============

Common cluster name inside given intranet::

  cluster.name: mycluster

Unique node name within claser::

  node.name: "node1"

Node types::

  node.master: true

  node.ingest: true

  node.data: true

Network interfaces to bind to::

  network.host: [_local_, node1.example.com]

Port definitions::

  http.port : 9200

  tcp.port : 9300

Override default locations::

  path.data: /path/to/data1,/path/to/data2

  path.logs: /path/to/logs

  path.plugins: /path/to/plugins

Explicit list of seed nodes in cluster::

  discovery.zen.ping.unicast.hosts: ["master1.example.com", "master2.example.com:9300"]

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html

  Node types.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html

  Discovery settings.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-transport.html

  Transport definitions.

REST syntax conventions

=======================

To get data in table form use ``/_cat`` endpoint::

  GET /_cat/nodes

To pretty print output append query::

  GET /_cat/nodes?pretty=1

Get base information

====================

Cluster health::

  GET /_cat/health?v

  GET /_cluster/health?pretty

  GET /_cluster/health?pretty&level=cluster

List of nodes in cluster (ip, RAM, CPU)::

  GET /_cat/nodes?v

  GET /_cat/master?v

  watch -d curl -s 'localhost:9200/_cat/nodes?v'

List cluster state::

  GET /_cluster/state?pretty

  GET /_cluster/allocation/explain

List of tasks executed in cluster::

  GET /_cat/tasks?v

  GET /_cat/tasks?detailed

  GET _tasks

List of indexes (status, health, size)::

  GET /_cat/indices

  GET /_cat/indices?v

  GET /_cat/indices?v&s=index

  GET /_cluster/health?pretty&level=indices

List of shards::

  GET /_cat/shards?v

List of mappings across all indexes::

  GET /_mapping

  GET /_all/_mapping

  GET /twitter,kimchy/_mapping

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html

  Get Mapping.

List of shards::

  GET /_cluster/health?pretty&level=shards

https://www.elastic.co/guide/en/elasticsearch/reference/current/_cluster_health.html

  Cluster Health.

https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html

  Cluster Allocation Explain API

https://www.elastic.co/guide/en/elasticsearch/reference/current/_list_all_indices.html

  List All Indices.

https://www.elastic.co/guide/en/elasticsearch/reference/current/cat.html

  cat APIs.

https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html

  Task Management API.

Managing indexes

================

Assign number of shards and replicas::

  curl -XPUT -d '{settings: {index: "number_of_shards": 3, "number_of_replicas": 1}}'

https://www.elastic.co/guide/en/elasticsearch/reference/current/_delete_an_index.html

  Delete an Index.

https://github.com/elastic/curator

  Manage/archive indices.

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/about.html

  Elasticsearch Curator helps you curate, or manage, your Elasticsearch indices and snapshots.

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/about-features.html

  Curator allows for many different operations to be performed to both indices and snapshots.

https://www.elastic.co/guide/en/elasticsearch/guide/current/retiring-data.html

  Retiring Data.

Lucene search syntax

====================

``TERM1 TERM2`` is same as ``TERM1 OR TERM2``.

``"TERM1 TERM2"`` is for phrase.

``"TERM1 TERM2"~5`` is for proximity.

``TERM~0.8`` is for fuzzy.

``*`` is for wildcard.

Boosting is done with ``^N`` syntax, like ``TERM1^10 TERM2``.

Range with ``[2017-01-01 TO 2017-02-29]``.

Logical ``AND``, ``OR``, ``NOT`` and parentheses for grouping.

Fields are set before colon, like ``FIELD:TERM``.

https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-query-string-query.html

  Query String Query

https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-simple-query-string-query.html

  Simple Query String Query

ES Query DSL

============

``_exists_:<field>`` for testing field existence.

.. note:: ``_missing_:<field>`` was removed from Kibana 5.x, use ``NOT _exists_:<field>``.

``-<field>:<val>`` or ``-<field>:"<val>"`` for excluding field value.

``+<field>:<val>`` or ``+<field>:"<val>"`` for including field value.

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html

  Exists Query

Performance

===========

https://www.elastic.co/guide/en/elasticsearch/reference/5.5/search-profile.html

  Profile API.

https://www.elastic.co/guide/en/elasticsearch/reference/current/_explain_analyze.html

  Explain Analyze.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-explain.html

  Explain API.

https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html

  Tune for disk usage.

https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html

  Tune for indexing speed.

https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html

  Tune for search speed.

https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster

  How many shards should I have in my Elasticsearch cluster?

https://www.elastic.co/guide/en/elasticsearch/reference/master/heap-size.html

  Setting the heap size.

Storage requirements

====================

https://www.elastic.co/blog/elasticsearch-storage-the-true-story

  The true story behind Elasticsearch storage requirements (2015).

https://www.elastic.co/blog/elasticsearch-storage-the-true-story-2.0

  Part 2.0: The true story behind Elasticsearch storage requirements (2015).

https://www.elastic.co/blog/minimize-index-storage-size-elasticsearch-6-0

  Space Saving Improvements in Elasticsearch 6.0 (2017).

https://www.elastic.co/blog/filebeat-modiles-access-logs-and-elasticsearch-storage-requirements

  Filebeat modules, access logs and Elasticsearch storage requirements.

JSON search syntax

==================

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html

  Search results pagination.

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

  Query String Query.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html

  Aggregation.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html

  Sort.

Alerting

========

https://github.com/Yelp/elastalert

  Easy & Flexible Alerting With Elasticsearch.

http://elastalert.readthedocs.io/en/latest/elastalert.html

  Easy & Flexible Alerting With Elasticsearch.

https://github.com/sirensolutions/sentinl/wiki/SENTINL-Alerts-in-Dashboard

  SENTINL Alerts in Dashboard.

https://github.com/sirensolutions/sentinl/wiki/SENTINL-Config-Example

  SENTINL Config Example

https://github.com/sirensolutions/sentinl/wiki/SENTINL-Tutorial

  SENTINL Tutorial

https://github.com/sirensolutions/sentinl/wiki/SENTINL-Watcher-Anatomy

  SENTINL Watcher Anatomy

https://github.com/sirensolutions/sentinl/wiki/SENTINL-Watcher-Examples

  SENTINL Watcher Examples