Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Basic Introduction to Elasticsearch

Basic Introduction to Elasticsearch

Yet another introductory presentation about Elasticsearch.

Bhaskar V. Karambelkar

December 12, 2014
Tweet

More Decks by Bhaskar V. Karambelkar

Other Decks in Technology

Transcript

  1. What is it ? • Full text search engine based

    on the Apache Lucene full text search library. • NoSQL Data Store for Structured/UnStructured data • Open Source but commercially supported by elasticsearch.com , and part of the ELK (Elasticsearch-LogStash-Kibana) product stack . • Competing Products - • None really in terms of feature-set and scalability • Apache Solr/SolrCloud, Cloudera Search (Hadoop + Solr) • Oracle Text • Splunk for machine data.
  2. Features • Full text search as well as structured query

    support. • All interaction via REST APIs (API Bindings available for all major languages) • Support fault-tolerant and automatic fail-over operations, as well as data replication out of box. • Various options for pre-processing data before being indexed. (Analyzers/Tokenizers/Filters) • Supports integration via rivers , ES-Hadoop, native integration. • Comes with sensible defaults out of box for ease of development/ deployment.
  3. Core Concepts • Cluster : 1+ nodes, by default all

    nodes are identical. Each cluster has a name that cannot change after being setup. • Node : A single JVM process. (It is possible to have 2+ nodes in a single box/vm) • Index : Equivalent to a DB Schema, contains have 1+ types. • Type : Equiv. to a DB Table, contains 1+ document. • Document : Equiv. to a DB Row. A JSON encoded structured/un- structured text. • Shard + Replicas : Index is broken into 1+ shards, where each shard lives on just one node in it’s entirety, and is replicated within the cluster as per the replication policy.
  4. How is it deployed ? • Scalable from get go.

    Just add more nodes to scale horizontally. (Current limit about 150 nodes/cluster). • Cluster has a master node elected amongst master eligible nodes. Auto failover to some other master-eligible node in case of master going down. • By Default all nodes are master-eligible and also act as data nodes and client nodes. • Nodes auto discover themselves using multicast. • 1 Node per box
  5. How SHOULD it be deployed ? • Dedicated Master/Data/Client nodes

    to avoid split brain and better load balancing. • Data nodes with locally attached disks. • Typical Data disk 8 cores, 64GB RAM, 4-6 1TB to 2 TB disks disks. Assign 30GB ram to VM leave rest to OS + FS Cache • Change default cluster name and use unicast discovery to avoid unexpected nodes from joining. • Nodes should be in a single subnet talk to each other over 1G /10G links. • Don’t grow cluster beyond 150 nodes, prefer separate clusters and use a “Tribe Node” for across cluster searches.
  6. Data Ingestion • HTTP PUT to /<index>/<type>/<id>, will • Auto

    create Index and Type • Use default Analyzer to analyze the input document • Store the input document at the given ID in the index along with the indexed data. • Replicate it as per replication policy • Make it available for immediate GET via ID, and in near- realtime for search.
  7. e.g. curl -XPUT 'localhost:9200/customer/external/1' \ -d '{"name": "John Doe"}' And

    the response: { "_index" : "customer", "_type" : "external", "_id" : "1", "_version" : 1, "created" : true }
  8. Types of APIs • Document : Index/Get/Delete/Update + Bulk •

    Search : Search/Count/Validate/Explain/MLT/Percolate • Indices : Index/Alias/Mappings Management +Monitoring • Cluster : Cluster wide health/state/status • Cat : APIs outputting compact and aligned text instead of JSON (scripting).
  9. Analysis • Analyzer contains 1 Tokenizer & 0+ Token Filters

    • Tokenizer can be preceded by 1+ Char Filter • Tokenizer breaks down strings in to stream of Tokens. e.g. “one two three” becomes ‘one’, ‘two’, ‘three’. Standard/Edge-ngram/keyword/pattern • Token Filters modify/delete/add tokens • Char Filters strip out /replace characters
  10. Mapping • Unstructured data works but structured is even better

    • Use Mapping API to pre populate the metadata about documents belonging to a certain index/type. • Used for specifying Analyzer, Field data types, transformations etc. • Highly recommended to use even if default mappings are good enough.
  11. QUERY DSL • All queries are JSONs • Simplest is

    query_string , field:’one or two or three’ • query string not recommended for prod. use. • query can comprise of search + filter. • search assigns a score , filter is just that a boolean gate. • Query DSL can be quite daunting at first, learn it like SQL.