Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Manage Your Content with Elasticsearch

Manage Your Content with Elasticsearch

Samantha Quiñones

January 29, 2016
Tweet

More Decks by Samantha Quiñones

Other Decks in Technology

Transcript

  1. Managing Your Content With
    Elasticsearch
    Samantha Quiñones / @ieatkillerbees

    View full-size slide

  2. About Me
    • Software Engineer & Data Nerd since 1997
    • Doing “media stuff” since 2012
    • Principal @ AOL since 2014
    • @ieatkillerbees
    • http://samanthaquinones.com

    View full-size slide

  3. What We’ll Cover
    • Intro to Elasticsearch
    • CRUD
    • Creating Mappings
    • Analyzers
    • Basic Querying & Searching
    • Scoring & Relevance
    • Aggregations Basics

    View full-size slide

  4. But First…
    • Download - https://www.elastic.co/downloads/elasticsearch
    • Clone - https://github.com/squinones/elasticsearch-tutorial.git

    View full-size slide

  5. What is Elasticsearch?
    • Near real-time (documents are available for search quickly after
    being indexed) search engine powered by Lucene
    • Clustered for H/A and performance via federation with shards and
    replicas

    View full-size slide

  6. What’s it Used For?
    • Logging (we use Elasticsearch to centralize traffic logs, exception
    logs, and audit logs)
    • Content management and search
    • Statistical analysis

    View full-size slide

  7. Installing Elasticsearch
    $ curl -L -O https://download.elastic.co/elasticsearch/release/org/elasticsearch/
    distribution/tar/elasticsearch/2.1.1/elasticsearch-2.1.1.tar.gz
    $ tar -zxvf elasticsearch*
    $ cd elasticsearch-2.1.1/bin
    $ ./elasticsearch

    View full-size slide

  8. Connecting to Elasticsearch
    • Via Java, there are two native clients which connect to an ES
    cluster on port 9300
    • Most commonly, we access Elasticsearch via HTTP API

    View full-size slide

  9. HTTP API
    curl -X GET "http://localhost:9200/?pretty"

    View full-size slide

  10. Data Format
    • Elasticsearch is a document-oriented database
    • All operations are performed against documents (object graphs
    expressed as JSON)

    View full-size slide

  11. Analogues
    Elasticsearch MySQL MongoDB
    Index Database Database
    Type Table Collection
    Document Row Document
    Field Column Field

    View full-size slide

  12. Index Madness
    • Index is an overloaded term.
    • As a verb, to index a document is store a document in an index.
    This is analogous to an SQL INSERT operation.
    • As a noun, an index is a collection of documents.
    • Fields within a document have inverted indexes, similar to how a
    column in an SQL table may have an index.

    View full-size slide

  13. Indexing Our First Document
    curl -X PUT "http://localhost:9200/test_document/test/1" -d '{ "name": "test_name" }’

    View full-size slide

  14. Retrieving Our First Document
    curl -X GET "http://localhost:9200/test_document/test/1"

    View full-size slide

  15. Let’s Look at Some Stackoverflow Posts!
    $ vi queries/bulk_insert_so_data.json

    View full-size slide

  16. Bulk Insert
    curl -X PUT "http://localhost:9200/_bulk" --data-binary "@queries/
    bulk_insert_so_data.json"

    View full-size slide

  17. First Search
    curl -X GET "http://localhost:9200/stack_overflow/_search"

    View full-size slide

  18. Query String Searches
    curl -X GET "http://localhost:9200/stack_overflow/_search?q=title:php"

    View full-size slide

  19. Query DSL
    curl -X POST "http://localhost:9200/stack_overflow/_search" -d
    '{
    "query" : {
    "match" : {
    "title" : "php"
    }
    }
    }'

    View full-size slide

  20. Compound Queries
    curl -X POST "http://localhost:9200/stack_overflow/_search" -d '{
    "query" : {
    "filtered": {
    "query" : {
    "match" : {
    "title" : "(php OR python) AND (flask OR laravel)"
    }
    },
    "filter": {
    "range": {
    "score": {
    "gt": 3
    }
    }
    }
    }
    }
    }'

    View full-size slide

  21. Full-Text Searching
    curl -X POST "http://localhost:9200/stack_overflow/_search" -d
    '{
    "query" : {
    "match" : {
    "title" : "php loop"
    }
    }
    }'

    View full-size slide

  22. Relevancy
    • When searching (in query context), results are scored by a
    relevancy algorithm
    • Results are presented in order from highest to lowest score

    View full-size slide

  23. Phrase Searching
    curl -X POST "http://localhost:9200/stack_overflow/_search" -d
    '{
    "query" : {
    "match" : {
    "title": {
    "query": "for loop",
    "type": "phrase"
    }
    }
    }
    }'

    View full-size slide

  24. Highlighting Searches
    curl -X POST "http://localhost:9200/stack_overflow/_search" -d
    '{
    "query" : {
    "match" : {
    "title": {
    "query": "for loop",
    "type": "phrase"
    }
    }
    },
    "highlight": {
    "fields" : {
    "title" : {}
    }
    }
    }'

    View full-size slide

  25. Aggregations
    • Run statistical operations over your data
    • Also near real-time!
    • Complex aggregations are abstracted away behind simple
    interfaces— you don’t need to be a statistician

    View full-size slide

  26. Analyzing Tags
    curl -X POST "http://localhost:9200/stack_overflow/_search" -d
    '{
    "size": 0,
    "aggs": {
    "all_tags": {
    "terms": {
    "field": "tags",
    "size": 0
    }
    }
    }
    }'

    View full-size slide

  27. Nesting Aggregations
    curl -X POST “http://localhost:9200/stack_overflow/_search" -d
    '{
    "size": 0,
    "aggs": {
    "all_tags": {
    "terms": {
    "field": "tags",
    "size": 0
    },
    "aggs": {
    "avg_score": {
    "avg": { "field": "score"}
    }
    }
    }
    }
    }'

    View full-size slide

  28. Under the Hood
    • Elasticsearch is designed from the ground-up to run in a distributed
    fashion.
    • Indices (collections of documents) are partitioned in to shards.
    • Shards can be stored on a single or multiple nodes.
    • Shards are balanced across the cluster to improve performance
    • Shards are replicated for redundancy and high availability

    View full-size slide

  29. What is a Cluster?
    • One or more nodes (servers) that work together to…
    • serve a dataset that exceeds the capacity of a single server…
    • provide federated indexing (writes) and searching (reads)…
    • provide H/A through sharing and replication of data

    View full-size slide

  30. What are Nodes?
    • Individual servers within a cluster
    • Can providing indexing and searching capabilities

    View full-size slide

  31. What is an Index?
    • An index is logically a collection of documents, roughly analogous
    to a database in MySQL
    • An index is in reality a namespace that points to one or more
    physical shards which contain data
    • When indexing a document, if the specified index does not exist, it
    will be created automatically

    View full-size slide

  32. What are Shards?
    • Low-level units that hold a slice of available data
    • A shard represents a single instance of lucene and is fully-
    functional, self-contained search engine
    • Shards are either primary or replicas and are assigned to nodes

    View full-size slide

  33. What is Replication?
    • Shards can have replicas
    • Replicas primarily provide redundancy for when shards/nodes fail
    • Replicas should not be allocated on the same node as the shard it
    replicates

    View full-size slide

  34. Default Topology
    • 5 primary shards per index
    • 1 replica per shard

    View full-size slide

  35. NODE
    Clustering & Replication
    NODE
    R1 P2 P3 R2 R3
    P4 R5 P1 R4 P5

    View full-size slide

  36. Cluster Health
    curl -X GET “http://localhost:9200/_cluster/health"
    curl -X GET "http://localhost:9200/_cat/health?v"

    View full-size slide

  37. _cat API
    • Display human-readable information about parts of the ES system
    • Provides some limited documentation of functions

    View full-size slide

  38. aliases
    > $ http GET ':9200/_cat/aliases?v'
    alias index filter routing.index routing.search
    posts posts_561729df8ce4e * - -
    posts.public posts_561729df8ce4e * - -
    posts.write posts_561729df8ce4e - - -
    Display all configured aliases

    View full-size slide

  39. allocation
    > $ http GET ':9200/_cat/allocation?v'
    shards disk.used disk.avail disk.total disk.percent host
    33 2.6gb 21.8gb 24.4gb 10 host1
    33 3gb 21.4gb 24.4gb 12 host2
    34 2.6gb 21.8gb 24.4gb 10 host3
    Show how many shards are allocated per node, with disk utilization info

    View full-size slide

  40. count
    > $ http GET ':9200/_cat/count?v'
    epoch timestamp count
    1453790185 06:36:25 182763
    > $ http GET ‘:9200/_cat/count/posts?v’
    epoch timestamp count
    1453790467 06:41:07 164169
    > $ http GET ‘:9200/_cat/count/posts.public?v’
    epoch timestamp count
    1453790472 06:41:12 164169=
    Display a count of documents in the cluster, or a specific index

    View full-size slide

  41. fielddata
    > $ http -b GET ':9200/_cat/fielddata?v'
    id host ip node
    total site_id published
    7tjeJNY3TMajqRkmYsJyrA host1 10.97.183.146 node1 1.1mb 170.1kb 996.5kb
    __xrpsKAQW6yyCY8luLQdQ host2 10.97.180.138 node2 1.6mb 329.3kb 1.3mb
    bdoNNXHXRryj22YqjnqECw host3 10.97.181.190 node3 1.1mb 154.7kb 991.7kb
    Shows how much memory is allocated to fielddata (metadata used for sorts)

    View full-size slide

  42. health
    > $ http -b GET ':9200/_cat/health?v'
    epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks
    1453829723 17:35:23 ampehes_prod_cluster green 3 3 100 50 0 0 0 0

    View full-size slide

  43. indices
    > $ http -b GET 'eventhandler-prod.elasticsearch.amppublish.aws.aol.com:9200/_cat/indices?v'
    health status index pri rep docs.count docs.deleted store.size pri.store.size
    green open posts_561729df8ce4e 5 1 468629 20905 4gb 2gb
    green open slideshows 5 1 3893 6 86mb 43mb

    View full-size slide

  44. master
    > $ http -b GET ':9200/_cat/master?v'
    id host ip node
    7tjeJNY3TMajqRkmYsJyrA host1 10.97.183.146 node1

    View full-size slide

  45. nodes
    > $ http -b GET ':9200/_cat/nodes?v'
    host ip heap.percent ram.percent load node.role master name
    127.0.0.1 127.0.0.1 50 100 2.47 d * Mentus

    View full-size slide

  46. pending tasks
    % curl 'localhost:9200/_cat/pending_tasks?v'
    insertOrder timeInQueue priority source
    1685 855ms HIGH update-mapping [foo][t]
    1686 843ms HIGH update-mapping [foo][t]
    1693 753ms HIGH refresh-mapping [foo][[t]]
    1688 816ms HIGH update-mapping [foo][t]
    1689 802ms HIGH update-mapping [foo][t]
    1690 787ms HIGH update-mapping [foo][t]
    1691 773ms HIGH update-mapping [foo][t]

    View full-size slide

  47. shards
    > $ http -b GET ':9200/_cat/shards?v'
    index shard prirep state docs store ip node
    posts_561729df8ce4e 2 r STARTED 94019 410.5mb 10.97.180.138 host1
    posts_561729df8ce4e 2 p STARTED 94019 412.7mb 10.97.181.190 host2
    posts_561729df8ce4e 0 p STARTED 93307 413.6mb 10.97.183.146 host3
    posts_561729df8ce4e 0 r STARTED 93307 415mb 10.97.180.138 host1
    posts_561729df8ce4e 3 p STARTED 94182 407.1mb 10.97.183.146 host2
    posts_561729df8ce4e 3 r STARTED 94182 403.4mb 10.97.180.138 host1
    posts_561729df8ce4e 1 r STARTED 94130 447.1mb 10.97.180.138 host1
    posts_561729df8ce4e 1 p STARTED 94130 447mb 10.97.181.190 host2
    posts_561729df8ce4e 4 r STARTED 93299 421.5mb 10.97.183.146 host3
    posts_561729df8ce4e 4 p STARTED 93299 398.8mb 10.97.181.190 host2

    View full-size slide

  48. segments
    > $ http -b GET ':9200/_cat/segments?v'
    index shard prirep ip segment generation docs.count docs.deleted size size.memory committed searchable version
    compound
    posts_561726fecd9c6 0 p 10.97.183.146 _a 10 24 0 227.7kb 69554 true true 4.10.4 true
    posts_561726fecd9c6 0 p 10.97.183.146 _b 11 108 0 659.1kb 103242 true true 4.10.4 false
    posts_561726fecd9c6 0 p 10.97.183.146 _c 12 7 0 90.7kb 54706 true true 4.10.4 true
    posts_561726fecd9c6 0 p 10.97.183.146 _d 13 6 0 82.2kb 49706 true true 4.10.4 true
    posts_561726fecd9c6 0 p 10.97.183.146 _e 14 8 0 119kb 67162 true true 4.10.4 true
    posts_561726fecd9c6 0 p 10.97.183.146 _f 15 1 0 35.9kb 32122 true true 4.10.4 true
    posts_561726fecd9c6 0 r 10.97.180.138 _a 10 24 0 227.7kb 69554 true true 4.10.4 true
    posts_561726fecd9c6 0 r 10.97.180.138 _b 11 108 0 659.1kb 103242 true true 4.10.4 false

    View full-size slide

  49. CRUD Operations

    View full-size slide

  50. Document Model
    • Documents represent objects
    • By default, all fields in all documents are analyzed, and indexed

    View full-size slide

  51. Metadata
    • _index - The index in which a document resides
    • _type - The class of object that a document represents
    • _id - The document’s unique identifier. Auto-generated when not
    provided

    View full-size slide

  52. Retrieving Documents
    curl -X GET "http://localhost:9200/test_document/test/1"
    curl -X HEAD “http://localhost:9200/test_document/test/1"
    curl -X HEAD "http://localhost:9200/test_document/test/2"

    View full-size slide

  53. Updating Documents
    curl -X PUT "http://localhost:9200/test_document/test/1" -d '{
    "name": "test_name",
    "conference": "php benelux"
    }'
    curl -X GET "http://localhost:9200/test_document/test/1"

    View full-size slide

  54. Explicit Creates
    curl -X PUT "http://localhost:9200/test_document/test/1/_create" -d '{
    "name": "test_name",
    "conference": "php benelux"
    }'

    View full-size slide

  55. Auto-Generated IDs
    curl -X POST "http://localhost:9200/test_document/test" -d '{
    "name": "test_name",
    "conference": "php benelux"
    }'

    View full-size slide

  56. Deleting Documents
    curl -X DELETE "http://localhost:9200/test_document/test/1"

    View full-size slide

  57. Bulk API
    • Perform many operations in a single request
    • Efficient batching of actions
    • Bulk queries take the form of a stream of single-line JSON objects
    that define actions and document bodies

    View full-size slide

  58. Bulk Actions
    • create - Index a document IFF it doesn’t exist already
    • index - Index a document, replacing it if it exists
    • update - Apply a partial update to a document
    • delete - Delete a document

    View full-size slide

  59. Bulk API Format
    { action: { metadata }}\n
    { request body }\n
    { action: { metadata }}\n
    { request body }\

    View full-size slide

  60. Sizing Bulk Requests
    • Balance quantity of documents with size of documents
    • Docs list the sweet-spot between 5-15 MB per request
    • AOL Analytics Cluster indexes 5000 documents per batch (approx
    7MB)

    View full-size slide

  61. Searching Documents
    • Structured queries - queries against concrete fields like “title” or
    “score” which return specific documents.
    • Full-text queries - queries that find documents which match a search
    query and return them sorted by relevance

    View full-size slide

  62. Search Elements
    • Mappings - Defines how data in fields are interpreted
    • Analysis - How text is parsed and processed to make it searchable
    • Query DSL - Elasticsearch’s query language

    View full-size slide

  63. About Queries
    • Leaf Queries - Searches for a value in a given field. These queries
    are standalone. Examples: match, range, term
    • Compound Queries - Combinations of leaf queries and other
    compound queries which combine operations together either
    logically (e.g. bool queries) or alter their behavior (e.g. score
    queries)

    View full-size slide

  64. Empty Search
    curl -X GET "http://localhost:9200/stack_overflow/_search"
    curl -X POST "http://localhost:9200/stack_overflow/_search" -d
    '{
    "query": { "match_all": {} }
    }'

    View full-size slide

  65. Timing Out Searches
    curl -X GET "http://localhost:9200/stack_overflow/_search?timeout=1s"
    curl -X POST "http://localhost:9200/stack_overflow/_search" -d '{
    "timeout": "1s",
    "query": { "match_all": {} }
    }'

    View full-size slide

  66. Multi-Index/Type Searches
    curl -X GET "http://localhost:9200/test_document,stack_overflow/_search"

    View full-size slide

  67. Multi-Index Use Cases
    • Dated indices for logging
    • Roll-off indices for content-aging
    • Analytic roll-ups

    View full-size slide

  68. Pagination
    curl -X GET "http://localhost:9200/stack_overflow/_search?size=5&from=5"
    curl -X POST "http://localhost:9200/stack_overflow/_search" -d '{
    "size": 5,
    "from": 5,
    "query": { "match_all": {} }
    }'

    View full-size slide

  69. Pagination Concerns
    • Since searches are distributed across multiple shards, paged
    queries must be sorted at each shard, combined, and resorted
    • The cost of paging in distributed data sets can increase
    exponentially
    • It is a wise practice to set limits to how many pages of results can
    be returned

    View full-size slide

  70. Full Text Queries
    • match - Basic term matching query
    • multi_match - Match which spans multiple fields
    • common_terms - Match query which preferences uncommon words
    • query_string - Match documents using a search “mini-dsl”
    • simple_query_string - A simpler version of query_string that never
    throws exceptions, suitable for exposing to users

    View full-size slide

  71. Term Queries
    • term - Search for an exact value
    • terms - Search for an exact value in multiple fields
    • range - Find documents where a value is in a certain range
    • exists - Find documents that have any non-null value in a field
    • missing - Inversion of `exists`
    • prefix - Match terms that begin with a string
    • wildcard - Match terms with a wildcard
    • regexp - Match terms against a regular expression
    • fuzzy - Match terms with configurable fuzziness

    View full-size slide

  72. Compound Queries
    • constant_score - Wraps a query in filter context, giving all results a constant score
    • bool - Combines multiple leaf queries with `must`, `should`, `must_not` and `filter` clauses
    • dis_max - Similar to bool, but creates a union of subquery results scoring each document with the
    maximum score of the query that produced it
    • function_score - Modifies the scores of documents returned by a query . Useful for altering the
    distribution of results based on recency, popularity, etc.
    • boosting - Takes a `positive` and `negative` query, returning the results of `positive` while
    reducing the scores of documents that also match `negative`
    • filtered - Combines a query clause in query context with one in filter context
    • limit - Perform the query over a limited number of documents in each shard

    View full-size slide

  73. What are Mappings?
    • Similar to schemas, they define the types of data found in fields
    • Determines how individual fields are analyzed & stored
    • Sets the format of date fields
    • Sets rules for mapping dynamic fields

    View full-size slide

  74. Mapping Types
    • Indices have one or more mapping types which group documents
    logically.
    • Types contain meta fields, which can be used to customize
    metadata like _index, _id, _type, and _source
    • Types can also list fields that have consistent structure across types.

    View full-size slide

  75. Data Types
    • Scalar Values - string, long, double, boolean
    • Special Scalars - date, ip
    • Structural Types - object, nested
    • Special Types - geo_shape, geo_point, completion
    • Compound Types - string arrays, nested objects

    View full-size slide

  76. Dynamic vs Explicit Mapping
    • Dynamic fields are not defined prior to indexing
    • Elasticsearch selects the most likely type for dynamic fields, based
    on configurable rules
    • Explicit fields are defined exactly prior to indexing
    • Types cannot accept data that is the wrong type for an explicit
    mapping

    View full-size slide

  77. Shared Fields
    • Fields that are defined in multiple mapping types must be identical
    if:
    • They have the same name
    • Live in the same index
    • Map to the same field internally

    View full-size slide

  78. Examining Mappings
    curl -X GET "http://localhost:9200/stack_overflow/post/_mapping"

    View full-size slide

  79. Dynamic Mappings
    • Mappings are generated when a type is created, if no mapping
    was previously specified.
    • Elasticsearch is good at identifying fields much of the time, but it’s
    far from perfect!
    • Fields can contain basic data-types, but importantly, mappings
    optimize a field for either structured (exact) or full-text searching

    View full-size slide

  80. Structured Data vs Full Text
    • Exact values contain exact strings which are not subject to natural
    language interpretation.
    • Full-text values must be interpreted in the context of natural
    language

    View full-size slide

  81. Exact Value
    • “[email protected]” is an email address in all contexts

    View full-size slide

  82. Natural Language
    • “us” can be interpreted differently in natural language
    • Abbreviation for “United States”
    • The English dative personal pronoun
    • An alternative symbol for µs
    • The French word us

    View full-size slide

  83. Analyzing Text
    • Elasticsearch is optimized for full text search
    • Text is analyzed in a two-step process
    • First, text is tokenized in to individual terms
    • Second, terms are normalized through a filter

    View full-size slide

  84. Analyzers
    • Analyzers perform the analysis process
    • Character filters clean up text, removing or modifying the text
    • Tokenizers break the text down in to terms
    • Token filters modify, remove, or add terms

    View full-size slide

  85. Standard Analyzer
    • General purpose analyzer that works for most natural language.
    • Splits text on word boundaries, removes punctuation, and
    lowercases all tokens.

    View full-size slide

  86. Standard Analyzer
    curl -X GET "http://localhost:9200/_analyze?analyzer=standard&text="Reverse+text+with
    +strrev($text)!""

    View full-size slide

  87. Whitespace Analyzer
    • Analyzer that splits on whitespace and lowercases all tokens

    View full-size slide

  88. Whitespace Analyzer
    curl -X GET "http://localhost:9200/_analyze?analyzer=whitespace&text="Reverse+text+with
    +strrev($text)!""

    View full-size slide

  89. Keyword Analyzer
    • Tokenizes the entire text as a single string.
    • Used for things that should be kept whole, like ID numbers, postal
    codes, etc

    View full-size slide

  90. Keyword Analyzer
    curl -X GET "http://localhost:9200/_analyze?analyzer=keyword&text="Reverse+text+with
    +strrev($text)!""

    View full-size slide

  91. Language Analyzers
    • Analyzers optimized for specific natural languages.
    • Reduce tokens to stems (jumper, jumped → jump)

    View full-size slide

  92. Language Analyzers
    curl -X GET "http://localhost:9200/_analyze?analyzer=english&text="Reverse+text+with
    +strrev($text)!""

    View full-size slide

  93. Analyzers
    • Analyzers are applied when documents are indexed
    • Analyzers are applied when a full-text search is performed against
    a field, in order to produce the correct set of terms to search for

    View full-size slide

  94. Character Filters
    • html_strip - Removes HTML from text
    • mapping - Filter based on a map of original → new ( { “ph”: “f” })
    • pattern_replace - Similar to mapping, using regular expressions

    View full-size slide

  95. Index Templates
    • Template mappings that are applied to newly created indices
    • Templates also contain index configuration information
    • Powerful when combined with dated indices

    View full-size slide

  96. Scoring
    • Scoring is based on a boolean model and scoring function
    • Boolean model applies AND/OR logic to an inverse index to
    produce a list of matching documents

    View full-size slide

  97. Term Frequency
    • Terms that appear frequently in a document increase the
    document’s relevancy score.
    • term_frequency(term in document) = √number_of_appearances

    View full-size slide

  98. Inverse Document Frequency
    • Terms that appear in many documents reduce a document’s
    relevancy score
    • inverse_doc_frequency(term) = 1 + log(number_of_docs /
    (frequency + 1))

    View full-size slide

  99. Field Length Normalization
    • Terms that appear in shorter fields increase the relevancy of a
    document.
    • norm(document) = 1 / √number_of_terms

    View full-size slide

  100. Example from the Docs
    • Given the text “quick brown fox” the term “fox” scores…
    • Term Frequency: 1.0
    • Inverse Doc Frequency: 0.30685282
    • Field Norm: 0.5
    • Score: 0.15342641

    View full-size slide

  101. Basic Relevancy
    {
    "size": 100,
    "query": {
    "filtered": {
    "query": {
    "match": {
    "contents": "miley cyrus"
    }
    },
    "filter": {
    "and": [ { "terms": { "site_id": [ 698 ] } } ]
    }
    }
    }
    }

    View full-size slide

  102. Non-Preferenced Result Recency

    View full-size slide

  103. Recency-Adjusted Query
    {
    "query": {
    "function_score": {
    "functions": [
    {
    "gauss": {
    "published": {
    "origin": "now",
    "scale": "10d",
    "offset": "1d",
    "decay": 0.3
    }
    }
    }
    ],
    "query": {
    "filtered": {
    "query": { "match": { "contents": "miley cyrus" } },
    "filter": { "and": [ { "terms": { "site_id": [ 698 ] } } ] }
    }
    }
    }
    }
    }

    View full-size slide

  104. Preferenced Result Recency

    View full-size slide

  105. Aggregations & Analytics

    View full-size slide

  106. Importing Energy Data
    curl -X PUT "http://localhost:9200/energy_use" --data-binary "@queries/
    mapping_energy.json"
    curl -X PUT "http://localhost:9200/_bulk" --data-binary "@queries/
    bulk_insert_energy_data.json"
    curl -X GET "http://localhost:9200/energy_use/_search"

    View full-size slide

  107. Average Energy Use
    curl -X POST "http://localhost:9200/energy_use/_search" -d '{
    "size": 0,
    "aggs": {
    "average_laundry_use": {
    "avg": {
    "field": "laundry"
    }
    },
    "average_kitchen_use": {
    "avg": {
    "field": "kitchen"
    }
    },
    "average_heater_use": {
    "avg": {
    "field": "heater"
    }
    },
    "average_other_use": {
    "avg": {
    "field": "other"
    }
    }
    }
    }'

    View full-size slide

  108. Multiple Aggregations
    curl -X POST “http://localhost:9200/energy_use/_search" -d '{
    "size": 0,
    "aggs": {
    "average_laundry_use": { "avg": { "field": "laundry" } },
    "min_laundry_use": { "min": { "field": "laundry"} },
    "max_laundry_use": { "max": { "field": "laundry"} }
    }
    }'

    View full-size slide

  109. Nesting Aggregations
    curl -X POST “http://localhost:9200/energy_use/_search" -d '{
    "size": 0,
    "aggs": {
    "by_date": {
    "terms": { "field": "date" },
    "aggs": {
    "average_laundry_use": { "avg": { "field": "laundry" } },
    "min_laundry_use": { "min": { "field": "laundry"} },
    "max_laundry_use": { "max": { "field": "laundry"} }
    }
    }
    }
    }'

    View full-size slide

  110. Stats/Extended Stats
    curl -X POST "http://localhost:9200/energy_use/_search" -d '{
    "size": 0,
    "aggs": {
    "by_date": {
    "terms": { "field": "date" },
    "aggs": {
    "laundry_stats": { "extended_stats": { "field": "laundry" } }
    }
    }
    }
    }'

    View full-size slide

  111. Bucket Aggregations
    • Date Histogram
    • Term/Terms
    • Geo*
    • Significant Terms

    View full-size slide

  112. Questions?
    Use Cases?
    Exploration Ideas?
    https://joind.in/talk/e2e4b

    View full-size slide