Taming Your Data with Elasticsearch - PHP Benelux

@DerekB_WI [email protected] Taming Your Data with Elasticsearch

Hello! I Am Derek Binkley Senior Engineer with TurnTo Networks
Volunteer with Community Justice @DerekB_WI [email protected]

Customer Generated Content

@DerekB_WI [email protected] Fast Searching Scalability Finding Value within a Sea
of Data

@DerekB_WI [email protected] What is it? open-source, RESTful, distributed search and
analytics engine built on Apache Lucene Elasticsearch Tool for querying and exploring data Kibana Beats and Logstash Tool for ingesting data from specific sources

@DerekB_WI [email protected] How is it stored? A grouping of JSON
documents with similar structure. Index Defines what is contained in a document Mapping A JSON document stores each data element. Document

@DerekB_WI [email protected] Storing Data

@DerekB_WI [email protected] Store new document POST

@DerekB_WI [email protected] Specify ID to update or insert PUT

@DerekB_WI [email protected] Created automatically or manually Updated automatically Mapping

@DerekB_WI [email protected] Mapping

@DerekB_WI [email protected] Define empty index Setup document structure https:/ /www.elastic.co/guide/en/
elasticsearch/reference/current/indices- put-mapping.html Put Mapping

@DerekB_WI [email protected] Storing Data with PHP

@DerekB_WI [email protected] Guzzle converts array to JSON body Put Mapping

@DerekB_WI [email protected] Guzzle converts array to JSON body Post

@DerekB_WI [email protected] Update Data

@DerekB_WI [email protected] Automatically assigned - POST Manually assigned - PUT
ID

@DerekB_WI [email protected] Replaces entire document if exists Adds new if
not exists PUT DOC

@DerekB_WI [email protected] Only updates named fields Update Fields

@DerekB_WI [email protected] Painless scripting language Script Update

@DerekB_WI [email protected] Searching Data

@DerekB_WI [email protected] Define query in JSON body match_all finds everything
Query Keyword

@DerekB_WI [email protected] Looking for best results Find a Match

@DerekB_WI [email protected] Results are scored Find a Match

@DerekB_WI [email protected] Results are scored Search Within Text

@DerekB_WI [email protected] Damerau-Levenshtein Distance Fuzziness

@DerekB_WI [email protected] more_like_this query Similar Documents

@DerekB_WI [email protected] Suggest Word Suggestions

@DerekB_WI [email protected] Paginating Data

@DerekB_WI [email protected] Skip 100 and limit results to 100. Skip
Results

@DerekB_WI [email protected] Only for first 10,000 hits Skip Results Organized
into shards Each shard is a Lucene index Move data around clusters

@DerekB_WI [email protected] Only stays open for specified time Scroll Through
Results

@DerekB_WI [email protected] Keep track with _scroll_id Scroll Through Results

@DerekB_WI [email protected] POST to scroll endpoint for next results. Scroll
Through Results

@DerekB_WI [email protected] Aggregating

@DerekB_WI [email protected] Query unique results or keywords What’s In a
Field

@DerekB_WI [email protected] Query unique results or keywords that get sorted
into “buckets” What’s In a Field

@DerekB_WI [email protected] Calculate summary values such as max, min, average
Metrics

@DerekB_WI [email protected] Group documents into buckets Buckets with Metrics

@DerekB_WI [email protected] Geo Points

@DerekB_WI [email protected] Complex mapping applications can be created by using
four types of queries Uses GeoJSON to define shape GeoShape Define top_left and bottom_right Geo Bounding Box Geo searches Previous example Geo Distance Define points to create a polygon Geo Polygon

@DerekB_WI [email protected] Find results with a distance of a point
Distance Search

@DerekB_WI [email protected] Filter by geo, aggregate by term Distance Aggregation

@DerekB_WI [email protected] Sort by distance Distance Sort

@DerekB_WI [email protected] Keeping in Sync

@DerekB_WI [email protected] Elasticsearch is read and search optimized at the
expense of expensive writes Use batch API to insert many records Batches Strategy for queuing up data for batching Message Queues Sync with database Batch by range Ranges of data

@DerekB_WI [email protected] Cannot update mapping manually Must setup destination index
Reindex mapping

@DerekB_WI [email protected] Can use alias to help with cutover Reindex
mapping

@DerekB_WI [email protected] ANY QUESTIONS? You can find me at @DerekB_WI
[email protected] derekb-wi.com Thanks!

@DerekB_WI [email protected] https:/ /joind.in/talk/5cced THANKS!

@DerekB_WI [email protected] https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up https://en.wikipedia.org/wiki/Damerau-Levenshtein_distance https://lucene.apache.org/ https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-getting-started.html http://geojson.org/ Resources https://www.elastic.co/blog/found-keeping-elasticsearch-in-sync https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

Taming Your Data with Elasticsearch - PHP Benelux

Taming Your Data with Elasticsearch - PHP Benelux

More Decks by derek-b

Other Decks in Technology

Featured

Transcript