Taming Your Data with Elasticsearch - PHP Benelux

D12eaf3ef46e4f0fc6b714fd2b7ffe3b?s=47 derek-b
January 25, 2019

Taming Your Data with Elasticsearch - PHP Benelux

Are you searching unstructured data or text fields? Do you need to aggregate and summarize your geo, financial, or other numeric data? Do you want to query your structured data in new and exciting ways? If so, Elasticsearch may be right for you. Let’s explore the many ways you can ask questions about your data and have it make sense to you and your users. We’ll sort through millions of rows in milliseconds and give you tools to take your data analysis to the next level. You will learn how to use PHP libraries and basic RESTful API calls to store, search, and aggregate your data.

D12eaf3ef46e4f0fc6b714fd2b7ffe3b?s=128

derek-b

January 25, 2019
Tweet

Transcript

  1. @DerekB_WI binkleyd@yahoo.com Taming Your Data with Elasticsearch

  2. Hello! I Am Derek Binkley Senior Engineer with TurnTo Networks

    Volunteer with Community Justice @DerekB_WI binkleyd@yahoo.com
  3. Customer Generated Content

  4. @DerekB_WI binkleyd@yahoo.com Fast Searching Scalability Finding Value within a Sea

    of Data
  5. @DerekB_WI binkleyd@yahoo.com What is it? open-source, RESTful, distributed search and

    analytics engine built on Apache Lucene Elasticsearch Tool for querying and exploring data Kibana Beats and Logstash Tool for ingesting data from specific sources
  6. @DerekB_WI binkleyd@yahoo.com How is it stored? A grouping of JSON

    documents with similar structure. Index Defines what is contained in a document Mapping A JSON document stores each data element. Document
  7. @DerekB_WI binkleyd@yahoo.com Storing Data

  8. @DerekB_WI binkleyd@yahoo.com Store new document POST

  9. @DerekB_WI binkleyd@yahoo.com Specify ID to update or insert PUT

  10. @DerekB_WI binkleyd@yahoo.com Created automatically or manually Updated automatically Mapping

  11. @DerekB_WI binkleyd@yahoo.com Mapping

  12. @DerekB_WI binkleyd@yahoo.com Define empty index Setup document structure https:/ /www.elastic.co/guide/en/

    elasticsearch/reference/current/indices- put-mapping.html Put Mapping
  13. @DerekB_WI binkleyd@yahoo.com Storing Data with PHP

  14. @DerekB_WI binkleyd@yahoo.com Guzzle converts array to JSON body Put Mapping

  15. @DerekB_WI binkleyd@yahoo.com Guzzle converts array to JSON body Put Mapping

  16. @DerekB_WI binkleyd@yahoo.com Guzzle converts array to JSON body Post

  17. @DerekB_WI binkleyd@yahoo.com Update Data

  18. @DerekB_WI binkleyd@yahoo.com Automatically assigned - POST Manually assigned - PUT

    ID
  19. @DerekB_WI binkleyd@yahoo.com Replaces entire document if exists Adds new if

    not exists PUT DOC
  20. @DerekB_WI binkleyd@yahoo.com Only updates named fields Update Fields

  21. @DerekB_WI binkleyd@yahoo.com Painless scripting language Script Update

  22. @DerekB_WI binkleyd@yahoo.com Searching Data

  23. @DerekB_WI binkleyd@yahoo.com Define query in JSON body match_all finds everything

    Query Keyword
  24. @DerekB_WI binkleyd@yahoo.com Looking for best results Find a Match

  25. @DerekB_WI binkleyd@yahoo.com Results are scored Find a Match

  26. @DerekB_WI binkleyd@yahoo.com Results are scored Search Within Text

  27. @DerekB_WI binkleyd@yahoo.com Results are scored Search Within Text

  28. @DerekB_WI binkleyd@yahoo.com Damerau-Levenshtein Distance Fuzziness

  29. @DerekB_WI binkleyd@yahoo.com more_like_this query Similar Documents

  30. @DerekB_WI binkleyd@yahoo.com Suggest Word Suggestions

  31. @DerekB_WI binkleyd@yahoo.com Suggest Word Suggestions

  32. @DerekB_WI binkleyd@yahoo.com Paginating Data

  33. @DerekB_WI binkleyd@yahoo.com Skip 100 and limit results to 100. Skip

    Results
  34. @DerekB_WI binkleyd@yahoo.com Only for first 10,000 hits Skip Results Organized

    into shards Each shard is a Lucene index Move data around clusters
  35. @DerekB_WI binkleyd@yahoo.com Only stays open for specified time Scroll Through

    Results
  36. @DerekB_WI binkleyd@yahoo.com Keep track with _scroll_id Scroll Through Results

  37. @DerekB_WI binkleyd@yahoo.com POST to scroll endpoint for next results. Scroll

    Through Results
  38. @DerekB_WI binkleyd@yahoo.com Aggregating

  39. @DerekB_WI binkleyd@yahoo.com Query unique results or keywords What’s In a

    Field
  40. @DerekB_WI binkleyd@yahoo.com Query unique results or keywords that get sorted

    into “buckets” What’s In a Field
  41. @DerekB_WI binkleyd@yahoo.com Calculate summary values such as max, min, average

    Metrics
  42. @DerekB_WI binkleyd@yahoo.com Calculate summary values such as max, min, average

    Metrics
  43. @DerekB_WI binkleyd@yahoo.com Group documents into buckets Buckets with Metrics

  44. @DerekB_WI binkleyd@yahoo.com Group documents into buckets Buckets with Metrics

  45. @DerekB_WI binkleyd@yahoo.com Geo Points

  46. @DerekB_WI binkleyd@yahoo.com Complex mapping applications can be created by using

    four types of queries Uses GeoJSON to define shape GeoShape Define top_left and bottom_right Geo Bounding Box Geo searches Previous example Geo Distance Define points to create a polygon Geo Polygon
  47. @DerekB_WI binkleyd@yahoo.com Find results with a distance of a point

    Distance Search
  48. @DerekB_WI binkleyd@yahoo.com Filter by geo, aggregate by term Distance Aggregation

  49. @DerekB_WI binkleyd@yahoo.com Filter by geo, aggregate by term Distance Aggregation

  50. @DerekB_WI binkleyd@yahoo.com Sort by distance Distance Sort

  51. @DerekB_WI binkleyd@yahoo.com Sort by distance Distance Sort

  52. @DerekB_WI binkleyd@yahoo.com Keeping in Sync

  53. @DerekB_WI binkleyd@yahoo.com Elasticsearch is read and search optimized at the

    expense of expensive writes Use batch API to insert many records Batches Strategy for queuing up data for batching Message Queues Sync with database Batch by range Ranges of data
  54. @DerekB_WI binkleyd@yahoo.com Cannot update mapping manually Must setup destination index

    Reindex mapping
  55. @DerekB_WI binkleyd@yahoo.com Can use alias to help with cutover Reindex

    mapping
  56. @DerekB_WI binkleyd@yahoo.com ANY QUESTIONS? You can find me at @DerekB_WI

    binkleyd@yahoo.com derekb-wi.com Thanks!
  57. @DerekB_WI binkleyd@yahoo.com https:/ /joind.in/talk/5cced THANKS!

  58. @DerekB_WI binkleyd@yahoo.com https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up https://en.wikipedia.org/wiki/Damerau-Levenshtein_distance https://lucene.apache.org/ https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-getting-started.html http://geojson.org/ Resources https://www.elastic.co/blog/found-keeping-elasticsearch-in-sync https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html