Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Taming Your Data with Elasticsearch - PHP Benelux

derek-b
January 25, 2019
Technology
1
290

Taming Your Data with Elasticsearch - PHP Benelux

Are you searching unstructured data or text fields? Do you need to aggregate and summarize your geo, financial, or other numeric data? Do you want to query your structured data in new and exciting ways? If so, Elasticsearch may be right for you. Let’s explore the many ways you can ask questions about your data and have it make sense to you and your users. We’ll sort through millions of rows in milliseconds and give you tools to take your data analysis to the next level. You will learn how to use PHP libraries and basic RESTful API calls to store, search, and aggregate your data.

derek-b

January 25, 2019
Tweet

More Decks by derek-b

See All by derek-b
Pardon the disruption: a user-first approach to software design
derekb
0
30
Teaching Through Code Review
derekb
0
85
Add Location-Based Searching to Your Site with Elasticsearch
derekb
0
17
Add Location-based Searching to Your PHP App with Elasticsearch
derekb
1
160
Map, Filter, and Reduce in PHP
derekb
0
330
You Version Your Code, Why Not Your Database?
derekb
0
48
What Raising 3 Kids Taught Me About Working With Users
derekb
0
180
Taming Your Data With Elasticsearch - PHP[world]
derekb
0
180
Browser Automation Testing with Headless Chrome
derekb
0
64

Other Decks in Technology

See All in Technology
応用から学ぶ強化学習
nearme_tech
0
210
Microsoft Build 2023 ふりかえり - IoT と AI 周りを中心に
mode86
0
160
「エンジニアリングで運用を改善する」ためのAmazon CloudWatch活用
mitsuaki96
1
870
Oracle Exadata Database Service on [email protected] ([email protected]) - UI スクリーン・キャプチャ集
oracle4engineer
PRO
0
430
『家族アルバム みてね』 AWSマルチリージョン構成における データベース運用 / FamilyAlbum Database in AWS multi-region
kohbis
1
180
安心して現場で学習できる環境として Flutter Web の管理サイトがおすすめな話
htsuruo
5
140
javapを使ってクラスファイルを読んでみよう！
yujisoftware
1
190
過去事例から見るモニタリングの大切さ #techfeed_live
whywaita
PRO
0
160
Exadata Database Cloud (Exadata Cloud Service) サービス技術詳細
oracle4engineer
PRO
0
29k
copilot-cli(Fargate)でRailsを運用するためのTips / JAWS-UG Okayama 2023
interu
1
360
MagicPod開発における テスト自動化とCI
nozomiito
0
150
net/http/httptest.Server のアプローチをテスト戦略に活用する / Go Conference 2023
k1low
7
1.1k

Featured

See All Featured
Debugging Ruby Performance
tmm1
68
11k
Web Components: a chance to create the future
zenorocha
304
41k
How to train your dragon (web standard)
notwaldorf
69
4.4k
Robots, Beer and Maslow
schacon
154
7.5k
Designing for Performance
lara
601
65k
YesSQL, Process and Tooling at Scale
rocio
158
13k
The Brand Is Dead. Long Live the Brand.
mthomps
48
4.1k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
16
5.8k
Git: the NoSQL Database
bkeepers
PRO
420
60k
What's in a price? How to price your products and services
michaelherold
234
10k
Three Pipe Problems
jasonvnalue
89
9.1k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
16
1.3k

Transcript

  1. @DerekB_WI [email protected]
    Taming Your Data
    with Elasticsearch

    View Slide

  2. Hello! I Am Derek Binkley
    Senior Engineer with TurnTo Networks
    Volunteer with Community Justice
    @DerekB_WI [email protected]

    View Slide

  3. Customer Generated Content

    View Slide

  4. @DerekB_WI [email protected]
    Fast Searching
    Scalability
    Finding Value within
    a Sea of Data

    View Slide

  5. @DerekB_WI [email protected]
    What is it?
    open-source, RESTful, distributed
    search and analytics engine built
    on Apache Lucene
    Elasticsearch
    Tool for querying and exploring
    data
    Kibana
    Beats and Logstash
    Tool for ingesting data from
    specific sources

    View Slide

  6. @DerekB_WI [email protected]
    How is it stored?
    A grouping of JSON documents
    with similar structure.
    Index
    Defines what is contained in a
    document
    Mapping
    A JSON document stores each
    data element.
    Document

    View Slide

  7. @DerekB_WI [email protected]
    Storing Data

    View Slide

  8. @DerekB_WI [email protected]
    Store new document
    POST

    View Slide

  9. @DerekB_WI [email protected]
    Specify ID to update or insert
    PUT

    View Slide

  10. @DerekB_WI [email protected]
    Created automatically or manually
    Updated automatically
    Mapping

    View Slide

  11. @DerekB_WI [email protected]
    Mapping

    View Slide

  12. @DerekB_WI [email protected]
    Define empty index
    Setup document structure
    https:/
    /www.elastic.co/guide/en/
    elasticsearch/reference/current/indices-
    put-mapping.html
    Put Mapping

    View Slide

  13. @DerekB_WI [email protected]
    Storing Data with PHP

    View Slide

  14. @DerekB_WI [email protected]
    Guzzle converts array to
    JSON body
    Put Mapping

    View Slide

  15. @DerekB_WI [email protected]
    Guzzle converts array to
    JSON body
    Put Mapping

    View Slide

  16. @DerekB_WI [email protected]
    Guzzle converts array to
    JSON body
    Post

    View Slide

  17. @DerekB_WI [email protected]
    Update Data

    View Slide

  18. @DerekB_WI [email protected]
    Automatically assigned - POST
    Manually assigned - PUT
    ID

    View Slide

  19. @DerekB_WI [email protected]
    Replaces entire document if exists
    Adds new if not exists
    PUT DOC

    View Slide

  20. @DerekB_WI [email protected]
    Only updates named fields
    Update Fields

    View Slide

  21. @DerekB_WI [email protected]
    Painless scripting language
    Script Update

    View Slide

  22. @DerekB_WI [email protected]
    Searching Data

    View Slide

  23. @DerekB_WI [email protected]
    Define query in JSON body
    match_all finds everything
    Query Keyword

    View Slide

  24. @DerekB_WI [email protected]
    Looking for best results
    Find a Match

    View Slide

  25. @DerekB_WI [email protected]
    Results are scored
    Find a Match

    View Slide

  26. @DerekB_WI [email protected]
    Results are scored
    Search Within Text

    View Slide

  27. @DerekB_WI [email protected]
    Results are scored
    Search Within Text

    View Slide

  28. @DerekB_WI [email protected]
    Damerau-Levenshtein
    Distance
    Fuzziness

    View Slide

  29. @DerekB_WI [email protected]
    more_like_this query
    Similar Documents

    View Slide

  30. @DerekB_WI [email protected]
    Suggest
    Word Suggestions

    View Slide

  31. @DerekB_WI [email protected]om
    Suggest
    Word Suggestions

    View Slide

  32. @DerekB_WI [email protected]
    Paginating Data

    View Slide

  33. @DerekB_WI [email protected]
    Skip 100 and limit results to
    100.
    Skip Results

    View Slide

  34. @DerekB_WI [email protected]
    Only for first 10,000 hits
    Skip Results Organized into shards
    Each shard is a Lucene index
    Move data around clusters

    View Slide

  35. @DerekB_WI [email protected]
    Only stays open for specified time
    Scroll Through Results

    View Slide

  36. @DerekB_WI [email protected]
    Keep track with _scroll_id
    Scroll Through Results

    View Slide

  37. @DerekB_WI [email protected]
    POST to scroll endpoint for next
    results.
    Scroll Through Results

    View Slide

  38. @DerekB_WI [email protected]
    Aggregating

    View Slide

  39. @DerekB_WI [email protected]
    Query unique results or keywords
    What’s In a Field

    View Slide

  40. @DerekB_WI [email protected]
    Query unique results or keywords that
    get sorted into “buckets”
    What’s In a Field

    View Slide

  41. @DerekB_WI [email protected]
    Calculate summary
    values such as max, min,
    average
    Metrics

    View Slide

  42. @DerekB_WI [email protected]
    Calculate summary
    values such as max, min,
    average
    Metrics

    View Slide

  43. @DerekB_WI [email protected]
    Group documents into buckets
    Buckets with Metrics

    View Slide

  44. @DerekB_WI [email protected]
    Group documents into buckets
    Buckets with Metrics

    View Slide

  45. @DerekB_WI [email protected]
    Geo Points

    View Slide

  46. @DerekB_WI [email protected]
    Complex mapping
    applications can be created
    by using four types of
    queries
    Uses GeoJSON to define shape
    GeoShape
    Define top_left and bottom_right
    Geo Bounding Box
    Geo searches
    Previous example
    Geo Distance
    Define points to create a polygon
    Geo Polygon

    View Slide

  47. @DerekB_WI [email protected]
    Find results with a distance of a point
    Distance Search

    View Slide

  48. @DerekB_WI [email protected]
    Filter by geo, aggregate by term
    Distance Aggregation

    View Slide

  49. @DerekB_WI [email protected]
    Filter by geo, aggregate by term
    Distance Aggregation

    View Slide

  50. @DerekB_WI [email protected]
    Sort by distance
    Distance Sort

    View Slide

  51. @DerekB_WI [email protected]
    Sort by distance
    Distance Sort

    View Slide

  52. @DerekB_WI [email protected]
    Keeping in Sync

    View Slide

  53. @DerekB_WI [email protected]
    Elasticsearch is read and
    search optimized at the
    expense of expensive writes
    Use batch API to insert many records
    Batches
    Strategy for queuing up data for batching
    Message Queues
    Sync with database
    Batch by range
    Ranges of data

    View Slide

  54. @DerekB_WI [email protected]
    Cannot update mapping manually
    Must setup destination index
    Reindex mapping

    View Slide

  55. @DerekB_WI [email protected]
    Can use alias to help with cutover
    Reindex mapping

    View Slide

  56. @DerekB_WI [email protected]
    ANY QUESTIONS?
    You can find me at
    @DerekB_WI
    [email protected]
    derekb-wi.com
    Thanks!

    View Slide

  57. @DerekB_WI [email protected]
    https:/
    /joind.in/talk/5cced
    THANKS!

    View Slide

  58. @DerekB_WI [email protected]
    https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up
    https://en.wikipedia.org/wiki/Damerau-Levenshtein_distance
    https://lucene.apache.org/
    https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-getting-started.html
    http://geojson.org/
    Resources
    https://www.elastic.co/blog/found-keeping-elasticsearch-in-sync
    https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

    View Slide