Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Add Location-Based Searching to Your Site with Elasticsearch

derek-b
November 30, 2020
17

Add Location-Based Searching to Your Site with Elasticsearch

In this talk, we'll take an in-depth look at how Elasticsearch supports geo-searching. We'll look at the different types of location searches and how to integrate them into your application. By the end of this talk, you will be able to add location-based searches that will impress your users.

derek-b

November 30, 2020
Tweet

Transcript

  1. @DerekB_WI
    Add Location-Based
    Searching to Your Site
    with Elasticsearch
    Presented by Derek Binkley

    View Slide

  2. @DerekB_WI
    Elasticsearch
    • Apache Lucene

    • REST calls

    • SDKs in many languages

    • Read optimized

    View Slide

  3. @DerekB_WI
    BELK Stack
    • Beats - Ingests data streams

    • Elasticsearch - You know for search

    • Logstash - Prebuilt dashboard for visualizing logs

    • Kibana - Browser based client

    View Slide

  4. @DerekB_WI
    Running
    • AWS - prebuilt

    • Hosted by Elastic

    • Install on OS

    • Docker *

    View Slide

  5. @DerekB_WI
    What Are We Building
    Today?

    View Slide

  6. @DerekB_WI
    WeAreDevelopers Project
    • Search text descriptions of locations

    • Search nearby locations

    • Add locations

    • Comment on existing locations

    • API

    View Slide

  7. @DerekB_WI
    docker-compose.yml
    version: '3'

    services:

    elasticsearch:

    image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2

    environment:

    - cluster.name=docker-cluster

    - bootstrap.memory_lock=true

    - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

    ulimits:

    memlock:

    soft: -1

    hard: -1

    ports:

    - "9200:9200"

    kibana:

    image: docker.elastic.co/kibana/kibana:6.3.2

    ports:

    - "5601:5601"

    View Slide

  8. @DerekB_WI
    How is it stored?
    • Document - A JSON document for each data element

    • Index - A grouping of documents with similar structure

    • Mapping - Defines what is contained in document

    View Slide

  9. Define our data

    View Slide

  10. Define our data

    View Slide

  11. @DerekB_WI
    Data Types
    • Text - Content is indexed and searchable

    • Keyword - Fixed content, think lookup table

    • Geo-point - latitude/longitude

    • Geo-shape - Define a polygon

    • long, integer, short, byte, double, float, half_float, scaled_floa
    t, date, date_nanos, boolean, binary

    • More at https://www.elastic.co/guide/en/elasticsearch/
    reference/current/mapping-types.html

    View Slide

  12. View Slide

  13. @DerekB_WI
    Queries
    GET location-list/location/_search

    {

    "query": {

    "match_all": {}

    }

    }

    View Slide

  14. @DerekB_WI
    Queries
    GET location-list/location/_search

    {

    "query": {

    "match": {"name": "Big River"}

    }

    }

    View Slide

  15. @DerekB_WI

    View Slide

  16. @DerekB_WI
    Queries
    GET location-list/location/_search

    {

    "query": {

    "bool": {

    "filter": {

    "term": {"feature_class": "L"}

    }

    }

    }

    }

    https://www.geonames.org/export/codes.html

    View Slide

  17. @DerekB_WI
    Queries
    GET location-list/location/_search

    {

    "query": {

    "bool": {

    "must":{

    "match": {"name":"Little Park"}

    },

    "filter": {

    "term": {"feature_class": "L"}

    }

    }

    }

    }

    View Slide

  18. View Slide

  19. @DerekB_WI
    Aggregations
    GET location-list/location/_search

    {

    "size": 0,

    "aggs": {

    "uniq_feature": {

    "terms": {"field":"feature_class"}

    }

    }

    }

    View Slide

  20. View Slide

  21. @DerekB_WI
    Benefits?
    • Fast: > 1 mil rows in 20ms

    • Search within text

    • Other search types, geo, more like this

    View Slide

  22. @DerekB_WI
    Load Data
    public function addLocation(Location $location)

    {

    $this->locations[] = $location;


    if (count($this->locations) >= 1000)

    {

    $this->repository

    ->bulkInsert('location-list', 'location', 

    $this->prepareBulkJson($this->locations));

    $this->locations = [];

    }

    }

    View Slide

  23. @DerekB_WI
    Load Data
    private function prepareBulkJson()

    {

    return array_map(function($location) {

    return [

    'geonameid' => $location->geonameid,

    'name' => $location->name,

    'asciiname' => $location->asciiname,

    'alternatenames' => $location->alternatenames,

    'location_point' => $location->latitude . ", "
    $location->longitude,

    'feature_class' => $location->featureClass,

    'feature_code' => $location->featureCode,

    'elevation' => $location->elevation,

    'timezone' => $location->timezone

    ];

    }, $this->locations);


    View Slide

  24. @DerekB_WI
    Load Data
    public function bulkInsert($indexName, $indexType, $data_lines)

    {

    $client = ClientBuilder::create()

    ->setHosts(['127.0.0.1:9200'])

    ->build();

    $json_body = [];

    foreach ($data_lines as $line)

    {

    $json_body[] = ['index' => [

    '_index' => $indexName,

    '_type' => $indexType

    ]

    ];

    $json_body[] = $line;

    }

    $responses = $client->bulk(['body' => $json_body]);

    }

    View Slide

  25. @DerekB_WI
    Load Data
    public function bulkInsert($indexName, $indexType, $data_lines)

    {

    $client = ClientBuilder::create()

    ->setHosts(['127.0.0.1:9200'])

    ->build();

    $json_body = [];

    foreach ($data_lines as $line)

    {

    $json_body[] = ['index' => ['_index' => $indexName,

    '_type' => $indexType]

    ];

    $json_body[] = $line;

    }

    $responses = $client->bulk(['body' => $json_body]);

    }

    View Slide

  26. @DerekB_WI
    Load Data
    public function bulkInsert($indexName, $indexType, $data_lines)

    {

    $client = ClientBuilder::create()

    ->setHosts(['127.0.0.1:9200'])

    ->build();

    $json_body = [];

    foreach ($data_lines as $line)

    {

    $json_body[] = ['index' => ['_index' => $indexName,

    '_type' => $indexType]

    ];

    $json_body[] = $line;

    }

    $responses = $client->bulk(['body' => $json_body]);

    }

    View Slide

  27. View Slide

  28. @DerekB_WI
    Search by Distance

    View Slide

  29. @DerekB_WI
    Text and Distance

    View Slide

  30. @DerekB_WI
    Closest Match

    View Slide

  31. @DerekB_WI
    Draw a Map

    View Slide

  32. View Slide

  33. @DerekB_WI
    Add to Mapping

    View Slide

  34. View Slide

  35. @DerekB_WI
    Thank You!

    View Slide