Add Location-Based Searching to Your Site with Elasticsearch

In this talk, we'll take an in-depth look at how Elasticsearch supports geo-searching. We'll look at the different types of location searches and how to integrate them into your application. By the end of this talk, you will be able to add location-based searches that will impress your users.

Transcript

1. @DerekB_WI
   Add Location-Based
   Searching to Your Site
   with Elasticsearch
   Presented by Derek Binkley
   

   View Slide

2. @DerekB_WI
   Elasticsearch
   • Apache Lucene
   
   • REST calls
   
   • SDKs in many languages
   
   • Read optimized
   

   View Slide

3. @DerekB_WI
   BELK Stack
   • Beats - Ingests data streams
   
   • Elasticsearch - You know for search
   
   • Logstash - Prebuilt dashboard for visualizing logs
   
   • Kibana - Browser based client
   

   View Slide

4. @DerekB_WI
   Running
   • AWS - prebuilt
   
   • Hosted by Elastic
   
   • Install on OS
   
   • Docker *
   

   View Slide

5. @DerekB_WI
   What Are We Building
   Today?
   

   View Slide

6. @DerekB_WI
   WeAreDevelopers Project
   • Search text descriptions of locations
   
   • Search nearby locations
   
   • Add locations
   
   • Comment on existing locations
   
   • API
   

   View Slide

7. @DerekB_WI
   docker-compose.yml
   version: '3'

   services:

   elasticsearch:

   image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2

   environment:

   - cluster.name=docker-cluster

   - bootstrap.memory_lock=true

   - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

   ulimits:

   memlock:

   soft: -1

   hard: -1

   ports:

   - "9200:9200"

   kibana:

   image: docker.elastic.co/kibana/kibana:6.3.2

   ports:

   - "5601:5601"
   
   

   View Slide

8. @DerekB_WI
   How is it stored?
   • Document - A JSON document for each data element
   
   • Index - A grouping of documents with similar structure
   
   • Mapping - Defines what is contained in document
   

   View Slide

9. Define our data
   

   View Slide

10. Define our data
    

    View Slide

11. @DerekB_WI
    Data Types
    • Text - Content is indexed and searchable
    
    • Keyword - Fixed content, think lookup table
    
    • Geo-point - latitude/longitude
    
    • Geo-shape - Define a polygon
    
    • long, integer, short, byte, double, float, half_float, scaled_floa
    t, date, date_nanos, boolean, binary
    
    • More at https://www.elastic.co/guide/en/elasticsearch/
    reference/current/mapping-types.html
    

    View Slide

12. View Slide

13. @DerekB_WI
    Queries
    GET location-list/location/_search

    {

    "query": {

    "match_all": {}

    }

    }
    

    View Slide

14. @DerekB_WI
    Queries
    GET location-list/location/_search

    {

    "query": {

    "match": {"name": "Big River"}

    }

    }
    
    

    View Slide

15. @DerekB_WI
    

    View Slide

16. @DerekB_WI
    Queries
    GET location-list/location/_search

    {

    "query": {

    "bool": {

    "filter": {

    "term": {"feature_class": "L"}

    }

    }

    }

    }
    
    https://www.geonames.org/export/codes.html
    

    View Slide

17. @DerekB_WI
    Queries
    GET location-list/location/_search

    {

    "query": {

    "bool": {

    "must":{

    "match": {"name":"Little Park"}

    },

    "filter": {

    "term": {"feature_class": "L"}

    }

    }

    }

    }
    

    View Slide

18. View Slide

19. @DerekB_WI
    Aggregations
    GET location-list/location/_search

    {

    "size": 0,

    "aggs": {

    "uniq_feature": {

    "terms": {"field":"feature_class"}

    }

    }

    }
    

    View Slide

20. View Slide

21. @DerekB_WI
    Benefits?
    • Fast: > 1 mil rows in 20ms
    
    • Search within text
    
    • Other search types, geo, more like this
    

    View Slide

22. @DerekB_WI
    Load Data
    public function addLocation(Location $location)

    {

    $this->locations[] = $location;

    

    if (count($this->locations) >= 1000)

    {

    $this->repository

    ->bulkInsert('location-list', 'location', 

    $this->prepareBulkJson($this->locations));
    
    $this->locations = [];

    }
    
    }
    
    

    View Slide

23. @DerekB_WI
    Load Data
    private function prepareBulkJson()

    {

    return array_map(function($location) {

    return [

    'geonameid' => $location->geonameid,

    'name' => $location->name,

    'asciiname' => $location->asciiname,

    'alternatenames' => $location->alternatenames,

    'location_point' => $location->latitude . ", "
    $location->longitude,

    'feature_class' => $location->featureClass,

    'feature_code' => $location->featureCode,

    'elevation' => $location->elevation,

    'timezone' => $location->timezone

    ];

    }, $this->locations);

    

    View Slide

24. @DerekB_WI
    Load Data
    public function bulkInsert($indexName, $indexType, $data_lines)
    
    {
    
    $client = ClientBuilder::create()
    
    ->setHosts(['127.0.0.1:9200'])
    
    ->build();
    
    $json_body = [];
    
    foreach ($data_lines as $line)
    
    {
    
    $json_body[] = ['index' => [
    
    '_index' => $indexName,
    
    '_type' => $indexType
    
    ]
    
    ];
    
    $json_body[] = $line;
    
    }
    
    $responses = $client->bulk(['body' => $json_body]);
    
    }
    
    

    View Slide

25. @DerekB_WI
    Load Data
    public function bulkInsert($indexName, $indexType, $data_lines)
    
    {
    
    $client = ClientBuilder::create()
    
    ->setHosts(['127.0.0.1:9200'])
    
    ->build();
    
    $json_body = [];
    
    foreach ($data_lines as $line)
    
    {
    
    $json_body[] = ['index' => ['_index' => $indexName,
    
    '_type' => $indexType]
    
    ];
    
    $json_body[] = $line;
    
    }
    
    $responses = $client->bulk(['body' => $json_body]);
    
    }
    
    

    View Slide

26. @DerekB_WI
    Load Data
    public function bulkInsert($indexName, $indexType, $data_lines)
    
    {
    
    $client = ClientBuilder::create()
    
    ->setHosts(['127.0.0.1:9200'])
    
    ->build();
    
    $json_body = [];
    
    foreach ($data_lines as $line)
    
    {
    
    $json_body[] = ['index' => ['_index' => $indexName,
    
    '_type' => $indexType]
    
    ];
    
    $json_body[] = $line;
    
    }
    
    $responses = $client->bulk(['body' => $json_body]);
    
    }
    
    

    View Slide

27. View Slide

28. @DerekB_WI
    Search by Distance
    

    View Slide

29. @DerekB_WI
    Text and Distance
    

    View Slide

30. @DerekB_WI
    Closest Match
    

    View Slide

31. @DerekB_WI
    Draw a Map
    

    View Slide

32. View Slide

33. @DerekB_WI
    Add to Mapping
    

    View Slide

34. View Slide

35. @DerekB_WI
    Thank You!
    

    View Slide