Add Location-Based Searching to Your Site with Elasticsearch

In this talk, we'll take an in-depth look at how Elasticsearch supports geo-searching. We'll look at the different types of location searches and how to integrate them into your application. By the end of this talk, you will be able to add location-based searches that will impress your users.

Transcript

1. @DerekB_WI Add Location-Based Searching to Your Site with Elasticsearch Presented by Derek Binkley
2. @DerekB_WI Elasticsearch • Apache Lucene • REST calls • SDKs in many languages • Read optimized
3. @DerekB_WI BELK Stack • Beats - Ingests data streams • Elasticsearch - You know for search • Logstash - Prebuilt dashboard for visualizing logs • Kibana - Browser based client
4. @DerekB_WI Running • AWS - prebuilt • Hosted by Elastic • Install on OS • Docker *
5. @DerekB_WI What Are We Building Today?
6. @DerekB_WI WeAreDevelopers Project • Search text descriptions of locations • Search nearby locations • Add locations • Comment on existing locations • API
7. @DerekB_WI docker-compose.yml version: '3'
 services:
 elasticsearch:
 image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
 environment:
 - cluster.name=docker-cluster
 - bootstrap.memory_lock=true
 - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
 ulimits:
 memlock:
 soft: -1
 hard: -1
 ports:
 - "9200:9200"
 kibana:
 image: docker.elastic.co/kibana/kibana:6.3.2
 ports:
 - "5601:5601"
8. @DerekB_WI How is it stored? • Document - A JSON document for each data element • Index - A grouping of documents with similar structure • Mapping - Defines what is contained in document
9. Define our data
10. Define our data
11. @DerekB_WI Data Types • Text - Content is indexed and searchable • Keyword - Fixed content, think lookup table • Geo-point - latitude/longitude • Geo-shape - Define a polygon • long, integer, short, byte, double, float, half_float, scaled_floa t, date, date_nanos, boolean, binary • More at https://www.elastic.co/guide/en/elasticsearch/ reference/current/mapping-types.html
13. @DerekB_WI Queries GET location-list/location/_search
 {
 "query": {
 "match_all": {}
 }
 }
14. @DerekB_WI Queries GET location-list/location/_search
 {
 "query": {
 "match": {"name": "Big River"}
 }
 }
15. @DerekB_WI
16. @DerekB_WI Queries GET location-list/location/_search
 {
 "query": {
 "bool": {
 "filter": {
 "term": {"feature_class": "L"}
 }
 }
 }
 } https://www.geonames.org/export/codes.html
17. @DerekB_WI Queries GET location-list/location/_search
 {
 "query": {
 "bool": {
 "must":{
 "match": {"name":"Little Park"}
 },
 "filter": {
 "term": {"feature_class": "L"}
 }
 }
 }
 }
19. @DerekB_WI Aggregations GET location-list/location/_search
 {
 "size": 0,
 "aggs": {
 "uniq_feature": {
 "terms": {"field":"feature_class"}
 }
 }
 }
21. @DerekB_WI Benefits? • Fast: > 1 mil rows in 20ms • Search within text • Other search types, geo, more like this
22. @DerekB_WI Load Data public function addLocation(Location $location)
 {
 $this->locations[] = $location;
 
 if (count($this->locations) >= 1000)
 {
 $this->repository
 ->bulkInsert('location-list', 'location', 
 $this->prepareBulkJson($this->locations)); $this->locations = [];
 } }
23. @DerekB_WI Load Data private function prepareBulkJson()
 {
 return array_map(function($location) {
 return [
 'geonameid' => $location->geonameid,
 'name' => $location->name,
 'asciiname' => $location->asciiname,
 'alternatenames' => $location->alternatenames,
 'location_point' => $location->latitude . ", " $location->longitude,
 'feature_class' => $location->featureClass,
 'feature_code' => $location->featureCode,
 'elevation' => $location->elevation,
 'timezone' => $location->timezone
 ];
 }, $this->locations);

24. @DerekB_WI Load Data public function bulkInsert($indexName, $indexType, $data_lines) { $client = ClientBuilder::create() ->setHosts(['127.0.0.1:9200']) ->build(); $json_body = []; foreach ($data_lines as $line) { $json_body[] = ['index' => [ '_index' => $indexName, '_type' => $indexType ] ]; $json_body[] = $line; } $responses = $client->bulk(['body' => $json_body]); }
25. @DerekB_WI Load Data public function bulkInsert($indexName, $indexType, $data_lines) { $client = ClientBuilder::create() ->setHosts(['127.0.0.1:9200']) ->build(); $json_body = []; foreach ($data_lines as $line) { $json_body[] = ['index' => ['_index' => $indexName, '_type' => $indexType] ]; $json_body[] = $line; } $responses = $client->bulk(['body' => $json_body]); }
26. @DerekB_WI Load Data public function bulkInsert($indexName, $indexType, $data_lines) { $client = ClientBuilder::create() ->setHosts(['127.0.0.1:9200']) ->build(); $json_body = []; foreach ($data_lines as $line) { $json_body[] = ['index' => ['_index' => $indexName, '_type' => $indexType] ]; $json_body[] = $line; } $responses = $client->bulk(['body' => $json_body]); }
28. @DerekB_WI Search by Distance
29. @DerekB_WI Text and Distance
30. @DerekB_WI Closest Match
31. @DerekB_WI Draw a Map
33. @DerekB_WI Add to Mapping
35. @DerekB_WI Thank You!