@DerekB_WI Location Based Searching with ElasticSearch

@DerekB_WI ElasticSearch • Apache Lucene • REST calls • SDKs in many languages • Read optimized

@DerekB_WI BELK Stack • Beats - Ingests data streams • Elasticsearch - You know for search • Logstash - Prebuilt dashboard for visualizing logs • Kibana - Browser based client

@DerekB_WI Running • AWS - prebuilt • Hosted by Elastic • Install on OS • Docker *

@DerekB_WI What Are We Building Today?

@DerekB_WI Project • Search nearby locations • Add locations • Comment on existing locations • API

@DerekB_WI PHP Client $ composer require elasticsearch/elasticsearch:7.4.*

@DerekB_WI docker-compose.yml version: '3'
 services:
 elasticsearch:
 image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
 environment:
 - cluster.name=docker-cluster
 - bootstrap.memory_lock=true
 - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
 ulimits:
 memlock:
 soft: -1
 hard: -1
 ports:
 - "9200:9200"
 kibana:
 image: docker.elastic.co/kibana/kibana:6.3.2
 ports:
 - "5601:5601"

@DerekB_WI How is it stored? • Document - A JSON document for each data element • Index - A grouping of documents with similar structure • Mapping - Defines what is contained in document

Define our data

Define our data

@DerekB_WI Data Types • Text - Content is indexed and searchable • Keyword - Fixed content, think lookup table • Geo-point - latitude/longitude • Geo-shape - Define a polygon • long, integer, short, byte, double, float, half_float, scaled_floa t, date, date_nanos, boolean, binary • More at https://www.elastic.co/guide/en/elasticsearch/ reference/current/mapping-types.html

No content

@DerekB_WI Queries GET location-list/location/_search
 {
 "query": {
 "match_all": {}
 }
 }

@DerekB_WI Queries GET location-list/location/_search
 {
 "query": {
 "match": {"name": "Big River"}
 }
 }

@DerekB_WI

@DerekB_WI Queries GET location-list/location/_search
 {
 "query": {
 "bool": {
 "filter": {
 "term": {"feature_class": "L"}
 }
 }
 }
 } https://www.geonames.org/export/codes.html

@DerekB_WI Queries GET location-list/location/_search
 {
 "query": {
 "bool": {
 "must":{
 "match": {"name":"Little Park"}
 },
 "filter": {
 "term": {"feature_class": "L"}
 }
 }
 }
 }

No content

@DerekB_WI Aggregations GET location-list/location/_search
 {
 "size": 0,
 "aggs": {
 "uniq_feature": {
 "terms": {"field":"feature_class"}
 }
 }
 }

No content

@DerekB_WI Benefits? • Fast - 2.4 mil rows in 15ms • Search within text • Other search types, geo, more like this

@DerekB_WI Load Data public function addLocation(Location $location)
 {
 $this->locations[] = $location;
 
 if (count($this->locations) >= 1000)
 {
 $this->repository
 ->bulkInsert('location-list', 'location', 
 $this->prepareBulkJson($this->locations)); $this->locations = [];
 } }

@DerekB_WI Load Data private function prepareBulkJson()
 {
 return array_map(function($location) {
 return [
 'geonameid' => $location->geonameid,
 'name' => $location->name,
 'asciiname' => $location->asciiname,
 'alternatenames' => $location->alternatenames,
 'location_point' => $location->latitude . ", " $location->longitude,
 'feature_class' => $location->featureClass,
 'feature_code' => $location->featureCode,
 'elevation' => $location->elevation,
 'timezone' => $location->timezone
 ];
 }, $this->locations);


@DerekB_WI Load Data public function bulkInsert($indexName, $indexType, $data_lines) { $client = ClientBuilder::create() ->setHosts(['127.0.0.1:9200']) ->build(); $json_body = []; foreach ($data_lines as $line) { $json_body[] = ['index' => [ '_index' => $indexName, '_type' => $indexType ] ]; $json_body[] = $line; } $responses = $client->bulk(['body' => $json_body]); }

@DerekB_WI Load Data public function bulkInsert($indexName, $indexType, $data_lines) { $client = ClientBuilder::create() ->setHosts(['127.0.0.1:9200']) ->build(); $json_body = []; foreach ($data_lines as $line) { $json_body[] = ['index' => ['_index' => $indexName, '_type' => $indexType] ]; $json_body[] = $line; } $responses = $client->bulk(['body' => $json_body]); }

@DerekB_WI Load Data public function bulkInsert($indexName, $indexType, $data_lines) { $client = ClientBuilder::create() ->setHosts(['127.0.0.1:9200']) ->build(); $json_body = []; foreach ($data_lines as $line) { $json_body[] = ['index' => ['_index' => $indexName, '_type' => $indexType] ]; $json_body[] = $line; } $responses = $client->bulk(['body' => $json_body]); }

No content

No content

@DerekB_WI Add to Mapping

No content

No content