Geospatial Data Structures in Elasticsearch and Apache Lucene

This talk covered everything you ever wanted to know about geo and Elasticsearch! Get advice on field mapping strategies, learn about geo aggregations and visualizations for exploratory spatial data analysis, as well as get insights into new spatial data structures being added to Lucene and Elasticsearch.

Transcript

1. ‹#› Geospatial Data Structures in Elasticsearch and Lucene Nick Knize @nknize
2. Geo capabilities are becoming more popular among Elasticsearch users 2
3. From combined geo with free text search… 3
4. … to monitoring network traffic to identify “bad actors” 4
5. Topics 5 Geo field types Geo indexing Geo search Geo Aggregations 1 2 3 4
6. PUT crime/incidents/_mapping { “properties” : { “location” : { “type” : “geo_point”, “ignore_malformed” : true, “geohash_precision” : 6, “geohash_prefix“ : true } } } define 6 geo_point Mappings
7. POST crime/incidents { “location” : { “lat” : 41.12, “lon” : -71.34 } } geo_point Mappings insert 7 POST crime/incidents { “location” : “41.12, -71.34” } POST crime/incidents { “location” : [[-71.34, 41.12], [-71.32, 41.21]] }
8. PUT police/precincts/_mapping { “properties” : { “coverage” : { “type” : “geo_shape”, “ignore_malformed” : false, “tree” : ”quadtree”, “precision” : “5m”, “distance_error_pct“ : 0.025, “orientation” : “ccw”, “points_only” : false } } } define 8 geo_shape Mappings
9. • Shapes are parsed using OGC and ISO standards definitions ‒OGC Simple Feature Access ‒ISO Geographic information — Spatial Schema (19107:2003) • Supports the following geo_shape types ‒Point, MultiPoint ‒LineString, MultiLineString ‒Polygon (with holes), MultiPolygon (with holes) ‒Envelope (box) 9 insert geo_shape Mappings
10. PUT police/precincts/ { “coverage” : { “type” : “polygon”, “coordinates” : [[ [40.7538588, -73.9762134], [40.7526327, -73.9742356], [40.7516774, -73.9656733], [40.7521246, -73.9763236], [40.7516733, -73.9723788], [40.7523556, -73.9732423], [40.7538588, -73.9762134] ]] } } insert geo_shape 10 geo_shape Mappings
11. ‹#› Geo Indexing 11
12. 12 postings based GeoPointField introduced in LUCENE 5.4 term postings (doc ids) 1 1, 2, 3, 4, 5 10 1, 2, 4 11 3, 5 100 1 101 2, 4 111 3, 5 1000 1 1010 4 1011 2 1110 3 1111 5 geo_point Indexing
13. postings based GeoPointField introduced in LUCENE 5.4 13 Encode lat, lon 64 bit integer (“z-curve”, or “morton code”) 1 2 3 Create prefix terms PRECISION_STEP (p) blocks up to 4p bits Add doc id to postings list (inverted index) geo_point Indexing
14. postings based GeoPointField introduced in LUCENE 5.4 14 0% 25% 50% 75% 100% Throughput Index Size Time Heap 12% 39% 28% 54% 66% 52% 80% 53% 2.1 2.2 2.3 geo_point Indexing
15. 15 Divide earth into 64 bit integer quad cells (QuadTree) 1 2 3 Create WITHIN and INTERSECT prefix terms (cells) (WITHIN == lowest res) (INTERSECT == highest res) Add doc id to postings list (inverted index) QuadPrefixTree and PackedQuadPrefixTree introduced in LUCENE 5.4 geo_shape Indexing
16. 16 QuadPrefixTree and PackedQuadPrefixTree introduced in LUCENE 5.4 • Max tree_levels == 32 (2 bits / cell) • distance_error_pct • “slop” factor to manage transient memory usage • % of the diagonal distance (degrees) of the shape • Default == 0 if precision set (2.0) • points_only • optimization for points only shape index • short-circuits recursion geo_shape Indexing
17. geo_shape vs. geo_point …why can’t we all just get along? 17
18. geo @experimental 18
19. 19 geo Indexing Tree Structure (e.g., Balanced K-d Tree Bkd-tree) coming LUCENE 6.0 a b c d e f g h i j k l m n o p
20. geo Indexing Tree Structure (e.g., Balanced K-d Tree Bkd-tree) coming LUCENE 6.0 20 Encode lat, lon 64 bit integer (“z-curve”, or “morton code”) 1 2 3 4 Start at root, recurse to ideal “leaf” bucket (either bounding box can contain point, or expand box) split on longest span dimension (ensure “squareness”) Write blocks (#children == M or on flush when: M/2 < #children < M
21. 21 points geo Indexing
22. 22 shapes geo Indexing • Dimensional Shapes represented using Minimum Bounding Ranges (MBR) ‒Ranges (1D) ‒Rectangles (2D) ‒Cubes (3D) ‒Hexadecant (4D)
23. 23 shapes geo Indexing m n o p
24. 24 shapes geo Indexing m n o p
25. 25 shapes geo Indexing m n o p
26. 26 shapes geo Indexing m n o p
27. 27 shapes geo Indexing m n o p
28. 28 shapes geo Indexing m n o p
29. 29 shapes geo Indexing m n o p
30. 30 source: wikipedia
31. 31 0% 25% 50% 75% 100% Index Size Index Time 49% 49% 100% 100% NumericField PointField 1D numerics geo Indexing
32. ‹#› Geo Search 32
33. 33 Divide earth into 64 bit integer quad cells (QuadTree) 1 2 3 Create WITHIN and INTERSECT prefix terms (cells) (WITHIN == lowest res) (INTERSECT == highest res) retrieve doc ids from postings list, use DocValues to post filter INTERSECT terms postings based GeoPointField introduced in LUCENE 5.4 geo_point Search
34. 34 geo_point Search postings based GeoPointField introduced in LUCENE 5.4 • New LUCENE Spatial Queries (5.4 / ES 2.2) • BoundingBox, Distance, DistanceRange, Polygon • PRECISION_STEP controls number of query terms (must match with index) • TwoPhaseIterator (2.3) • Delays boundary confirmation so other query (filters, conjunctions) can pre-filter
35. geo_point Search 35 0% 25% 50% 75% 100% BoundingBox Distance DistanceRange Polygon 11% 21% 26% 36% 51% 45% 70% 82% 2.1 2.2 2.3 postings based GeoPointField introduced in LUCENE 5.4
36. geo_shape Search • Supports the following geo_shape types ‒ Point, MultiPoint ‒ LineString, MultiLineString ‒ Polygon (with holes), MultiPolygon (with holes) ‒ Envelope (box) • Shapes are parsed using OGC (SFA) and ISO (19107:2003) standards definitions • Supports relational queries ‒ INTERSECTS, DISJOINT, WITHIN, CONTAINS 36 geo_shape field
37. 37 Recursively Traverse Query terms 1 2 OR the DocIDs from each term’s Postings List INTERSECTS geo_shape Search
38. 38 (MUST, EXISTS) 1 2 (MUST_NOT, INTERSECTS) DISJOINT = !INTERSECTS geo_shape Search BooleanQuery
39. geo_shape Search WITHIN 39 Buffer Shape by PERCENT_DISTANCE (Provides EXCLUDE terms outside the shape perimeter) 1 2 3 4 Traverse query terms of the buffered shape Compute Relation between term and unbuffered Query Shape Accept docs whose terms INTERSECT / WITHIN but MUST_NOT contain DISJOINT terms
40. 40 geo_shape Search • PERCENT_DISTANCE • -1, traverses entire map. Costly, but more accurate • >0, buffered shape. Faster at the cost of accuracy. • ES 2.3 uses distance_error_pct or 2.5% if set to 0. WITHIN
41. 41 Recursively Traverse Query terms 1 2 AND the DocIDs from each term’s Postings List CONTAINS - NEW in 2.2 geo_shape Search
42. geo @experimental 42
43. geo Search Tree Structure (e.g., Balanced K-d Tree Bkd-tree) coming LUCENE 6.0 43 Begin with root node bounding box 1 2 3 4 RELATE with search criteria (WITHIN, CONTAIN, INTERSECT) DFS traverse (until Internal Node WITHIN or Leaf Node WITHIN or INTERSECTS) Collect doc IDs (WITHIN == ALL INTERSECTS == post filter)
44. 44 geo Search Tree Structure for points (e.g., Balanced K-d Tree Bkd-tree) coming LUCENE 6.0 Leaf cell is fully within polygon (salmon) - return all docs Leaf cell crosses the boundary (gray) - two-phase check 1 2
45. 45 geo Search Tree Structure for shapes (e.g., R* Tree) coming LUCENE 6.x/? a b c d e f g h i j k l m n o p m n o p a! a! a! x
46. 46 geo Search Tree Structure for shapes (e.g., R* Tree) coming LUCENE 6.x/? a b c d e f g h i j k l m n o p m n o p a! a! x x a! a! a b c d e g
47. 47 geo Search Tree Structure for shapes (e.g., R* Tree) coming LUCENE 6.x/? source: wikipedia
48. 48 0% 25% 50% 75% 100% Search Time Search Time Heap Usage 15% 76% 100% 100% NumericField PointField geo Search 1D numerics
49. ‹#› Geo Aggregations 49
50. ‹#› 50 GeoDistance Agg { "aggs" : { “sf_rings" : { "geo_distance" : { "field" : "location", "origin" : [32.95, -96.82], "ranges" : [ { "to" : 50 }, { "from" : 50, "to" : 100 }, { "from" : 100, "to" : 300} ] } } } }
51. ‹#› 51 GeoDistance Agg
52. ‹#› 52 GeoGrid Agg { "aggs" : { “crime_cells" : { "geohash_grid" : { "field" : "location", "precision" : 8 } } } }
53. ‹#› 53 GeoGrid Agg
54. ‹#› 54 GeoCentroid Agg "query" : { "match" : { "crime" : "burglary" } }, "aggs" : { "towns" : { "terms" : { "field" : "town" }, "aggs" : { "centroid" : { "geo_centroid" : { "field" : “location" } } } } }
55. ‹#› 55 GeoCentroid Agg
56. ‹#› 56 GeoCentroid Agg
57. Questions? 19 Also find us at the AMA Booth
58. ‹#› Please attribute Elastic with a link to elastic.co Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 58