MongoDB DC 2012: Why We Chose MongoDB to Put Big Data "On The Map"

Nicholas Knize, Thermopylae Sciences and Technology
Traditional Geospatial Information Systems (GIS) have heavily depended on the use of Relational Databases (RDBMS) for indexing. Relational Databases place a priority on long running transactions, pre-defined fixed normalized schemas, and large joins which make them ill equipped to support Big Data scenarios of data volume, variability, and velocity. Relational database theory is not optimal for geospatial applications and decent relational geospatial databases are expensive and typically difficult to maintain. Learn how Thermopylae Sciences and Technology applied the non-relational (NoSQL) implementation of MongoDB for tackling dynamic geospatial data at massive scale. During the presentation we will cover: Why NoSQL technology over Relational Databases for scaling geospatial date, why MongoDB (what are the enhancements to the spatial indexer?) and where is it being used.

Transcript

1. WHY$WE$CHOSE$MONGODB$TO$$ PUT$BIG2DATA$‘ON$THE$MAP’$ JUNE$2012$ $ $ $ $ $ $ $ @nknize$ +Nicholas$Knize$
2. “The%3D%UDOP%allows%near%real%2me%visibility%of%all%SOUTHCOM%Directorates%informa2on%in%one% loca2on…this%capability%allows%for%unprecedented%situa2onal%awareness%and%informa2on%sharing”% % % % % % % % % % % % %EGen.%Doug%Frasier% TST PRODUCTS ACCOMPLISHING$THE$IMPOSSIBLE$
3. •  Expose$enterprise$data$in$a$geo2temporal$user$defined$ environment$ •  Provide$a$flexible$and$scalable$spaUal$indexing$framework$ for$heterogeneous$data$$ •  Visualize$spaUally$referenced$data$on$3D$globe$&$2D$maps$ •  Manage$real2Ume$data$feeds$and$mobile$messaging$$ •  View$data$over$geo2recUfied$imagery$with$3D$terrain$ •  Support$mission$planning$and$simulaUon$ •  Provide$real2Ume$collaboraUon$and$sharing$ ISPATIAL OVERVIEW ACCOMPLISHING$THE$IMPOSSIBLE$
4. •  Horizontally$scalable$–$Large$volume$/$elasUc$ •  VerUcally$scalable$–$Heterogeneous$data$types$(“Data$Stack”)$ •  Smartly$Distributed$–$Reduce$the$distance$bits$must$travel$ •  Fault$Tolerant$–$ReplicaUon$Strategy$and$Consistency$model$ •  High$Availability$–$Node$recovery$ •  Fast$–$Reads$or$writes$(can’t$always$have$both)$ BIG DATA STORAGE CHARACTERISTICS ACCOMPLISHING$THE$IMPOSSIBLE$ $$$$Desired$Data$Store$CharacterisUc$for$‘Big$Data’$
5. •  Cassandra$ –  Nice$Bring$Your$Own$Index$(BYOI)$design$ –  …$but$Java,$Java,$Java…$Memory$management$can$be$an$issue$ –  Adding$new$nodes$can$be$a$pain$(Token$Changes,$nodetool)$ –  Key2Value$store…good$for$simple$data$models$ •  Hbase$ –  Nice$BigTable$model$ –  Theory$grounded$heavily$in$C.A.P,$inflexible$trade2offs$ –  Complicated$setup$and$maintenance$$$ •  CouchDB$ –  Provides$some$GeoSpaUal$funcUonality$ –  HEAVILY$dependent$on$Map2Reduce$model$(complicated$design)$ –  Erlang$based$–$poor$mulU2threaded$heap$management$ $ NOSQL OPTIONS ACCOMPLISHING$THE$IMPOSSIBLE$ Subset$of$Evaluated$NoSQL$OpUons$
6. $$$$Why$MongoDB$for$Thermopylae?$ •  Documents$based$on$Javascript$Object$NotaUon$(JSON)$–$A$GEOJSON$ match$made$in$heaven!$ $ •  C++$2$No$Garbage$CollecUon$Overhead!$$Efficient$memory$management$ design$reduces$disk$swapping$and$paging$ •  Disk$storage$is$memory$mapped,$enabling$fast$swapping$when$necessary$$ $ •  Built$in$auto2failover$with$replica$sets$and$fast$recovery$with$journaling$ •  Tunable$Consistency$–$Consistency$defined$at$applicaUon$layer$ •  Schema$Flexible$–$friendly$properUes$of$SQL$enable$easy$port$ •  Provided$iniUal$spaUal$indexing$support$–$Point$based$limited!$ $ WHY TST LIKES MONGODB ACCOMPLISHING$THE$IMPOSSIBLE$
7. MONGODB SPATIAL INDEXER ACCOMPLISHING$THE$IMPOSSIBLE$ $$$...$The$SpaUal$Indexer$wasn’t$quite$right$ •  MongoDB$(like$nearly$all$relaUonal$DBs)$uses$a$b2Tree$$ –  Data$structure$for$storing$sorted$data$in$log$Ume$ –  Great$for$indexing$numerical$and$text$documents$(anribute$data)$ –  Cannot$store$mulU2dimension$(>2)$data$–$NOT$COMPLEX$GEOMETRY$ FRIENDLY$
8. DIMENSIONALITY REDUCTION ACCOMPLISHING$THE$IMPOSSIBLE$ How$does$MongoDB$solve$the$dimensionality$problem?$$ •  Space$Filling$(Z)$Curve$$ –  A$conUnuous$line$that$ intersects$every$point$in$a$ two2dimensional$plane$ •  Use$Geohash$to$ represent$lat/lon$values$ –  Interleave$the$bits$of$a$ lat/long$pair$ –  Base32$encode$the$result$
9. GEOHASH BTREE ISSUES ACCOMPLISHING$THE$IMPOSSIBLE$ •  Neighbors$aren’t$so$ close!$ –  Neighboring$points$on$the$ Geoid$may$end$up$on$ opposite$ends$of$the$ plane$ –  Impacts$search$efficiency$ •  What$about$Geometry?$ –  Doesn’t$support$>$2D$ –  Mongo$uses$MulU2 LocaUon$documents$ which$really$just$indexes$ mulUple$points$that$link$ back$to$a$single$document$ $$$$Issues$with$the$Geohash$b2Tree$approach$
10. Case 3: Case 4: Multi-Location Document (aka. Polygon) Search Polygon Case 1: Case 2: Success! Success! Fail! Fail! Mongo$MulU2locaUon$Document$Clipping$Issues$ ($within$search$doesn’t$always$work$w/$mulU2locaUon)$ MULTI-LOCATION CLIPPING ACCOMPLISHING$THE$IMPOSSIBLE$
11. •  Constrain$the$system$to$single$point$searches$ –  MulU2dimension$support$will$be$exponenUally$complex$(won’t$scale)$ $ $ •  Interpolate$points$along$the$edge$of$the$shape$ –  MulU2dimension$support$will$be$exponenUally$complex$(won’t$scale)$ •  Customize$the$spaUal$indexer$ –  Selected$approach$ SOLUTIONS TO GEOHASH PROBLEM ACCOMPLISHING$THE$IMPOSSIBLE$ $$$$PotenUal$SoluUons$
12. CUSTOM TUNED SPATIAL INDEXER ACCOMPLISHING$THE$IMPOSSIBLE$ Thermopylae$Custom$Tuned$MongoDB$$$$$$for$Geo$ TST$Leverage’s$Gunman’s$1984$Research$in$R/R*$Trees$ •  R2Trees$organize$any2dimensional$data$by$represenUng$ the$data$as$a$minimum$bounding$box.$$ •  Each$node$bounds$it
    s$children.$$A$node$can$have$many$ objects$in$it$(max:$m$$$min:$$ceil(m/2)%)$ •  Splits$and$merges$opUmized$by$minimizing$overlaps$ •  The$leaves$point$to$the$actual$objects$(stored$on$disk$ probably)$ •  Height$balanced$–$search$is$always$O(log$n)$$
13. SpaUal$Indexing$at$Scale$with$R2Trees$ RTREE THEORY ACCOMPLISHING$THE$IMPOSSIBLE$ SpaUal$data$represented$as$minimum$bounding$rectangles$(22dimension),$ cubes$(32dimension),$hexadecant$(42dimension)! ! Index$represented$as:$$$<I,$DiskLoc>$$where:$ ! !I$=$(I0 ,$I1 ,$…$In )$$$:$$n$=$number$of$dimensions$ !Each$I$is$a$set$in$the$form$of$[min,max]$describing$MBR$range$along$a$dimension$ $ $! !
14. R*-Tree Spatial Index Example •  Sample insertion result for 4th order tree •  Objectives: 1.  Minimize area 2.  Minimize overlaps 3.  Minimize margins 4.  Maximize inner node utilization a b c d e f g h i j k l m n o p R*-TREE INDEX OBJECTIVES ACCOMPLISHING$THE$IMPOSSIBLE$
15. Insert •  Similar to insertion into B+-tree but may insert into any leaf; leaf splits in case capacity exceeded. –  Which leaf to insert into? –  How to split a node? R*-TREE INSERT EXAMPLE ACCOMPLISHING$THE$IMPOSSIBLE$
16. Insert—Leaf Selection •  Follow a path from root to leaf. •  At each node move into subtree whose MBR area increases least with addition of new rectangle. m n o p
17. Insert—Leaf Selection •  Insert into m. m
18. Insert—Leaf Selection •  Insert into n. n
19. Insert—Leaf Selection •  Insert into o. o
20. Insert—Leaf Selection •  Insert into p. p
21. m n o p a! a! a! x a b c d e f g h i j k l m n o p Query •  Start at root •  Find all overlapping MBRs •  Search subtrees recursively
22. Query •  Search m. m n o p a! a! x x a b c d e f g h i j k l m n o p a! a! a b c d e g
23. R*2Tree$Leverages$B2Tree$Base$Data$Structures$(buckets)$ R*-TREE MONGODB IMPLEMENTATION ACCOMPLISHING$THE$IMPOSSIBLE$
24. Geo2Sharding$–$(in%work)$ $ $Scalable$Distributed$R*$Tree$(SD2r*Tree)$ Balanced$binary$tree,$ distributed$on$a$set$of$ servers:$ $ •  Each$internal$node$has$ exactly$two$children$ $ •  Each$leaf$node$stores$a$ subset$of$the$indexed$ dataset$ $ •  At$each$node,$the$height$ of$the$subtrees$differ$by$ at$most$one$ $ •  Each$server$stores$one$ data$node$and$one$ “rouUng”$node$ GEO-SHARDING ACCOMPLISHING$THE$IMPOSSIBLE$
25. d0! d1! r1! d0! Data!Node! Spa.al!! Coverage! a! a! b! c! c! b! d0! r1! a! b! c! c! b! d2! d1! e! d! d! r2! e! SD2r*Tree$Data$Structure$IllustraUon$$ •  di$ =$Data$Node$(Chunk)$ •  ri$ =$Coverage$Node$ $ Leveraged$work$from$Litwin,$Mouza,$Rigaux$2007$ SD-r*Tree DATA STRUCTURE ACCOMPLISHING$THE$IMPOSSIBLE$
26. SD2r*Tree$Structure$DistribuUon$ d0! r1! a! b! c! c! b! d2! d1! e! d! d! r2! e! r2! d1! d2! d0! r1! GeoShard!2! GeoShard!3! GeoShard!1! mongos! SD-r*TREE STRUCTURE DISTRIBUTION ACCOMPLISHING$THE$IMPOSSIBLE$
27. GeoSharding$AlternaUve$–$3D$/$4D$Hilbert$Scanning$Order$ GEO-SHARDING ALTERNATIVE ACCOMPLISHING$THE$IMPOSSIBLE$
28. Next$Steps:$Beyond$42Dimensions$2$X2Tree$ (Berchtold,$Keim,$Kriegel$–$1996)$$ Normal Internal Nodes Supernodes Data Nodes •  Avoid$MBR$overlaps$ $ •  Avoid$node$splits$(main$cause$for$high$overlap)$ $ •  Introduce$new$node$structure:$Supernodes!–$Large$Directory$nodes$of$variable$size$ BEYOND 4-DIMENSIONS ACCOMPLISHING$THE$IMPOSSIBLE$
29. X-TREE PERFORMANCE ACCOMPLISHING$THE$IMPOSSIBLE$ X2Tree$Performance$Results$ (Berchtold,$Keim,$Kriegel$–$1996)$$
30. T2Sciences$Custom$Tuned$SpaUal$Indexer$ •  OpUmized$SpaUal$Search$–$Finds$intersecUng$MBR$and$recurses$into$ those$nodes$ $ •  OpUmized$SpaUal$Inserts$–$Uses$the$Hilbert$Value$of$MBR$centroid$to$ guide$search$$ –  28%$reducUon$in$number$of$nodes$touched$ •  OpUmize$Deletes$–$Leverages$R*$split/merge$approach$for$rebalancing$ tree$when$nodes$become$over/under2full$ •  Low$maintenance$–$Leverages$MongoDB’s$automaUc$data$compacUon$ and$parUUoning$ CONCLUSION ACCOMPLISHING$THE$IMPOSSIBLE$
31. Example$Use$Case$–$OSINT$(Foursquare$Data)$ •  Sample Foursquare data set mashed with Government Intel Data •  1 million Geo Document test (points and polys) •  4 server replica set •  ~350ms query response •  ~300% improvement over PostGIS EXAMPLE ACCOMPLISHING$THE$IMPOSSIBLE$
32. Community$Support$ •  Thermopylae$contributes$fixes$to$the$codebase$ –  hnp://github.com/mongodb$ •  TST$will$work$with$10gen$to$fold$into$the$baseline$ $ •  AcUve$developer$collaboraUon$ –  IRC:$#mongodb$$$freenode.net$ $ FIND US ACCOMPLISHING$THE$IMPOSSIBLE$
33. $ THANK$YOU$ QuesUons?$ $ Nicholas$Knize$ nknize@t2sciences.com$ THANK YOU ACCOMPLISHING$THE$IMPOSSIBLE$
34. $ Backup$ $
35. Thermopylae$Sciences$&$Technology$–$Who$are$we?$ •  Advanced$technology$w/$160+$employees$ •  Core$customers$in$naUonal$security,$venues$and$ events,$military$and$police,$and$city$planning$ •  Partnered$with$Google$and$imagery$providers$ •  Long$term$relaUonship$focused$–$TS/SCI$Staff$ $$$$$$$$TST$+$10gen$+$Google$=$Game2changing$approach$ WHO ARE THESE GUYS? ACCOMPLISHING$THE$IMPOSSIBLE$ ENTERPRISE PARTNER
36. Key$Customers$2$Government $$ •  US$Dept$of$State$Bureau$of$DiplomaUc$Security$ –  Build$and$support$30$TB$Google$Earth$Globe$with$mulU2 terabytes$of$individual$globes$sent$to$embassies$throughout$ the$world.$$Integrated$Google$Earth$and$iSpaUal$framework.$ •  US$Army$Intelligence$Security$Command$ –  Provide$experUse$in$managing$technology$integraUon$–$ prime$contractor$providing$operaUons,$intelligence,$and$IT$ support$worldwide.$$Partners$include$IBM,$Lockheed$MarUn,$ Google,$MIT,$Carnegie$Mellon.$$Integrated$Google$Earth$and$ iSpaUal$framework.$ •  US$Southern$Command$ –  Coordinate$Intelligence$management$systems$spaUal$data$ collecUon,$indexing,$and$distribuUon.$$Integrated$Google$ Earth,$iSpaUal,$and$iHarvest.$ –  Index$large$volume$imagery$and$expose$it$for$different$ services$(Air$Force,$Navy,$Army,$Marines,$Coast$Guard)$ $ GOVERNMENT CUSTOMERS ACCOMPLISHING$THE$IMPOSSIBLE$
37. COMMERCIAL CUSTOMERS ACCOMPLISHING$THE$IMPOSSIBLE$ Key$Customers$2$Commercial$$ Cleveland! Cavaliers! USGIF! Las!Vegas! Motor!Speedway! Bal.more! Grand!Prix! iSpaUal$framework$serves$thousands$of$mobile$devices$
38. •  Banle$tested,$Banle$proven$–$RelaUonal$Model$dates$back$to$1969$ •  Plethora$of$RelaUonal$Experience$–$Full2Time$DBAs,$Training$&$Certs$ •  Company$Backed$–$Safe$choice$for$business$/$mission$criUcal$systems$ •  Fewer$AlternaUves$–$Non2relaUonal$is$a$5$year$old$know2it2all$ •  Mostly$Standardized$–$SQL$ISO/IEC$9075$Accepted$Standard$ •  TheoreUcally$Sound$–$Based$on$100$years$of$First2Order$Logic$theory$ RDBMS STRENGTHS ACCOMPLISHING$THE$IMPOSSIBLE$ $$$$RDBMS$Strengths$
39. •  Atomicity$–$If$one$fails,$we$all$fail!$$$$ •  Consistency$–$All$data$constraints$(normalized$schema)$cascades,$ triggers,$etc.$must$be$met$before$transacUon$succeeds.$(LATENCY)$ •  IsolaUon$–$SynchronizaUon,$no$operaUon$can$see$a$transacUon$that$ hasn’t$yet$completed$ •  Durability$–$Once$a$transacUon$is$commined$it$will$remain$commined$ even$in$power$loss$crashes$or$other$hardware$errors.$ ACID THEORY ACCOMPLISHING$THE$IMPOSSIBLE$ $$$$RelaUonal$on$ACID$
40. $ •  Writes$are$accomplished$using$in2place$update$on$disk$(crazy$disk$ swapping$rate)$ $ •  Table$joins,$updates,$and$large$queries$quickly$outgrow$disk$cache$ requiring$many$random$disk$seeks$(performance$bonleneck!!)$ •  Strict$consistency$requirements$impacts$scalability$(e.g.$Postgres$ uses$MulUversion$Consistency,$commonly$resulUng$in$stale$data)$ •  As$data$centers$grow,$the$probability$of$node$failure$(due$to$Disk$ Writes,$Consistency,$and$Atomic$operaUons)$increases$ $ RDBMS WEAKNESSES ACCOMPLISHING$THE$IMPOSSIBLE$ RDBMS$Weaknesses$
41. Why$NoSQL?!?$ (CAVEATS)$ •  Use$the$right$tool$for$the$job$ WHY NOSQL? ACCOMPLISHING$THE$IMPOSSIBLE$ •  Understand$your$needs!$ •  RelaUonal$is$not$always$bad$ Engineering!with!Constraints! Unbounded!Engineering!