Using MongoDB for location based and other geotargeting services - Jorgen Nilsson

MongoMunich 2011

Transcript

1. The communicative web agency Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 måndag 17 oktober 11
2. Using MongoDB for location based and other geotargeting services måndag 17 oktober 11
3. Web professional Jörgen Nilsson Modondo - A web agency in southern Sweden Interface design Front-end development Shallow back-end development måndag 17 oktober 11
4. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 MongoDB at Modondo måndag 17 oktober 11
5. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 MongoDB at Modondo Publify - A centralized decentralized CMS in the cloud... or space “MonkeyApp” - Project management specialized for web projects Modondo.com - Our own web site måndag 17 oktober 11
6. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 ChitChat - Message system built as a mountable engine for Ruby on Rails Ett Bra Vin - A small wine project (ettbravin.se - Swedish only) The Daily List - A simple to-do browser application using hosted MongoDB services (MongoLab and MongoHQ) My personal web site, a never ending project Querycus - The handy document browser Labs and personal projects måndag 17 oktober 11
7. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Brief introduction to Geospatial indexing and querying måndag 17 oktober 11
8. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Query types and commands $near and $within are the two query types used with find() $near - Used for finding locations close and around a specific place (coordinate) $within - Used for finding locations within a bounding area geoNear - Similar to find() but returns distances and diagnostics for near queries which is good for calculations or debugging. måndag 17 oktober 11
9. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Query types and commands $within $box - Search within a box based on SouthWest and NorthEast coordinates $center - Search from a specific point and within a certain radius $polygon - Search within a polygon shape, you could be brave and call it a freeform måndag 17 oktober 11
10. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Query types and commands $near Default limit of 100 returned documents $maxDistance måndag 17 oktober 11
11. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Querying - $box måndag 17 oktober 11
12. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Querying - $center måndag 17 oktober 11
13. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Querying - Polygon måndag 17 oktober 11
14. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 The basics - Getting started Model your document Apply the index Populate your collection Make a search Make another search and celebrate with a beer Enjoy the fun :) måndag 17 oktober 11
15. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Model your document A simple place document could look like this and it will also be used in the following examples { formatted_address: "", kind: "", location: {"lat": "", "lng": ""} } We have chosen to use descriptive keys instead of relying on the order in the array All examples can be run with db.places.find(); måndag 17 oktober 11
16. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Model your document or like this... # Pure Ruby places = { formatted_address: "", kind: "", location: {"lat": "", "lng": ""} } places.create_index([["location", Mongo::GEO2D]]) måndag 17 oktober 11
17. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Model your document or like this... # MongoID - Ruby on Rails class Places include Mongoid::Document field :formatted_address, type: String field :kind, type: String field :location, type: Hash index [[ :location, Mongo::GEO2D ]], min: 180, max: 180 end måndag 17 oktober 11
18. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Model your document or like this # MongoKit - Python class Places(Document): collection_name = "places" structure = { 'formatted_address':unicode, 'kind':unicode, 'location':[] } indexes = [ { 'fields': ('location', INDEX_GEO2D), 'unique':False, }, ] required_fields = ['formatted_address', 'location '] måndag 17 oktober 11
19. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Querying - What’s around me? Find all hotels near me in Munich # Using array { "location" : { "$near" : [ 48.139, 11.580 ] }, kind: "hotel"} # Using hash {"location" : {"$near": {lat: 48.139, lng: 11.580}}, kind: "hotel"} måndag 17 oktober 11
20. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Querying - Never know what’s inside Find all hotels in Munich using $box # Using array {"location" : {"$within": {"$box": [[48.03, 11.44], [48.25, 11.75]]}}, kind: "hotel"} # Using hash {"location" : {"$within": {"$box": [{lat: 48.03, lng: 11.44}, {lat: 48.25, lng: 11.75}]}}, kind: "hotel"} måndag 17 oktober 11
21. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Querying - Never know what’s inside # Using hash {"location": {"$within": {"$center": [{lat: 48.13, lng: 11.58}, 20]}}} # Using array {"location": {"$within": {"$center": [[48.13, 11.58], 20]}}} Find all hotels in Munich using $center måndag 17 oktober 11
22. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Querying - Compound Geospatial Indexes If you are querying often on a location and other attributes at the same time, create a compound index Makes filtering faster db.places.ensureIndex( { location : "2d" , kind : 1 } ); måndag 17 oktober 11
23. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Querying $multi-location $spherical model måndag 17 oktober 11
24. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Some points to remember Default limit of documents that is returned by a geo query is 100 documents. You should always limit the amount of documents to what you really need. Create index using ensureIndex and passing “2d” as value. Use unordered queries whenever possible The Earth is Round but Maps are Flat måndag 17 oktober 11
25. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 More than just maps You can create and specify your own grid Can be used for various kind of games or in all places you need a grid and coordination system. Why not use it for a Dungeon and Dragons game? Or as Word2 - build a giant word game måndag 17 oktober 11
26. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Use case Find your holiday måndag 17 oktober 11
27. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Use case - Service description Find your next holiday is a service where users can rent or exchange homes with each other. Users should be able to search the entire world for holiday homes and view them on a map. The use case description has been very simplified to better illustrate the design and solution. måndag 17 oktober 11
28. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Use case - Background - Stats 40 000+ users world wide 40 000+ location objects 18 languages 4000+ visits a day 60 000+ page views per day måndag 17 oktober 11
29. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Use case - Background - Tech stack Ubuntu server Ruby on Rails Ruby 1.9.2 Nginx + Phusion, (WEBrick for local dev) PostgresSQL MongoDB MongoID RVM måndag 17 oktober 11
30. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Use case - Why MongoDB? No special datatype needed. Just store longitude and latitude as floats in array and apply the GEO2D index. Short and concise quires for finding items close by or within a bound in the form of rectangle, circle and polygon shape (geoNear and Bounds). Stored values are human readable and can be accessed and output into various contexts. måndag 17 oktober 11
31. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Use case - Why MongoDB Works great with MongoID – An Ruby Object- Document-Mapper (ODM) No special configurations needed, it works right out of the box Great for keyword searches Easy to expand the data, no need to worry for NULL values måndag 17 oktober 11
32. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Use case scenarios All of the following examples are based on the usage of Ruby on Rails together with MongoDB and MongoID. It also uses and relies heavily on Google Maps JavaScript API v3 måndag 17 oktober 11
33. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Scenario 1 - Adding a property The user adds the property to the system by entering the property’s address in a input field By using Googles Places API we get a nice autocomplete and get the coordinates for the property and place a maker on the map The user can adjust the marker on the map and we get instant feedback on the coordinates The user clicks on the “save location” button and everything is stored down to MongoDB måndag 17 oktober 11
34. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Scenario 1 - Adding a property måndag 17 oktober 11
35. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Scenario 1 - Adding a property The stored document looks like this so far class Place include Mongoid::Document include Mongoid::Timestamps field :user_id, type: Integer field :street, type: String field :postal_town, type: String field :postal_code, type: String field :country, type: String field :formatted_address, type: String field :location, type: Hash index [[ :location, Mongo::GEO2D ]], min: 180, max: 180 end måndag 17 oktober 11
36. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Scenario 2 - Adding locations near our property Now when we have a property we can add places that is connected or are relevant to our property. The connected places are added to the property by adding markers to a Google map. Each connected location are embedded into our current document as surroundings and takes advantage of the Multi-Location feature available in version 2.0 of MongoDB måndag 17 oktober 11
37. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Scenario 2 - Adding locations near our property The markers are for illustrative purposes and the locations are therefore not real and accurate. måndag 17 oktober 11
38. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Scenario 2 - Adding locations near our property Our property document now looks like this class Place include Mongoid::Document include Mongoid::Timestamps field :user_id, type: Integer field :street, type: String field :postal_town, type: String field :postal_code, type: String field :country, type: String field :formatted_address, type: String field :location, type: Hash field embeds_many :surroundings index [[ :location, Mongo::GEO2D ]], min: 180, max: 180 end måndag 17 oktober 11
39. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Scenario 2 - Adding locations near our property class surroundings include Mongoid::Document include Mongoid::Timestamps field :kind, type: String field :formatted_address, type: String field :location, type: Hash embedded_in :places index [[ :location, Mongo::GEO2D ]], min: 180, max: 180 end Our embedded document looks like this måndag 17 oktober 11
40. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Scenario 3 - Searching for a property It is now super easy to find properties near or around Munich måndag 17 oktober 11
41. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Scenario 4 - Looking at a property Nearly effortless we can list all interesting surroundings and use various filters to show and hide them måndag 17 oktober 11
42. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Lessons learned måndag 17 oktober 11
43. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 When searching for New Zealand you would get the following bounds coordinates back. South West Lat: -52.72, Lng: 165.74 North East Lat: -28.87, Lng: -175.06 Negative and positive latitude coordinates that overlaps Should I read that way or that way? Lessons learned - The kiwi issue måndag 17 oktober 11
44. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Lessons learned - The kiwi issue måndag 17 oktober 11
45. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Lessons learned - The kiwi issue måndag 17 oktober 11
46. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Lessons learned - The kiwi issue You have to construct two boxes One that goes from 165.74 to 179.99 and one that goes from 180.00 to -175.06 You simply use an “or” statement applied by the following condition. If southwest longitude > north east longitude måndag 17 oktober 11
47. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Lessons learned - Point not in range To narrow index, make sure the values are within the given range in the index. Default indexing values are -180, 180, which you could say corresponds to the earths bounds. Expand the index min and max values Check for incorrect values for the index Make sure there aren’t any spaces, tabs or other odd characters around the coordinates if importing data måndag 17 oktober 11
48. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Lessons learned - Be consistent Use keys for latitude and longitude Automate as much as possible Don’t let users enter their own geo data Always validate your input values måndag 17 oktober 11
49. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Lessons learned - Conventions Modondo are using the following format for storing geo coordinates. “location” : {“lat”: 51.5171003, “lng” : “-0.1406875999”} It is quite self explanatory, it is easer to read and understand both for clients and other developers that wasn’t around during the development It is the same “format” and order that Google Maps are using for their API måndag 17 oktober 11
50. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Resources måndag 17 oktober 11
51. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Resources There are not that much information or documentation about Geospatial indexing and what you can with it in MongoDB. Why? Maybe because it so easy? I’ve collected a few resources that I see as good starting points. Hope it helps. måndag 17 oktober 11
52. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Resources Geospatial Indexing - (Official docs) http://www.mongodb.org/display/DOCS/Geospatial+Indexing Geospatial Indexing with MongoDB - (Video) Greg Studer, Software Engineer, 10gen http://www.10gen.com/presentation/mongosf-2011/geospatial-indexing-mongodb Storing and Querying location data with MongoDB - (Video) Grant Goodale, WordSquared http://www.10gen.com/presentation/mongosf2011/wordsquared mongodb-user - (Forum / Google group) http://groups.google.com/group/mongodb-user måndag 17 oktober 11
53. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Resources MongoID: Querying (Criteria) - (Docs) http://mongoid.org/docs/querying/criteria.html MongoDB's Geospatial Interactive Tutorial Karl Seguin http://mongly.com/geo/index Using MongoDB for location based and other geotargeting services - (Blog post) Jörgen Nilsson, Web Professional, Modondo http://blog.modondo.com/2011/08/using-mongodb-for-location-based-and-other-geotargeting-services/ måndag 17 oktober 11
54. Using MongoDB for location based and other geotargeting services - Jörgen Nilsson - MongoMunich 2011 Questions? måndag 17 oktober 11
55. Thank you! jorgen@modondo.com coffeepunk A presentation about geotargeting and location based services using MongoDB Jörgen Nilsson of Modondo by måndag 17 oktober 11