Using RethinkDB and ArcGIS to create amazing map visualizations

In this talk, we'll take public data to create a visualization of restaurant density across the US. We'll use use RethinkDB to clean, analyze and store our data. Then, we'll feed the data into ArcGIS in order create an interactive visualization. After this talk, you'll get a sense of why a realtime database with geospatial features is an excellent tool for anything "geo" related.

Transcript

1. RethinkDB+ArcGIS Creating amazing geospatial visualizations GeoDev Meetup Sacramento, California June 1, 2015
2. Jorge Silva @thejsj Developer Evangelist @ RethinkDB
3. Introduction What is RethinkDB?
4. What is RethinkDB? • Open source database for building realtime web applications • NoSQL database that stores schemaless JSON documents • Distributed database that is easy to scale
5. Built for Realtime Apps • Subscribe to change notifications from database queries • No more polling — the database pushes changes to your app • Reduce the amount of plumbing needed to stream live updates
6. RethinkDB Structure Database → Table → Document MySQL: Database → Table → Row MongoDB: Database → Collection → Document
7. Sample Document { "name": "Will Riker", "position": "Commander", "height": 193, "birthdate": Mon Aug 19 2335, "ships": [ { "name": "USS Pegasus" }, { "name": "USS Potemkin" }, { "name": "USS Enterprise" }, ], ... }
8. Introduction to ReQL RethinkDB Query Language
9. Introduction to ReQL • ReQL embeds natively into your programming language • Compose ReQL queries by chaining commands
10. Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Number of unique last names
11. Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Access a database table
12. Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Isolate a document property
13. Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Consolidate duplicate values
14. Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Display the number of items
15. Sample ReQL Queries r.table("user") .filter(r.row(“age").gt(30)) r.table(“post") .eqJoin(“uId”, r.table(“user”)) .zip() r.table("fellowship") .filter({ species: "hobbit" }) .update({ species: "halfling" })
16. Running Queries https://github.com/thejsj/arcgis-rethinkdb
17. Additional ReQL Features • Date and time functions for time data • Support for storing binary objects • r.http allows you to execute http requests from the database
18. Realtime Updates Working with Changefeeds
19. Subscribe to change notifications on database queries Changefeeds
20. r.table("users").changes() Track changes on the users table Changefeeds
21. Changefeeds r.table("users").changes() r.table("users") .insert({name: "Bob"}) Changefeed output: { new_val: { id: '362ae837-2e29-4695-adef-4fa415138f90', name: 'Bob', ... }, old_val: null }
22. Changefeeds r.table("users").changes() r.table("users") .filter({name: "Bob"}).delete() Changefeed output: { new_val: null, old_val: { id: '362ae837-2e29-4695-adef-4fa415138f90', name: 'Bob', ... } }
23. Changefeeds r.table("users").changes() r.table("users") .get("362ae837-2e29-4695-adef-4fa415138f90") .update({name: "Bobbby"}) Changefeed output: { new_val: { id: '362ae837-2e29-4695-adef-4fa415138f90', name: 'Bobby' }, old_val: { id: '362ae837-2e29-4695-adef-4fa415138f90', name: 'Bob' } }
24. Changefeeds r.table("players") .orderBy({index: r.desc("score")}) .limit(3).changes() Track top three players by score Chain the changes command to an actual ReQL query:
25. Using Changefeeds
26. Advanced ReQL Geospatial Features
27. Introduction to Geospatial • Geospatial queries are integrated into ReQL • With changefeeds, you can listen to changes in your geospatial data
28. Geospatial capabilities • Geometry types: point, line, polygon • Geometry Manipulation: fill, polygonSub • Non-Indexed Operations: distance, intersects, includes • Indexed operations: getNearest, getIntersecting
29. Geospatial ReQL Query r.table("starbucks_locations") .getIntersecting(cityOfSacramento, { index: 'location'}) .orderBy(function (row) { return r.point(-122.482, 38.578) .distance(row('location')); }) Closest Starbucks to Hacker Lab in SAC
30. Geospatial ReQL Query r.table("starbucks_locations") .getIntersecting(cityOfSacramento, { index: 'location'}) .orderBy(function (row) { return r.point(-122.482, 38.578) .distance(row('location')); }) Access the database table
31. Geospatial ReQL Query r.table("starbucks_locations") .getIntersecting(cityOfSacramento, { index: 'location'}) .orderBy(function (row) { return r.point(-122.482, 38.578) .distance(row('location')); }) Get all points within polygon
32. Geospatial ReQL Query r.table("starbucks_locations") .getIntersecting(cityOfSacramento, { index: 'location'}) .orderBy(function (row) { return r.point(-122.482, 38.578) .distance(row('location')); }) Order by distance to point
33. Geospatial ReQL Query r.table("starbucks_locations") .getIntersecting(cityOfSacramento, { index: 'location'}) .orderBy(function (row) { return r.point(-122.482, 38.578) .distance(row('location')); }) Return the distance to a point
34. Running Geospatial Queries https://github.com/thejsj/arcgis-rethinkdb
35. Demo Application Using RethinkDB with ArcGIS
36. Question Which counties in the USA have the highest density of restaurants?
37. Demo
38. County + Restaurant data
39. Restaurants in county query return r.table("counties") // 1. Get number of restaurants .merge(...) // 2. Calculate restaurant density .merge(...) Get restaurant density
40. Restaurants in county query Get all counties return r.table("counties") // 1. Get number of restaurants .merge(...) // 2. Calculate restaurant density .merge(...)
41. Restaurants in county query return r.table("counties") // 1. Get number of restaurants .merge({ restaurants: // .. }) // 2. Calculate restaurant density .merge(...) Merge a property into county
42. Restaurants in county query return r.table("counties") // 1. Get number of restaurants .merge({ restaurants: r.table('restaurants') .getIntersecting(row('geometry'), { index: 'geometry' }) .count() }) // 2. Calculate restaurant density .merge(...) Get number of restaurants in county
43. Restaurants in county query Get restaurants table return r.table("counties") // 1. Get number of restaurants .merge({ restaurants: r.table('restaurants') .getIntersecting(row('geometry'), { index: 'geometry' }) .count() }) // 2. Calculate restaurant density .merge(...)
44. Restaurants in county query Get intersecting geometries return r.table("counties") // 1. Get number of restaurants .merge({ restaurants: r.table('restaurants') .getIntersecting(row('geometry'), { index: 'geometry' }) .count() }) // 2. Calculate restaurant density .merge(...)
45. Restaurants in county query Count intersecting geometries return r.table("counties") // 1. Get number of restaurants .merge({ restaurants: r.table('restaurants') .getIntersecting(row('geometry'), { index: 'geometry' }) .count() }) // 2. Calculate restaurant density .merge(...)
46. Restaurants in county query return r.table("counties") // 1. Get number of restaurants .merge(...) // 2. Calculate restaurant density .merge({ stat: row('restaurants').div(row('area')) }) Add new `stat` property
47. Restaurants in county query return r.table("counties") // 1. Get number of restaurants .merge(...) // 2. Calculate restaurant density .merge({ stat: row('restaurants').div(row('area')) }) Divide `restaurants` by `area`
48. Processed data
49. Client/ArcGIS • Data is queried and sent through a WebSocket connection • Polygon geometry is converted into ArcGIS Graphic and added to map • Color (Symbol) is applied dynamically based on max
50. Summary • ReQL allows for complex geospatial queries • RethinkDB makes building realtime applications a lot easier
51. Additional Resources • RethinkDB website:
 http://rethinkdb.com • RethinkDB cookbook:
 http://rethinkdb.com/docs/cookbook • RethinkDB installation:
 http://rethinkdb.com/docs/install/
52. Questions?