MongoNYC 2012: Real Time Data Analytics

MongoNYC 2012: Real Time Data Analytics, Scott Hernandez, 10gen. There are many ways to build dashboards and reports with detailed aggregations. In this talk we will outline one way of using pre-aggregating as data is collected in order to provide real-time views. You may be familiar with options which include post data collection processing, like map/reduce or the new aggregation framework; we will investigate some of the downsides which we will need to be addressed when defining the use case for counters and pre-aggregation with MongoDB.
Come learn how to build these types of reports and see how easy it can be.

Transcript

1. Real Time Data Analytics Pre-aggregation with counters © Copyright 2010 10gen Inc.
2. Goals • Dashboards • (Known) Reports • Real-time numbers
3. Framework • Know your metrics/counter • Prepared reports • Calculate during write • Fast queries • Always up to date • Record time-series collections
4. Dashboard
5. Roads not traveled • Map/Reduce • Reprocess raw data • Now possible to do partial reduce • Aggregation Framework (aggregate in 2.2) • Also reprocess data on operation (initial release) • Optimizations to come • More costly during reads
6. Not Appropriate For • Ad-hoc aggregations (unknown metrics) • One-off reports • Possibly complex calculations
7. Processing • Event received • Split into many updates w/$inc • Aggregate • Input Field(s) • Time periods (hourly, monthly, annual) • Defined Metrics
8. Example Data: github > db.events.findOne() { "repository" : { "url" : "https://github.com/vidageek/games", ... "open_issues" : 25, "watchers" : 6, "pushed_at" : "2012/03/10 08:34:00 -0800", "language" : "Java" }, "actor_attributes" : {...}, "created_at" : "2012/03/11 15:20:24 -0700", "public" : true, "actor" : "juliano", "payload" : {...}, "url" : "https://github.com/...", "type" : "CommitCommentEvent” }
9. Define Metrics • “actor” • “repository.name” • “repository.language” • “type” PushEvent, IssuesEvent, WatchEvent, GistEvent • “payload.ref” efs/heads/improved_history, refs/heads/master, refs/heads/signs
10. Aggregations TimePeriod, type # TimePeriod, author # TimePeriod, project #
11. Stats Collections stats_[hourly/daily/monthly].actors stats_[hourly/daily/monthly].projects stats_[hourly/daily/monthly].langs stats_[hourly/daily/monthly].types
12. Stats > db.stats_hourly.types.find({"_id.type":"GistEvent"}) { "_id" : { "p" : ISODate("2012-05-21T00:00:00Z"), "type" : "GistEvent” }, "hour" : { "2" : { "count" : 65 }, "3" : { "count" : 2 }, "7" : { ”count" : 130}, "8" : { "count" : 5 } }, "total" : { ”count" : 202 } }
13. Updates Increment Query: { ”p" : Date(…), "actor" : "neoplastic"}} Update: { "$inc" : { "h.21.c" : 1 , "t.c" : 1}} Upsert : true
14. Query/Graphing • Select by grouping (by date, by type/value) • Documents hold many data points
15. The Whys • Multiple data points per document • Documents hold many timed points • Good for graphs by time, or types • Nested for improved performance
16. Questions
17. © Copyright 2010 10gen Inc. try at try.mongodb.org
18. © Copyright 2010 10gen Inc. drivers at mongodb.org REST ActionScript3 C# and .NET Clojure ColdFusion Delphi Erlang F# Go: gomongo Groovy Haskell Javascript Lua C C# C++ Erlang Haskell Java Javascript Perl PHP Python Ruby node.js Objective C PHP PowerShell Blog post Python Ruby Scala Scheme (PLT) Smalltalk: Dolphin Smalltalk Community Supported mongodb.org Supported
19. @mongodb © Copyright 2010 10gen Inc. conferences, appearances, and meetups http://www.10gen.com/events http://bit.ly/mongofb Facebook | Twitter | LinkedIn http://linkd.in/joinmongo download at mongodb.org support, training, and this talk brought to you by