Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
MongoDB for Analytics - John Nunemaker, Ordered List
Search
mongodb
November 01, 2011
Technology
9
7.6k
MongoDB for Analytics - John Nunemaker, Ordered List
MongoChicago 2011
mongodb
November 01, 2011
Tweet
Share
More Decks by mongodb
See All by mongodb
NoSQL Now! 2012
mongodb
17
3.1k
MongoDB 2.2 At the Silicon Valley MongoDB User Group
mongodb
7
1.2k
Turning off the LAMP Hunter Loftis, Skookum Digital Works
mongodb
1
1.4k
Mobilize Your MongoDB! Developing iPhone and Android Apps in the Cloud Grant Shipley, Red Hat
mongodb
0
440
Beanstalk Data - MongoDB In Production Chris Siefken, CTO Beanstalk Data
mongodb
0
410
New LINQ support in C#/.NET driver Robert Stam, 10gen
mongodb
9
40k
Welcome and Keynote Aaron Heckman, 10gen
mongodb
0
390
Webinar Introduction to MongoDB's Java Driver
mongodb
0
1.1k
Webinar Intro to Schema Design
mongodb
4
1.7k
Other Decks in Technology
See All in Technology
現代CSSフレームワークの内部実装とその仕組み
poteboy
8
3.5k
20240416_devopsdaystokyo
kzkmaeda
1
210
2024/4/26 コンピュータ歴史博物館解説告知
toshi_atsumi
0
210
AOAI をきっかけに 社内の Azure 管理を見直した話
recruitengineers
PRO
1
240
KubeCon EU 2024 Recap “Kubernetes Policy Time Machine: Where to Next?”
ryysud
0
200
Azure犬駆動開発の記録/GlobalAzureFukuoka2024_20240420
nina01
1
200
Databricks における 『MLOps』
databricksjapan
2
160
On Your Data を超えていく!
hirotomotaguchi
2
650
Cracking the KubeCon CfP
inductor
2
220
Cloud Native Java with Spring Boot (CNCF Aarhus, April 2024)
thomasvitale
1
160
本当のAWS基礎
toru_kubota
0
480
MLOpsの「壁」を乗り越える、LINEヤフーの Data Quality as Code
lycorptech_jp
PRO
4
270
Featured
See All Featured
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
227
16k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
501
140k
A designer walks into a library…
pauljervisheath
200
23k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
Six Lessons from altMBA
skipperchong
21
3k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
352
28k
GraphQLとの向き合い方2022年版
quramy
32
12k
Why You Should Never Use an ORM
jnunemaker
PRO
51
8.6k
Testing 201, or: Great Expectations
jmmastey
28
6.3k
Stop Working from a Prison Cell
hatefulcrawdad
266
19k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
17
1.4k
Designing for Performance
lara
601
67k
Transcript
Ordered List John Nunemaker MongoChi 2011 October 18, 2011 MongoDB
for Analytics A loving conversation with @jnunemaker
Background As presented through interpretive dance
None
None
None
~1 month Of evenings and weekends
~4 dog years Since public launch
~6 tiny servers 2 web, 2 app, 2 db
~1-2 Million Page views per day
None
None
Implementation Imma show you how we do what we do
baby
Doing It Live No aggregate querying
get('/track.gif') do Hit.record(...) TrackGif end
class Hit def record site.atomic_update(site_updates) Resolution.record(self) Technology.record(self) Location.record(self) Referrer.record(self) Content.record(self)
Search.record(self) Notification.record(self) View.record(self) end end
class Resolution def record(hit) query = {'_id' => "..."} update
= {'$inc' => {}} update['$inc']["sx.#{hit.screenx}"] = 1 update['$inc']["bx.#{hit.browserx}"] = 1 update['$inc']["by.#{hit.browsery}"] = 1 collection(hit.created_on) .update(query, update, :upsert => true) end end end
Pros
Pros Space
Pros Space RAM
Pros Space RAM Reads
Pros Space RAM Reads Live
Cons
Cons Writes
Cons Writes Constraints
Cons Writes Constraints More Forethought
Cons Writes Constraints More Forethought No raw data
Time Frame Minute, hour, month, day, year, forever?
# of Variations One document vs many
Single Document Per Time Frame
None
{ "t" => 336381, "u" => 158951, "2011" => {
"02" => { "18" => { "t" => 9, "u" => 6 } } } }
{ '$inc' => { 't' => 1, 'u' => 1,
'2011.02.18.t' => 1, '2011.02.18.u' => 1, } }
Single Document For all ranges in time frame
None
{ "_id" =>"...:10", "bx" => { "320" => 85, "480"
=> 318, "800" => 1938, "1024" => 5033, "1280" => 6288, "1440" => 2323, "1600" => 3817, "2000" => 137 }, "by" => { "480" => 2205, "600" => 7359,
"600" => 7359, "768" => 4515, "900" => 3833, "1024"
=> 2026 }, "sx" => { "320" => 191, "480" => 179, "800" => 195, "1024" => 1059, "1280" => 5861, "1440" => 3533, "1600" => 7675, "2000" => 1279 } }
{ '$inc' => { 'sx.1440' => 1, 'bx.1280' => 1,
'by.768' => 1, } }
Many Documents Search terms, content, referrers...
None
[ { "_id" => "<oid>:<hash>", "t" => "ruby class variables",
"sid" => BSON::ObjectId('<oid>'), "v" => 352 }, { "_id" => "<oid>:<hash>", "t" => "ruby unless", "sid" => BSON::ObjectId('<oid>'), "v" => 347 }, ]
Writes {'_id' => "#{site_id}:#{hash}"}
Reads [['sid', 1], ['v', -1]]
Growth The best laid plans of mice and men
Partition Hot Data Currently using collections for time frames
Bigger, Faster Server More CPU, RAM, Disk Space
Users Sites Content Referrers Terms Engines Resolutions Locations Users Sites
Content Referrers Terms Engines Resolutions Locations
Partition by Function Spread writes across a few servers
Users Sites Content Referrers Terms Engines Resolutions Locations
Partition by Server Spread writes across a ton of servers,
way down the road, not worried yet
Ordered List Thank you!
[email protected]
John Nunemaker MongoChi 2011 October
18, 2011 @jnunemaker