MongoDB for Analytics

Presented at MongoChicago on November 13, 2012.

Transcript

1. GitHub
   John Nunemaker
   MongoChicago 2012
   November 12, 2012
   MongoDB for Analytics
   A loving conversation with @jnunemaker
   

   View Slide

2. Background
   How hernias can be good for you
   

   View Slide

3. View Slide

4. View Slide

5. 1 month
   Of evenings and weekends
   

   View Slide

6. 18 months
   Since public launch
   

   View Slide

7. 10-15 Million
   Page views per day
   

   View Slide

8. 2.7 Billion
   Page views to date
   

   View Slide

9. 13 tiny servers
   2 web, 6 app, 3 db, 2 queue
   

   View Slide

10. requests/sec
    

    View Slide

11. ops/sec
    

    View Slide

12. cpu %
    

    View Slide

13. lock %
    

    View Slide

14. Implementation
    How we do what we do
    

    View Slide

15. Doing It (mostly) Live
    No aggregate querying
    

    View Slide

16. View Slide

17. View Slide

18. get('/track.gif') do
    track_service.record(...)
    TrackGif
    end
    

    View Slide

19. class TrackService
    def record(attrs)
    message = MessagePack.pack(attrs)
    @client.set(@queue, message)
    end
    end
    

    View Slide

20. class TrackProcessor
    def run
    loop { process }
    end
    def process
    record @client.get(@queue)
    end
    def record(message)
    attrs = MessagePack.unpack(message)
    Hit.record(attrs)
    end
    end
    

    View Slide

21. http://bit.ly/rt-kestrel
    

    View Slide

22. class Hit
    def record
    site.atomic_update(site_updates)
    Resolution.record(self)
    Technology.record(self)
    Location.record(self)
    Referrer.record(self)
    Content.record(self)
    Search.record(self)
    Notification.record(self)
    View.record(self)
    end
    end
    

    View Slide

23. class Resolution
    def record(hit)
    query = {'_id' => "..."}
    update = {'$inc' => {}}
    update['$inc']["sx.#{hit.screenx}"] = 1
    update['$inc']["bx.#{hit.browserx}"] = 1
    update['$inc']["by.#{hit.browsery}"] = 1
    collection(hit.created_on)
    .update(query, update, :upsert => true)
    end
    end
    end
    

    View Slide

24. Pros
    

    View Slide

25. Pros
    Space
    

    View Slide

26. Pros
    Space
    RAM
    

    View Slide

27. Pros
    Space
    RAM
    Reads
    

    View Slide

28. Pros
    Space
    RAM
    Reads
    Live
    

    View Slide

29. Cons
    

    View Slide

30. Cons
    Writes
    

    View Slide

31. Cons
    Writes
    Constraints
    

    View Slide

32. Cons
    Writes
    Constraints
    More Forethought
    

    View Slide

33. Cons
    Writes
    Constraints
    More Forethought
    No raw data
    

    View Slide

34. http://bit.ly/rt-counters
    http://bit.ly/rt-counters2
    

    View Slide

35. Time Frame
    Minute, hour, month, day, year, forever?
    

    View Slide

36. # of Variations
    One document vs many
    

    View Slide

37. Single Document
    Per Time Frame
    

    View Slide

38. View Slide

39. {
    "t" => 336381,
    "u" => 158951,
    "2011" => {
    "02" => {
    "18" => {
    "t" => 9,
    "u" => 6
    }
    }
    }
    }
    

    View Slide

40. {
    '$inc' => {
    't' => 1,
    'u' => 1,
    '2011.02.18.t' => 1,
    '2011.02.18.u' => 1,
    }
    }
    

    View Slide

41. Single Document
    For all ranges in time frame
    

    View Slide

42. View Slide

43. {
    "_id" =>"...:10",
    "bx" => {
    "320" => 85,
    "480" => 318,
    "800" => 1938,
    "1024" => 5033,
    "1280" => 6288,
    "1440" => 2323,
    "1600" => 3817,
    "2000" => 137
    },
    "by" => {
    "480" => 2205,
    "600" => 7359,
    

    View Slide

44. "600" => 7359,
    "768" => 4515,
    "900" => 3833,
    "1024" => 2026
    },
    "sx" => {
    "320" => 191,
    "480" => 179,
    "800" => 195,
    "1024" => 1059,
    "1280" => 5861,
    "1440" => 3533,
    "1600" => 7675,
    "2000" => 1279
    }
    }
    

    View Slide

45. {
    '$inc' => {
    'sx.1440' => 1,
    'bx.1280' => 1,
    'by.768' => 1,
    }
    }
    

    View Slide

46. Many Documents
    Search terms, content, referrers...
    

    View Slide

47. View Slide

48. [
    {
    "_id" => ":",
    "t" => "ruby class variables",
    "sid" => BSON::ObjectId(''),
    "v" => 352
    },
    {
    "_id" => ":",
    "t" => "ruby unless",
    "sid" => BSON::ObjectId(''),
    "v" => 347
    },
    ]
    

    View Slide

49. Writes
    {'_id' => "#{sid}:#{hash}"}
    

    View Slide

50. Reads
    [['sid', 1], ['v', -1]]
    

    View Slide

51. Growth
    Don’t say shard, don’t say shard...
    

    View Slide

52. Partition Hot Data
    Currently using collections for time frames
    

    View Slide

53. [
    "content.2011.7",
    "content.2011.8",
    "content.2011.9",
    "content.2011.10",
    "content.2011.11",
    "content.2011.12",
    "content.2012.1",
    "content.2012.2",
    "content.2012.3",
    "content.2012.4",
    ]
    

    View Slide

54. [
    "resolutions.2011",
    "resolutions.2012",
    ]
    

    View Slide

55. Move
    

    View Slide

56. Move
    BigintMove
    

    View Slide

57. Move
    BigintMove
    MakeYouWannaMove
    

    View Slide

58. Move
    BigintMove
    MakeYouWannaMove
    DaMove
    

    View Slide

59. Move
    BigintMove
    MakeYouWannaMove
    DaMove
    SmoothMove
    

    View Slide

60. Move
    BigintMove
    MakeYouWannaMove
    DaMove
    SmoothMove
    NightMove
    

    View Slide

61. Move
    BigintMove
    MakeYouWannaMove
    DaMove
    SmoothMove
    NightMove
    DanceMove
    

    View Slide

62. Bigger, Faster Server
    More CPU, RAM, Disk Space
    

    View Slide

63. Users
    Sites
    Content
    Referrers
    Terms
    Engines
    Resolutions
    Locations
    Users
    Sites
    Content
    Referrers
    Terms
    Engines
    Resolutions
    Locations
    

    View Slide

64. Partition by Function
    Spread writes across a few servers
    

    View Slide

65. Users
    Sites
    Content
    Referrers
    Terms
    Engines
    Resolutions
    Locations
    

    View Slide

66. Partition by Server
    Spread writes across a ton of servers,
    way down the road, not worried yet
    

    View Slide

67. GitHub
    Thank you!
    john@github.com
    John Nunemaker
    MongoChicago 2012
    November 12, 2012
    @jnunemaker
    

    View Slide