Finding the right NoSQL DB for the job - The path to a non-RDBMS solution at Traackr

A walkthrough of Traackr's experience in choosing a NoSQL solution and how we ended up up switching from HBase to MongoDB. This deck goes through some in depth technical aspects, like schema design and our use of secondary indexes.

Transcript

1. Finding  the  right  NoSQL  DB  for  
   the  job  
   The  path  to  a  non-­‐RDBMS  solu>on  at  
   

   View Slide

2. Who  we  are  
   •  A  search  
   engine  
   •  A  people  
   search  engine  
   •  An  influencer  
   search  engine  
   •  Subscrip>on-­‐
   based  
   

   View Slide

3. George  Stathis  
   VP  Engineering  
   14+  years  of  experience  
   building  full-­‐stack  web  
   soKware  systems  with  a  past  
   focus  on  e-­‐commerce  and  
   publishing.  Currently  
   responsible  for  building  
   engineering  capability  to  
   enable  Traackr's  growth  goals.  
   

   View Slide

4. What’s  this  talk  about?  
   •  Why  we  picked  a  NoSQL  database  
   •  How  we  picked  a  NoSQL  database  
   •  My  NoSQL  does  not  do  the  job!  What  now?!  
   •  Nirvana  =  the  right  tool  for  the  job  
   

   View Slide

5. Why  did  we  pick  a  NoSQL  DB?  
   

   View Slide

6. There  are  some  misconcep>ons  
   around  NoSQL  only  being  appropriate  
   when  one  needs  to  achieve  
   “Web  Scale”    
   

   View Slide

7. I  need  web  scale!  
   hXp://www.youtube.com/watch?v=b2F-­‐DItXtZs  
   

   View Slide

8. Traackr  picked  NoSQL;  are  we    
   “Web  Scale”?  
   

   View Slide

9. •  In  terms  of  users/traffic?  
   Do  we  fit  the  “Web  scale”  profile?  
   

   View Slide

10. Source:  compete.com  
    

    View Slide

11. Source:  compete.com  
    

    View Slide

12. Source:  compete.com  
    

    View Slide

13. Source:  compete.com  
    

    View Slide

14. Source:  highscalability.com  
    

    View Slide

15. View Slide

16. •  In  terms  of  users/traffic?  
    •  In  terms  of  the  amount  of  data?  
    Do  we  fit  the  “Web  scale”  profile?  
    

    View Slide

17. PRIMARY>  use  traackr  
    switched  to  db  traackr  
    PRIMARY>  db.stats()  
    {  
     "db"  :  "traackr",  
     "collec>ons"  :  12,  
     "objects"  :  68226121,  
     "avgObjSize"  :  2972.0800625760330,  
     "dataSize"  :  202773493971,  
     "storageSize"  :  221491429671,  
     "numExtents"  :  199,  
     "indexes"  :  33,  
     "indexSize"  :  27472394891,  
     "fileSize"  :  266623699968,  
     "nsSizeMB"  :  16,  
     "ok"  :  1  
    }  
    That’s  a  quarter  of  a  
    terabyte  …  
    

    View Slide

18. Wait!  What?  My  
    Synology  NAS  at  home  
    can  hold  2TB!  
    

    View Slide

19. No  need  for  us  to  track  the  en>re  web  
    Web  Content  
    Influencer  
    Content  
    Not  at  scale  :-­‐)  
    

    View Slide

20. •  In  terms  of  users/traffic?  
    •  In  terms  of  the  amount  of  data?  
    Do  we  fit  the  “Web  scale”  profile?  
    

    View Slide

21. Alternate  view  of  “Web  Scale”  
    Web  data  is:  
    Heterogeneous  
    Unstructured  (text)  
    

    View Slide

22. Source:  hXp://www.opte.org/  
    Visualiza>on  of  the  Internet,  Nov.  23rd  2003  
    

    View Slide

23. Data  sources  are  
    isolated  islands  of  rich  
    data  with  lose  links  to  
    one  another    
    

    View Slide

24. How  do  we  build  a  database  that  
    models  all  possible  en>>es  found  on  
    the  web?  
    

    View Slide

25. Modeling  the  web:  the  RDBMS  way  
    

    View Slide

26. Source:  socialbuXerflyclt.com  
    

    View Slide

27. or  
    

    View Slide

28. View Slide

29. {  
         "realName":  "David  Chancogne",  
         ">tle":  "CTO",  
         "descrip>on":  "Web.  Geek.\r\nTraackr:  hXp://traackr.com\r\nPropz:  hXp://propz.me",  
         "primaryAffilia>on":  "Traackr",  
         "email":  "dchancogne@traackr.com",  
         "loca>on":  "Cambridge,  MA,  United  States",  
         "siteReferences":  [  
               {  
                     "siteUrl":  "hXp://twiXer.com/dchancogne",  
                     "metrics":  [  
                           {  
                                 "value":  216,  
                                 "name":  "twiXer_followers_count"  
                           },  
                           {  
                                 "value":  2107,  
                                 "name":  "twiXer_statuses_count"  
                           }  
                     ]  
               },  
               {  
                     "siteUrl":  "hXp://traackr.com/blog/author/david",  
                     "metrics":  [  
                           {  
                                 "value":  21,  
                                 "name":  "google_inbound_links"  
                           }  
                     ]  
               }  
         ]  
    }  
    Influencer  data  as  JSON  
    

    View Slide

30. “In  the  old  world  of  data  analysis  you  knew  
    exactly  which  ques>ons  you  wanted  to  ask,  
    which  drove  a  very  predictable  collec>on  and  
    storage  model.  In  the  new  world  of  data  analysis  
    your  ques>ons  are  going  to  evolve  and  change  
    over  >me  and  as  such  you  need  to  be  able  to  
    collect,  store  and  analyze  data  without  being  
    constrained  by  resources.”  
    —  Werner  Vogels,  CTO/VP  Amazon.com  
    

    View Slide

31. NoSQL  =  schema  flexibility  
    

    View Slide

32. •  In  terms  of  users/traffic?  
    •  In  terms  of  the  amount  of  data?  
    Do  we  fit  the  “Web  scale”  profile?  
    

    View Slide

33. •  In  terms  of  users/traffic?  
    •  In  terms  of  the  amount  of  data?  
    •  In  terms  of  the  variety  of  the  data  
    Do  we  fit  the  “Web  scale”  profile?  
    ✓  
    

    View Slide

34. Traackr’s  Datastore  Requirements  
    •  Schema  flexibility  
    •  Good  at  storing  lots  of  variable  length  text  
    •  Batch  processing  op>ons  
    ✓  
    

    View Slide

35. Requirement:  text  storage  
    Variable  text  length:  
    <  big  variance  <  
    140  
    character  
    tweets  
    mul>-­‐page  
    blog  posts  
    

    View Slide

36. Requirement:  text  storage  
    RDBMS’  answer  to  variable  text  length:  
    Plan  ahead  for  largest  value  
    CLOB/BLOB  
    

    View Slide

37. Requirement:  text  storage  
    Issues  with  CLOB/BLOG  for  us:  
    No  clue  what  largest  value  is  
    CLOB/BLOB  for  tweets  =  wasted  space  
    

    View Slide

38. Requirement:  text  storage  
    NoSQL  solu>ons  are  great  for  text:  
    No  length  requirements  (automated  
    chunking)  
    Limited  space  overhead  
    

    View Slide

39. Traackr’s  Datastore  Requirements  
    •  Schema  flexibility  
    •  Good  at  storing  lots  of  variable  length  text  
    •  Batch  processing  op>ons  
    ✓  
    ✓  
    

    View Slide

40. Requirement:  batch  processing  
    Some  NoSQL  
    solu>ons  come  
    with  MapReduce  
    Source:  hXp://code.google.com/  
    

    View Slide

41. Requirement:  batch  processing  
    MapReduce  +  RDBMS:  
    Possible  but  proprietary  solu>ons  
    Usually  involves  expor>ng  data  from  
    RDBMS  into  a  NoSQL  system  anyway.  
    Defeats  data  locality  benefit  of  MR  
    

    View Slide

42. Traackr’s  Datastore  Requirements  
    •  Schema  flexibility  
    •  Good  at  storing  lots  of  variable  length  text  
    •  Batch  processing  op>ons  
    ✓  
    ✓  
    A  NoSQL  op>on  is  the  right  fit  
    ✓  
    

    View Slide

43. How  did  we  pick  a  NoSQL  DB?  
    

    View Slide

44. Bewildering  number  of  op>ons  
    Key/Value  Databases  
    •  Distributed  hashtables  
    •  Designed  for  high  load  
    •  In-­‐memory  or  on-­‐disk  
    •  Eventually  consistent  
    Column  Databases  
    •  Spread  sheet  like  
    •  Key  is  a  row  id  
    •  AXributes  are  columns  
    •  Columns  can  be  grouped  
    into  families  
    Document  Databases  
    •  Like  Key/Value  
    •  Value  =  Document  
    •  Document  =  JSON/BSON  
    •  JSON  =  Flexible  Schema  
    Graph  Databases  
    •  Graph  Theory  G=(E,V)  
    •  Great  for  modeling  
    networks  
    •  Great  for  graph-­‐based  
    query  algorithms    
    

    View Slide

45. Bewildering  number  of  op>ons  
    Key/Value  Databases  
    •  Distributed  hashtables  
    •  Designed  for  high  load  
    •  In-­‐memory  or  on-­‐disk  
    •  Eventually  consistent  
    Column  Databases  
    •  Spread  sheet  like  
    •  Key  is  a  row  id  
    •  AXributes  are  columns  
    •  Columns  can  be  grouped  
    into  families  
    Document  Databases  
    •  Like  Key/Value  
    •  Value  =  Document  
    •  Document  =  JSON/BSON  
    •  JSON  =  Flexible  Schema  
    Graph  Databases  
    •  Graph  Theory  G=(E,V)  
    •  Great  for  modeling  
    networks  
    •  Great  for  graph-­‐based  
    query  algorithms    
    

    View Slide

46. Trimming  op>ons  
    Key/Value  Databases  
    •  Distributed  hashtables  
    •  Designed  for  high  load  
    •  In-­‐memory  or  on-­‐disk  
    •  Eventually  consistent  
    Column  Databases  
    •  Spread  sheet  like  
    •  Key  is  a  row  id  
    •  AXributes  are  columns  
    •  Columns  can  be  grouped  
    into  families  
    Document  Databases  
    •  Like  Key/Value  
    •  Value  =  Document  
    •  Document  =  JSON/BSON  
    •  JSON  =  Flexible  Schema  
    Graph  Databases  
    •  Graph  Theory  G=(E,V)  
    •  Great  for  modeling  
    networks  
    •  Great  for  graph-­‐based  
    query  algorithms    
    Graph  Databases:  while  we  can  model  
    our  domain  as  a  graph  we  don’t  want  to  
    pigeonhole  ourselves  into  this  structure.  
    We’d  rather  use  these  tools  for  
    specialized  data  analysis  but  not  as  the  
    main  data  store.  
    

    View Slide

47. Trimming  op>ons  
    Key/Value  Databases  
    •  Distributed  hashtables  
    •  Designed  for  high  load  
    •  In-­‐memory  or  on-­‐disk  
    •  Eventually  consistent  
    Column  Databases  
    •  Spread  sheet  like  
    •  Key  is  a  row  id  
    •  AXributes  are  columns  
    •  Columns  can  be  grouped  
    into  families  
    Document  Databases  
    •  Like  Key/Value  
    •  Value  =  Document  
    •  Document  =  JSON/BSON  
    •  JSON  =  Flexible  Schema  
    Graph  Databases  
    •  Graph  Theory  G=(E,V)  
    •  Great  for  modeling  
    networks  
    •  Great  for  graph-­‐based  
    query  algorithms    
    Memcache:  memory-­‐based,  
    we  need  true  persistence  
    

    View Slide

48. Trimming  op>ons  
    Key/Value  Databases  
    •  Distributed  hashtables  
    •  Designed  for  high  load  
    •  In-­‐memory  or  on-­‐disk  
    •  Eventually  consistent  
    Column  Databases  
    •  Spread  sheet  like  
    •  Key  is  a  row  id  
    •  AXributes  are  columns  
    •  Columns  can  be  grouped  
    into  families  
    Document  Databases  
    •  Like  Key/Value  
    •  Value  =  Document  
    •  Document  =  JSON/BSON  
    •  JSON  =  Flexible  Schema  
    Graph  Databases  
    •  Graph  Theory  G=(E,V)  
    •  Great  for  modeling  
    networks  
    •  Great  for  graph-­‐based  
    query  algorithms    
    Amazon  SimpleDB:  not  willing  to  
    store  our  data  in  a  proprietary  
    datastore.  
    

    View Slide

49. Trimming  op>ons  
    Key/Value  Databases  
    •  Distributed  hashtables  
    •  Designed  for  high  load  
    •  In-­‐memory  or  on-­‐disk  
    •  Eventually  consistent  
    Column  Databases  
    •  Spread  sheet  like  
    •  Key  is  a  row  id  
    •  AXributes  are  columns  
    •  Columns  can  be  grouped  
    into  families  
    Document  Databases  
    •  Like  Key/Value  
    •  Value  =  Document  
    •  Document  =  JSON/BSON  
    •  JSON  =  Flexible  Schema  
    Graph  Databases  
    •  Graph  Theory  G=(E,V)  
    •  Great  for  modeling  
    networks  
    •  Great  for  graph-­‐based  
    query  algorithms    
    Not  willing  to  store  our  data  in  a  
    proprietary  datastore.  
    Redis  and  LinkedIn’s  Project  
    Voldermort:  no  query  filters,  
    beXer  used  as  queues  or  
    distributed  caches  
    

    View Slide

50. Trimming  op>ons  
    Key/Value  Databases  
    •  Distributed  hashtables  
    •  Designed  for  high  load  
    •  In-­‐memory  or  on-­‐disk  
    •  Eventually  consistent  
    Column  Databases  
    •  Spread  sheet  like  
    •  Key  is  a  row  id  
    •  AXributes  are  columns  
    •  Columns  can  be  grouped  
    into  families  
    Document  Databases  
    •  Like  Key/Value  
    •  Value  =  Document  
    •  Document  =  JSON/BSON  
    •  JSON  =  Flexible  Schema  
    Graph  Databases  
    •  Graph  Theory  G=(E,V)  
    •  Great  for  modeling  
    networks  
    •  Great  for  graph-­‐based  
    query  algorithms    
    CouchDB:  no  ad-­‐hoc  queries;  
    maturity  in  early  2010  made  us  
    shy  away  although  we  did  try  
    early  prototypes.  
    

    View Slide

51. Trimming  op>ons  
    Key/Value  Databases  
    •  Distributed  hashtables  
    •  Designed  for  high  load  
    •  In-­‐memory  or  on-­‐disk  
    •  Eventually  consistent  
    Column  Databases  
    •  Spread  sheet  like  
    •  Key  is  a  row  id  
    •  AXributes  are  columns  
    •  Columns  can  be  grouped  
    into  families  
    Document  Databases  
    •  Like  Key/Value  
    •  Value  =  Document  
    •  Document  =  JSON/BSON  
    •  JSON  =  Flexible  Schema  
    Graph  Databases  
    •  Graph  Theory  G=(E,V)  
    •  Great  for  modeling  
    networks  
    •  Great  for  graph-­‐based  
    query  algorithms    
    Cassandra:  in  early  2010,  
    maturity  ques>ons,  no  secondary  
    indexes  and  no  batch  processing  
    op>ons  (came  later  on).  
    

    View Slide

52. Trimming  op>ons  
    Key/Value  Databases  
    •  Distributed  hashtables  
    •  Designed  for  high  load  
    •  In-­‐memory  or  on-­‐disk  
    •  Eventually  consistent  
    Column  Databases  
    •  Spread  sheet  like  
    •  Key  is  a  row  id  
    •  AXributes  are  columns  
    •  Columns  can  be  grouped  
    into  families  
    Document  Databases  
    •  Like  Key/Value  
    •  Value  =  Document  
    •  Document  =  JSON/BSON  
    •  JSON  =  Flexible  Schema  
    Graph  Databases  
    •  Graph  Theory  G=(E,V)  
    •  Great  for  modeling  
    networks  
    •  Great  for  graph-­‐based  
    query  algorithms    
    MongoDB:  in  early  2010,  maturity  
    ques>ons,  adop>on  ques>ons  
    and  no  batch  processing  op>ons.  
    

    View Slide

53. Trimming  op>ons  
    Key/Value  Databases  
    •  Distributed  hashtables  
    •  Designed  for  high  load  
    •  In-­‐memory  or  on-­‐disk  
    •  Eventually  consistent  
    Column  Databases  
    •  Spread  sheet  like  
    •  Key  is  a  row  id  
    •  AXributes  are  columns  
    •  Columns  can  be  grouped  
    into  families  
    Document  Databases  
    •  Like  Key/Value  
    •  Value  =  Document  
    •  Document  =  JSON/BSON  
    •  JSON  =  Flexible  Schema  
    Graph  Databases  
    •  Graph  Theory  G=(E,V)  
    •  Great  for  modeling  
    networks  
    •  Great  for  graph-­‐based  
    query  algorithms    
    Riak:  very  close  but  in  early  2010,  
    we  had  adop>on  ques>ons.  
    

    View Slide

54. Trimming  op>ons  
    Key/Value  Databases  
    •  Distributed  hashtables  
    •  Designed  for  high  load  
    •  In-­‐memory  or  on-­‐disk  
    •  Eventually  consistent  
    Column  Databases  
    •  Spread  sheet  like  
    •  Key  is  a  row  id  
    •  AXributes  are  columns  
    •  Columns  can  be  grouped  
    into  families  
    Document  Databases  
    •  Like  Key/Value  
    •  Value  =  Document  
    •  Document  =  JSON/BSON  
    •  JSON  =  Flexible  Schema  
    Graph  Databases  
    •  Graph  Theory  G=(E,V)  
    •  Great  for  modeling  
    networks  
    •  Great  for  graph-­‐based  
    query  algorithms    
    HBase:  came  across  as  the  most  mature  
    at  the  >me,  with  several  deployments,  a  
    healthy  community,  "out-­‐of-­‐the  box"  
    secondary  indexes  through  a  contrib  and  
    support  for  batch  processing  using  
    Hadoop/MR  .  
    

    View Slide

55. Climbing  the  learning  curve  
    

    View Slide

56. When  Big-­‐Data  =  Big  Architectures  
    Source:  hXp://www.larsgeorge.com/2009/10/hbase-­‐architecture-­‐101-­‐storage.html  
    Must  have  a  Hadoop  HDFS  
    cluster  of  at  least  2x  replica>on  
    factor  nodes  
    Must  have  an  odd  
    number  of    
    Zookeeper  quorum  
    nodes  
    Then  you  can  run  your  Hbase  
    nodes  but  it’s  recommended  to  
    co-­‐locate  regionservers  with  
    hadoop  datanodes  so  you  have  
    to  manage  resources.  
    Master/slave  architecture  
    means  a  single  point  of  failure,  
    so  you  need  to  protect  your  
    master.  
    And  then  we  also  have  to  
    manage  the  MapReduce  
    processes  and  resources  in  the  
    Hadoop  layer.  
    

    View Slide

57. Source:  socialbuXerflyclt.com  
    

    View Slide

58. Jokes  aside,  no  one  said  open  source  
    was  easy  to  use  
    

    View Slide

59. To  be  expected  
    •  Hadoop/Hbase  are  
    designed  to  move  
    mountains  
    •  If  you  want  to  move  big  
    stuff,  be  prepared  to  
    some>mes  use  big  
    equipment  
    

    View Slide

60. What  it  means  to  a  startup  
    Development  capacity  before  
    Development  capacity  aKer  
    Congrats,  you  
    are  now  a  
    sysadmin…  
    

    View Slide

61. Whatever,  we  can  do  it!  
    Source:  hXp://knowyourmeme.com/memes/honey-­‐badger  
    

    View Slide

62. Mapping  an  A-­‐List  to  a  column  store  
    Name  
    Ranks   References  to  influencer  records  
    

    View Slide

63. Mapping  an  A-­‐List  to  a  column  store  
    Unique  
    key    
    “aXributes”  
    column  family  
    for  general  
    aXributes  
    “influencerId”  column  family  
    for  influencer  ranks  and  foreign  keys  
    

    View Slide

64. Mapping  an  A-­‐List  to  a  column  store  
    Qualifiers  (basically  aXribute  names)  
    

    View Slide

65. Mapping  an  A-­‐List  to  a  column  store  
    “name”  aXribute  
    Influencer  ranks  
    can  be  aXribute  
    names  as  well  
    

    View Slide

66. Mapping  an  A-­‐List  to  a  column  store  
    Alist  name  
    value  
    Influencer  id  values  assigned  to  each  rank  (basically  
    foreign  keys  to  an  influencer  table)  
    

    View Slide

67. Mapping  an  A-­‐List  to  a  column  store  
    Can  get  preXy  long  so  needs  indexing  and  pagina>on  
    

    View Slide

68. Problem:  no  out-­‐of-­‐the-­‐box  row-­‐based  
    indexing  and  pagina>on  
    

    View Slide

69. Whatever,  it’s  open-­‐source!  
    Source:  hXp://knowyourmeme.com/memes/honey-­‐badger  
    

    View Slide

70. Jumping  right  into  the  code  
    

    View Slide

71. MapReduce  for  batch  scoring  
    •  Need  to  re-­‐score  our  influencer  
    database  once  a  week  
    •  M/R  cranked  through  it  in  15  mins  
    

    View Slide

72. Source:  hXp://www.charliesheentshirts.info/  
    

    View Slide

73. a  few  months  later…  
    

    View Slide

74. Need  to  upgrade  to  Hbase  0.90  
    •  Making  sure  to  remain  on  recent  code  base  
    •  Performance  improvements  
    •  Mostly  to  get  the  latest  bug  fixes  
    No  thanks!  
    

    View Slide

75. Looks  like  something  is  missing  
    

    View Slide

76. View Slide

77. Our  DB  indexes  depend  on  this!  
    

    View Slide

78. Let’s  get  this  straight  
    •  Hbase  no  longer  comes  with  secondary  
    indexing  out-­‐of-­‐the-­‐box  
    •  It’s  been  moved  out  of  the  trunk  to  GitHub  
    •  Where  only  one  other  company  besides  us  
    seems  to  care  about  it  
    

    View Slide

79. Only  one  other  
    maintainer  
    besides  us  
    

    View Slide

80. What  it  means  to  a  startup  
    Development  capacity  
    Congrats,  you  are  
    now  an  hbase  
    maintainer…  
    

    View Slide

81. Source:  socialbuXerflyclt.com  
    

    View Slide

82. Whatever,  we’ll  roll  our  own  indexing!  
    Source:  hXp://knowyourmeme.com/memes/honey-­‐badger  
    

    View Slide

83. Homegrown  Hbase  Indexes  
    Rows  have  id  prefixes  that  can  be  
    efficiently  scanned  using  STARTROW  
    and  STOPROW  filters  
    Row  ids  for  Posts  
    

    View Slide

84. Homegrown  Hbase  Indexes  
    Find  posts  for  
    influencer_id_1234    
    Row  ids  for  Posts  
    

    View Slide

85. Homegrown  Hbase  Indexes  
    Find  posts  for  
    influencer_id_5678  
    Row  ids  for  Posts  
    

    View Slide

86. Homegrown  Hbase  Indexes  
    •  No  longer  depending  on  
    unmaintained  code  
    •  Work  with  out-­‐of-­‐the-­‐box  Hbase  
    installa>on  
    

    View Slide

87. What  it  means  to  a  startup  
    Development  capacity  
    You  are  back  but  
    you  s>ll  need  to  
    maintain  indexing  
    logic  
    

    View Slide

88. Source:  hXp://www.charliesheentshirts.info/  
    Applica>on  layer  indexes  are  slow  and  
    briXle.  The  DB  should  be  doing  this,  not  
    us.  
    Sort  of…  
    

    View Slide

89. a  few  months  later…  
    

    View Slide

90. Cracks  in  the  data  model  
    huffingtonpost.com  
    huffingtonpost.com  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_1.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_2.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_3.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post1.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post2.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post3.html  
    writes  for  
    authored  by  
    published  under  
    writes  for  
    authored  by  
    published  under  
    

    View Slide

91. Cracks  in  the  data  model  
    huffingtonpost.com  
    huffingtonpost.com  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_1.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_2.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_3.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post1.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post2.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post3.html  
    writes  for  
    authored  by  
    published  under  
    writes  for  
    authored  by  
    published  under  
    Denormalized/duplicated  
    for  fast  run>me  access  
    and  storage  of  influencer-­‐
    to-­‐site  rela>onship  
    proper>es  
    

    View Slide

92. Cracks  in  the  data  model  
    huffingtonpost.com  
    huffingtonpost.com  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_1.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_2.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_3.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post1.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post2.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post3.html  
    writes  for  
    authored  by  
    published  under  
    writes  for  
    authored  by  
    published  under  
    Content  aXribu>on  logic  could  some>mes  
    mis-­‐aXribute  posts  because  of  the  
    duplicated  data.  
    

    View Slide

93. Cracks  in  the  data  model  
    huffingtonpost.com  
    huffingtonpost.com  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_1.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_2.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_3.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post1.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post2.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post3.html  
    writes  for  
    authored  by  
    published  under  
    writes  for  
    authored  by  
    published  under  
    Exacerbated  when  we  started  tracking  
    people’s  content  on  a  daily  basis  in  
    mid-­‐2011  
    

    View Slide

94. Fixing  the  cracks  in  the  data  model  
    huffingtonpost.com  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_1.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_2.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_3.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post1.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post2.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post3.html  
    writes  for  
    authored  by  
    published  under  
    writes  for  
    authored  by  
    published  under  
    

    View Slide

95. Fixing  the  cracks  in  the  data  model  
    huffingtonpost.com  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_1.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_2.html  
    hXp://www.huffingtonpost.com/arianna-­‐huffington/post_3.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post1.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post2.html  
    hXp://www.huffingtonpost.com/shaun-­‐donovan/post3.html  
    writes  for  
    authored  by  
    published  under  
    writes  for  
    authored  by  
    published  under  
    Normalize  the  sites  
    

    View Slide

96. Fixing  the  cracks  in  the  data  model  
    •  Normaliza>on  requires  stronger  
    secondary  indexing  
    •  Our  applica>on  layer  indexing  would  
    need  revisi>ng…again!  
    

    View Slide

97. What  it  means  to  a  startup  
    Development  capacity  
    Psych!  You  are  back  
    to  wri>ng  indexing  
    code.  
    

    View Slide

98. Source:  socialbuXerflyclt.com  
    

    View Slide

99. Whatever,  we’ll  change  our  NoSQL!  
    Source:  hXp://knowyourmeme.com/memes/honey-­‐badger  
    

    View Slide

100. Traackr’s  Datastore  Requirements  
     (Revisited)  
     •  Schema  flexibility  
     •  Good  at  storing  lots  of  variable  length  text  
     •  Batch  processing  op>ons  (maybe)  
     •  Out-­‐of-­‐the-­‐box  SECONDARY  INDEX  support!  
     •  Simple  to  use  and  administer  
     

     View Slide

101. NoSQL  picking  –  Round  2  
     Key/Value  Databases  
     •  Distributed  hashtables  
     •  Designed  for  high  load  
     •  In-­‐memory  or  on-­‐disk  
     •  Eventually  consistent  
     Column  Databases  
     •  Spread  sheet  like  
     •  Key  is  a  row  id  
     •  AXributes  are  columns  
     •  Columns  can  be  grouped  
     into  families  
     Document  Databases  
     •  Like  Key/Value  
     •  Value  =  Document  
     •  Document  =  JSON/BSON  
     •  JSON  =  Flexible  Schema  
     Graph  Databases  
     •  Graph  Theory  G=(E,V)  
     •  Great  for  modeling  
     networks  
     •  Great  for  graph-­‐based  
     query  algorithms    
     

     View Slide

102. NoSQL  picking  –  Round  2  
     Key/Value  Databases  
     •  Distributed  hashtables  
     •  Designed  for  high  load  
     •  In-­‐memory  or  on-­‐disk  
     •  Eventually  consistent  
     Column  Databases  
     •  Spread  sheet  like  
     •  Key  is  a  row  id  
     •  AXributes  are  columns  
     •  Columns  can  be  grouped  
     into  families  
     Document  Databases  
     •  Like  Key/Value  
     •  Value  =  Document  
     •  Document  =  JSON/BSON  
     •  JSON  =  Flexible  Schema  
     Graph  Databases  
     •  Graph  Theory  G=(E,V)  
     •  Great  for  modeling  
     networks  
     •  Great  for  graph-­‐based  
     query  algorithms    
     Nope!  
     

     View Slide

103. NoSQL  picking  –  Round  2  
     Key/Value  Databases  
     •  Distributed  hashtables  
     •  Designed  for  high  load  
     •  In-­‐memory  or  on-­‐disk  
     •  Eventually  consistent  
     Column  Databases  
     •  Spread  sheet  like  
     •  Key  is  a  row  id  
     •  AXributes  are  columns  
     •  Columns  can  be  grouped  
     into  families  
     Document  Databases  
     •  Like  Key/Value  
     •  Value  =  Document  
     •  Document  =  JSON/BSON  
     •  JSON  =  Flexible  Schema  
     Graph  Databases  
     •  Graph  Theory  G=(E,V)  
     •  Great  for  modeling  
     networks  
     •  Great  for  graph-­‐based  
     query  algorithms    
     Graph  Databases:  we  looked  at  
     Neo4J  a  bit  closer  but  passed  again  
     for  the  same  reasons  as  before.  
     

     View Slide

104. NoSQL  picking  –  Round  2  
     Key/Value  Databases  
     •  Distributed  hashtables  
     •  Designed  for  high  load  
     •  In-­‐memory  or  on-­‐disk  
     •  Eventually  consistent  
     Column  Databases  
     •  Spread  sheet  like  
     •  Key  is  a  row  id  
     •  AXributes  are  columns  
     •  Columns  can  be  grouped  
     into  families  
     Document  Databases  
     •  Like  Key/Value  
     •  Value  =  Document  
     •  Document  =  JSON/BSON  
     •  JSON  =  Flexible  Schema  
     Graph  Databases  
     •  Graph  Theory  G=(E,V)  
     •  Great  for  modeling  
     networks  
     •  Great  for  graph-­‐based  
     query  algorithms    
     Memcache:  s>ll  no  
     

     View Slide

105. NoSQL  picking  –  Round  2  
     Key/Value  Databases  
     •  Distributed  hashtables  
     •  Designed  for  high  load  
     •  In-­‐memory  or  on-­‐disk  
     •  Eventually  consistent  
     Column  Databases  
     •  Spread  sheet  like  
     •  Key  is  a  row  id  
     •  AXributes  are  columns  
     •  Columns  can  be  grouped  
     into  families  
     Document  Databases  
     •  Like  Key/Value  
     •  Value  =  Document  
     •  Document  =  JSON/BSON  
     •  JSON  =  Flexible  Schema  
     Graph  Databases  
     •  Graph  Theory  G=(E,V)  
     •  Great  for  modeling  
     networks  
     •  Great  for  graph-­‐based  
     query  algorithms    
     Amazon  SimpleDB:  s>ll  no.  
     

     View Slide

106. NoSQL  picking  –  Round  2  
     Key/Value  Databases  
     •  Distributed  hashtables  
     •  Designed  for  high  load  
     •  In-­‐memory  or  on-­‐disk  
     •  Eventually  consistent  
     Column  Databases  
     •  Spread  sheet  like  
     •  Key  is  a  row  id  
     •  AXributes  are  columns  
     •  Columns  can  be  grouped  
     into  families  
     Document  Databases  
     •  Like  Key/Value  
     •  Value  =  Document  
     •  Document  =  JSON/BSON  
     •  JSON  =  Flexible  Schema  
     Graph  Databases  
     •  Graph  Theory  G=(E,V)  
     •  Great  for  modeling  
     networks  
     •  Great  for  graph-­‐based  
     query  algorithms    
     Not  willing  to  store  our  data  in  a  
     proprietary  datastore.  
     Redis  and  LinkedIn’s  Project  
     Voldermort:  s>ll  no  
     

     View Slide

107. NoSQL  picking  –  Round  2  
     Key/Value  Databases  
     •  Distributed  hashtables  
     •  Designed  for  high  load  
     •  In-­‐memory  or  on-­‐disk  
     •  Eventually  consistent  
     Column  Databases  
     •  Spread  sheet  like  
     •  Key  is  a  row  id  
     •  AXributes  are  columns  
     •  Columns  can  be  grouped  
     into  families  
     Document  Databases  
     •  Like  Key/Value  
     •  Value  =  Document  
     •  Document  =  JSON/BSON  
     •  JSON  =  Flexible  Schema  
     Graph  Databases  
     •  Graph  Theory  G=(E,V)  
     •  Great  for  modeling  
     networks  
     •  Great  for  graph-­‐based  
     query  algorithms    
     CouchDB:  more  mature  but  s>ll  
     no  ad-­‐hoc  queries.  
     

     View Slide

108. NoSQL  picking  –  Round  2  
     Key/Value  Databases  
     •  Distributed  hashtables  
     •  Designed  for  high  load  
     •  In-­‐memory  or  on-­‐disk  
     •  Eventually  consistent  
     Column  Databases  
     •  Spread  sheet  like  
     •  Key  is  a  row  id  
     •  AXributes  are  columns  
     •  Columns  can  be  grouped  
     into  families  
     Document  Databases  
     •  Like  Key/Value  
     •  Value  =  Document  
     •  Document  =  JSON/BSON  
     •  JSON  =  Flexible  Schema  
     Graph  Databases  
     •  Graph  Theory  G=(E,V)  
     •  Great  for  modeling  
     networks  
     •  Great  for  graph-­‐based  
     query  algorithms    
     Cassandra:  matured  quite  a  bit,  added  
     secondary  indexes  and  batch  processing  
     op>ons  but  more  restric>ve  in  its’  use  than  
     other  solu>ons.  AKer  the  Hbase  lesson,  
     simplicity  of  use  was  now  more  important.  
     

     View Slide

109. NoSQL  picking  –  Round  2  
     Key/Value  Databases  
     •  Distributed  hashtables  
     •  Designed  for  high  load  
     •  In-­‐memory  or  on-­‐disk  
     •  Eventually  consistent  
     Column  Databases  
     •  Spread  sheet  like  
     •  Key  is  a  row  id  
     •  AXributes  are  columns  
     •  Columns  can  be  grouped  
     into  families  
     Document  Databases  
     •  Like  Key/Value  
     •  Value  =  Document  
     •  Document  =  JSON/BSON  
     •  JSON  =  Flexible  Schema  
     Graph  Databases  
     •  Graph  Theory  G=(E,V)  
     •  Great  for  modeling  
     networks  
     •  Great  for  graph-­‐based  
     query  algorithms    
     Riak:  strong  contender  s>ll  but  
     adop>on  ques>ons  remained.  
     

     View Slide

110. NoSQL  picking  –  Round  2  
     Key/Value  Databases  
     •  Distributed  hashtables  
     •  Designed  for  high  load  
     •  In-­‐memory  or  on-­‐disk  
     •  Eventually  consistent  
     Column  Databases  
     •  Spread  sheet  like  
     •  Key  is  a  row  id  
     •  AXributes  are  columns  
     •  Columns  can  be  grouped  
     into  families  
     Document  Databases  
     •  Like  Key/Value  
     •  Value  =  Document  
     •  Document  =  JSON/BSON  
     •  JSON  =  Flexible  Schema  
     Graph  Databases  
     •  Graph  Theory  G=(E,V)  
     •  Great  for  modeling  
     networks  
     •  Great  for  graph-­‐based  
     query  algorithms    
     MongoDB:  matured  by  leaps  and  bounds,  increased  
     adop>on,  support  from  10gen,  advanced  indexing  
     out-­‐of-­‐the-­‐box  as  well  as  some  batch  processing  
     op>ons,  breeze  to  use,  well  documented  and  fit  into  
     our  exis>ng  code  base  very  nicely.  
     

     View Slide

111. Immediate  Benefits  
     •  No  more  maintaining  custom  applica>on-­‐layer  
     secondary  indexing  code  
     

     View Slide

112. What  it  means  to  a  startup  
     Development  capacity  
     Yay!  I’m  back!  
     

     View Slide

113. Immediate  Benefits  
     •  No  more  maintaining  custom  applica>on-­‐layer  
     secondary  indexing  code  
     •  Single  binary  installa>on  greatly  simplifies  
     administra>on  
     

     View Slide

114. What  it  means  to  a  startup  
     Development  capacity  
     Honestly,  I  thought  
     I’d  never  see  you  
     guys  again!  
     

     View Slide

115. Immediate  Benefits  
     •  No  more  maintaining  custom  applica>on-­‐layer  
     secondary  indexing  code  
     •  Single  binary  installa>on  greatly  simplifies  
     administra>on  
     •  Our  NoSQL  could  now  support  our  domain  
     model  
     

     View Slide

116. many-­‐to-­‐many  
     rela>onship  
     

     View Slide

117. Modeling  an  influencer  
     Embedded  list  of  
     references  to  sites  
     augmented  with  
     influencer-­‐specific  
     site  aXributes  (e.g.  
     percent  contribu>on  
     to  content)    
     {  
          ”_id":  "770cf5c54492344ad5e45„791ae5d52”,  
          "realName":  "David  Chancogne",  
          ">tle":  "CTO",  
          "descrip>on":  "Web.  Geek.\r\nTraackr:  hXp://traackr.com\r\nPropz:  hXp://propz.me",  
          "primaryAffilia>on":  "Traackr",  
          "email":  "dchancogne@traackr.com",  
          "loca>on":  "Cambridge,  MA,  United  States",  
          "siteReferences":  [  
                {  
                      "siteId":  "b31236da306270dc2b5db34e943af88d",  
                      "contribu>on":  0.25    
                },  
                {  
                      "siteId":  "602dc370945d3b3480fff4f2a541227c",  
                      "contribu>on":  1.0    
                }  
          ]  
     }  
     

     View Slide

118. Modeling  an  influencer  
     siteId  indexed  for  
     “find  influencers  
     connected  to  site  X”  
     > db.influencers.ensureIndex({siteReferences.siteId: 1});!
     > db.influencers.find({siteReferences.siteId: "602dc370945d3b3480fff4f2a541227c"});!
     {  
          ”_id":  "770cf5c54492344ad5e45„791ae5d52”,  
          "realName":  "David  Chancogne",  
          ">tle":  "CTO",  
          "descrip>on":  "Web.  Geek.\r\nTraackr:  hXp://traackr.com\r\nPropz:  hXp://propz.me",  
          "primaryAffilia>on":  "Traackr",  
          "email":  "dchancogne@traackr.com",  
          "loca>on":  "Cambridge,  MA,  United  States",  
          "siteReferences":  [  
                {  
                      "siteId":  "b31236da306270dc2b5db34e943af88d",  
                      "contribu>on":  0.25    
                },  
                {  
                      "siteId":  "602dc370945d3b3480fff4f2a541227c",  
                      "contribu>on":  1.0    
                }  
          ]  
     }  
     

     View Slide

119. Embedded  list  of  
     influencer  references  
     augmented  with  
     “usernames”  (useful  
     for  content  
     aXribu>on)  
     {  
          ”_id":  "0001e86f73cc3975a29e6a98a41a4280”,  
          ”url":  "hXp://traackr.com/blog/",  
          "metrics":  [  
                {  
                    "name":  "google_inbound_links",  
                    "value":  5432  
                }  
            ],  
          "authors":  [  
                {  
                    "username":  "dchancogne",  
                    "influencerId":  "770cf5c54492344ad5e45„791ae5d52"  
                },  
              {  
                    "username":  ”gstathis",  
                    "influencerId":  "0001e86f73cc3975a29e6a98a41a4280"  
                }  
            ]  
     }  
     Modeling  a  site  
     

     View Slide

120. Modeling  a  site  
     Indexed  for  “find  
     sites  associated  to  
     influencer  X”  
     > db.sites.ensureIndex({authors.influencerId: 1});!
     > db.sites.find({authors.influencerId: "0001e86f73cc3975a29e6a98a41a4280"});!
     {  
          ”_id":  "0001e86f73cc3975a29e6a98a41a4280”,  
          ”url":  "hXp://traackr.com/blog/",  
          "metrics":  [  
                {  
                    "name":  "google_inbound_links",  
                    "value":  5432  
                }  
            ],  
          "authors":  [  
                {  
                    "username":  "dchancogne",  
                    "influencerId":  "770cf5c54492344ad5e45„791ae5d52"  
                },  
              {  
                    "username":  ”gstathis",  
                    "influencerId":  "0001e86f73cc3975a29e6a98a41a4280"  
                }  
            ]  
     }  
     

     View Slide

121. Other  index  uses  
     Support  for  alternate  site  URLs  (a.k.a.  URL  aliases):  
     {!
     "_id": "0001e86f73cc3975a29e6a98a41a4280",!
     "url_hash_list": [!
     {!
     "url": "http://traackr.com/blog",!
     "hash": "770cf5c54492344ad5e45fb791ae5d52"!
     },!
     {!
     "url": "http://blog.traackr.com/",!
     "hash": "0001e86f73cc3975a29e6a98a41a4280"!
     }!
     ]!
     }!
      
     Indexed  for  “find  
     sites  associated  to  
     influencer  X”  
     Index  on  MD5  hash  
     of  URL  
     

     View Slide

122. Other  Benefits  
     •  Ad  hoc  queries  and  reports  became  easier  to  write  with  JavaScript:  
     no  need  for  a  Java  developer  to  write  map  reduce  code  to  extract  
     the  data  in  a  usable  form  like  it  was  needed  with  Hbase.  
     

     View Slide

123. Ad  hoc  report  example  
     // File Name: retweetTotal.js!
     // Purpose: report the count of twitter URLs for which we have!
     // computed the the number of total retweets!
     print( "NUMBER OF TWITTER URLS where retweetTotal IS SET:" );!
     print( db.sites.find( { platformName: "twitter.com", !
     retweetTotal: { $exists: true } } ).count() );!
     !
     •  Easy  to  execute  JS  report  script  remotely:  
     !
     > mongo :/traackr --quiet retweetTotal.js!
     !
     •  Run  as  a  cron  job,  pipe  the  output  to  a  file  and  email  it  out  
     •  Also,  more  complex  MR-­‐based  reports  are  easily  accessible  to  
     someone  with  some  JavaScript  knowledge  
     

     View Slide

124. Other  Benefits  (cont.)  
     •  Ad  hoc  queries  and  reports  became  easier  to  write  with  JavaScript:  
     no  need  for  a  Java  developer  to  write  map  reduce  code  to  extract  
     the  data  in  a  usable  form  like  it  was  needed  with  Hbase.  
     •  Simpler  backups:  Hbase  mostly  relied  on  HDFS  redundancy;  intra-­‐
     cluster  replica>on  is  available  but  experimental  and  a  lot  more  
     involved  to  setup.  
      
     

     View Slide

125. Same  binary  can  be  deployed  several  
     >mes  for  replica>on  &  backups  
     

     View Slide

126. Same  binary  can  be  deployed  several  
     >mes  for  replica>on  &  backups  
     Different  Availability  
     Zones  for  beXer  SPOF  
     tolerance  
     

     View Slide

127. Same  binary  can  be  deployed  several  
     >mes  for  replica>on  &  backups  
     priority  0  for  backup  
     server  so  that  it  never  
     gets  elected  as  
     primary  
     

     View Slide

128. Same  binary  can  be  deployed  several  
     >mes  for  replica>on  &  backups  
     Using  xfs_freeze  
     before  taking  backups  
     

     View Slide

129. Same  binary  can  be  deployed  several  
     >mes  for  replica>on  &  backups  
     EBS  snapshots  as  
     backups  are  portable  
     to  new  instances  (e.g.  
     QA)  
     

     View Slide

130. Other  Benefits  (cont.)  
     •  Ad  hoc  queries  and  reports  became  easier  to  write  with  JavaScript:  
     no  need  for  a  Java  developer  to  write  map  reduce  code  to  extract  
     the  data  in  a  usable  form  like  it  was  needed  with  Hbase.  
     •  Simpler  backups:  Hbase  mostly  relied  on  HDFS  redundancy;  intra-­‐
     cluster  replica>on  is  available  but  experimental  and  a  lot  more  
     involved  to  setup.  
     •  Great  documenta>on  
     •  Great  adop>on  and  community  
     

     View Slide

131. Mongo  cursors  for  batch  scoring  
     •  Mongo  is  fast  enough  for  our  data  size  to  
     be  able  to  serially  score  the  DB  faster  
     than  the  MapReduce  jobs  did  in  parallel.  
     •  When  we  grow  larger,  MapReduce  is  s>ll  
     available  as  an  op>on  
     

     View Slide

132. looks  like  we  found  the  right  fit!  
     

     View Slide

133. We  have  more  of  this  
     Development  capacity  
     

     View Slide

134. And  less  of  this  
     Source:  socialbuXerflyclt.com  
     

     View Slide

135. Source:  hXp://www.charliesheentshirts.info/  
     for  now…  
     

     View Slide

136. Addi>onal  takeaways  
     •  Fearless  refactoring  
     •  Ease  of  use  and  administra>on  cannot  be  
     overstated  for  a  small  startup  
     

     View Slide

137. Q&A  
     

     View Slide