Automated testing with MongoDB and Offscale

Omer Gertel, OffScale Co-Founder and CTO

In this talk we will explore several techniques to integrate MongoDB into the automated build and test cycle. Schemaless databases like MongoDB allows for more flexibly during development, but require more rigorous tests for different data types. As code evolves, it needs to support objects from older versions or migrate objects as the expected object structure changes. Among the different strategies for automated tests that include the database, we will show how OffScale, git for databases, can help you quickly set up data sets for tests.

Transcript

1. Automated  tes+ng  with     MongoDB  and  OffScale   Get  your  database  under  control   Omer  Gertel   CTO  @  OffScale  
2. Managing   databases  is   complicated  
3. Many  people   are  involved   Developers   QA   DBAs   IT  
4. Production Dev QA Integration MulAple   environments  
5. MulAple   copies  and   versions  
7. We  Help  Companies…     Iterate  Faster  &   Release  Be-er  Products       By…     Shortening  product  cycles   with  DB  management   automa=on  
8. Production Dev QA Integration Copy  data   Write  code   Write  unit  tests   Clone  Schema   Copy  data   Test  the  applicaAon Clone  schema   Copy  data   Run  integraAon  
9. Copy  data   Write  code   Write  unit  tests   Clone  Schema   Copy  data   Test  the  applicaAon Clone  schema   Copy  data   Run  integraAon   DataGrove   Git  for  databases Production Dev QA Staging
10. For  developers,  by  developers   $ datagrove tag -n=new_tag $ datagrove list $ datagrove load -n=old_tag $ datagrove unload •  Works  automagically   •  Easy  to  integrate  with  exisAng  tools   •  Your  personal  database  Ame  machine  for  FREE  
12. Automated  TesAng   • IntegraAon  tests   • Build  servers  and  conAnuous  integraAon   • MigraAons   – Some  MongoDB  specific  issues  
13. Automated  Tests   •  Isolated   •  Repeatable   •  Researchable   •  Fast  
14. Automated  Tests   •  For  each  test:   – Populate  database   – Run  test   – Drop  database  
15. What  happens  if  a  test  breaks?   •  Does  it  affect  other  tests?   •  Does  it  stop  the  build  cycle?   •  How  do  you  discover  what  went  wrong?   •  Is  the  bug  related  to  the  data?   •  How  can  you  reproduce  the  bug?   •  What  happens  when  you  push  a  fix?  
16. Automated  Tests   •  For  each  test:   – Load  dataset   – Run  test   – If  test  fails:   •  Tag  dataset  at  end  of  run  
17. Schema  MigraAons  
18. Schema  MigraAons  
19. •  Migrate  ALL  the  objects   – Test  before  you  run   •  Support  older  structures  in  code   – Test  code  against  old  objects   – Have  datasets  ready  tagged  by  version   Schema  MigraAons   Object  Structure  &  Index  Changes  
20.        Make  MongoDB  a  first   class  ciAzen  of  the   development  cycle   Version  management  that  doesn’t   suck  
21. ExisAng  MongoDB  Support   For  single  node  environments   $ datagrove load - n=tag_name Create  virtual  mount  point:     /media/offscale   (1-­‐10  sec)   mongod  -­‐-­‐dpath=  /media/offscale   /media/offscale   Mongod   path-­‐to-­‐old-­‐data   path-­‐to-­‐store-­‐ changes   (opAonal)  
22. ExisAng  MongoDB  Support   For  single  node  environments   $ datagrove tag - n=tag_name Tag  point  in  Ame  in  virtual  drive   (1-­‐10  sec)   {fsync:1,lock:1}     (using  Java  driver)   /media/offscale   Mongod   path-­‐to-­‐old-­‐data   path-­‐to-­‐store-­‐ changes   (opAonal)   db.$cmd.sys.unlock.findOne()  
23. Replica  Set   Primary   Slave   Slave   Replica  Set   Primary   Slave   Slave   Upcoming  MongoDB  Cluster  Support    Conceptual  architecture  for  taking  a  snapshot   Replica  Set   Primary   Slave   Slave   One  node  managed  under   OffScale,  per  replica  set,   allows  for  backup,  recovery   and  creaAng  new  nodes   ConfiguraAon   cluster   Cfg  1   Cfg  2   Cfg  3   One  or  more  nodes   managed  by  OffScale  for   backup  and  recovery  of   configuraAon  servers   $ datagrove tag-n=ta Issue  datagrove  tag  for   configuraAon  server   Pause  shard   balancing   Resume  shard   balancing   Issue  datagrove  tag,   one  shard  at  a  Ame  
24. •  Data  is  a  first  class   ciAzen  in  your   environment   •  TesAng  is  important  to   keep  your  code  working   •  Iterate  faster  &  release   beher  products   Automated  tesAng  with     OffScale  &  MongoDB  
25. Now  available  at   hhp://off-­‐scale.com/   Get  your  database  under  control