Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automated testing with MongoDB and Offscale

mongodb
May 10, 2012
590

Automated testing with MongoDB and Offscale

Omer Gertel, OffScale Co-Founder and CTO

In this talk we will explore several techniques to integrate MongoDB into the automated build and test cycle. Schemaless databases like MongoDB allows for more flexibly during development, but require more rigorous tests for different data types. As code evolves, it needs to support objects from older versions or migrate objects as the expected object structure changes. Among the different strategies for automated tests that include the database, we will show how OffScale, git for databases, can help you quickly set up data sets for tests.

mongodb

May 10, 2012
Tweet

Transcript

  1. Automated  tes+ng  with     MongoDB  and  OffScale   Get

     your  database  under  control   Omer  Gertel   CTO  @  OffScale  
  2. We  Help  Companies…     Iterate  Faster  &   Release

     Be-er  Products       By…     Shortening  product  cycles   with  DB  management   automa=on  
  3. Production Dev QA Integration Copy  data   Write  code  

    Write  unit  tests   Clone  Schema   Copy  data   Test  the  applicaAon Clone  schema   Copy  data   Run  integraAon  
  4. Copy  data   Write  code   Write  unit  tests  

    Clone  Schema   Copy  data   Test  the  applicaAon Clone  schema   Copy  data   Run  integraAon   DataGrove   Git  for  databases Production Dev QA Staging
  5. For  developers,  by  developers   $ datagrove tag -n=new_tag $

    datagrove list $ datagrove load -n=old_tag $ datagrove unload •  Works  automagically   •  Easy  to  integrate  with  exisAng  tools   •  Your  personal  database  Ame  machine  for  FREE  
  6. Automated  TesAng   • IntegraAon  tests   • Build  servers  and  conAnuous

     integraAon   • MigraAons   – Some  MongoDB  specific  issues  
  7. Automated  Tests   •  For  each  test:   – Populate  database

      – Run  test   – Drop  database  
  8. What  happens  if  a  test  breaks?   •  Does  it

     affect  other  tests?   •  Does  it  stop  the  build  cycle?   •  How  do  you  discover  what  went  wrong?   •  Is  the  bug  related  to  the  data?   •  How  can  you  reproduce  the  bug?   •  What  happens  when  you  push  a  fix?  
  9. Automated  Tests   •  For  each  test:   – Load  dataset

      – Run  test   – If  test  fails:   •  Tag  dataset  at  end  of  run  
  10. •  Migrate  ALL  the  objects   – Test  before  you  run

      •  Support  older  structures  in  code   – Test  code  against  old  objects   – Have  datasets  ready  tagged  by  version   Schema  MigraAons   Object  Structure  &  Index  Changes  
  11.        Make  MongoDB  a  first   class  ciAzen

     of  the   development  cycle   Version  management  that  doesn’t   suck  
  12. ExisAng  MongoDB  Support   For  single  node  environments   $

    datagrove load - n=tag_name Create  virtual  mount  point:     /media/offscale   (1-­‐10  sec)   mongod  -­‐-­‐dpath=  /media/offscale   /media/offscale   Mongod   path-­‐to-­‐old-­‐data   path-­‐to-­‐store-­‐ changes   (opAonal)  
  13. ExisAng  MongoDB  Support   For  single  node  environments   $

    datagrove tag - n=tag_name Tag  point  in  Ame  in  virtual  drive   (1-­‐10  sec)   {fsync:1,lock:1}     (using  Java  driver)   /media/offscale   Mongod   path-­‐to-­‐old-­‐data   path-­‐to-­‐store-­‐ changes   (opAonal)   db.$cmd.sys.unlock.findOne()  
  14. Replica  Set   Primary   Slave   Slave   Replica

     Set   Primary   Slave   Slave   Upcoming  MongoDB  Cluster  Support    Conceptual  architecture  for  taking  a  snapshot   Replica  Set   Primary   Slave   Slave   One  node  managed  under   OffScale,  per  replica  set,   allows  for  backup,  recovery   and  creaAng  new  nodes   ConfiguraAon   cluster   Cfg  1   Cfg  2   Cfg  3   One  or  more  nodes   managed  by  OffScale  for   backup  and  recovery  of   configuraAon  servers   $ datagrove tag-n=ta Issue  datagrove  tag  for   configuraAon  server   Pause  shard   balancing   Resume  shard   balancing   Issue  datagrove  tag,   one  shard  at  a  Ame  
  15. •  Data  is  a  first  class   ciAzen  in  your

      environment   •  TesAng  is  important  to   keep  your  code  working   •  Iterate  faster  &  release   beher  products   Automated  tesAng  with     OffScale  &  MongoDB