The  economic  (in)feasibility  of     on-­‐premise  dev/test   The  demand  and  supply  mismatch  of   development  and  tes4ng  resources   Navin  R.  Thadani,  @navinthadani   Ravello  Systems,  Inc.     Ravello  Systems,  Inc.  |  GlueCon  2013   

On-­‐premise  development  &  test  sucks   Sprint 1   Sprint   2   Release   Candidate   Product  late  to   market  and   poor  quality   Data  center/private  cloud  capacity   Project  Demand   Resources   Resources   Sprint 1   Sprint   2   Release   Candidate   Ravello  Systems,  Inc.  |  GlueCon  2013   

#1  Lack  of  internal  capacity   #2  Difficult  to  replicate/ provision  environments  quickly   Ravello  Systems,  Inc.  |  GlueCon  2013   

A  simple  applicaSon   router   firewall   load   balancer   be1   OS   db2   OS   fe2   OS   fe1   OS   be2   OS   db1   OS   Front  end   Back  end   db  4er   q   OS   Ravello  Systems,  Inc.  |  GlueCon  2013   

A  simple  development  process   Unit  test   environments   Integra4on  test   environments   System  test   environments   for  every   commit   for  every   commit   batched     Ravello  Systems,  Inc.  |  GlueCon  2013   

Fundamental  quesSon   How  much  capacity  do  you  need   to  opSmally  develop  and  test   your  applica4on?     Ravello  Systems,  Inc.  |  GlueCon  2013   

AssumpSon   As  far  as  possible,  you  want  to   test  on  replicas  of  the  produc4on   environment   Ravello  Systems,  Inc.  |  GlueCon  2013   

A  queuing  theory  perspecSve   C2   C3   Test   environments   C6   C7   C8   C1   Arrival  rate  =  λ   commits/hour   C5   Service  rate  =  μ   tests/hour   M-­‐D-­‐s  queue   Average  Sme   spent  in  the   queue   Probability  of   waiSng  more   than  5   minutes   Overall   resource   uSlizaSon   Ravello  Systems,  Inc.  |  GlueCon  2013   

Dev/test  efficiency  fronSer   Number  of  tesSng  environments   Cost   Cost  of  tes4ng   environments   Cost  of  “wai4ng”   4me   Total  cost   X   TE1   Ravello  Systems,  Inc.  |  GlueCon  2013   

8  person  development  team     5,000  commits  (in  12  months)     300,000  lines  of  Java   Ravello  Systems,  Inc.  |  GlueCon  2013   

Some  more  numbers   Unit  tests   Integra4on  tests   System  test   environments   λ     4/hr   μ     8/hr   μ     5/hr   Ravello  Systems,  Inc.  |  GlueCon  2013   

Average  wait  Sme   1 2 3 4 5 6 7 8 9 10 1 2 0 0 0 0 0 0 0 0 0 2 4 0 0 0 0 0 0 0 0 0 3 9 1 0 0 0 0 0 0 0 0 4 24 1 0 0 0 0 0 0 0 0 5 oo 2 0 0 0 0 0 0 0 0 Number  of  integraSon  tesSng  environments   Commit  rate   P 0 =1/ λ µ ( )n n! + λ µ ( )S S! 1 1− λ / Sµ " # $ % & ' n=0 S−1 ∑ ) * + + + + , - . . . . L q = λ / µ ( )S λ / Sµ ( ) S! 1-λ/Sµ ( )2 P 0 W q = L q λ Or  simple  simulaSon    h_ps://github.com/geertj     Dust  off  your  text  books  ;)   Ravello  Systems,  Inc.  |  GlueCon  2013   

Dev/test  efficiency  fronSer  (λ=4/hr)   $0   $50,000   $100,000   $150,000   $200,000   $250,000   $300,000   $350,000   $400,000   $450,000   $500,000   1   2   3   4   5   6   7   8   9   10   11   12   OpSmal  point   •  2  x  central  UT   •  2  x  IT     •  1  x  ST   •  2  x  manual  QA   Number  of  tes4ng  environments   Cost   Cost  of  “wai4ng”   Capex  +  opex   Total  cost  fron4er   Ravello  Systems,  Inc.  |  GlueCon  2013   

Reality  bites  (badly)   Commits  per  hour   0   5   10   15   20   25   30   10/2/12   11/2/12   12/2/12   1/2/13   2/2/13   3/2/13   4/2/13   Average  4   commits/  hour   Peak  27  commits/   hour   Ravello  Systems,  Inc.  |  GlueCon  2013   

Dev/test  efficiency  fronSer   Number  of  tesSng  environments   Cost   Cost  of  tes4ng   environments   Cost  of   “wai4ng”  4me   For  commit   rate  L1   For  commit   rate  L2   TE1   TE2   X1   X2   Ravello  Systems,  Inc.  |  GlueCon  2013   

Average  wait  Sme   Number  of  integraSon  tesSng  environments   1 2 3 4 5 6 7 8 9 10 1 2 0 0 0 0 0 0 0 0 0 2 4 0 0 0 0 0 0 0 0 0 3 9 1 0 0 0 0 0 0 0 0 4 24 1 0 0 0 0 0 0 0 0 5 100 2 0 0 0 0 0 0 0 0 6 100 4 1 0 0 0 0 0 0 0 7 100 6 1 0 0 0 0 0 0 0 8 100 11 1 0 0 0 0 0 0 0 9 100 26 2 0 0 0 0 0 0 0 10 100 100 3 1 0 0 0 0 0 0 11 100 100 4 1 0 0 0 0 0 0 12 100 100 7 1 0 0 0 0 0 0 13 100 100 12 2 0 0 0 0 0 0 14 100 100 26 2 1 0 0 0 0 0 15 100 100 100 3 1 0 0 0 0 0 16 100 100 100 5 1 0 0 0 0 0 17 100 100 100 7 1 0 0 0 0 0 18 100 100 100 12 2 1 0 0 0 0 19 100 100 100 27 3 1 0 0 0 0 20 100 100 100 100 3 1 0 0 0 0 21 100 100 100 100 5 1 0 0 0 0 22 100 100 100 100 7 2 1 0 0 0 23 100 100 100 100 13 2 1 0 0 0 24 100 100 100 100 26 3 1 0 0 0 25 100 100 100 100 100 4 1 0 0 0 26 100 100 100 100 100 5 1 0 0 0 27 100 100 100 100 100 8 2 1 0 0 28 100 100 100 100 100 13 2 1 0 0 29 100 100 100 100 100 28 3 1 0 0 30 100 100 100 100 100 100 4 1 0 0 Commit  rate   Ravello  Systems,  Inc.  |  GlueCon  2013   

Dev/test  efficiency  fronSer  (λ=27/hr)   $0   $50,000   $100,000   $150,000   $200,000   $250,000   $300,000   $350,000   $400,000   $450,000   $500,000   1   2   3   4   5   6   7   8   9   10   11   12   OpSmal  point   •  5  x  central  UT   •  8  x  IT     •  1  x  ST   •  3  x  manual  QA   Number  of  tes4ng  environments   Cost   Cost  of  “wai4ng”   Capex  +     Opex   Total  cost  fron4er   Ravello  Systems,  Inc.  |  GlueCon  2013   

$100K  –  substandard  lab   WaiSng,  inefficiency,  resource   contenSon  …     $250K  –  good  lab  service   Efficient  development   Hidden  cost  >>$1m  (inefficiency)   Average  uSlizaSon  (<5%)   Ravello  Systems,  Inc.  |  GlueCon  2013   

A  new  fronSer?     $0   $50,000   $100,000   $150,000   $200,000   $250,000   $300,000   $350,000   $400,000   $450,000   $500,000   1   2   3   4   5   6   7   8   9   10   11   12   Public  Cloud   25x  lower   cost   Internal  data   center   Number  of  tes4ng  environments   Ravello  Systems,  Inc.  |  GlueCon  2013   

Reality  check   $0   $50,000   $100,000   $150,000   $200,000   $250,000   $300,000   $350,000   $400,000   $450,000   $500,000   1   2   3   4   5   6   7   8   9   10   11   12   Change  applicaSon  to  run  on-­‐premise  (producSon)   and  cloud  (development  and  test)     ?   Public  Cloud   Internal  data   center   Number  of  tes4ng  environments   Ravello  Systems,  Inc.  |  GlueCon  2013   

Your  opSons   1x   >>   2.5x   0.1x   <0.1x   Wai4ng  4me   Capex  +   Opex   Modify  your  app  to  run  in   the  cloud  and  internal  data   center  +  automa4on     Internal  sub   opSmal  lab   Internal  opSmal   lab   “Migrate”  to   public  cloud   How  we  have   solved  it       Ravello  Systems,  Inc.  |  GlueCon  2013   

DemonstraSon   Ravello  Systems,  Inc.  |  GlueCon  2013   

Ravello  Systems,  Inc.  |  GlueCon  2013