ESEM maintenance defect model validation

Transcript

1. Jet Propulsion Laboratory! Empirical  and  Face  Validity  of   So4ware  Maintenance  Defect   Models  Used  at  JPL   Bill  Taber,  Caltech/JPL   Dan  Port,  UH   Copyright  ©  2014  all  rights  reserved.  This  research  was  carried  out  at  the  Jet  Propulsion   Laboratory,  California  InsHtute  of  Technology  under  a  contract  with  the  NaHonal  AeronauHcs  and   Space  AdministraHon.    
2. Jet Propulsion Laboratory! What?   •  Defects  in  soOware  are  inevitable,  even  at  JPL,  so  …   –  We  need  to  plan  for  their  discovery  and  repair   –  It  is  not  enough  to  say  “I  need  a  maintenance  budget.”    Need  to  be   able  to  jus?fy  the  budget  and  match  predicHons  with  reality.   •  EssenHal  to  this  are  pragma?c  models  that  can  easily  be   implemented,  calibrated  and  validated     –  AssumpHons  need  only  be  reasonably  valid  for  our  environment   –  Must  be  able  to  explain  models  to  stakeholders.     •  Simple  (if  it  works)  is  always  beYer.   –  Conclusions  need  to  fit  our  observaHons  from  over  a  decade  of  defect   discovery.   •  Stakeholders  won’t  believe  (and  shouldn’t)  believe  a  model  unless  it  fits   the  data,  is  consistent  their  experience,  and  is  grounded.   –  For  this  kind  of  engineering  work:  Good  enough  is  perfect.   •  If  theoreHcal  nuances  don’t  maYer,  they  really  don’t  maYer.    
3. Jet Propulsion Laboratory! Luck  was  on  our  side   •  At  JPL  we  had  the  good  fortune  of  having  a  “hydrogen  atom”   of  soOware  maintenance  present  itself.   –  Two  systems  under  going  repair  only  beginning  in  2006  and  2008.   •  No  new  development,  new  development  on  an  upgrade  system   –  Complete  defect  histories  tracked  prior  to  the  onset  of  repair  only   mode   •  The  defect  tracker  bugzilla  was  adopted  for  all  mission  design  and  and   navigaHon  soOware  in  2002  and  is  sHll  used  today.   –  Defects  repaired  quickly  aOer  their  discovery  (thanks  to  the   adopHon  of  bugzilla  and  a  rapid  response  process)   –  Uniform  daily  use       •  Based  on  uniform  staff  levels  of  users  and  system  accounHng  history.   –  The  system  was  free  of  the  perturbing  forces  of  sporadic   development  that  are  part  of  most  maintenance  systems  allowing   for  the  development  of  a  simple  defect  decay  model.  
4. Jet Propulsion Laboratory! The  Models:  NegaHve  ExponenHal   •  The  rate  of  change  in  the  expected  number  of  defects  in  the  system  at  Hme   t,  N(t),  is  proporHonal    to  the  number  of  defects  in  the  system.       •  Let  T  be  any  reference  epoch,  then  N  saHsfies   •  We  cannot  observe  the  iniHal  number  of  defects  N(t)  We  can  observe  A(t)   —  the  accumulated  number  of  defects  at  Hme  t.         •  Use  the  model  for  N  relaHve  to  the  reference  epoch  T.   –  The  number  of  accumulated  defects  is  just  the  number  that  the  total  defect   count  has  been  reduced  since  the  reference  epoch  N(T)  –  N(t).       •  Model  depends  only  on  Hme  elapsed  since  any  reference  Hme.       •  The  Hme  it  takes  to  reduce  the  defects  by  some  factor  depends  only  upon   the  esHmable  decay  constant  α       A(t) = N(T) 1− e−α (t −T ) ( ) N(t) N(t) = N (T)e−α (t −T).
5. Jet Propulsion Laboratory! Empirical  Validity:  Legacy  NavigaHon  System    R^2  =  0.96267  
6. Jet Propulsion Laboratory! Empirical  Validity:  Legacy  Mission  Design  System   0" 50" 100" 150" 200" 250" 0" 100" 200" 300" 400" 500" 600" 700" Defects'Reported' Work'Days'Since'April'21,'2008' Legacy'Mission'Design'Defect'Accumula@on' Accumula0on" 0" 50" 100" 150" 200" 250" 0" 100" 200" 300" 400" 500" 600" 700" Defects'Reported' Work'Days'Since'April'21,'2008' Legacy'Mission'Design'Defect'Accumula@on' Accumula0on" Best"Fit" R^2  =  0.99305  
7. Jet Propulsion Laboratory! PracHcal  ApplicaHons   •  The  Poisson  distribuHon  together  with  the   exponenHal  decay  model  allows  us  to:   (1)  Tell  projects  what  the  range  of  defects  they  are   likely  to  find  in  a  new  release   (3)  How  long  it  will  take  to  find  them   (4)  Establish  credibility   •  Our  assumpHons  have  been  challenged  in  the  past  and   we  have  been  able  to  win  a  gentlemen’s  bet  that  we   can  predict  what  stakeholders  will  actually  find.   •  Our  models  are  now  widely  accepted  by  our   stakeholders.  No  one  will  bet  against  our  models.  
8. Jet Propulsion Laboratory! AYracHveness  of  These  Models   •  Our  models  are:   –  Simple   •  It  requires  a  minimal  set  of  parameters   –  PracHcal   •  It  is  easy  to  implement  and  fit   –  PragmaHc   •  It  are  good  enough  when  trying  to  answer  reliability   quesHons  about  our  soOware.   –  Useful   •  Allows  us  to  defend  budgets   •  Allows  us  to  characterize  risk   •  Allows  us  to  give    good  quanHtaHve  answers  to  quesHons   about  the  reliability  of  our  soOware.  
9. Jet Propulsion Laboratory! Moral:   All  models  are  wrong,     but  some  are  useful  anyway.  
10. Jet Propulsion Laboratory! APPLICATIONS  TO  MAINTAINED   SYSTEMS  WITH  NEW  DEVELOPEMNT   Backup  Material  
11. Jet Propulsion Laboratory! ApplicaHon  to  the  JPL  system  under  development   •  The  model  can  be  used  to  understand  defect  accumulaHon   in  the  new  system  with  planned  conHnued  development   –  Defects  repaired  as  soon  as  they  are  discovered   –  Uniform  daily  use   –  Uniform  producHon  of  new  code  resulHng  a  uniform  rate,  D,  of   producing  new  defects.   •  The  number  of  defects  in  the  system  will  stabilize  to  the   rate  of  producHon  of  defects  D  divided  by  the  discovery   rate  α.   •  The  defect  accumulaHon  will  become  linear  over  Hme  with   a  rate  of  discovery  equal  to  the  rate  of  producHon.  
12. Jet Propulsion Laboratory! Empirical  ValidaHon:  New  System  
13. Jet Propulsion Laboratory! Planning  for  New  Defects   •  An  examinaHon  of  our  released  defects  shows  that  defects   tend  to  be  distributed  widely  across  the  code  base  are   unrelated  (recall  defects  are  reported  bugs,  not  code  issues).   •  The  previous  chart  implies  a  relaHvely  constant  defect   density.   •  Under  this  assumpHon  we  assume  that  defects  per  release   follow  a  Poisson  distribuHon.       –  if  the  expected  number  defects  is    β,  then     –  Under  the  exponenHal  decay  model,  the  distribuHon  for  the  number   of  defects  remains  Poisson  with  the  the  expected  number  of  defects   decaying  exponenHally   Pr(n defects) = βn n! e−β = P β (n)
14. Jet Propulsion Laboratory! Empirical  ValidaHon:  Model  Fit   •  NO  WAY,  TOO  SIMPLE!  How  can  we  validate  this?   –  Solving  for  the  best  fit  exponenHal  decay  constant  and  defect  density,  we  can   simulate  the  history  of  defect  discoveries  for  the  new  system  and  compare  this  to   the  actual  history   0" 500" 1000" 1500" 2000" 2500" 0" 500" 1000" 1500" 2000" 2500" 3000" 3500" Accumulated*Defects* Work*Days*Since*August*19,*2001* Theore>cal*Vs*Actual*Defects*Reported*for*Monte* Actual"Defect"History" Theory" R^2  =  0.9893  
15. Jet Propulsion Laboratory! Empirical  ValidaHon:  Model  PotenHal   •  This  is  just  one  possible  history.  Maybe  we  just  got  lucky  and  this  is  not   the  way  new  defects  actually  accumulate.     •  Compute  distribuHon  of  all  possible  histories  based  on  the  model  and  see   if  the  actual  actual  history  lies  within  it.   0" 0.005" 0.01" 0.015" 0.02" 0.025" 0.03" 0.035" 0.04" 0" 0.2" 0.4" 0.6" 0.8" 1" 1.2"
    0" "125.40834" "250.81668" "376.22502" " 501.6333599" " 627.0416999" " 752.4500399" " 877.8583799" " 1003.26672" " 1128.67506" "1254.0834" " 1379.49174" " 1504.90008" " 1630.30842" Empirical)Distribu/on)of)theoreical)histories) Actual" History" pdf" Indeed  it  does!     And  not  even  a  rare  one.