Code Coverage and CI/CD for an iOS app

This talk was given by me at PhillyCocoa monthly meeting on how to develop a CI/CD framework for an iOS app. This talk describes the process involved in developing unit tests and reporting code coverage data for the Mint iOS app.

1. iOS  Unit  Tes+ng  and  Code   Coverage     Amit  Rao   Mint  iOS  Developer   rao.amit@gmail.com  
2. Key  Takeaway   •  Move  Quality  close  to  the  developer  desktop   with  unit  tes+ng,  code  coverage,  sta+c   analysis,  code  reviews  and  warning  free   builds.      
3. 3   Assump+ons  and  Pre-­‐Requisites   •  Basic  working  knowledge  of  XCTest   •  Xcode  5.1.1   •  Jenkins  (1.564  or  above)  installed  and  basic  working  knowledge  of  Jenkins  job   crea<on  and  configura<on   •  At  least  a  couple  of  unit  test  already  wriBen     •  Unit  tests  can  be  invoked  successfully  from  command  line  
4. High  Level  Steps   •  Turn  on  code  coverage  flags  in  Xcode  Project   •  Add  Logic  to  generate  code  coverage  files  in  source  code   •  Install  Code  Coverage  Tool  and  Plugin  for  Jenkins  Server   •  Configure  Unit  Test  Job  with  Code  Coverage    
5. Unit  Tes+ng  Resources   •  hXp://qualitycoding.org/     •  OCMock  (hXp://ocmock.org/reference/)   •  OCHamcrest/OCMockito   •  CoverStory  
6. Unit  Tes+ng  CI  Methodolgy   •  Develop  unit  tests  while  developing  app  code   •  Use  CoverStory  to  view  coverage  on  desktop   •  Post  check-­‐in  verify  unit  tests  and  code   coverage  report  via  CI  system.      
7. 7   Step  1   Turn  on  Code  Coverage  Flags  in  xCode   Project  
8. 8   Turn  on  Code  Coverage  Flags  in  xCode   Project   Set  the  following  flags  for  the  main  target  to  ‘Yes’:   •    Generate  Test  Coverage  Files   •    Instrument  Program  Flow  
9. 9   Step  2   Add  Logic  to  Generate  Code  Coverage   Files  in  Source  Code    
10. 10   Add  Logic  to  Generate  Code  Coverage   Files  in  Source  Code   •  As  of  Xcode  4.x,  code  coverage  files  are  no  longer  auto  generated  by  xCode.   •  __gcov_flush();  needs  to  be  called  in  the  applica+on  delegate  a%er  all  tests  are  run  
11. 11   Add  Logic  to  Generate  Code  Coverage  Files  in  Source   Code   •  To  do  so:   –  Create  ‘test  observer’  to  no+fy  applica+on  delegate  when  tests  are  done.    *The  test   observer  should  be  a  member  of  the  unit  test  target   –  Eg:  
12. 12   Add  Logic  to  Generate  Code  Coverage  Files  in  Source   Code   •  Add  observer  when  applica+on  delegate  loads     •  Enhance  applica+onWillTerminate  of  applica+on  delegate  to  call  __gcov_flush();   •  Note  might  be  a  good  idea  to  use  #ifdef  to  only  call  this  during  unit  tes+ng  
13. 13   Step  3     Install  Code  Coverage  Tool  and   Plugins  to  Jenkins  Server    
14. 14   Install  Code  Coverage  Tool  and  Plugins  to  Jenkins   Server   •  Gcovr   –  genera+ng  summarized  code  coverage  results  from  code  genera+on  files.     –  produces  either  compact  human-­‐readable  summary  reports,  machine  readable  XML   reports  or  a  simple  HTML  summary.   •  Cobertura  plugin   –  calculates  the  percentage  of  code  accessed  by  tests.  It  can  be  used  to  iden+fy  which   parts  of  the  program  are  lacking  test  coverage   –  Renders  the  xml  report  generated  by  Gcovr  to  a  navigable  code  coverage  dashboard  
15. 15     Install  gcovr       •  Download  the  latest  gcovr  installa+on  on  server  hos+ng  Jenkins  from   hXps://github.com/gcovr/gcovr/releases   •  Get  the  tar.gz  file  of  the  latest  release  and  uncompress  it  in  a  temp  directory   •  Copy  the  gcovr-­‐X.X/scripts/gcovr  to  an  accessible  directory  e.g.  /usr/local/bin  or  /usr/ bin  or  /Applica+ons   –  Note:  Gcovr  is  installed  on  /Applica+ons/Scripts  on  the  TT  iPad  build  server   • Run  gcovr  –-­‐version  to  ensure  installa+on   > <your path>/gcovr --version gcovr 3.1 Note:    if  gcovr  is  in  your  path,  you  can  just  type  gcovr  -­‐-­‐version    
16. 16   Install  Cobertura  plugin        
17. 17   Add  Code  Coverage  Plugins  to  Jenkins   -­‐  Install  Cobertura  plugin         Start  typing  the  word  Cobertura  in  the  Filter  text  box  and  the  list  below  will  expand  and  show  the   Cobertura  (do  not  hit  return  aoer  typing  the  word  Cobertura  
18. 18   Add  Code  Coverage  Plugins  to  Jenkins   -­‐  Install  Cobertura  plugin        
19. 19   Step  4     Configure  Unit  Test  Job  with  Code   Coverage  
20. 20   Configure  Unit  Test  Job  with  Code  Coverage   •  Crea+ng  a  separate  job  from  main  build  is  a  good  idea  while  configuring  and  fine   tuning  the  process  
21. 21   Configure  Unit  Test  Job  with  Code  Coverage      
22. 22   Configure  Unit  Test  Job  with  Code  Coverage      
23. 23   Configure  Unit  Test  Job  with  Code   Coverage       •  Note:    at  this  point,  we  will  be  wri+ng  a  lot  of  instruc+ons  that  can  be   executed  on  the  command  line   •  it  would  be  a  good  idea  to  run  these  commands  on  the  build  server  which   hosts  Jenkins  to  make  sure  they  work  before  configuring  them  onto  Jenkins  
24. 24   Configure  Unit  Test  Job  with  Code  Coverage  –   run  unit  test   •  *Change  directory  to  loca+on  of  xcode  project   cd /Users/devmac/Developer/Mint/trunk_svn1.7/Gala     •  Run  xcodebuild  command  line   xcodebuild –project Gala.xcodeproj -scheme ’Mint.com Debug-Internal' -sdk iphonesimulator7.1 clean test | ocunit2junit •  *Note:    You  don’t  have  to  change  directory  and  instead  use   an  absolute  path  but  I’d  like  to  break  things  up  a  bit  so  the   commands  are  shorter  and  more  readable   •  Ocunit2junit  is  a  ruby  script  that  generates  the  unit  test   summary  report  
25. 25   Configure  Unit  Test  Job  with  Code  Coverage  –  run  gcovr   to  generate  Cobertura  xml  report   •  Change  directory  to  loca+on  of  code  coverage   files   cd "`xcodebuild –project Gala.xcodeproj - showBuildSettings -sdk iphonesimulator7.1 test | grep OBJECT_FILE_DIR_normal |grep -oEi "/.*" | awk '{print $0"/i386"}'`”   Note:  The  code  generated  files  are  placed  in  a  xCode  DerivedData  folder  which   is  xCode  assigns.    The  script  above  is  one  way  to  determine  the  DerivedData   folder.  The  method  I  use  is  to  run  a  script  called  “exportenv.sh”  which   generates  a  file  called  “env.sh”  with  an  environment  variable  to  the   DerivedData  folder.  For  more  details  please  refer  this  link.  
26. 26   Configure  Unit  Test  Job  with  Code  Coverage  –   run  gcovr  to  generate  Cobertura  xml  report   •  Run  gcovr  and  create  the  coverage.xml  file   /Applications/scripts/gcovr --object- directory . --exclude 'NS*’ --exclude '.*Frameworks.*' --exclude '.*include.*’ --xml > ${WORKSPACE}/coverage.xml   •  Note  exclude  what  you  don’t  need!   •  Note  {WORKSPACE}  is  a  Jenkins  env  variable  that   indicates  the  Jenkin  user’s  workspace    
27. 27   Configure  Unit  Test  Job  with  Code  Coverage   •  Puvng  it  together  …  
28. 28   Configure  Cobertura  plugin  to  render  coverage   report   •  Add  a  post  build  ac+on  
29. 29   Configure  Cobertura  plugin  to  generate  coverage  and   unit  tests  report  
30. 30   Run  unit  test  build  job   Now  lets  give  our  unit  test  job  a  run  by  selec+ng  ‘Build  Now’   If  successful,  an  entry  will  show  up  under  Build  History,  for  our  example,  lets   look  at  #45  Jun  5,  2014  3:04:37    
31. 31   Run  unit  test  build  job  
32. 32   Run  unit  test  build  job  
33. 33   Run  unit  test  build  job   Drilling  down  to  individual  files    Green  sec+on  indicates  code  that  are  covered  by  unit  test    Pink  sec+on  indicates  code  that  are  not  covered  
34. 34   Add  step  to  invoke  unit  test  build  from  main   build  
35. Demo