Stay C.A.L.M.S.: A local company's journey into DevOps

A fast walk-through of our journey to DevOps at Solana Beach based OneHealth.

Links:

No such thing as a DevOps team
- http://continuousdelivery.com/2012/10/theres-no-such-thing-as-a-devops-team/
- http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/
- http://www.slideshare.net/jezhumble/scaling-devops

Enter C.A.L.M.S.
- John Willis, July 2010 https://www.chef.io/blog/2010/07/16/what-devops-means-to-me/
- James Turnbull, Feb 2010 http://www.kartar.net/2010/02/what-devops-means-to-me/
- Jez Humble, Sept 2010 http://www.slideshare.net/jezhumble/devops-and-agile-release-management

Culture: Spotify Squads
- https://www.scribd.com/doc/113617905/Scaling-Agile-Spotify

Automation: Infrastructure as Code
- http://www.jedi.be/blog/2013/05/24/Infrastructure%20as%20Code/

Reduce Risk of Release
- http://www.slideshare.net/jezhumble/devops-and-agile-release-management
- http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

Theory of Constraints
- http://en.wikipedia.org/wiki/Theory_of_constraints

Our Implementation
- http://www.slideshare.net/JarrodOverson/continuous-delivery-for-the-web-platform

Continous Delivery Maturity Model
- http://info.thoughtworks.com/rs/thoughtworks2/images/continuous_delivery_a_maturity_assessment_modelfinal.pdf
- http://www.infoq.com/articles/Continuous-Delivery-Maturity-Model

Continuous Delivery Patterns
- http://continuousdelivery.com/patterns/
- http://refcardz.dzone.com/refcardz/continuous-delivery-patterns

Pattern: Feature Flags /Dark Launching
- http://en.wikipedia.org/wiki/Feature_toggle
- http://www.facebook.com/note.php?note_id=96390263919

Pattern: Control your Environment
- https://github.com/rcrowley/freight

Measure
- http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

Monitoring Sucks
- http://obfuscurity.com/2011/07/Monitoring-Sucks-Do-Something-About-It

Transcript

1. ì   A  local  company’s  journey  to  DevOps   Stay  C.A.L.M.S.   Sander  van  Zoest   @svanzoest  
2. ì   It  is  a  human  and  management  problem   What  is  DevOps?  
3. No  such  thing  as  DevOps  team   ì  DevOps  is  not  a  new  name  for  System   AdministraAon   ì  It  is  about  collaboraAon  between  funcAonal  teams.  
4. Enter  C.A.L.M.S.   ì  Culture   ì  AutomaAon   ì  Lean   ì  Measure   ì  Sharing  
5. Company  Culture   Industrial   FuncAonal  Silos   Blame  runs  downhill  in  public,   and  uphill  at  the  water-­‐cooler   Avoid  Risk  and  Failure   Never  revise  policy  on  success   (RetrospecAve)   Heteronomy       Agile   Cross-­‐funcAonal  delivery   teams   Team  Ownership  (Trust)   Fail  Fast,  Learning   Opportunity   ConAnuous  Process  Review   Autonomy  and  Alignment    
6. Culture:  Breaking  the  Silos   ì  SeaAng  Arrangements   ì  Cubicles  vs  Open  Spaces   ì  Headphones   ì  Sales  versus  Development  tasks   ì  Unbiased  Project  Manager  (reports  to  Product  or  Engineering?)   ì  Matrix  Teams   ì  SpoAfy’s  Squads  
7. Culture:  Spotify  Squads  
8. Squads  own  a  minimum   viable  product   How  do  split  into  squads?   Horizontal   Based  on  Product  Features   VerAcal   Based  on  Product  Layers   Is  Customer  Service  a  Squad?   Is  Account  Management?   Is  OperaAons?  
9. Fear  of  Failure   Understand  you  can  never   know  everything.   Account  for  unknown   unknowns   Focus  on  unknowns  first   Fail  fast   Foster  creaAvity  and   innovaAon   Learn   Create  a  safety  net     Never  ending  2  week  sprints   InAmidaAng  teammates   Constant  context  switching   Perceived  failure  in  lack  of  progress   Lack  of  personal  safety  in  admiZng   failure  of  a  task   Limited  Long  term  vision   Low  Self-­‐Esteem   PerfecAonism  
10. ì  Project  Manager:  "What  is  the  status  of   project  X?  Did  you  ever  complete  that?"   ì  So]ware  Engineer:  "Oh,  that's  done."   ì  Project  Manager:  "Great,  so  it  is  live  in   produc>on  then?"   ì  So]ware  Engineer:  "Ehhh...no,  it  s>ll   needs  to  be  tested,  integrated  and   deployed.  I  am  just  dev  done”.     It  isn't  rodeo  done,  unAl  it  has   passed  acceptance  tesAng  in   producAon.     Rodeo  Done  
11. Automation   ì  Infrastructure  as  Code   ì  Creates  Consistency   ì  Avoids  RepeAAve  (boring)  Tasks   ì  Makes  room  for  improvement  and  learning   ì  Removes  Ambiguity  (Blame  Culture)   ì  Safety  Net  to  try  new  things  
12. ì   Service  Life  Cycle  
13. Automation:  Baby  steps   ì  First  determine  what  your  product  life  cycle  is.   ì  We  started  with  deploying  the  dev  environment.   ì  Make  it  as  close  to  producAon  as  possible     ì  Avoid  separate  ways  of  doing  things.     ì  Create  a  single  flow  to  producAon.   ì  Goal   ì  Fully  automated  deployment  to  producAon   ì  Get  it  down  to  a  single  bucon  press  
14. Dev  Environment  Setup:  Goal   ì  How  long  does  it  take  for  a  new  hire  to  setup  their   environment?  How  long  unAl  they  commit  to   producAon?   ì  In  2010  it  took  2  months  setup  the  env  and  update   docs.   ì  In  2014  they  would  push  to  producAon  within  48   hours.   ì  What  should  be  other  goals?  How  would  you   measure  them?  
15. Dev  Environment  Setup:  How   ì  Leverage  VirtualizaAon:  We  used  Vagrant  with   Virtualbox   ì  It  allowed  us  to  DRY  up  use  the  same  tools  on  dev   as  producAon.   ì  Don’t  worry  about  what  tools,  focus  on   repeatability.  Shell  scripts?  Doesn’t  macer.  As  long   as  it  is  reliable  and  repeatable.   ì  Once  it  is,  measure,  review  and  iterate  
16. Dev  Environment  Setup:  Shell  Scripts   ì  Now  that  it  is  repeatable,  how  would  you  apply  it  to   producAon?   ì  Do  any  of  the  packages  and/or  configs  change?   ì  Did  you  check  them  in  version  control?   ì  Next  convert  them  into  a  more  flexible  tool.   ì  We  chose  Chef  
17. ì   Release  Cycles  
18. Release  Frequently  
19. Old  Release  Process   Feature  1   Feature  2   Feature  3   Feature  4   Feature  5   Feature  6   Feature  7   Release  Regression  TesAng   Release  Regression  TesAng  
20. Goal:  Reduce  Risk  of  Release  
21. ì  IdenAfy  the  system's  constraint   ì  IntegraAon  TesAng  and  Release  Management   ì  Development  completed.  Needs  to  be  tested  (waterfall)   ì  Decide  how  to  exploit  the  system's  constraint   ì  Build  tests  at  the  same  Ame  as  developing  the  code  (alignment/ focus)   ì  Minimize  changes  tested  at  once.  Minimize  backlog  of  releases.   ì  Subordinate  everything  else  to  the  above  decision   ì  Align  the  whole  system  or  organizaAon  to  support  the  decision   made  above ì  Elevate the system's constraint ì  Measure,  rinse  and  repeat Theory  of  Constraints  
22. Release  Summary  2013   Release   Date   US   DE     5.6   20-­‐Dec   3   20   5.7   4-­‐Jan   8   49   5.8   9-­‐Jan   0   4     5.9   17-­‐Jan   2   12   5.1   25-­‐Jan   3   26   5.11   7-­‐Feb   4   21   5.12   14-­‐Feb   3   14   5.13   18-­‐Mar   5   29   6   15-­‐Apr   19   169   0   50   100   150   200   250   300   350   400   450   20-­‐Dec   20-­‐Jan   20-­‐Feb   20-­‐Mar   User  Story/Defect  
23. ì  Making  sure  your  so]ware  is  always  producAon   ready  throughout  its  enAre  lifecycle   ì  reduce  the  cost,  Ame,  and  risk  of  delivering   incremental  changes  to  users   ì  Everybody  is  responsible  for  delivery   What  is  continuous  delivery?  
24. ©  Copyright  2012  OneHealth   SoluIons,  Inc.   1/25/15   24   Continuous  Delivery  Process  Flow  
25. Our  Implementation  
26. ì  Quicker  turn  around  from  Concept  to  Deployment   ì  Easier  to  quickly  fix  issues  in  producAon   ì  Focus  on  Roll  forward,  Not  backwards   ì  More  focus  on  automated  tests   ì  More  focus  on  process  improvement   ì  Quality  focus  rather  than  individual  throughput   focus   Findings  &  Learnings  
27. Continuous  Delivery  Maturity  Model  
28. Continuous  Delivery  Patterns   ì  ConAnuous  IntegraAon   ì  aka  Develop  on  Mainline   ì  Feature  Toggles   ì  Branch  by  AbstracAon   ì  Dark  Launching   ì  Database  Versioning   ì  Database  Backwards   CompaAbility   ì  ProducAon  Immune   System   ì  Blue-­‐Green  Deployments   ì  Canary  Releasing   ì  Expand/Contract   ì  A/B  TesAng  
29. Pattern:  Feature  Flags  /  Dark  Launching   ì  Allows  code  to  be  released  but  not  visible  to   everyone   ì  Works  in  parallel  with  A/B  TesAng   ì  Allows  for  feature  iteraAon  by  product  owner  in   while  using  it  in  producAon   ì  Allows  Customer  Service  and  Account  Management   to  get  familiar  with  feature  and  communicate  to   customers  at  their  own  pace  
30. Pattern:  Develop  on  Mainline   ì  We  explored  many  branching  models   ì  Git  Flow   ì  Pull  Request   ì  Etc.   We  currently  standardize  on  Pull  Requests  to  allow  for   peer  code  review.  
31. Pattern:  System  Composition   LocaAon  (Data  Center)   ì  for  gateway,  NTP,  APT  Mirror,  etc.   Cluster   ì  for  the  smallest  logical  horizontal  scalable  node   ì  E.g.  Front  End,  Data  Store   Environment   ì  Dev,  QA,  CI,  Staging,  Sandbox,  ProducAon   Realm   ì  Data  type  or  source:  IdenAty,  Content,  BI  
32. Pattern:  System  Composition   Translates  to  monitoring,  graphing,   reporAng,  alerAng  
33. Pattern:  Vendoring  during  Development   ì  Starts  by  creaAng  a  vendor  directory  in  your  project   ì  You  ulAmately  only  need  to  vendor  your   dependencies  in  your  build  arAfacts   ì  We  use:   ì  Composer/SaAs  for  PHP   ì  Bundler  for  Ruby   ì  Berkshelf  for  Chef  Cookbooks   ì  NPM  for  Node/JavaScript  
34. Pattern:  Build  Artifacts  for  Deployment   Build   Process   Version  Control   ArAfacts  
35. Pattern:  Separate  Code  from  Config   Dev   CI   Staging   ProducAon   Build   ArAfact   Same  ArAfact   Same  ArAfact   Build   Cookbook   Same  Cookbook   Same  Cookbook   Specific   Environment   ConfiguraAon   Specific   Environment   ConfiguraAon   Specific   Environment   ConfiguraAon   Specific   Environment   ConfiguraAon  
36. Pattern:  Control  your  Environment   ì  Setup  your  own  OS  Package  Repository   ì  Setup  controlled  mirrors  of  all  so]ware  you  depend   upon  in  producAon   ì  We  used  apropos  and  freight  for  our  debian   packages  
37. Lean   ì  Remember  team  ownership   ì  CollaboraAvely  determine   the  true  problem  (root   cause)   ì  Pick  tools  useful  by  everyone   ì  Communicate  Reasoning     ì  Eliminate  waste   ì  Amplify  learning   ì  Decide  as  late  as  possible   ì  Deliver  as  fast  as  possible   ì  Empower  the  team   ì  Build  integrity  in   ì  See  the  whole  
38. Measure   ì  Build  Confidence   ì  Provides  Context   ì  Incidents   ì  Frequency   ì  Severity   ì  Root  Cause   ì  Time-­‐To-­‐Detect  (TTD)   ì  Time-­‐To-­‐Resolve  (TTR)   ì  Coordinate  Responses   ì  Measure  Progress   ì  Track  Change   ì  Value  Stream  Mapping   ì  Graph  It   ì  Alert  on  it  
39. Monitoring  Sucks   Take  a  chapter  from  the  "Infrastructure  as   Code"  movement  and  appreciate  the   benefits  that  come  from  interoperable   pieces  that  are  inexpensive,  scalable  and   easily  automated  with  a  licle  code  and  a   stable  API.  -­‐-­‐  Jason  Dixon  (obfuscurity)  
40. Monitoring:  Tracking   ì  ApplicaAon  Data:   ì  Statsd   ì  ApplicaAon  Logs  and  Events   ì  Deployment  Data   ì  Jenkins   ì  Chef  Report  Handler   ì  System  Data   ì  Sensu   ì  etc  
41. Monitoring:  Graphing   ì  Graphite   ì  ELK  Stack   ì  ElasAcSearch   ì  Logstash   ì  Kibana   The  key  thing  is  to  build  systems  that  can  be  easily   maintained  and  accept  new  arbitrary  data,  so  you  can  focus   on  the  data  and  presentaAon.     Not  the  creaAon  of  dashboards.  
42. ì  Does  two  things:   ì  Store  numeric  Ame-­‐series  data   ì  Render  graphics  of  this  data  on  demand   ì  Consists  of  three  things:   ì  Receiver  Daemon  -­‐  Carbon   ì  Disk  Storage  Library  -­‐  Whisper   ì  Web  ApplicaAon  –  Graphite  /Graphana   ì  Used  for  ApplicaAon  and  System  metrics   Monitoring:  What  is  Graphite?  
43. Monitoring:  What  is  an  ELK  Stack?   ì  ElasAcSearch   ì  Logstash   ì  Kibana   We  feed  it  via  rsyslog  and   allows  querying  of  the  logs  and   alerAng  on  log  data.  
44. Monitoring:  Alerting   ì  Tried  Nagios,  Opsview,  etc.   ì  Landed  at  Sensu   ì  Event  Messaging  Bus   ì  Maps  monitoring  to  Chef  cookbooks   ì  AutomaAc  Client  detecAon  and  decommission   ì  Easy  integraAon  with  other  so]ware   ì  Graphite   ì  Email   ì  Slack  (ChatOps)  
45. Share   ì  Good  News  and  Bad  News   ì  Creates  Trust  and   Confidence   ì  Helps  collaboraAon   ì  Within  your  team   ì  Between  departments   ì  Between  companies  
46. Share:  Real-­‐Time  Communication   ì  Dashboards   ì  Hallway  Monitors   ì  Via  Email   ì  Over  the  Cube   ì  ChatOps   ì  Focus  on  informaAon   sharing   ì  System  IntegraAons   ì  ConAnuously  tune  to   improve  signal  to  noise  raAo.  
47. Share:  Community   ì  Open  Source   ì  Bug  Reports   ì  Code  Patches   ì  DocumentaAon   ì  Speak  at  User  Groups,   Conferences   ì  Contribute  back  to  the   community   ì  Open  Source  re-­‐usable   generic  tools  and  methods   ì  Write  a  blog   ì  Join  mailing  lists,  etc.   ì  Join  San  Diego  DevOps   ì  SDDevOps.org  
48. San  Diego  DevOps  Speakers  &  Members  
49. Summary   ì  Every  Journey  is  different   ì  You  need  to  determine  what  is  right  for  you   ì  Learn,  Implement,  Measure,  Iterate  and  Share