Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Stay C.A.L.M.S.: A local company's journey into DevOps

Stay C.A.L.M.S.: A local company's journey into DevOps

A fast walk-through of our journey to DevOps at Solana Beach based OneHealth.


No such thing as a DevOps team
- http://continuousdelivery.com/2012/10/theres-no-such-thing-as-a-devops-team/
- http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/
- http://www.slideshare.net/jezhumble/scaling-devops

Enter C.A.L.M.S.
- John Willis, July 2010 https://www.chef.io/blog/2010/07/16/what-devops-means-to-me/
- James Turnbull, Feb 2010 http://www.kartar.net/2010/02/what-devops-means-to-me/
- Jez Humble, Sept 2010 http://www.slideshare.net/jezhumble/devops-and-agile-release-management

Culture: Spotify Squads
- https://www.scribd.com/doc/113617905/Scaling-Agile-Spotify

Automation: Infrastructure as Code
- http://www.jedi.be/blog/2013/05/24/Infrastructure%20as%20Code/

Reduce Risk of Release
- http://www.slideshare.net/jezhumble/devops-and-agile-release-management
- http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

Theory of Constraints
- http://en.wikipedia.org/wiki/Theory_of_constraints

Our Implementation
- http://www.slideshare.net/JarrodOverson/continuous-delivery-for-the-web-platform

Continous Delivery Maturity Model
- http://info.thoughtworks.com/rs/thoughtworks2/images/continuous_delivery_a_maturity_assessment_modelfinal.pdf
- http://www.infoq.com/articles/Continuous-Delivery-Maturity-Model

Continuous Delivery Patterns
- http://continuousdelivery.com/patterns/
- http://refcardz.dzone.com/refcardz/continuous-delivery-patterns

Pattern: Feature Flags /Dark Launching
- http://en.wikipedia.org/wiki/Feature_toggle
- http://www.facebook.com/note.php?note_id=96390263919

Pattern: Control your Environment
- https://github.com/rcrowley/freight

- http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

Monitoring Sucks
- http://obfuscurity.com/2011/07/Monitoring-Sucks-Do-Something-About-It


Sander van Zoest

January 21, 2015


  1. ì   A  local  company’s  journey  to  DevOps   Stay

     C.A.L.M.S.   Sander  van  Zoest   @svanzoest  
  2. ì   It  is  a  human  and  management  problem  

    What  is  DevOps?  
  3. No  such  thing  as  DevOps  team   ì  DevOps  is

     not  a  new  name  for  System   AdministraAon   ì  It  is  about  collaboraAon  between  funcAonal  teams.  
  4. Enter  C.A.L.M.S.   ì  Culture   ì  AutomaAon   ì 

    Lean   ì  Measure   ì  Sharing  
  5. Company  Culture   Industrial   FuncAonal  Silos   Blame  runs

     downhill  in  public,   and  uphill  at  the  water-­‐cooler   Avoid  Risk  and  Failure   Never  revise  policy  on  success   (RetrospecAve)   Heteronomy       Agile   Cross-­‐funcAonal  delivery   teams   Team  Ownership  (Trust)   Fail  Fast,  Learning   Opportunity   ConAnuous  Process  Review   Autonomy  and  Alignment    
  6. Culture:  Breaking  the  Silos   ì  SeaAng  Arrangements   ì 

    Cubicles  vs  Open  Spaces   ì  Headphones   ì  Sales  versus  Development  tasks   ì  Unbiased  Project  Manager  (reports  to  Product  or  Engineering?)   ì  Matrix  Teams   ì  SpoAfy’s  Squads  
  7. Culture:  Spotify  Squads  

  8. Squads  own  a  minimum   viable  product   How  do

     split  into  squads?   Horizontal   Based  on  Product  Features   VerAcal   Based  on  Product  Layers   Is  Customer  Service  a  Squad?   Is  Account  Management?   Is  OperaAons?  
  9. Fear  of  Failure   Understand  you  can  never   know

     everything.   Account  for  unknown   unknowns   Focus  on  unknowns  first   Fail  fast   Foster  creaAvity  and   innovaAon   Learn   Create  a  safety  net     Never  ending  2  week  sprints   InAmidaAng  teammates   Constant  context  switching   Perceived  failure  in  lack  of  progress   Lack  of  personal  safety  in  admiZng   failure  of  a  task   Limited  Long  term  vision   Low  Self-­‐Esteem   PerfecAonism  
  10. ì  Project  Manager:  "What  is  the  status  of   project

     X?  Did  you  ever  complete  that?"   ì  So]ware  Engineer:  "Oh,  that's  done."   ì  Project  Manager:  "Great,  so  it  is  live  in   produc>on  then?"   ì  So]ware  Engineer:  "Ehhh...no,  it  s>ll   needs  to  be  tested,  integrated  and   deployed.  I  am  just  dev  done”.     It  isn't  rodeo  done,  unAl  it  has   passed  acceptance  tesAng  in   producAon.     Rodeo  Done  
  11. Automation   ì  Infrastructure  as  Code   ì  Creates  Consistency

      ì  Avoids  RepeAAve  (boring)  Tasks   ì  Makes  room  for  improvement  and  learning   ì  Removes  Ambiguity  (Blame  Culture)   ì  Safety  Net  to  try  new  things  
  12. ì   Service  Life  Cycle  

  13. Automation:  Baby  steps   ì  First  determine  what  your  product

     life  cycle  is.   ì  We  started  with  deploying  the  dev  environment.   ì  Make  it  as  close  to  producAon  as  possible     ì  Avoid  separate  ways  of  doing  things.     ì  Create  a  single  flow  to  producAon.   ì  Goal   ì  Fully  automated  deployment  to  producAon   ì  Get  it  down  to  a  single  bucon  press  
  14. Dev  Environment  Setup:  Goal   ì  How  long  does  it

     take  for  a  new  hire  to  setup  their   environment?  How  long  unAl  they  commit  to   producAon?   ì  In  2010  it  took  2  months  setup  the  env  and  update   docs.   ì  In  2014  they  would  push  to  producAon  within  48   hours.   ì  What  should  be  other  goals?  How  would  you   measure  them?  
  15. Dev  Environment  Setup:  How   ì  Leverage  VirtualizaAon:  We  used

     Vagrant  with   Virtualbox   ì  It  allowed  us  to  DRY  up  use  the  same  tools  on  dev   as  producAon.   ì  Don’t  worry  about  what  tools,  focus  on   repeatability.  Shell  scripts?  Doesn’t  macer.  As  long   as  it  is  reliable  and  repeatable.   ì  Once  it  is,  measure,  review  and  iterate  
  16. Dev  Environment  Setup:  Shell  Scripts   ì  Now  that  it

     is  repeatable,  how  would  you  apply  it  to   producAon?   ì  Do  any  of  the  packages  and/or  configs  change?   ì  Did  you  check  them  in  version  control?   ì  Next  convert  them  into  a  more  flexible  tool.   ì  We  chose  Chef  
  17. ì   Release  Cycles  

  18. Release  Frequently  

  19. Old  Release  Process   Feature  1   Feature  2  

    Feature  3   Feature  4   Feature  5   Feature  6   Feature  7   Release  Regression  TesAng   Release  Regression  TesAng  
  20. Goal:  Reduce  Risk  of  Release  

  21. ì  IdenAfy  the  system's  constraint   ì  IntegraAon  TesAng  and

     Release  Management   ì  Development  completed.  Needs  to  be  tested  (waterfall)   ì  Decide  how  to  exploit  the  system's  constraint   ì  Build  tests  at  the  same  Ame  as  developing  the  code  (alignment/ focus)   ì  Minimize  changes  tested  at  once.  Minimize  backlog  of  releases.   ì  Subordinate  everything  else  to  the  above  decision   ì  Align  the  whole  system  or  organizaAon  to  support  the  decision   made  above ì  Elevate the system's constraint ì  Measure,  rinse  and  repeat Theory  of  Constraints  
  22. Release  Summary  2013   Release   Date   US  

    DE     5.6   20-­‐Dec   3   20   5.7   4-­‐Jan   8   49   5.8   9-­‐Jan   0   4     5.9   17-­‐Jan   2   12   5.1   25-­‐Jan   3   26   5.11   7-­‐Feb   4   21   5.12   14-­‐Feb   3   14   5.13   18-­‐Mar   5   29   6   15-­‐Apr   19   169   0   50   100   150   200   250   300   350   400   450   20-­‐Dec   20-­‐Jan   20-­‐Feb   20-­‐Mar   User  Story/Defect  
  23. ì  Making  sure  your  so]ware  is  always  producAon   ready

     throughout  its  enAre  lifecycle   ì  reduce  the  cost,  Ame,  and  risk  of  delivering   incremental  changes  to  users   ì  Everybody  is  responsible  for  delivery   What  is  continuous  delivery?  
  24. ©  Copyright  2012  OneHealth   SoluIons,  Inc.   1/25/15  

    24   Continuous  Delivery  Process  Flow  
  25. Our  Implementation  

  26. ì  Quicker  turn  around  from  Concept  to  Deployment   ì 

    Easier  to  quickly  fix  issues  in  producAon   ì  Focus  on  Roll  forward,  Not  backwards   ì  More  focus  on  automated  tests   ì  More  focus  on  process  improvement   ì  Quality  focus  rather  than  individual  throughput   focus   Findings  &  Learnings  
  27. Continuous  Delivery  Maturity  Model  

  28. Continuous  Delivery  Patterns   ì  ConAnuous  IntegraAon   ì  aka

     Develop  on  Mainline   ì  Feature  Toggles   ì  Branch  by  AbstracAon   ì  Dark  Launching   ì  Database  Versioning   ì  Database  Backwards   CompaAbility   ì  ProducAon  Immune   System   ì  Blue-­‐Green  Deployments   ì  Canary  Releasing   ì  Expand/Contract   ì  A/B  TesAng  
  29. Pattern:  Feature  Flags  /  Dark  Launching   ì  Allows  code

     to  be  released  but  not  visible  to   everyone   ì  Works  in  parallel  with  A/B  TesAng   ì  Allows  for  feature  iteraAon  by  product  owner  in   while  using  it  in  producAon   ì  Allows  Customer  Service  and  Account  Management   to  get  familiar  with  feature  and  communicate  to   customers  at  their  own  pace  
  30. Pattern:  Develop  on  Mainline   ì  We  explored  many  branching

     models   ì  Git  Flow   ì  Pull  Request   ì  Etc.   We  currently  standardize  on  Pull  Requests  to  allow  for   peer  code  review.  
  31. Pattern:  System  Composition   LocaAon  (Data  Center)   ì  for

     gateway,  NTP,  APT  Mirror,  etc.   Cluster   ì  for  the  smallest  logical  horizontal  scalable  node   ì  E.g.  Front  End,  Data  Store   Environment   ì  Dev,  QA,  CI,  Staging,  Sandbox,  ProducAon   Realm   ì  Data  type  or  source:  IdenAty,  Content,  BI  
  32. Pattern:  System  Composition   Translates  to  monitoring,  graphing,   reporAng,

  33. Pattern:  Vendoring  during  Development   ì  Starts  by  creaAng  a

     vendor  directory  in  your  project   ì  You  ulAmately  only  need  to  vendor  your   dependencies  in  your  build  arAfacts   ì  We  use:   ì  Composer/SaAs  for  PHP   ì  Bundler  for  Ruby   ì  Berkshelf  for  Chef  Cookbooks   ì  NPM  for  Node/JavaScript  
  34. Pattern:  Build  Artifacts  for  Deployment   Build   Process  

    Version  Control   ArAfacts  
  35. Pattern:  Separate  Code  from  Config   Dev   CI  

    Staging   ProducAon   Build   ArAfact   Same  ArAfact   Same  ArAfact   Build   Cookbook   Same  Cookbook   Same  Cookbook   Specific   Environment   ConfiguraAon   Specific   Environment   ConfiguraAon   Specific   Environment   ConfiguraAon   Specific   Environment   ConfiguraAon  
  36. Pattern:  Control  your  Environment   ì  Setup  your  own  OS

     Package  Repository   ì  Setup  controlled  mirrors  of  all  so]ware  you  depend   upon  in  producAon   ì  We  used  apropos  and  freight  for  our  debian   packages  
  37. Lean   ì  Remember  team  ownership   ì  CollaboraAvely  determine

      the  true  problem  (root   cause)   ì  Pick  tools  useful  by  everyone   ì  Communicate  Reasoning     ì  Eliminate  waste   ì  Amplify  learning   ì  Decide  as  late  as  possible   ì  Deliver  as  fast  as  possible   ì  Empower  the  team   ì  Build  integrity  in   ì  See  the  whole  
  38. Measure   ì  Build  Confidence   ì  Provides  Context  

    ì  Incidents   ì  Frequency   ì  Severity   ì  Root  Cause   ì  Time-­‐To-­‐Detect  (TTD)   ì  Time-­‐To-­‐Resolve  (TTR)   ì  Coordinate  Responses   ì  Measure  Progress   ì  Track  Change   ì  Value  Stream  Mapping   ì  Graph  It   ì  Alert  on  it  
  39. Monitoring  Sucks   Take  a  chapter  from  the  "Infrastructure  as

      Code"  movement  and  appreciate  the   benefits  that  come  from  interoperable   pieces  that  are  inexpensive,  scalable  and   easily  automated  with  a  licle  code  and  a   stable  API.  -­‐-­‐  Jason  Dixon  (obfuscurity)  
  40. Monitoring:  Tracking   ì  ApplicaAon  Data:   ì  Statsd  

    ì  ApplicaAon  Logs  and  Events   ì  Deployment  Data   ì  Jenkins   ì  Chef  Report  Handler   ì  System  Data   ì  Sensu   ì  etc  
  41. Monitoring:  Graphing   ì  Graphite   ì  ELK  Stack  

    ì  ElasAcSearch   ì  Logstash   ì  Kibana   The  key  thing  is  to  build  systems  that  can  be  easily   maintained  and  accept  new  arbitrary  data,  so  you  can  focus   on  the  data  and  presentaAon.     Not  the  creaAon  of  dashboards.  
  42. ì  Does  two  things:   ì  Store  numeric  Ame-­‐series  data

      ì  Render  graphics  of  this  data  on  demand   ì  Consists  of  three  things:   ì  Receiver  Daemon  -­‐  Carbon   ì  Disk  Storage  Library  -­‐  Whisper   ì  Web  ApplicaAon  –  Graphite  /Graphana   ì  Used  for  ApplicaAon  and  System  metrics   Monitoring:  What  is  Graphite?  
  43. Monitoring:  What  is  an  ELK  Stack?   ì  ElasAcSearch  

    ì  Logstash   ì  Kibana   We  feed  it  via  rsyslog  and   allows  querying  of  the  logs  and   alerAng  on  log  data.  
  44. Monitoring:  Alerting   ì  Tried  Nagios,  Opsview,  etc.   ì 

    Landed  at  Sensu   ì  Event  Messaging  Bus   ì  Maps  monitoring  to  Chef  cookbooks   ì  AutomaAc  Client  detecAon  and  decommission   ì  Easy  integraAon  with  other  so]ware   ì  Graphite   ì  Email   ì  Slack  (ChatOps)  
  45. Share   ì  Good  News  and  Bad  News   ì 

    Creates  Trust  and   Confidence   ì  Helps  collaboraAon   ì  Within  your  team   ì  Between  departments   ì  Between  companies  
  46. Share:  Real-­‐Time  Communication   ì  Dashboards   ì  Hallway  Monitors

      ì  Via  Email   ì  Over  the  Cube   ì  ChatOps   ì  Focus  on  informaAon   sharing   ì  System  IntegraAons   ì  ConAnuously  tune  to   improve  signal  to  noise  raAo.  
  47. Share:  Community   ì  Open  Source   ì  Bug  Reports

      ì  Code  Patches   ì  DocumentaAon   ì  Speak  at  User  Groups,   Conferences   ì  Contribute  back  to  the   community   ì  Open  Source  re-­‐usable   generic  tools  and  methods   ì  Write  a  blog   ì  Join  mailing  lists,  etc.   ì  Join  San  Diego  DevOps   ì  SDDevOps.org  
  48. San  Diego  DevOps  Speakers  &  Members  

  49. Summary   ì  Every  Journey  is  different   ì  You

     need  to  determine  what  is  right  for  you   ì  Learn,  Implement,  Measure,  Iterate  and  Share