Scaling puppet

Scaling a multi-tenant puppet-system from a technical and organizational side

Transcript

1. Pascal  Hahn pascal.hahn@nokia.com  /  ph@lxd.bz Head  of  Automated  Deployment Nokia  Loca?on  and  Commerce   Scaling  a  mul+-­‐tenant  puppet-­‐ system  from  a  technical  and   organiza+onal  side 1
2. -­‐ Head  of  Thor  team  (Nokia’s  Automated  Deployment) -­‐ At  Nokia  for  about  a  year  and  a  half -­‐ Started  as  Principal  Engineer,  got  sucked  into  Management -­‐ Before  Nokia,  Site  Reliability  Engineer  at  Google,  also   developing  Automa?on  (no  puppet) Who  am  I 2
3. Isn’t  Nokia  just  building   cellphones? 3
4. No 4 4
5. What  does  Nokia  do  with  puppet? -­‐ Deploy  wide  range  of  services,  mostly  focused  around  loca?on -­‐ Mul?ple  environments  and  datacenters  around  the  world  (labs,   produc?on,  tes?ng,  development,  …) -­‐ Strict  versioning  and  isola?on  using  environments  +  ENC -­‐ Deployment  as  code,  lots  of  automated  tes?ng  to  make  sure  the   code  works -­‐ Custom  func?ons  –  for  example:  encrypt  confiden?al  data -­‐ RESTful  API  to  manage  yum  repositories  globally -­‐ RESTful  API  to  change  puppet  configura?on 5
6. -­‐ Services  deployed  to  Legacy  system  in  various  ways,  massive  pain -­‐ Developed  Puppet-­‐system  V1,  became  unmaintainable  quickly -­‐ Lots  of  thinking -­‐ Developed  Puppet-­‐system  V2  (=Thor) -­‐ Launched  in  06/2011 Time-­‐lapse 6
7. What  happened  since  then 7
8. -­‐ ~120  different  internal  customer-­‐teams  (~90  in  june) -­‐ Customers  all  over  the  globe -­‐ Machine  fleet  exploded -­‐ Went  from  underdog-­‐solu?on  to  de-­‐facto  standard If  you  build  it  they  will  come 8
9. Problems  we  hit  along  the  way 9
10. Puppet  scales, To  a  point 10
11. -­‐ 1  primary  puppet-­‐master  per  cell -­‐ Listens  on  8140  and  loadbalances  compiles -­‐ >=  3  puppetmasters  total -­‐ Cert  signing  on  primary  master -­‐ Slaves  sync  certs  from  master Catalog  compiles  are  expensive 11
12. -­‐ 8  x  16  hypervisors  maximum -­‐ Standardized  hardware -­‐ Standardized  network-­‐setup -­‐ Repeatable  and  testable  setup Fix-­‐size  `scaling-­‐units`  /  cells 12
13. Small  manual  steps  hurt 13
14. -­‐ 2  jenkins  jobs  per  project,  per  release -­‐ Manual  work  for  new  projects  and  releases -­‐ Results  in  lots  of  clicking  in  Jenkins  and  confused  users Build  Pipeline 14
15. So  we  automated  it 15
16. In  a  repeatable  way 16
17. From  one  end  … 17
18. …  to  … 18
19. ...  the  other 19
20. More  customers,  more  problems 20
21. Luckily,  most  customers  have   similar  issues  and  complaints 21
22. `Integra+on  is  taking  a  lot  of  +me` 22
23. -­‐ Default  set  of  nagios  alerts,  on  every  host -­‐ Machine  metrics  in  trending  system  (Graphite  +  collectd) -­‐ Default  logs  in  Splunk Basic  integra+on  comes  for  free 23
24. APIs  to  extend  integra+on  easily 24
25. `Puppet  is  hard` 25
26. Fully  working  environment  as   star+ng-­‐point 26
27. -­‐ Apache  module  fully  integrated  with  shared  services -­‐ PHP  and  Tomcat  base  projects -­‐ Modules  are  security-­‐team  approved Standard  building-­‐blocks 27
28. Scaling  the  development  team 28
29. -­‐ Hiring  puppet  or  ruby  (systems)  specialists  is  hard -­‐ Hire  smart  ,  generalist,  system  engineers Hiring 29
30. -­‐ Subject-­‐maier-­‐experts  are  good -­‐ As  long  as  they  share  knowledge -­‐ Whole  team  needs  to  know  all  pieces 1  Area    ==  1  Developer 30
31. -­‐ High  quality  documenta?on -­‐ Users-­‐help-­‐users -­‐ IRC  is  great! -­‐ Onsite,  face-­‐2-­‐face  trainings Support  is  taking  a  lot  of  +me 31
32. -­‐ Open-­‐source  core  components,  build  pipeline  and  pm  manager  1. -­‐ Revisit  tes?ng  methodology -­‐ Introduce  permission  model  in  backend  /  replace  legacy  backend -­‐ Wrap  up  AWS  support -­‐ Hire  more  engineers  (Nokia  is  hiring:  hip://devops.nokia.com) Next  steps 32