Scaling Automation in an Enterprise

Presented at the March infracoders meetup on 10th March 2015.
http://www.meetup.com/Infrastructure-Coders/events/218901403/

Transcript

1. Scaling  Automa.on  in  an   Enterprise   infracoders  March  Meetup     Michael  Wu,  Australia  Post  
2. Australia  Post   •  200+  year  old  government  owned  corpora.on   •  LeBers,  parcels  and  retail   •  Digital  channels  a  larger  focus  
3.    Digital  Delivery  Centre     •  Established  2.5  years  ago   •  Agile  teams,  prac.cing  SAFe   •  All  consumer  facing  digital  applica.ons   •  Java  focused,  Linux  plaKorms  and  API  First  
4.    Digital  Delivery  Centre     Parcel  Tracking   Post  Office   Loca.ons   Postage   Calculator   Travel  Services  
5.    A  liBle  over  a  year  ago…     •  Everything  with  managed  service  provider   •  No  configura.on  management   •  Impeded  by  inconsistent  quality   •  Not  able  to  quickly  adapt  to  change  
6. So  we  began  the  journey…     Several  months  of:   – Lots  of  tech  spikes,  trial  and  error   – Lots  and  lots  of  workshops  with  different  teams   – MVP,  with  bolt  ons  as  things  become  more  clear  
7.   Support  diverse  applica.ons,   environments  and  teams   •  ~40  applica.ons   •  Differing  levels  of  maturity   •  Differing  states  of  “ac.veness”   •  Differing  teams,  technologies  and  outputs  
8. Skip  forward  several  months…  
9. Two  Artefacts   repo   automa.on-­‐code   configura.on   artefact   automa.on-­‐code-­‐2.3.0.tgz   configura.on-­‐2.3.0-­‐1.tgz   purpose   Automa.on  logic   Configura.on  that  describes  the   stack  and  its  environment   technologies   •  AWS  CloudForma.on  for   infrastructure  provisioning   •  Puppet  for  Configura.on   Management   •  Python  scripts  for   deployment  process   •  Puppet  Hiera   Automa.on   Code   2.3.0   Automa.on   Config   2.3.0-­‐1  
10. Provisioning,  how  it  fits…  
11. Provisions  “applica.on  stacks”   Amazon S3 Web  Stack  with  Nginx   Web  Stack  with  Apache   App Stack with Tomcat/JRE Every  Stack…   CloudWatch Monitoring  metrics…   Logging…   In-­‐transit  encryp.on…   Dynamo  DB  
12. Configura.on   •  Separa.on  of  code  and  configura.on  a  no-­‐ brainer   •  Key  to  not  repea.ng  yourself   •  Mature  tools  exist:  Hiera,  Data  Bags  
13. Hiera  101   •  Hiera  is  a  key/value  lookup  tool   •  For  Puppet  configura.on  data  
14. Typical  uses  of  Hiera…   # environment/development.yaml example name_servers: - '10.0.0.10' - '10.0.1.10’ ntp_servers: - ‘0.us.pool.ntp.org iburst’ - ‘1.us.pool.ntp.org iburst’ - ‘2.us.pool.ntp.org iburst’ newrelic_licence: ENC[PKCS7,MIIBmQYJKoZIhvcNAQcDoIIBijCCAYYCAw DQYJKoZIhvcNAQEBBQAEggEAXysZOMY0W8/iwL4sU=]
15. Can  also  be  used  to  define   infrastructure   # environment/test.yaml example vpc_id: 'vpc-847d2c4e6’ private_application_subnet_a: 'subnet-a1598f48' private_application_subnet_b: 'subnet-51e9a52c’ private_application_subnets: - "%{hiera('private_application_subnet_a')}" - "%{hiera('private_application_subnet_b')}” # application/parceltracking.yaml instance_ami: ‘ami-9fc5130a5’ application_elb_health_check: ‘SSL:8443’ application_auto_scale_group: '2,2,2’ # min, max, desired rds_multi_az: ‘false’
16. Hiera  driving  CloudForma.on   Key   Value   ELBHealthCheck     SSL:8443   ELBSSLCert   star.ap-­‐southeast-­‐2.elb.amazonaws.com   InstanceAmi   ami-­‐1f34e762c   InstanceType   t2.small   InstanceVolumeSize   10   AutoScaleGroup   2,2,2   AppTierSubnets   subnet-­‐98fc7a26,subnet-­‐e52b5fe7   hiera-­‐py   Hiera  
17. Define  applica.on  artefacts  and  config   # application/parceltracking.yaml example application_config::artefact_hash: ’parceltracking': deploy_location: "%{hiera('tomcat_deploy_path')}/track" target: 'tomcat' extension: 'war' file_format: '<name>-<version>.<extension>' # application/parceltracking.yaml example application_config::config_hash: db: jdbc: url: "jdbc:mysql://%{::main_db_address}:3306/track" username: "tracking" password: ENC[PKCS7,MIIBeQYJK...] driver: "com.mysql.jdbc.Driver" recaptcha: url: "http://www.google.com/recaptcha/api/verify" privatekey: ENC[PKCS7,MIIBmQYJKoZIhvc...]
18. As  One     Trea.ng  all  automa.on  and  configura.on  the   same  –  Infrastructure,  ConfiguraDon   Management  and  ApplicaDon  –  can  greatly   simplify  your  life  
19. Simplify  processes   I  want  to  deploy  applica.on  and  infrastructure   changes     1.  Code,  commit,  push  and  pull  request   2.  Update  stack  with  new  version  of  code/config   3.  Scale  up  autoscale  group  (2  >  4)   4.  Scale  down  autoscale  group  (4  >  2)   5.  Done!   2.1   2.1   2.2   2.2   But  it’s  also  the  exact  same  process  for:     •  Modifying  applica.on  configura.on   •  Patching  for  security  vulnerabili.es   •  Cer.ficate  rota.on   •  Stack  enhancements    
20. Automated  Stacks   •  With  two  artefacts  /  repos   •  Infrastructure,  configura.on  management  and   applica.on  deployment   •  Designed  to  work  for  all  applica.ons  and   teams  
21. Solving  for  diversity  is  hard   Leads  to  talk  of  standardisa.on   …  paBerns   …  stencils   …  plaKorms  
22. PlaKorms…  as  a  Service  
23. Where  are  we  Today?   •  100%  in  AWS,  thanks  to  lis-­‐and-­‐shis   •  150+  applica.on  stacks  across  dev,  test,  prod   •  25%  produc.on  applica.ons  on  new  stacks    
24. What’s  next…   •  API  for  stacks   •  Scaling  with  more  services   Email:  michael.wu2@auspost.com.au   LinkedIn:  au.linkedin.com/in/michaelwu10