How To Make Our Staging Deployment 3x Faster

Bagaimana proses improvement di KMK, bagaimana tim devops meng-improve proses deployment ke staging

Transcript

1. HOWTO MAKE OUR STAGING DEPLOYMENT 3x FASTER KMKLABS TechTalk
2. STAGING DEPLOYMENTS
3. STAGING DEPLOYMENTS Vidio-child-SCM-Poll VIDIO-TEST-PARENT Vidio Vidio-child-Tag-Rev staging_deploy_vidio
5. Hydra • 68% Ansible • Python Script, Bash Script, Doc, Config
6. ./hydra/canary/canary.py ENVIRONMENT
7. update_bastion list_migration_updates provision webserver deploy webserver update webserver canary_test tunnel_canary elb disable_worker_cron_jobs autoscaling deploy workerserver update workerserver start_worker_cron_jobs canary_terminate
8. update_bastion git pull, peru sync list_migration_updates list all db migration provision webserver deploy webserver update webserver canary_test tunnel_canary elb disable_worker_cron_jobs autoscaling deploy workerserver update workerserver start_worker_cron_jobs canary_terminate
9. update_bastion git pull, peru sync list_migration_updates list all db migration provision webserver launch ec2 instance deploy webserver install services & dependencies update webserver update code, compile assets canary_test check port 80 ready or not tunnel_canary elb disable_worker_cron_jobs autoscaling deploy workerserver update workerserver start_worker_cron_jobs canary_terminate
10. update_bastion git pull, peru sync list_migration_updates list all db migration provision webserver launch ec2 instance deploy webserver install services & dependencies update webserver update code, compile assets canary_test check port 80 ready or not tunnel_canary test by QA elb test using real traffic/users disable_worker_cron_jobs autoscaling deploy workerserver update workerserver start_worker_cron_jobs canary_terminate
11. update_bastion git pull, peru sync list_migration_updates list all db migration provision webserver launch ec2 instance deploy webserver install services & dependencies update webserver update code, compile assets canary_test check port 80 ready or not tunnel_canary test by QA elb test using real traffic/users disable_worker_cron_jobs temporary remove cron jobs autoscaling create autoscaling group deploy workerserver install services & dependencies update workerserver update code start_worker_cron_jobs re-activate cron jobs canary_terminate terminate canary instance
12. update_bastion git pull, peru sync list_migration_updates list all db migration provision webserver launch ec2 instance deploy webserver install services & dependencies update webserver update code, compile assets canary_test check port 80 ready or not tunnel_canary test by QA elb test using real traffic/users disable_worker_cron_jobs temporary remove cron jobs autoscaling create autoscaling group deploy workerserver install services & dependencies update workerserver update code start_worker_cron_jobs re-activate cron jobs canary_terminate terminate canary instance
13. staging deployment automated using jenkins
14. update_bastion git pull, peru sync list_migration_updates list all db migration provision webserver launch ec2 instance deploy webserver install services & dependencies update webserver update code, compile assets canary_test check port 80 ready or not tunnel_canary test by QA elb test using real traffic/users disable_worker_cron_jobs temporary remove cron jobs autoscaling create autoscaling group deploy workerserver install services & dependencies update workerserver update code start_worker_cron_jobs re-activate cron jobs canary_terminate terminate canary instance Deployment using Canary script
15. update_bastion git pull, peru sync list_migration_updates list all db migration provision webserver launch ec2 instance deploy webserver install services & dependencies update webserver update code, compile assets canary_test check port 80 ready or not tunnel_canary test by QA elb test using real traffic/users disable_worker_cron_jobs temporary remove cron jobs autoscaling create autoscaling group deploy workerserver install services & dependencies update workerserver update code start_worker_cron_jobs re-activate cron jobs canary_terminate terminate canary instance Staging Deployment using Jenkins
16. seconds minutes update_bastion 32.56 0.54 provision webserver 65.55 1.09 provision workerserver 3.19 0.05 deploy webserver 370.92 6.18 deploy workerserver 167.82 2.80 update webserver 1045.03 17.42 update workerserver 70.99 1.18 autoscaling 163.62 2.73 sleep 600.00 10.00 canary_terminate 11.87 0.20 others 168.45 2.81 TOTAL 2700.00 45.00 Staging Deployment using Jenkins
17. #1
 Provision using daily base image
 (5 Jan 2016)
18. seconds minutes second s minutes update_bastion 32.56 0.54 31.82 0.53 provision webserver 65.55 1.09 78.39 1.31 provision workerserver 3.19 0.05 2.80 0.05 deploy webserver 370.92 6.18 244.69 4.08 deploy workerserver 167.82 2.80 160.84 2.68 update webserver 1045.03 17.42 407.35 6.79 update workerserver 70.99 1.18 65.41 1.09 autoscaling 163.62 2.73 78.68 1.31 sleep 600.00 10.00 600.00 10.00 canary_terminate 11.87 0.20 11.91 0.20 others 168.45 2.81 178.11 2.97 TOTAL 2700.00 45.00 1860.00 31.00 Staging Deployment using Jenkins #1
19. #2 Remove “sleep” before canary_terminate (6 Jan 2016)
20. seconds minutes seconds minutes update_bastion 32.56 0.54 31.43 0.52 provision webserver 65.55 1.09 82.70 1.38 provision workerserver 3.19 0.05 3.26 0.05 deploy webserver 370.92 6.18 243.48 4.06 deploy workerserver 167.82 2.80 180.27 3.00 update webserver 1045.03 17.42 392.55 6.54 update workerserver 70.99 1.18 70.87 1.18 autoscaling 163.62 2.73 142.41 2.37 sleep 600.00 10.00 0.00 0.00 canary_terminate 11.87 0.20 11.83 0.20 others 168.45 2.81 221.00 3.68 TOTAL 2700.00 45.00 1380.00 23.00 Staging Deployment using Jenkins #2
21. #3 Skip deploy webserver, 
 deploy workerserver, 
 if there is no change in Hydra (11 Jan 2016)
22. seconds minutes seconds minutes update_bastion 32.56 0.54 31.08 0.52 provision webserver 65.55 1.09 85.52 1.43 provision workerserver 3.19 0.05 2.95 0.05 deploy webserver 370.92 6.18 0.00 0.00 deploy workerserver 167.82 2.80 0.00 0.00 update webserver 1045.03 17.42 448.88 7.48 update workerserver 70.99 1.18 72.30 1.21 autoscaling 163.62 2.73 72.37 1.21 sleep 600.00 10.00 0.00 0.00 canary_terminate 11.87 0.20 11.81 0.20 others 168.45 2.81 235.09 3.92 TOTAL 2700.00 45.00 960.00 16.00 Staging Deployment using Jenkins #3
23. • Staging deployment vidio, before speedup process: 40 min ~ 50 min • After speedup process: 12 min ~ 18 min • That’s 3x faster!
24. Ansible / Jenkins Code Demo
25. Any Question?