Responsive Infrastructure With Autoscaling

At Bugsnag, we receive error reports from our customers when their mobile, server, or JavaScript applications crash. This means that we often experience large fluctuations in load, as customers ship code containing bugs and then subsequently release fixes. Based on these load characteristics, we've designed our services to use autoscaling to keep Bugsnag performant and available. In this talk, I cover some of the architectural choices we've made to get the most out of autoscaling.

Presented at the AWS Loft in San Francisco.

Transcript

1. Responsive Infrastructure with Autoscaling
2. @keeganlow
3. autoscaling overview + autoscaling at Bugsnag
4. autoscaling overview
5. common load patterns
6. common load patterns
7. common load patterns
8. common load patterns
9. common load patterns
10. common load patterns
11. common load patterns
12. common load patterns
13. common load patterns
14. common load patterns
15. common load patterns
16. common load patterns
17. autoscaling!
18. a real-world example
19. netflix cpu load http://techblog.netflix.com/2012/01/auto-scaling-in-amazon-cloud.html
20. http://techblog.netflix.com/2012/01/auto-scaling-in-amazon-cloud.html netflix active servers
21. cpu load servers http://techblog.netflix.com/2012/01/auto-scaling-in-amazon-cloud.html
22. a few autoscaling concepts…
23. beyond CPU usage as a scaling trigger
24. safe guards
25. what is
26. cross-platform error monitoring
28. load characteristics
29. common load patterns
30. reqs/min over 2 weeks
31. reqs/min over 2 weeks 2x increase
32. Bad Deploy: reqs/min
33. Concurrent Issues: reqs/min
34. Concurrent Issues: reqs/min
35. Concurrent Issues: reqs/min
36. autoscaling is essential for our performance availability customer happiness
37. architecture
38. event servers error payloads event workers datastores dashboard / api
39. Event Workers process events from event server queues
40. event server: go server + queue event worker
41. event server: go server + queue event worker
42. DB event server: go server + queue event worker
43. cloudwatch: total queue length
44. DB event server: go server + queue event worker
45. DB event server: go server + queue event worker event worker event worker
46. DB event server: go server + queue event worker event worker event worker
47. DB event server: go server + queue event worker
48. low worker cpu usage? scale down
49. deploying
50. the general problem…
51. take 1: copy code from running machine
52. take 1: copy code from running machine deploy to all machines…
53. take 1: copy code from running machine deploy to all machines… autoscaling event!
54. take 1: copy code from running machine deploy “finishes” deploy to all machines… autoscaling event! ???
55. Deploying with NFS build server deploy existing hosts sv restart mount -t nfs
56. Autoscaling with NFS build server new hosts mount -t nfs start service
57. Autoscaling with NFS issues with this approach…
58. Autoscaling with NFS single point of failure
59. Autoscaling with NFS need build server to autoscale
60. Elastic File System (EFS)
61. saving money with spot instances
62. on demand instances + spot instances event worker fleet
63. spot instances name your own price cost < on demand ~6min spin up
64. spot instances design for interruption
65. on demand instances predictable cost in general: cost > spot ~2min spin up
66. on demand instances interruptions much less likely
67. 100% spot instances: limited availability (unless bid $$$)
68. 100% on demand instances: not optimizing costs
69. our goals take advantage of spot instance cost benefit do not sacrifice availability
70. group 1: spot instances scale up aggressively scale down slowly
71. group 2: on demand instances scale up conservatively scale down aggressively
72. thanks to autoscaling…
73. 1 day 490,000,000 errors
74. 1 day 490,000,000 errors no one noticed
75. 15 billion crashes processed
76. 15 billion crashes processed 99.99999% availability
77. benefits of autoscaling minimize infrastructure costs
78. benefits of autoscaling increase availability
79. benefits of autoscaling increase customer happiness
80. @keeganlow