Building Koi Pond: Simulating Millions of Slack Clients

Talk for Craft Conf, June 2021

Transcript

1. M a ude Lem a ire (@qcm a ude) Building Koi Pond: Simul a ting Millions of Sl a ck Clients 1
2. A little bit about me • I started at Slack in October 2016. • Most of my work has been focused on big refactors, and scaling and performance problems 🚀. • In 2020, I published an O’Reilly book 📕. • After 4 years in San Francisco 🌁, I relocated to New Hampshire 🌲. 2
3. Once upon a time … we had a big customer. We’ll call them Big Corp. Big Corp HQ 3
4. Big Corp was already pushing the boundaries of our system. Both of us were nervous to opt them into more features. Servers ful fi lling Big Corp requests 4
5. But Big Corp really liked Slack, and they wanted to use it more. So we needed to load test*. *unfortunately, we didn’t have any tooling to do that. 5
6. https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V https://slack.com/api/conversations.history?channel=C5JN6JH7V Our fi rst tool was simple. It blasted API requests to our servers. Read more about API Blaster on our blog here. 6
7. Unfortunately, it didn’t test a critical component of our infrastructure: the websocket stack. 7
8. High-level overview of Slack’s architecture Clients (Desktop App, Browser, iOS, Android) Permanent Data Stores Backend Systems (Business Logic) Real-time Services RTM API long-lived JSON web socket Edge API HTTPS Web API HTTPS 8
9. Surface area tested via API Blaster Clients (Desktop App, Browser, iOS, Android) Permanent Data Stores Backend Systems (Business Logic) Real-time Services RTM API long-lived JSON web socket Edge API HTTPS Web API HTTPS 9
10. Posting a message to Slack Clients (Desktop App, Browser, iOS, Android) Permanent Data Stores Backend Systems (Business Logic) Real-time Services RTM API long-lived JSON web socket Edge API HTTPS Web API HTTPS chat.postMessage “Good morning!” to #general 10
11. Message propagation mechanism Clients (Desktop App, Browser, iOS, Android) Permanent Data Stores Backend Systems (Business Logic) Real-time Services RTM API long-lived JSON web socket Web API HTTPS chat.postMessage “Good morning!” to #general 11
12. Even Bigger Corp. came along. Even Bigger Corp HQ 12
13. Like with Big Corp., Even Bigger Corp. also really liked Slack. They wanted a channel to house all of their users, about 100,000 of which were active on a weekly basis. *unfortunately, API blaster wouldn’t cut it. 13
14. So we built Puppet Show. 14
15. Clients (Desktop App, Browser, iOS, Android) Clients (Desktop App, Browser, iOS, Android) Clients (Desktop App, Browser, iOS, Android) Clients (Desktop App, Browser, iOS, Android) Puppet Show within Slack’s architecture Headless Chrome browsers logged into slack.com/messages Permanent Data Stores Backend Systems (Business Logic) Real-time Services RTM API long-lived JSON web socket Edge API HTTPS Web API HTTPS Puppets 15
16. About a year later, just a few weeks into the global pandemic, we set a goal. Support 1M users in the same Slack team by June 2021 🚀. 16
17. Lightweight client simulators Lightweight client simulators Lightweight client simulators We designed and began building Koi Pond. Lightweight client simulators Clients (Desktop App, Browser, iOS, Android) Clients (Desktop App, Browser, iOS, Android) Clients (Desktop App, Browser, iOS, Android) Clients (Desktop App, Browser, iOS, Android) Headless Chrome browsers logged into slack.com/messages Lightweight client simulators 17
18. Keeper • A koi is a Slack client simulation (a single Go routine). • A school is a collection of up to 5,000 koi (a single Go program). • The keeper manages the schools and keeps track of the overall load test state and parameters. Koi School School School Koi Koi 18
19. 19 Permanent Data Stores Backend Systems (Business Logic) Real-time Services RTM API long-lived JSON web socket Edge API HTTPS Web API HTTPS Koi within Slack’s architecture Koi
20. How about a live demo? 20
21. behaviours & formations We de f ine two kinds of work for koi to execute: 21
22. Behaviours • An action koi take with a given per-minute frequency. • Usually consists of a single API call, like conversations.history. • Made up of two components: a frequency de f inition, and an implementation. 22
23. { “behaviors”: [ ... , “conversations.history”: { "frequency": 0.10811145124716563, }, ] ... 23 Behaviours (cont.)
24. Behaviours (cont.) { “sequences”: [ ... , “conversations.history”: { "doc": "Calls the conversations.history API endpoint with a random channel", "steps": [ { "doc": "Get a random channel. Halt if no channel is found, otherwise call conversations.history", "pre_actions": [ "{{ set \"channel\" (randomChannel false) }}", "{{ if (not (get \"channel\")) }}{{ halt \"no random channel found\" }}{{ end }}", "{{ setCurrentChannel (get \"channel\") }}" ], ... 24
25. Behaviours (cont.) { “sequences”: [ ... , “conversations.history”: { . .. , "request": { "type": "api", "method": "conversations.history", "args": { "channel": "{{ (get \"channel\").ID }}", "limit": "42", "ignore_replies": "true", "include_pin_count": "true", }, .. . 25
26. Behaviours (cont.) { “sequences”: [ ... , “conversations.history”: { . .. , "post_actions": [ "{{ addMessages (get \"channel\").ID .messages }}", "{{ set \"user_ids\" (extractStrings .messages \"user\") }}", "{{ addUserIDs (get \"user_ids\") }}" ] } } ] .. . 26
27. Formations • Multiple actions a certain percentage of koi should take within a speci f ic period of time to simulate real-world thundering herd behaviour. • Made up of a single sequence (set of steps). 27
28. Formations (cont.) { "formations": [ { "name": "Announcement Channel", "begin_within_secs": 360, "percent": 1.0, "sequence": { "steps": [ { "pre_actions": [ "{{ set \"channel_id\" \"C016DR3DC11\" }}" ], "request": { "type": "api", "method": “conversations.history” }, "delay_after_ms": 5000 }, 28
29. Formations (cont.) { "formations": [ { "steps": [ ... { "request": { "type": "api", "method": “reactions.add” }, "args": { "channel": "{{ (get \"channel\").ID }}", "timestamp": “{{ get \”message_ts\"}}“, "name": “{{ pickString (stringArray \”fish\” \”ocean\" \”whale\”}}”, }, } ], }, } 🐟 🌊 🐳 29
30. Let’s check back in with our koi. 30
31. 31 Were we successful? Development starts First behaviour-only test run 50,000 koi July 2020 Scale up to 500,000 koi Per koi cost: 0.001$ per day September 2020 Formations introduced October 2020 Successful test run with 1M koi 🎉 December 2020 Successful test run with 1.7M koi April 2021 Successful test run with 2M koi 🚀 July 2020 June 2020 *timeline not to scale
32. Check out our recent blog post for more details on the many bugs 🐛 we’ve found and f ixed, and future development plans 🔮. As always, Slack is hiring. 👩💻 32