How Optimizely scaled its REST API with asyncio

Description
With developers, an awesome product isn’t everything, or at least we found that out. More than a product you need a platform. But what is a platform? Learn tips and tricks about building a public API using the latest and greatest tools: OpenAPI, Python 3 and asyncio.

Abstract
With developers, an awesome product isn’t everything, or at least we found that out. More than a product, you need a platform. But what is a platform? And how do you build one alongside all the other weird old code you might have inherited?

In a perfect world, Jeff Dean would ride in on a Unicorn with stone tablets inscribed with UML diagrams of an application architecture so perfect you’d weep with joy. In the real world however that tends not to happen, so the process is a bit more iterative. Sometimes you try things that don’t work or seem silly in retrospect.

Part of our platform journey has been building a public API with our favorite hand picked technologies: OpenAPI and Python 3, especially asyncio. Building a rock solid, fast, and scalable Python application was not all rainbows and unicorns along the way. Through this process, we learned quite a bit about describing REST APIs and how modern Python concurrency mechanisms behave in the real world.

We’ll share how we used OpenAPI to power our public API and improve the developer experience for both internal engineers and external consumers.

We started off with a mix of multiprocessing and asyncio, and have since migrated completely to asyncio. Also, we’ll share a few of the things we've learned in the process, which we hope will help you if you choose to use asyncio.

Bios
Nick DiRienzo is a software engineer at Optimizely. He enjoys writing code in Python, making developers’ lives better, and eating Mission burritos.

Vinay Tota is currently a member of the Application Platform team at Optimizely. A long time ago wanted to be a theoretical physicist but now has way more fun hacking code. More recently he's worked on building high throughput low latency systems for ad serving using Python.

Transcript

1. How Optimizely scaled 
 its REST API with asyncio
2. Nick DiRienzo @nickdirienzo Software Engineer Optimizely @Optimizely Vinay Tota @vinaytota Senior Software Engineer Optimizely
3. We’ll talk about: • Optimizely’s API story • What makes an API “good” • Optimizely’s API gateway • Improving it with asyncio • Lessons learned
4. Where we started Where we started
5. Where we started
6. Optimizely Classic was an A/B Testing Solution
7. Optimizely X is an experimentation platform
8. Optimizely X Web Visual Editor Based Experimentation
9. Optimizely X Full Stack Code Based Experimentation Optimizely X Web Visual Editor Based Experimentation
10. Optimizely X Full Stack Code Based Experimentation Optimizely X Web Visual Editor Based Experimentation Experimentation Platform
13. What is happening?
17. How can we turn a good web UI into a good API?
18. What makes an API “good”?
19. Easy to use APIs What makes an API “good”?
20. More than mapping a data model to an HTTP endpoint
21. API is the UI for developers
22. TODO: Add results slide.
23. Getting Optimizely Results 1. Get Experiment data
24. Getting Optimizely Results 1. Get Experiment data 2. Get Layer data
25. Getting Optimizely Results 1. Get Experiment data 2. Get Layer data (whatever that is)
26. Getting Optimizely Results 1. Get Experiment data 2. Get Layer data (whatever that is) 3. Get Project Settings
27. Getting Optimizely Results 1. Get Experiment data 2. Get Layer data (whatever that is) 3. Get Project Settings 4. Get results data
28. 1. Get Experiment data 2. Get Layer data (whatever that is) 3. Get Project Settings 4. Get results data GET /v2/experiments/:id/results Getting Optimizely Results
29. Easy to use APIs What makes an API “good”? Ensured Trust
30. Build a business on our APIs
31. How do we ensure trust? No surprises
32. How do we ensure trust? No surprises Reliable platform
33. How do we ensure trust? No surprises Reliable platform Performant platform
34. Trust can be made with: Interface Stability
36. Interface stability means... Beta & Stable APIs
37. Interface stability means... Beta & Stable APIs No breaking changes
38. Interface stability means... Beta & Stable APIs No breaking changes Versioning
39. Trust can be made with: Interface Stability Reliability
40. 99.999%
42. Trust can be made with: Interface Stability Reliability Performance
43. Performance is mission critical
44. Performance can be a
 dealbreaker
45. Trust can be made with: Interface Stability Reliability Performance
46. Easy to use APIs What makes an API “good”? Ensured Trust Documentation
47. Write documentation
48. Write documentation
49. Generate documentation
50. Generate documentation with swagger-ui to ensure docs are...
51. Generate documentation with swagger-ui to ensure docs are... Up to Date
52. Generate documentation with swagger-ui to ensure docs are... Up to Date Comprehensive
53. Generate documentation with swagger-ui to ensure docs are... Up to Date Comprehensive Accurate
55. Easy to use APIs What makes an API “good”? Ensured Trust Documentation
56. How can we provide
 “good” APIs? How can we provide “good” APIs?
57. Getting Optimizely Results 1. Get Experiment data 2. Get Layer data (whatever that is) 3. Get Project Settings 4. Get results data
58. 1. Get Experiment data 2. Get Layer data (whatever that is) 3. Get Project Settings 4. Get results data /v2/experiments/:id/results Getting Optimizely Results
59. We’ve got data all over
60. We’ve got data all over Experiments
61. We’ve got data all over Experiments Projects
62. We’ve got data all over Experiments Projects Results
63. We’ve got data all over Experiments Projects Results API Gateway
64. Benefits of an API gateway Operations through the API look a lot like their web UI counterparts

65. Benefits of an API gateway Operations through the API look a lot like their web UI counterparts Allows us to orchestrate APIs instead of data models

66. Benefits of an API gateway Operations through the API look a lot like their web UI counterparts Allows us to orchestrate APIs instead of data models
 
 Pulls problems away from business logic into a higher-level
 

67. Benefits of an API gateway Operations through the API look a lot like their web UI counterparts Allows us to orchestrate APIs instead of data models
 
 Pulls problems away from business logic into a higher-level
 
 Internal and external consumers go through the same application
 

68. api.optimizely.com What we’ve got
69. Python 3 + Pyramid + pyramid_swagger
70. We need to do a bunch of things in parallel...
71. Pyramid and Multiprocessing with multiprocessing.Pool() as pool, multiprocessing.Manager() as manager:
 response_queue = manager.Queue()
 try:
 # send our requests get back responses from the response_queue
 finally:
 pool.join()
 pool.terminate()
72. Pyramid and Multiprocessing with multiprocessing.Pool() as pool, multiprocessing.Manager() as manager:
 response_queue = manager.Queue()
 try:
 # send our requests get back responses from the response_queue
 finally:
 pool.join()
 pool.terminate()
73. Pyramid and Multiprocessing with multiprocessing.Pool() as pool, multiprocessing.Manager() as manager:
 response_queue = manager.Queue()
 try:
 # send our requests get back responses from the response_queue
 finally:
 pool.join()
 pool.terminate()
74. Pyramid and Multiprocessing with multiprocessing.Pool() as pool, multiprocessing.Manager() as manager:
 response_queue = manager.Queue()
 try:
 # send our requests get back responses from the response_queue
 finally:
 pool.join()
 pool.terminate()
75. An Unhealthy Situation AWS Elastic Load Balancer 5XX
76. An Unhealthy Situation Textbook case for non blocking IO AWS Elastic Load Balancer 5XX
77. Not the Experience We Want
78. Blocking IO: “Normal” Python Application Kernel Start
79. Blocking IO: “Normal” Python Application Kernel Start read()
80. Blocking IO: “Normal” Python Application Kernel Start read() Initiate read
81. Blocking IO: “Normal” Python Application Kernel Start read() Initiate read Receive response
82. Blocking IO: “Normal” Python Application Kernel Start read() Initiate read read() returns Receive response
83. Blocking IO: “Normal” Python Application Kernel Start read() Initiate read read() returns Receive response
84. Nonblocking IO Application Kernel Start
85. Nonblocking IO Application Kernel Start read()
86. Nonblocking IO Application Kernel Start read() Initiate read, returns immediately
87. Nonblocking IO Application Kernel Start read() Initiate read, returns immediately poll()until data has arrived
88. Nonblocking IO Application Kernel Start read() Initiate read, returns immediately Receive response poll()until data has arrived
89. Nonblocking IO Application Kernel Start read() Initiate read, returns immediately read() Receive response poll()until data has arrived
90. Nonblocking IO Application Kernel Start read() Initiate read, returns immediately read() Receive response poll()until data has arrived
91. Nonblocking IO Application Kernel poll(5)
92. Nonblocking IO Application Kernel poll(5) poll(5)
93. Nonblocking IO Application Kernel poll(5) poll(5) poll(5)
94. Nonblocking IO Application Kernel Receive response poll(5) poll(5) poll(5)
95. Nonblocking IO Application Kernel read() Receive response poll(5) poll(5) poll(5)
96. Nonblocking IO Application Kernel read() Receive response poll(5) poll(5) poll(5)
97. Non Blocking IO Tw isted N odeJS Tornado 2002 N etty 2004 2009 2006 EventM achine 2008 Eventlet 2015 Asyncio 2012 G o/G orutines
98. What’s it good for? Proxy
99. What’s it good for? Proxy
100. What’s it good for? Push Updates/Chat
101. What’s it good for? Push Updates/Chat
102. How do you schedule this work?
103. What framework?
104. Enter Asyncio Python 3.4
105. async def i_take_a_while(): print('slow func start') await asyncio.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_take_a_while() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_take_a_while i_run_fast
106. async def i_take_a_while(): print('slow func start') await asyncio.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_take_a_while() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_take_a_while i_run_fast
107. async def i_take_a_while(): print('slow func start') await asyncio.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_take_a_while() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_take_a_while i_run_fast fast func start
108. async def i_take_a_while(): print('slow func start') await asyncio.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_take_a_while() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_take_a_while i_run_fast fast func start
109. async def i_take_a_while(): print('slow func start') await asyncio.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_take_a_while() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_take_a_while i_run_fast slow func start fast func start
110. async def i_take_a_while(): print('slow func start') await asyncio.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_take_a_while() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_take_a_while i_run_fast slow func start fast func start
111. async def i_take_a_while(): print('slow func start') await asyncio.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_take_a_while() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_take_a_while i_run_fast slow func start fast func start
112. async def i_take_a_while(): print('slow func start') await asyncio.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_take_a_while() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_take_a_while i_run_fast slow func start fast func start 1 sec after start fast func done
113. async def i_take_a_while(): print('slow func start') await asyncio.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_take_a_while() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_take_a_while i_run_fast slow func start fast func start 1 sec after start fast func done
114. async def i_take_a_while(): print('slow func start') await asyncio.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_take_a_while() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_take_a_while i_run_fast slow func start fast func start 1 sec after start fast func done
115. async def i_take_a_while(): print('slow func start') await asyncio.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_take_a_while() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_take_a_while i_run_fast slow func start fast func start 1 sec after start fast func done 3 sec after start slow func done
116. Blocking -> Asyncio Gunicorn Pyramid Pyramid View
117. Blocking -> Asyncio Gunicorn Pyramid Pyramid View Switch to asyncio worker class http:// docs.gunicorn.org/en/stable/design.html#asyncio- workers
118. Blocking -> Asyncio Gunicorn Pyramid Pyramid View Switch to aiopyramid https://github.com/housleyjk/aiopyramid
119. Blocking -> Asyncio Gunicorn Pyramid Pyramid View We can rewrite the views incrementally? Right?
120. async def i_block(): print('slow func start') time.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_block() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_block i_run_fast
121. async def i_block(): print('slow func start') time.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_block() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_block i_run_fast fast func start
122. async def i_block(): print('slow func start') time.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_block() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_block i_run_fast fast func start
123. async def i_block(): print('slow func start') time.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_block() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_block i_run_fast block func start fast func start
124. async def i_block(): print('slow func start') time.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_block() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_block i_run_fast block func start fast func start
125. async def i_block(): print('slow func start') time.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_block() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_block i_run_fast block func start fast func start 3 sec later block func done
126. async def i_block(): print('slow func start') time.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_block() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_block i_run_fast block func start fast func start 3 sec later block func done
127. async def i_block(): print('slow func start') time.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_block() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_block i_run_fast block func start fast func start 3 sec later block func done
128. async def i_block(): print('slow func start') time.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_block() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_block i_run_fast block func start fast func start 3 sec later block func done
129. async def i_block(): print('slow func start') time.sleep(3) print('slow func done') async def i_run_fast(): print('fast func start') await asyncio.sleep(1) print('fast func done') async def run_me(): await asyncio.gather( i_run_fast(), i_block() ) loop = asyncio.get_event_loop()
 loop.run_until_complete(run_me())
 loop.close() Event loop i_block i_run_fast block func start fast func start 3 sec later block func done fast func done
130. Blocking and Nonblocking IO Don’t Mix
131. Blocking and Nonblocking IO Don’t Mix
132. asyncio all the things
133. asyncio all the things
134. How Do You Make Big Changes?
135. How Do You Make Big Changes?
136. How Do You Make Big Changes?
137. Code is Social
138. Code is Social • Better make sure it actually works!
139. Code is Social • Better make sure it actually works! • Pull Request Open 3+ weeks
140. Code is Social • Better make sure it actually works! • Pull Request Open 3+ weeks • Everyone on team added as a reviewer
141. Code is Social • Better make sure it actually works! • Pull Request Open 3+ weeks • Everyone on team added as a reviewer • 63 comments on Pull Request
142. Code is Social • Better make sure it actually works! • Pull Request Open 3+ weeks • Everyone on team added as a reviewer • 63 comments on Pull Request • Meetings for questions/concerns
143. Happily ever after AWS Elastic Load Balancer 5XX
144. Happily ever after AWS Elastic Load Balancer 5XX
145. Happily ever after 99th percentile latency
146. Happily ever after 99th percentile latency
147. asyncio
148. • asyncio from the start is ideal asyncio
149. • asyncio from the start is ideal • But can add it later if you have to asyncio
150. • asyncio from the start is ideal • But can add it later if you have to • Still early, but support will only improve asyncio
151. Good APIs are Hard

152. Good APIs are Hard
 • Lots of details to get right
153. Good APIs are Hard
 • Lots of details to get right • Using the right technologies helps
154. Good APIs are Hard
 • Lots of details to get right • Using the right technologies helps • Always keep iterating and improving
155. Come visit us at our booth! @Optimizely Try out Optimizely: optimizely.com/get-started
156. Q&A