Codex: Scaling Code Evaluation

A brief overview of how Codecademy has scaled their code evaluation system to handle up to 5,000 submissions a second for a variety of languages, frameworks, and APIs.

Transcript

1. CODEX MIT - SEPTEMBER 2013 How we scale code evaluation Timor Tsentsiper, Engineer timor@codecademy.com
2. WE’RE HIRING: http://codecademy.com/jobs Codecademy • Interactive programming education, learn by doing. • Community of millions around the globe. • Try it out at: http://codecademy. com
3. Who am I? • Backend Engineer ◦ APIs, Services ◦ Infrastructure ◦ DevOps • Mobile (native and web) • Some pure front end experience as well (ew) WE’RE HIRING: http://codecademy.com/jobs
4. THE PROBLEM
5. Problem statement • Millions of users • ~25 million code submissions a day • 4 different languages: Javascript, Ruby, Python, PHP + (insert future here) ? • API lessons • Framework lessons WE’RE HIRING: http://codecademy.com/jobs
6. CONSTRAINTS
7. Constraints • Has to be performant ( < 100ms RS) • Secure • Scale to millions of users • Accessed globally • Easy to deploy • Easy to extend • Lightweight JS client library WE’RE HIRING: http://codecademy.com/jobs
8. OUR SOLUTION
9. Solution: Codex Codex components: • Front end Javascript client • Back end service: ▪ NodeJS adapter • Runners • Guiding principle: run code passed over a websocket WE’RE HIRING: http://codecademy.com/jobs
10. Solution: Javascript Client • Simple (250 lines of code) • Handles web sockets (sockm/sockjs) • Negotiates the codex protocol (simple JSON) • All messages are events on the connection object, with an optional payload: client requests a spawn event [{"type": "spawn", "data": "ruby"}] WE’RE HIRING: http://codecademy.com/jobs
11. Solution: Javascript Client • Reconnects (in case codex disconnects or the runners crash) • Websocket state • Timeouts • Authentication • Logging of events and errors WE’RE HIRING: http://codecademy.com/jobs
12. Solution: Javascript Client WE’RE HIRING: http://codecademy.com/jobs Client codex-n 1. Establish 2. Authenticate 3. Spawn Ruby OK Auth OK Ready 4. Run Ruby Result
13. Solution: Backend service • Behind a load balancer ◦ AWS hosted ◦ HAProxy (round robin software lb) ◦ HAProxy handles websockets out of the box • N-number codex-hosts (this changes as the service scales up or down) WE’RE HIRING: http://codecademy.com/jobs
14. Solution: Backend service WE’RE HIRING: http://codecademy.com/jobs cdx-1 cdx-n HAProxy cdx-2 Client socket communication
15. Solution: Backend service • Each host: ◦ Standard ubuntu install ◦ LXC ▪ NodeJS Adapter • Runners ◦ Iptables (for security, port forwarding etc) ◦ Apache proxy for outbound connections for various API providers ◦ Gems/python libs/c libs etc WE’RE HIRING: http://codecademy.com/jobs
16. Solution: Backend service WE’RE HIRING: http://codecademy.com/jobs Codex host LXC IP Tables Internet IP Tables Apache Proxy NodeJs Runner Runner Runner
17. Solution: The LXC WE’RE HIRING: http://codecademy.com/jobs • Adapter isolation • OS level virtualization, ve with its own process and network space (uses linux kernel groups, think Heroku Dyno rather than VMware) • User separation (unix file system permissions, home directories) • Port restrictions • Resource limiting (rlimit and nice, no core dumps, limit child procs, etc)
18. Solution: Adapter WE’RE HIRING: http://codecademy.com/jobs • Connection management (sockets) • Runner management ◦ Process management ▪ Kill/respwan ◦ Lifetime of the runner ◦ Infinite loop detection • File management • Authentication • Error handling (response back to the user/browser)
19. Solution: Runner WE’RE HIRING: http://codecademy.com/jobs • Use eval, exec etc (language dependent) to run passed in code • Do sanity checks/clean up • Ability to run correctness tests, think of these like unit tests for user submissions • Adapter communicates with the runners over STDIO • Runners read from STDIN, respond on STDOUT and/or STDERR
20. Solution: Adapter ⇔ Runner WE’RE HIRING: http://codecademy.com/jobs • Example: ◦ Adapter: ◦ Runner: ◦ Runner: ◦ Runner: RUN\r\nprint "hello world" \n\r\n OUTPUT hello world\r OUTPUT \n\r RESULT None\r
21. Thank You! Timor Tsentsiper timor@codecademy.com