How edX uses Ansible

NYC Ansiblefest 5/20/14

Transcript

1. How edX uses Ansible http://github.com/edx/configuration open-source configuration management for an open-source learning platform
2. What is edX? • A website for learning • A website for creating courses • A platform for research and analytics • A community of learners and developers creating courses and extending the edX open source platform
3. Taking a course as a student • Interactives • Inline discussions • Peer assessment • Code grading
4. Creating a course as an instructor • GUI for creating different problems and publishing courses • XML export and import • Instructor view for grade reports and viewing student progress
5. Interacting with students and instructors • General purpose forums for every course • Inline discussions on each problem
6. Try out the platform! • Point your browser to https://www.sandbox. edx.org • Login as staff@example.com / edx • This public sandbox runs on a single EC2 server that is recreated weekly • One of the many examples of Ansible automation @ edX
7. edX and Ansible • ~ 50 custom Ansible roles • ~ 700 Ansible tasks • ~ 150 forks of our configuration repo • edX Ansible plays are run every day at edX and around the globe to provision and deploy the platform
8. Why Ansible? • edX started out with puppet and fabric ◦ Even the most simple tasks felt like we were constantly fighting the tool ◦ We wanted one tool for code updates and base OS configuration While we were having issues with Puppet Stanford was adopting the edX platform and having similar issues with Chef. edX started using Ansible (1.3) in September, 2013
9. Roles, plays and vars - edX conventions • Ansible plays are short, simple and often limited to calling a single role • Ansible roles are self-contained ◦ Role name prefix for variable namespace ◦ Defaults defined in the role, overridden using extra vars.
10. Updating code and configuration • Tasks that update application code are put in a single deploy.yml task file • The deploy task file is included with tags, - include deploy.yml tags: deploy
11. Ansible variables • Role variables are defined in defaults/main. yml • Vars that are overridden are typed in ALL CAPS • Internal role variables are lowercase • Defaults assume a single server localhost installation
12. edX operational requirements • Universities that run edX want control over the tools and the data • Isolate deployments, student privacy is extremely important • Operates in the cloud but strives to be cloud agnostic • Official releases of our configuration scripts with Vagrant images and public AMIs
13. edX uses three different types of installations • Devstack - Using Vagrant, code directories are exported from the host filesystem to the guest • Fullstack - Every service in production installed on a single server • A small number of multi-cluster redundant configurations (stage, production, loadtest)
14. Devstack - optimized for development • curl the Vagrantfile • `vagrant up` • Run the dev server • See changes instantly on the guest VM • Develop locally on the host using an IDE
15. Fullstack - All edX services • Single server configured the same as production • One-click single instance provisioning for launching sandboxes • Ansible handles creation, termination and config • Official vagrant images and public AMIs
16. Nginx with basic auth OS Services • MySQL server • Mongo server • RabbitMQ • ElasticSearch • Memcache edX services (supervisor) • student courseware (gunicorn/django) • course authoring (gunicorn/django) • workers (django) • forum (sinatra) • xserver (gunicorn/wsgi) • certificates (python) Single self contained edX installation on a single Ubuntu server - edxapp - forums - xserver - certs sandbox.yml
17. Installing edX in production • git checkouts instead of packages • python virtual environments for every service • pre-built images for AWS auto-scaling • AWS virtual private clouds used for multiple edX installations • VPC layout described using cloudformation • Utilizing AWS resources where possible (RDS, ELB, S3, Elasticcache, SES)
18. • Every cluster has a corresponding Ansible play for configuration • A single ops VPC (lower left) connects to the other VPCs for running config updates and creating deploy images
19. Configuration repos for multiple installations • A single generic open source repo contains all of the edX roles • A private repo contains roles internal to edX • A private repo containing one yaml file per environment for variable overrides • For code version updates, vars are set on the command-line
20. How edX uses Ansible for server administration • Jenkins server that connects to other VPCs and runs ansible plays. • Inventory is fed directly into Jenkins via an inventory script that gathers tag information • All Ansible tasks are run from Jenkins for auditing and repeatability
21. Ansible Inventory in a Jenkins dropdown Feed the ouput of ec2.py into Jenkins with the Dynamic Choice Paramater plugin and a simple groovy script.
22. Selecting inventory in Jenkins • Using a special first_in prefix to ec2.py group names so tasks can be run on one server in a tag group • Use serial with pre and post tasks for rolling updates group names returned by ec2.py
23. Running Ansible from Jenkins Plays are selected from a drop-down and run on any cluster or individual server returned by the inventory script
24. Image based deployments • Using images and autoscaling makes it easier to scale • Ansible was the perfect tool to make this work with very little additional effort • Netflix’s Asgard has proven extremely useful for managing autoscaling groups and, cut overs and rollbacks.
25. Creating an AMI with Jenkins and Ansible • A python script launches an instance and runs ansible on bringup in user-data • Progress is sent back to the user via sqs using a custom Ansible callback plugin
26. The callback plugin allows for our own custom Ansible status output After every Ansible run the longest run tasks are displayed with a time summary
27. Releasing with Asgard Servers are built with the AMIs using Netflix’s Asgard. Traffic is sent to the new servers, monitored and the old cluster is removed.
28. What’s next for edX and Ansible? • Initiate deployments from hipchat • Moving cloudformation provisioning to Ansible using VPC modules • Productizing provisioning in other cloud environments • Automating canary workflows
29. THANKS!! • Questions? • For more information ◦ jarv on freenode (#edx-code and #ansible) ◦ The edx-ops email list ◦ edX configuration repo ◦ Stand up your own edX website!