DEMYSTIFYING THE OPENEDX ARCHITECTURE BITSAVANT © 2016 Bitsavant - All Rights Reserved 

WHO ARE WE A BIT ABOUT US Principal Security Engineer Based in the Bay Area Hardware & Software Deployment Manager Boston, MA © 2016 Bitsavant - All Rights Reserved 

WHY ARE WE PRESENTING HOW WE GOT INTO THIS ▸ Circa 2015, a business opportunity had us scouring the market for an LMS that would underpin our commercial offering. ▸ Started seriously looking into Open edX and we liked what we saw. ▸ Deploying and running a production-grade platform however was a different matter and had us dig deeper. © 2016 Bitsavant - All Rights Reserved 

TODAY’S TALK WHAT WE WANT TO COVER ▸ Our journey on the way to understanding this awesome platform. ▸ Unfortunate, and sometimes frequent, encounters with sinking ships. ▸ Issues we faced along the way with various deployment topologies. ▸ Reversing for fun and profit. ▸ Deep-dives of select components. © 2016 Bitsavant - All Rights Reserved 

THE DIFFERENT FACES OF OPEN EDX STARTING WITH THE END-USER 

THE DIFFERENT FACES OF OPEN EDX MOVING ON TO ARCHITECTS 

THE DIFFERENT FACES OF OPEN EDX AND FINALLY IMPLEMENTERS 

THE REALITY AWS ANYONE? 

THE REALITY BACK IN THE REAL WORLD…. 

THE REALITY DOES THIS LOOK FAMILIAR 

THAT FEELING… 

THE RESULT MOST OF US ENDED UP HERE wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/ install/sandbox.sh -O - | bash © 2016 Bitsavant - All Rights Reserved 

DEPLOYMENT MODELS THE ALL-IN-ONE WONDER ▸ Mostly works out of the box with a clean deploy run. ▸ After some minor work, we’re betting most people get this to work. ▸ Troubleshooting problems or customization isn’t trivial and requires efforts digging around. ▸ Component interactions are on the complex side given the self-contained nature of this use-case. © 2016 Bitsavant - All Rights Reserved 

MOVING ON THIS WAS FINE, UNTIL…. ▸ We always knew that the fullstack instance was not for production purposes (why we avoided it in the first place). ▸ To revisit the distributed deployment, we wanted to first be very intimate with the foundations of the platform. © 2016 Bitsavant - All Rights Reserved 

DIGGING FOR MORE SOME USEFUL RESOURCES ▸ Great starting point, but someone deploying this will have a hard time mapping component layout to the architecture, hardly the Rosetta stone this clueless duo needed. 

WHEN IN DOUBT, REVERSE. Us, whenever we’re confused. HOW WE DID IT © 2016 Bitsavant - All Rights Reserved 

HOW WE DID IT REVERSING FOR FUN AND EDUCATION ▸ We took a multi-pronged approach. ▸ Break down the single-box installation and figure out the minimum component baseline needed to setup a working cluster. ▸ In parallel, grok the distributed production topology prior to customizing it to our liking. © 2016 Bitsavant - All Rights Reserved 

HOW WE DID IT TOOLS OF THE TRADE ▸ Code review. ▸ Plain old bug hunting by scrutinizing logs. ▸ For figuring out application calls we leveraged a number of tools including Sysdig, Slack (for support), and old- style system sleuthing. ▸ Needless to say, system administration fundamentals come in handy here to figure out what the system is trying to do, and why it’s able to do it (or not). © 2016 Bitsavant - All Rights Reserved 

INITIAL RESULTS SOME PROGRESS MADE © 2016 Bitsavant - All Rights Reserved 

ANSIBLE DEPENDENCIES COMMON SUPERVISOR ORACLEJDK EDX_SERVICE EDXAPP_ COMMON HADOOP_ COMMON NLTK © 2016 Bitsavant - All Rights Reserved 

SOMETHING STILL AMISS 

STAYING LOCAL LESS IS MORE ▸ Realized that we could simplify the core set of components even further. ▸ Further experimentation with distributed deployments using local resources. ▸ Ansible playbook clean-ups inline with our simplification drive. ▸ We ended up abstracting the architecture down to components we thought most essential to our understanding of the platform. © 2016 Bitsavant - All Rights Reserved 

SIMPLER App © 2016 Bitsavant - All Rights Reserved 

COMPONENT DEEP-DIVE FOCUS: NGINX ▸ Entry point into the public-facing portion of the platform. ▸ Provides proxied and load-balanced access to static and dynamic assets. ▸ Stateless and horizontally scalable. Process Username Workers Proto/Port Function nginx www-data 4 TCP *:80 LMS nginx www-data 4 TCP *:18010 CMS nginx www-data 4 TCP *:18040 xqueue nginx www-data 4 TCP *:18080 forum nginx www-data 4 TCP *:18090 certs ▸ /edx/var/log/nginx/{access,error}.log ▸ service nginx {status, reload, restart, stop} TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved 

COMPONENT DEEP-DIVE FOCUS: NGINX App LMS TCP:80 CMS TCP:18010 xqueue TCP:18040 forum TCP:18080 certs TCP:18090 LMS TCP:8000 CMS TCP:8010 xqueue TCP:8040 UNIX Domain Socket Call Static Assets Student Login Instructor Login © 2016 Bitsavant - All Rights Reserved 

COMPONENT DEEP-DIVE FOCUS: GUNICORN ▸ Python WSGI HTTP server that NGiNX routes most calls through to the application servers (Django). ▸ Dynamic worker management. ▸ Stateless and horizontally scalable. Process Username Workers Proto/Port Function gunicorn www-data 8 TCP *:8000 LMS gunicorn www-data 4 TCP *:8010 CMS gunicorn www-data 4 TCP *:8040 xqueue ▸ /edx/var/log/{cms, lms, xqueue}/edx.log ▸ /edx/var/log/supervisor/{cms, lms, xqueue}-stderr.log TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved 

COMPONENT DEEP-DIVE FOCUS: EDX APP ▸ Bundle of core application components. ▸ Mostly implemented as Django apps. ▸ Includes a Ruby component serving particular workflows. Config Files Function cms.auth.json API Keys, Usernames, Passwords … cms.env.json Component URL’s and various CMS Configs cms_gunicorn.py Gunicorn Configuration File lms.auth.json API Keys, Usernames, Passwords … lms.env.json Component URL’s and various LMS Configs lms_gunicorn.py Gunicorn Configuration File ▸ /edx/var/log/{cms, lms, xqueue}/edx.log ▸ /edx/var/log/supervisor/{cms, lms, xqueue}-stderr.log TROUBLESHOOTING App elasticsearch rabbitmq mongodb memcache mysql nginx www-data certs xqueue forum edx_notes_api insights analytics_api notifier sandbox edxapp supervisor ACCOUNTS © 2016 Bitsavant - All Rights Reserved 

EDXAPP PACKAGES base_requirements_file - %51.56 github_requirements_file - %15.23 edxapp_common_debian_pkgs - %5.47 edxapp_debian_pkgs - %4.3 server_utils_debian_pkgs - %3.52 paver_requirements_file - %2.73 sandbox_base_requirements - %2.34 common_debian_pkgs - %2.34 local_requirements_file - %1.56 sandbox_local_requirements - %1.56 pre_requirements_file - %1.17 common_pip_pkgs - %1.17 security_debian_pkgs - %1.17 aws_pip_pkgs - %1.17 © 2016 Bitsavant - All Rights Reserved 

COMPONENT DEEP-DIVE FOCUS: RABBITMQ ▸ Underpins queueing infrastructure. ▸ Large jobs are run through the queue, coordinated by Celery. ▸ Should be well-resourced and available so it doesn’t turn into a hidden SPoF. Process Username Proto/Port Function beam.smp rabbitmq localhost *:5672 Message Queue beam.smp rabbitmq TCP *:15672 Message Queue beam.smp rabbitmq TCP *:38246 Message Queue beam.smp rabbitmq TCP *:55672 Message Queue epmd rabbitmq TCP localhost:4369 Port Mapper ▸ /var/log/rabbitmq/* TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved 

COMPONENT DEEP-DIVE FOCUS: MEMCACHED ▸ High-speed distributed KV store for object- caching. ▸ Caches user sessions, course structure info, e.t.c. Process Username Proto/Port Function memcached memcache TCP localhost:11211 KV Store ▸ /var/log/memcached.log TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved 

COMPONENT DEEP-DIVE FOCUS: MONGODB ▸ NoSQL DB backend. ▸ Used mainly for storing course and forum data. ▸ Relatively easy to scale. Process Username Proto/Port Function mongod mongodb TCP *:27017 DB ▸ /edx/var/log/mongo/mongodb.log TROUBLESHOOTING App Collections 7 DBs 4 DB/COLLECTION COUNTS © 2016 Bitsavant - All Rights Reserved 

COMPONENT DEEP-DIVE FOCUS: MYSQL ▸ Main DB for user data such user profiles, enrollment status, e.t.c. ▸ Holds authentication/authorization data as well. ▸ Also holds analytics results. ▸ Too many edxapp tables to get into (250+). Process Username Proto/Port Function mysqld mysql TCP localhost:3306 DB ▸ /var/log/mysql/error.log TROUBLESHOOTING App reports
 0 xqueue
 11 edxapp
 278 edx_
 notes_api
 10 ecommerce
 0 dashboard
 23 analytics-api
 12 DB/TABLE COUNTS © 2016 Bitsavant - All Rights Reserved 

CONCLUSION LAST THOUGHTS ▸ This talk is just the start! ▸ We plan a similar application-centric exercise to cover core business logic. ▸ Keep an eye out for our repo (github.com/bitsavant) for upcoming cheatsheets, Ansible playbooks, and other stuff. ▸ Our contact details are at www.bitsavant.com. We’d love to hear from you! © 2016 Bitsavant - All Rights Reserved