Demystifying The Open edX Architecture

Transcript

1. DEMYSTIFYING THE OPENEDX ARCHITECTURE BITSAVANT © 2016 Bitsavant - All

   Rights Reserved

2. WHO ARE WE A BIT ABOUT US Principal Security Engineer

   Based in the Bay Area Hardware & Software Deployment Manager Boston, MA © 2016 Bitsavant - All Rights Reserved

3. WHY ARE WE PRESENTING HOW WE GOT INTO THIS ▸

   Circa 2015, a business opportunity had us scouring the market for an LMS that would underpin our commercial offering. ▸ Started seriously looking into Open edX and we liked what we saw. ▸ Deploying and running a production-grade platform however was a different matter and had us dig deeper. © 2016 Bitsavant - All Rights Reserved

4. TODAY’S TALK WHAT WE WANT TO COVER ▸ Our journey

   on the way to understanding this awesome platform. ▸ Unfortunate, and sometimes frequent, encounters with sinking ships. ▸ Issues we faced along the way with various deployment topologies. ▸ Reversing for fun and profit. ▸ Deep-dives of select components. © 2016 Bitsavant - All Rights Reserved

5. THE DIFFERENT FACES OF OPEN EDX STARTING WITH THE END-USER

6. THE DIFFERENT FACES OF OPEN EDX MOVING ON TO ARCHITECTS

7. THE DIFFERENT FACES OF OPEN EDX AND FINALLY IMPLEMENTERS

8. THE REALITY AWS ANYONE?

9. THE REALITY BACK IN THE REAL WORLD….

10. THE REALITY DOES THIS LOOK FAMILIAR

11. THAT FEELING…

12. THE RESULT MOST OF US ENDED UP HERE wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/

    install/sandbox.sh -O - | bash © 2016 Bitsavant - All Rights Reserved

13. DEPLOYMENT MODELS THE ALL-IN-ONE WONDER ▸ Mostly works out of

    the box with a clean deploy run. ▸ After some minor work, we’re betting most people get this to work. ▸ Troubleshooting problems or customization isn’t trivial and requires efforts digging around. ▸ Component interactions are on the complex side given the self-contained nature of this use-case. © 2016 Bitsavant - All Rights Reserved

14. MOVING ON THIS WAS FINE, UNTIL…. ▸ We always knew

    that the fullstack instance was not for production purposes (why we avoided it in the first place). ▸ To revisit the distributed deployment, we wanted to first be very intimate with the foundations of the platform. © 2016 Bitsavant - All Rights Reserved

15. DIGGING FOR MORE SOME USEFUL RESOURCES ▸ Great starting point,

    but someone deploying this will have a hard time mapping component layout to the architecture, hardly the Rosetta stone this clueless duo needed.

16. WHEN IN DOUBT, REVERSE. Us, whenever we’re confused. HOW WE

    DID IT © 2016 Bitsavant - All Rights Reserved

17. HOW WE DID IT REVERSING FOR FUN AND EDUCATION ▸

    We took a multi-pronged approach. ▸ Break down the single-box installation and figure out the minimum component baseline needed to setup a working cluster. ▸ In parallel, grok the distributed production topology prior to customizing it to our liking. © 2016 Bitsavant - All Rights Reserved

18. HOW WE DID IT TOOLS OF THE TRADE ▸ Code

    review. ▸ Plain old bug hunting by scrutinizing logs. ▸ For figuring out application calls we leveraged a number of tools including Sysdig, Slack (for support), and old- style system sleuthing. ▸ Needless to say, system administration fundamentals come in handy here to figure out what the system is trying to do, and why it’s able to do it (or not). © 2016 Bitsavant - All Rights Reserved

19. INITIAL RESULTS SOME PROGRESS MADE © 2016 Bitsavant - All

    Rights Reserved

20. ANSIBLE DEPENDENCIES COMMON SUPERVISOR ORACLEJDK EDX_SERVICE EDXAPP_ COMMON HADOOP_ COMMON

    NLTK © 2016 Bitsavant - All Rights Reserved

21. SOMETHING STILL AMISS

22. STAYING LOCAL LESS IS MORE ▸ Realized that we could

    simplify the core set of components even further. ▸ Further experimentation with distributed deployments using local resources. ▸ Ansible playbook clean-ups inline with our simplification drive. ▸ We ended up abstracting the architecture down to components we thought most essential to our understanding of the platform. © 2016 Bitsavant - All Rights Reserved

23. SIMPLER App © 2016 Bitsavant - All Rights Reserved

24. COMPONENT DEEP-DIVE FOCUS: NGINX ▸ Entry point into the public-facing

    portion of the platform. ▸ Provides proxied and load-balanced access to static and dynamic assets. ▸ Stateless and horizontally scalable. Process Username Workers Proto/Port Function nginx www-data 4 TCP *:80 LMS nginx www-data 4 TCP *:18010 CMS nginx www-data 4 TCP *:18040 xqueue nginx www-data 4 TCP *:18080 forum nginx www-data 4 TCP *:18090 certs ▸ /edx/var/log/nginx/{access,error}.log ▸ service nginx {status, reload, restart, stop} TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved

25. COMPONENT DEEP-DIVE FOCUS: NGINX App LMS TCP:80 CMS TCP:18010 xqueue

    TCP:18040 forum TCP:18080 certs TCP:18090 LMS TCP:8000 CMS TCP:8010 xqueue TCP:8040 UNIX Domain Socket Call Static Assets Student Login Instructor Login © 2016 Bitsavant - All Rights Reserved

26. COMPONENT DEEP-DIVE FOCUS: GUNICORN ▸ Python WSGI HTTP server that

    NGiNX routes most calls through to the application servers (Django). ▸ Dynamic worker management. ▸ Stateless and horizontally scalable. Process Username Workers Proto/Port Function gunicorn www-data 8 TCP *:8000 LMS gunicorn www-data 4 TCP *:8010 CMS gunicorn www-data 4 TCP *:8040 xqueue ▸ /edx/var/log/{cms, lms, xqueue}/edx.log ▸ /edx/var/log/supervisor/{cms, lms, xqueue}-stderr.log TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved

27. COMPONENT DEEP-DIVE FOCUS: EDX APP ▸ Bundle of core application

    components. ▸ Mostly implemented as Django apps. ▸ Includes a Ruby component serving particular workflows. Config Files Function cms.auth.json API Keys, Usernames, Passwords … cms.env.json Component URL’s and various CMS Configs cms_gunicorn.py Gunicorn Configuration File lms.auth.json API Keys, Usernames, Passwords … lms.env.json Component URL’s and various LMS Configs lms_gunicorn.py Gunicorn Configuration File ▸ /edx/var/log/{cms, lms, xqueue}/edx.log ▸ /edx/var/log/supervisor/{cms, lms, xqueue}-stderr.log TROUBLESHOOTING App elasticsearch rabbitmq mongodb memcache mysql nginx www-data certs xqueue forum edx_notes_api insights analytics_api notifier sandbox edxapp supervisor ACCOUNTS © 2016 Bitsavant - All Rights Reserved

28. EDXAPP PACKAGES base_requirements_file - %51.56 github_requirements_file - %15.23 edxapp_common_debian_pkgs -

    %5.47 edxapp_debian_pkgs - %4.3 server_utils_debian_pkgs - %3.52 paver_requirements_file - %2.73 sandbox_base_requirements - %2.34 common_debian_pkgs - %2.34 local_requirements_file - %1.56 sandbox_local_requirements - %1.56 pre_requirements_file - %1.17 common_pip_pkgs - %1.17 security_debian_pkgs - %1.17 aws_pip_pkgs - %1.17 © 2016 Bitsavant - All Rights Reserved

29. COMPONENT DEEP-DIVE FOCUS: RABBITMQ ▸ Underpins queueing infrastructure. ▸ Large

    jobs are run through the queue, coordinated by Celery. ▸ Should be well-resourced and available so it doesn’t turn into a hidden SPoF. Process Username Proto/Port Function beam.smp rabbitmq localhost *:5672 Message Queue beam.smp rabbitmq TCP *:15672 Message Queue beam.smp rabbitmq TCP *:38246 Message Queue beam.smp rabbitmq TCP *:55672 Message Queue epmd rabbitmq TCP localhost:4369 Port Mapper ▸ /var/log/rabbitmq/* TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved

30. COMPONENT DEEP-DIVE FOCUS: MEMCACHED ▸ High-speed distributed KV store for

    object- caching. ▸ Caches user sessions, course structure info, e.t.c. Process Username Proto/Port Function memcached memcache TCP localhost:11211 KV Store ▸ /var/log/memcached.log TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved

31. COMPONENT DEEP-DIVE FOCUS: MONGODB ▸ NoSQL DB backend. ▸ Used

    mainly for storing course and forum data. ▸ Relatively easy to scale. Process Username Proto/Port Function mongod mongodb TCP *:27017 DB ▸ /edx/var/log/mongo/mongodb.log TROUBLESHOOTING App Collections 7 DBs 4 DB/COLLECTION COUNTS © 2016 Bitsavant - All Rights Reserved

32. COMPONENT DEEP-DIVE FOCUS: MYSQL ▸ Main DB for user data

    such user profiles, enrollment status, e.t.c. ▸ Holds authentication/authorization data as well. ▸ Also holds analytics results. ▸ Too many edxapp tables to get into (250+). Process Username Proto/Port Function mysqld mysql TCP localhost:3306 DB ▸ /var/log/mysql/error.log TROUBLESHOOTING App reports
 0 xqueue
 11 edxapp
 278 edx_
 notes_api
 10 ecommerce
 0 dashboard
 23 analytics-api
 12 DB/TABLE COUNTS © 2016 Bitsavant - All Rights Reserved

33. CONCLUSION LAST THOUGHTS ▸ This talk is just the start!

    ▸ We plan a similar application-centric exercise to cover core business logic. ▸ Keep an eye out for our repo (github.com/bitsavant) for upcoming cheatsheets, Ansible playbooks, and other stuff. ▸ Our contact details are at www.bitsavant.com. We’d love to hear from you! © 2016 Bitsavant - All Rights Reserved