Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Demystifying The Open edX Architecture
Search
bitsavant
June 14, 2016
Technology
3
1.6k
Demystifying The Open edX Architecture
Slides for Bitsavant's talk at Open edX 2016, held at Stanford, CA.
bitsavant
June 14, 2016
Tweet
Share
Other Decks in Technology
See All in Technology
FOSS4G 2025 KANSAI QGISで点群データをいろいろしてみた
kou_kita
0
170
SpringBoot x TestContainerで実現するポータブル自動結合テスト
demaecan
0
120
Fabric + Databricks 2025.6 の最新情報ピックアップ
ryomaru0825
1
160
【5分でわかる】セーフィー エンジニア向け会社紹介
safie_recruit
0
26k
プロダクトエンジニアリング組織への歩み、その現在地 / Our journey to becoming a product engineering organization
hiro_torii
0
140
Amazon Bedrockで実現する 新たな学習体験
kzkmaeda
2
680
モバイル界のMCPを考える
naoto33
0
350
2025-06-26 GitHub CopilotとAI駆動開発:実践と導入のリアル
fl_kawachi
1
240
生成AI活用の組織格差を解消する 〜ビジネス職のCursor導入が開発効率に与えた好循環〜 / Closing the Organizational Gap in AI Adoption
upamune
5
4.5k
AI専用のリンターを作る #yumemi_patch
bengo4com
3
1.8k
mrubyと micro-ROSが繋ぐロボットの世界
kishima
2
380
登壇ネタの見つけ方 / How to find talk topics
pinkumohikan
5
580
Featured
See All Featured
Code Review Best Practice
trishagee
69
18k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
GitHub's CSS Performance
jonrohan
1031
460k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.4k
Facilitating Awesome Meetings
lara
54
6.4k
Build The Right Thing And Hit Your Dates
maggiecrowley
36
2.8k
The Cost Of JavaScript in 2023
addyosmani
51
8.5k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Faster Mobile Websites
deanohume
307
31k
Transcript
DEMYSTIFYING THE OPENEDX ARCHITECTURE BITSAVANT © 2016 Bitsavant - All
Rights Reserved
WHO ARE WE A BIT ABOUT US Principal Security Engineer
Based in the Bay Area Hardware & Software Deployment Manager Boston, MA © 2016 Bitsavant - All Rights Reserved
WHY ARE WE PRESENTING HOW WE GOT INTO THIS ▸
Circa 2015, a business opportunity had us scouring the market for an LMS that would underpin our commercial offering. ▸ Started seriously looking into Open edX and we liked what we saw. ▸ Deploying and running a production-grade platform however was a different matter and had us dig deeper. © 2016 Bitsavant - All Rights Reserved
TODAY’S TALK WHAT WE WANT TO COVER ▸ Our journey
on the way to understanding this awesome platform. ▸ Unfortunate, and sometimes frequent, encounters with sinking ships. ▸ Issues we faced along the way with various deployment topologies. ▸ Reversing for fun and profit. ▸ Deep-dives of select components. © 2016 Bitsavant - All Rights Reserved
THE DIFFERENT FACES OF OPEN EDX STARTING WITH THE END-USER
THE DIFFERENT FACES OF OPEN EDX MOVING ON TO ARCHITECTS
THE DIFFERENT FACES OF OPEN EDX AND FINALLY IMPLEMENTERS
THE REALITY AWS ANYONE?
THE REALITY BACK IN THE REAL WORLD….
THE REALITY DOES THIS LOOK FAMILIAR
THAT FEELING…
THE RESULT MOST OF US ENDED UP HERE wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/
install/sandbox.sh -O - | bash © 2016 Bitsavant - All Rights Reserved
DEPLOYMENT MODELS THE ALL-IN-ONE WONDER ▸ Mostly works out of
the box with a clean deploy run. ▸ After some minor work, we’re betting most people get this to work. ▸ Troubleshooting problems or customization isn’t trivial and requires efforts digging around. ▸ Component interactions are on the complex side given the self-contained nature of this use-case. © 2016 Bitsavant - All Rights Reserved
MOVING ON THIS WAS FINE, UNTIL…. ▸ We always knew
that the fullstack instance was not for production purposes (why we avoided it in the first place). ▸ To revisit the distributed deployment, we wanted to first be very intimate with the foundations of the platform. © 2016 Bitsavant - All Rights Reserved
DIGGING FOR MORE SOME USEFUL RESOURCES ▸ Great starting point,
but someone deploying this will have a hard time mapping component layout to the architecture, hardly the Rosetta stone this clueless duo needed.
WHEN IN DOUBT, REVERSE. Us, whenever we’re confused. HOW WE
DID IT © 2016 Bitsavant - All Rights Reserved
HOW WE DID IT REVERSING FOR FUN AND EDUCATION ▸
We took a multi-pronged approach. ▸ Break down the single-box installation and figure out the minimum component baseline needed to setup a working cluster. ▸ In parallel, grok the distributed production topology prior to customizing it to our liking. © 2016 Bitsavant - All Rights Reserved
HOW WE DID IT TOOLS OF THE TRADE ▸ Code
review. ▸ Plain old bug hunting by scrutinizing logs. ▸ For figuring out application calls we leveraged a number of tools including Sysdig, Slack (for support), and old- style system sleuthing. ▸ Needless to say, system administration fundamentals come in handy here to figure out what the system is trying to do, and why it’s able to do it (or not). © 2016 Bitsavant - All Rights Reserved
INITIAL RESULTS SOME PROGRESS MADE © 2016 Bitsavant - All
Rights Reserved
ANSIBLE DEPENDENCIES COMMON SUPERVISOR ORACLEJDK EDX_SERVICE EDXAPP_ COMMON HADOOP_ COMMON
NLTK © 2016 Bitsavant - All Rights Reserved
SOMETHING STILL AMISS
STAYING LOCAL LESS IS MORE ▸ Realized that we could
simplify the core set of components even further. ▸ Further experimentation with distributed deployments using local resources. ▸ Ansible playbook clean-ups inline with our simplification drive. ▸ We ended up abstracting the architecture down to components we thought most essential to our understanding of the platform. © 2016 Bitsavant - All Rights Reserved
SIMPLER App © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: NGINX ▸ Entry point into the public-facing
portion of the platform. ▸ Provides proxied and load-balanced access to static and dynamic assets. ▸ Stateless and horizontally scalable. Process Username Workers Proto/Port Function nginx www-data 4 TCP *:80 LMS nginx www-data 4 TCP *:18010 CMS nginx www-data 4 TCP *:18040 xqueue nginx www-data 4 TCP *:18080 forum nginx www-data 4 TCP *:18090 certs ▸ /edx/var/log/nginx/{access,error}.log ▸ service nginx {status, reload, restart, stop} TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: NGINX App LMS TCP:80 CMS TCP:18010 xqueue
TCP:18040 forum TCP:18080 certs TCP:18090 LMS TCP:8000 CMS TCP:8010 xqueue TCP:8040 UNIX Domain Socket Call Static Assets Student Login Instructor Login © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: GUNICORN ▸ Python WSGI HTTP server that
NGiNX routes most calls through to the application servers (Django). ▸ Dynamic worker management. ▸ Stateless and horizontally scalable. Process Username Workers Proto/Port Function gunicorn www-data 8 TCP *:8000 LMS gunicorn www-data 4 TCP *:8010 CMS gunicorn www-data 4 TCP *:8040 xqueue ▸ /edx/var/log/{cms, lms, xqueue}/edx.log ▸ /edx/var/log/supervisor/{cms, lms, xqueue}-stderr.log TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: EDX APP ▸ Bundle of core application
components. ▸ Mostly implemented as Django apps. ▸ Includes a Ruby component serving particular workflows. Config Files Function cms.auth.json API Keys, Usernames, Passwords … cms.env.json Component URL’s and various CMS Configs cms_gunicorn.py Gunicorn Configuration File lms.auth.json API Keys, Usernames, Passwords … lms.env.json Component URL’s and various LMS Configs lms_gunicorn.py Gunicorn Configuration File ▸ /edx/var/log/{cms, lms, xqueue}/edx.log ▸ /edx/var/log/supervisor/{cms, lms, xqueue}-stderr.log TROUBLESHOOTING App elasticsearch rabbitmq mongodb memcache mysql nginx www-data certs xqueue forum edx_notes_api insights analytics_api notifier sandbox edxapp supervisor ACCOUNTS © 2016 Bitsavant - All Rights Reserved
EDXAPP PACKAGES base_requirements_file - %51.56 github_requirements_file - %15.23 edxapp_common_debian_pkgs -
%5.47 edxapp_debian_pkgs - %4.3 server_utils_debian_pkgs - %3.52 paver_requirements_file - %2.73 sandbox_base_requirements - %2.34 common_debian_pkgs - %2.34 local_requirements_file - %1.56 sandbox_local_requirements - %1.56 pre_requirements_file - %1.17 common_pip_pkgs - %1.17 security_debian_pkgs - %1.17 aws_pip_pkgs - %1.17 © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: RABBITMQ ▸ Underpins queueing infrastructure. ▸ Large
jobs are run through the queue, coordinated by Celery. ▸ Should be well-resourced and available so it doesn’t turn into a hidden SPoF. Process Username Proto/Port Function beam.smp rabbitmq localhost *:5672 Message Queue beam.smp rabbitmq TCP *:15672 Message Queue beam.smp rabbitmq TCP *:38246 Message Queue beam.smp rabbitmq TCP *:55672 Message Queue epmd rabbitmq TCP localhost:4369 Port Mapper ▸ /var/log/rabbitmq/* TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: MEMCACHED ▸ High-speed distributed KV store for
object- caching. ▸ Caches user sessions, course structure info, e.t.c. Process Username Proto/Port Function memcached memcache TCP localhost:11211 KV Store ▸ /var/log/memcached.log TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: MONGODB ▸ NoSQL DB backend. ▸ Used
mainly for storing course and forum data. ▸ Relatively easy to scale. Process Username Proto/Port Function mongod mongodb TCP *:27017 DB ▸ /edx/var/log/mongo/mongodb.log TROUBLESHOOTING App Collections 7 DBs 4 DB/COLLECTION COUNTS © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: MYSQL ▸ Main DB for user data
such user profiles, enrollment status, e.t.c. ▸ Holds authentication/authorization data as well. ▸ Also holds analytics results. ▸ Too many edxapp tables to get into (250+). Process Username Proto/Port Function mysqld mysql TCP localhost:3306 DB ▸ /var/log/mysql/error.log TROUBLESHOOTING App reports 0 xqueue 11 edxapp 278 edx_ notes_api 10 ecommerce 0 dashboard 23 analytics-api 12 DB/TABLE COUNTS © 2016 Bitsavant - All Rights Reserved
CONCLUSION LAST THOUGHTS ▸ This talk is just the start!
▸ We plan a similar application-centric exercise to cover core business logic. ▸ Keep an eye out for our repo (github.com/bitsavant) for upcoming cheatsheets, Ansible playbooks, and other stuff. ▸ Our contact details are at www.bitsavant.com. We’d love to hear from you! © 2016 Bitsavant - All Rights Reserved