Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Demystifying The Open edX Architecture
Search
bitsavant
June 14, 2016
Technology
3
1.4k
Demystifying The Open edX Architecture
Slides for Bitsavant's talk at Open edX 2016, held at Stanford, CA.
bitsavant
June 14, 2016
Tweet
Share
Other Decks in Technology
See All in Technology
Google Cloud Next '24でブログを10本書いた方法と勉強会を沸かせた方法
yasumuusan
0
310
web-application-security
matsuihidetoshi
0
170
DevOpsメトリクスとアウトカムの接続にトライ!開発プロセスを通して計測できるメトリクスの活用方法
ham0215
2
240
今年のRubyKaigiはProfiler Year🤘
osyoyu
0
170
開発パフォーマンスを最大化するための開発体制
ham0215
2
440
どうするコスト最適化のトレードオフ
tetsuyaooooo
1
530
AOAI をきっかけに 社内の Azure 管理を見直した話
recruitengineers
PRO
1
300
Vertex AI を中心に 生成AIのアップデートを共有します
kaz1437
0
310
Gitlab本から学んだこと - そーだいなるプレイバック / gitlab-book
soudai
4
440
Terraformあれやこれ/terraform-this-and-that
emiki
8
1.4k
ゼロから始めるVue.jsコミュニティ貢献 / first-vuejs-community-contribution-link-and-motivation
lmi
1
130
オーナーシップを持つ領域を明確にする
konifar
13
3.2k
Featured
See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
501
140k
Building Adaptive Systems
keathley
31
1.9k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
121
39k
Navigating Team Friction
lara
178
13k
Optimizing for Happiness
mojombo
370
69k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
18
6.9k
Writing Fast Ruby
sferik
621
60k
The World Runs on Bad Software
bkeepers
PRO
61
6.7k
YesSQL, Process and Tooling at Scale
rocio
164
13k
The MySQL Ecosystem @ GitHub 2015
samlambert
243
12k
Code Review Best Practice
trishagee
55
15k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
155
14k
Transcript
DEMYSTIFYING THE OPENEDX ARCHITECTURE BITSAVANT © 2016 Bitsavant - All
Rights Reserved
WHO ARE WE A BIT ABOUT US Principal Security Engineer
Based in the Bay Area Hardware & Software Deployment Manager Boston, MA © 2016 Bitsavant - All Rights Reserved
WHY ARE WE PRESENTING HOW WE GOT INTO THIS ▸
Circa 2015, a business opportunity had us scouring the market for an LMS that would underpin our commercial offering. ▸ Started seriously looking into Open edX and we liked what we saw. ▸ Deploying and running a production-grade platform however was a different matter and had us dig deeper. © 2016 Bitsavant - All Rights Reserved
TODAY’S TALK WHAT WE WANT TO COVER ▸ Our journey
on the way to understanding this awesome platform. ▸ Unfortunate, and sometimes frequent, encounters with sinking ships. ▸ Issues we faced along the way with various deployment topologies. ▸ Reversing for fun and profit. ▸ Deep-dives of select components. © 2016 Bitsavant - All Rights Reserved
THE DIFFERENT FACES OF OPEN EDX STARTING WITH THE END-USER
THE DIFFERENT FACES OF OPEN EDX MOVING ON TO ARCHITECTS
THE DIFFERENT FACES OF OPEN EDX AND FINALLY IMPLEMENTERS
THE REALITY AWS ANYONE?
THE REALITY BACK IN THE REAL WORLD….
THE REALITY DOES THIS LOOK FAMILIAR
THAT FEELING…
THE RESULT MOST OF US ENDED UP HERE wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/
install/sandbox.sh -O - | bash © 2016 Bitsavant - All Rights Reserved
DEPLOYMENT MODELS THE ALL-IN-ONE WONDER ▸ Mostly works out of
the box with a clean deploy run. ▸ After some minor work, we’re betting most people get this to work. ▸ Troubleshooting problems or customization isn’t trivial and requires efforts digging around. ▸ Component interactions are on the complex side given the self-contained nature of this use-case. © 2016 Bitsavant - All Rights Reserved
MOVING ON THIS WAS FINE, UNTIL…. ▸ We always knew
that the fullstack instance was not for production purposes (why we avoided it in the first place). ▸ To revisit the distributed deployment, we wanted to first be very intimate with the foundations of the platform. © 2016 Bitsavant - All Rights Reserved
DIGGING FOR MORE SOME USEFUL RESOURCES ▸ Great starting point,
but someone deploying this will have a hard time mapping component layout to the architecture, hardly the Rosetta stone this clueless duo needed.
WHEN IN DOUBT, REVERSE. Us, whenever we’re confused. HOW WE
DID IT © 2016 Bitsavant - All Rights Reserved
HOW WE DID IT REVERSING FOR FUN AND EDUCATION ▸
We took a multi-pronged approach. ▸ Break down the single-box installation and figure out the minimum component baseline needed to setup a working cluster. ▸ In parallel, grok the distributed production topology prior to customizing it to our liking. © 2016 Bitsavant - All Rights Reserved
HOW WE DID IT TOOLS OF THE TRADE ▸ Code
review. ▸ Plain old bug hunting by scrutinizing logs. ▸ For figuring out application calls we leveraged a number of tools including Sysdig, Slack (for support), and old- style system sleuthing. ▸ Needless to say, system administration fundamentals come in handy here to figure out what the system is trying to do, and why it’s able to do it (or not). © 2016 Bitsavant - All Rights Reserved
INITIAL RESULTS SOME PROGRESS MADE © 2016 Bitsavant - All
Rights Reserved
ANSIBLE DEPENDENCIES COMMON SUPERVISOR ORACLEJDK EDX_SERVICE EDXAPP_ COMMON HADOOP_ COMMON
NLTK © 2016 Bitsavant - All Rights Reserved
SOMETHING STILL AMISS
STAYING LOCAL LESS IS MORE ▸ Realized that we could
simplify the core set of components even further. ▸ Further experimentation with distributed deployments using local resources. ▸ Ansible playbook clean-ups inline with our simplification drive. ▸ We ended up abstracting the architecture down to components we thought most essential to our understanding of the platform. © 2016 Bitsavant - All Rights Reserved
SIMPLER App © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: NGINX ▸ Entry point into the public-facing
portion of the platform. ▸ Provides proxied and load-balanced access to static and dynamic assets. ▸ Stateless and horizontally scalable. Process Username Workers Proto/Port Function nginx www-data 4 TCP *:80 LMS nginx www-data 4 TCP *:18010 CMS nginx www-data 4 TCP *:18040 xqueue nginx www-data 4 TCP *:18080 forum nginx www-data 4 TCP *:18090 certs ▸ /edx/var/log/nginx/{access,error}.log ▸ service nginx {status, reload, restart, stop} TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: NGINX App LMS TCP:80 CMS TCP:18010 xqueue
TCP:18040 forum TCP:18080 certs TCP:18090 LMS TCP:8000 CMS TCP:8010 xqueue TCP:8040 UNIX Domain Socket Call Static Assets Student Login Instructor Login © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: GUNICORN ▸ Python WSGI HTTP server that
NGiNX routes most calls through to the application servers (Django). ▸ Dynamic worker management. ▸ Stateless and horizontally scalable. Process Username Workers Proto/Port Function gunicorn www-data 8 TCP *:8000 LMS gunicorn www-data 4 TCP *:8010 CMS gunicorn www-data 4 TCP *:8040 xqueue ▸ /edx/var/log/{cms, lms, xqueue}/edx.log ▸ /edx/var/log/supervisor/{cms, lms, xqueue}-stderr.log TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: EDX APP ▸ Bundle of core application
components. ▸ Mostly implemented as Django apps. ▸ Includes a Ruby component serving particular workflows. Config Files Function cms.auth.json API Keys, Usernames, Passwords … cms.env.json Component URL’s and various CMS Configs cms_gunicorn.py Gunicorn Configuration File lms.auth.json API Keys, Usernames, Passwords … lms.env.json Component URL’s and various LMS Configs lms_gunicorn.py Gunicorn Configuration File ▸ /edx/var/log/{cms, lms, xqueue}/edx.log ▸ /edx/var/log/supervisor/{cms, lms, xqueue}-stderr.log TROUBLESHOOTING App elasticsearch rabbitmq mongodb memcache mysql nginx www-data certs xqueue forum edx_notes_api insights analytics_api notifier sandbox edxapp supervisor ACCOUNTS © 2016 Bitsavant - All Rights Reserved
EDXAPP PACKAGES base_requirements_file - %51.56 github_requirements_file - %15.23 edxapp_common_debian_pkgs -
%5.47 edxapp_debian_pkgs - %4.3 server_utils_debian_pkgs - %3.52 paver_requirements_file - %2.73 sandbox_base_requirements - %2.34 common_debian_pkgs - %2.34 local_requirements_file - %1.56 sandbox_local_requirements - %1.56 pre_requirements_file - %1.17 common_pip_pkgs - %1.17 security_debian_pkgs - %1.17 aws_pip_pkgs - %1.17 © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: RABBITMQ ▸ Underpins queueing infrastructure. ▸ Large
jobs are run through the queue, coordinated by Celery. ▸ Should be well-resourced and available so it doesn’t turn into a hidden SPoF. Process Username Proto/Port Function beam.smp rabbitmq localhost *:5672 Message Queue beam.smp rabbitmq TCP *:15672 Message Queue beam.smp rabbitmq TCP *:38246 Message Queue beam.smp rabbitmq TCP *:55672 Message Queue epmd rabbitmq TCP localhost:4369 Port Mapper ▸ /var/log/rabbitmq/* TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: MEMCACHED ▸ High-speed distributed KV store for
object- caching. ▸ Caches user sessions, course structure info, e.t.c. Process Username Proto/Port Function memcached memcache TCP localhost:11211 KV Store ▸ /var/log/memcached.log TROUBLESHOOTING App © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: MONGODB ▸ NoSQL DB backend. ▸ Used
mainly for storing course and forum data. ▸ Relatively easy to scale. Process Username Proto/Port Function mongod mongodb TCP *:27017 DB ▸ /edx/var/log/mongo/mongodb.log TROUBLESHOOTING App Collections 7 DBs 4 DB/COLLECTION COUNTS © 2016 Bitsavant - All Rights Reserved
COMPONENT DEEP-DIVE FOCUS: MYSQL ▸ Main DB for user data
such user profiles, enrollment status, e.t.c. ▸ Holds authentication/authorization data as well. ▸ Also holds analytics results. ▸ Too many edxapp tables to get into (250+). Process Username Proto/Port Function mysqld mysql TCP localhost:3306 DB ▸ /var/log/mysql/error.log TROUBLESHOOTING App reports 0 xqueue 11 edxapp 278 edx_ notes_api 10 ecommerce 0 dashboard 23 analytics-api 12 DB/TABLE COUNTS © 2016 Bitsavant - All Rights Reserved
CONCLUSION LAST THOUGHTS ▸ This talk is just the start!
▸ We plan a similar application-centric exercise to cover core business logic. ▸ Keep an eye out for our repo (github.com/bitsavant) for upcoming cheatsheets, Ansible playbooks, and other stuff. ▸ Our contact details are at www.bitsavant.com. We’d love to hear from you! © 2016 Bitsavant - All Rights Reserved