Learning and Loving the edX Technology Stack

Ansible and operations training on full edx stack (MITx Residential Variant)

Transcript

1. Learning and Loving the edX Technology Stack Carson Gee and Brandon DeRosier 1 / 32
2. What We Hope to Cover Basic ansible usage in our world Using edx/configuration to your advantage Operations at scale (demo of tasks on full scale stack) 2 / 32
3. What You'll Need mitxstack downloaded and working ansible installed (1.5.5) https://github.com/edx/configuration cloned Some patience 3 / 32
4. Basic Ansible Usage We'll be covering: Static inventory one off commands intro to using modules using ansible to manage edx-platform 4 / 32
5. Create a simple inventory for mitxstack Create an inf inventory file that will specifies all the connection information you need to connect to mitxstack. Hints: IP is in Vagrantfile Vagrant ssh private key is at ~ / . v a g r a n t . d / i n s e c u r e _ p r i v a t e _ k e y http://docs.ansible.com/intro_inventory.html 5 / 32
6. Inventory Solution [ m i t x s t a c k ] v 1 a n s i b l e _ s s h _ h o s t = 1 9 2 . 1 6 8 . 3 3 . 1 0 a n s i b l e _ s s h _ u s e r = v a g r a n t a n s i b l e _ s s h _ p r i v a t e _ k e y _ f i l e = ~ / . v a g r a n t . d / i n s e c u r e _ p r i v a t e _ k e y Get at http://goo.gl/aM7nXA Can test with: a n s i b l e a l l - i m i t x s t a c k . i n i - m p i n g Should get a p o n g back 6 / 32
7. Fix ShellShock Bug on mitxstack Repair the bash vulnerability in mitxstack using a n s i b l e and your newly minted inventory file. Check if vulnerable first with: e n v x = ' ( ) { : ; } ; e c h o v u l n e r a b l e ' b a s h - c " e c h o t h i s i s a t e s t " Hint Apt module makes this easy: http://docs.ansible.com/apt_module.html 7 / 32
8. Shellshock solution a n s i b l e a l l - i m i t x s t a c k . i n i - m a p t - a ' n a m e = b a s h s t a t e = l a t e s t u p d a t e _ c a c h e = t r u e ' - s Get at http://goo.gl/VQp2xA Much easier than: v a g r a n t s s h s u d o a p t - g e t u p d a t e - y s u d o a p t - g e t i n s t a l l - - o n l y - u p g r a d e b a s h and you can run it on your entire fleet with the same one line 8 / 32
9. Restart nginx in mitxstack Use ansible to stop nginx, confirm it is down, and then start it. Hint Service module makes this pretty slick: http://docs.ansible.com/service_module.html 9 / 32
10. Nginx Solution a n s i b l e a l l - i m i t x s t a c k . i n i - m s e r v i c e - a ' n a m e = n g i n x s t a t e = s t o p p e d ' - s Go to http://192.168.33.10 to verify it times out or for bonus points: a n s i b l e a l l - i m i t x s t a c k . i n i - m w a i t _ f o r - a ' p o r t = 8 0 d e l a y = 1 t i m e o u t = 5 ' should fail a n s i b l e a l l - i m i t x s t a c k . i n i - m s e r v i c e - a ' n a m e = n g i n x s t a t e = s t a r t e d ' - s and refresh browser to verify everything is back in mitxstack or reuse the w a i t _ f o r command Get at http://goo.gl/GV8V3Z 10 / 32
11. Cat a file or files on mitxstack Maybe this is easier to do by hand with one host, but with a couple or dozens this makes this chore much easier. Use ansible to cat out / v a r / l o g / s y s l o g and then cat out all the files in / e d x / v a r / l o g / s u p e r v i s o r / that start with lms and end with log Hint The shell module in ansible is the great swiss army knife of ansible 11 / 32
12. cat Solution a n s i b l e a l l - i m i t x s t a c k . i n i - m s h e l l - a ' c a t / v a r / l o g / s y s l o g ' - s a n s i b l e a l l - i m i t x s t a c k . i n i - m s h e l l - a ' c a t / e d x / v a r / l o g / s u p e r v i s o r / l m s * . l o g ' - s should take care of cat'ing all the files to your terminal. This will also work with multiple servers and will output by server so piping this into l e s s is handy. You can even redirect it to a file on a loop and tail that file for multiserver multifile tail. e.g.: w h i l e t r u e ; d o \ a n s i b l e - i m i t x s t a c k . i n i - m s h e l l \ - a ' c a t / e d x / v a r / l o g / s u p e r v i s o r / l m s * . l o g ' a l l - s > t a i l . l o g ; \ s l e e p 1 ; d o n e & t a i l - f t a i l . l o g Get at: http://goo.gl/blA64b 12 / 32
13. Upgrade edx-platform without the script edx-platform, as well as the other components of the stack, can be easily updated using an update script which runs ansible. For example: s u d o / e d x / b i n / u p d a t e e d x - p l a t f o r m m i t x - r e l e a s e Using ansible from outside your vagrant box, update your edx- platform to m i t x - h o t f i x - 2 0 1 4 0 9 1 9 , our latest hotfix branch. Hint: You'll need to copy s e r v e r - v a r s . y m l file from the vagrant box to your host machine. It's all in the script. 13 / 32
14. Upgrade solution You need to copy the / e d x / a p p / e d x _ a n s i b l e / s e r v e r - v a r s . y m l file from the vagrant box to your host machine, then run the e d x a p p playbook as follows: a n s i b l e - p l a y b o o k e d x - e a s t / e d x a p p . y m l - i m i t x s t a c k . i n i - e @ s e r v e r - v a r s . y m l - e \ ' e d x _ p l a t f o r m _ v e r s i o n = m i t x - h o t f i x - 2 0 1 4 0 9 1 9 ' - - t a g s d e p l o y Get at http://goo.gl/PqpHJz class: title, center, middle 14 / 32
15. From Basics to Roles Jumping way ahead in ansible to roles, with some playbooks along the way. Logs are nice, but pretty logs are better: Install Kibana/Logstash/Elasticsearch Add some more MITx flavor to the mitxstack with gitreload: Sidebar on the sysadmin dashboard 15 / 32
16. Playbooks and Roles and Something or Other Roles are like packages in ansible Playbooks run roles edx/configuration is full of treasures (65 roles currently) From here on out, we'll be running out of the edx/configuration repo in the p l a y b o o k s directory 16 / 32
17. Three Services, so Hard (ELK stack) Install the elasticsearch, logstash, and kibana stack on your mitxstack with kibana being available on port 10000. Hints KIBANA_NGINX_PORT is the var with port set The play to install the full ELK stack is already there and log(gy) playbooks run with a n s i b l e - p l a y b o o k and not a n s i b l e 17 / 32
18. Install ELK solution The play to use is log_server.yml which is very simple: # B u i l d a k i b a n a / l o g s t a s h / e l a s t i c s e a r c h s e r v e r f o r c a p t u r i n g a n d # a n a l y z i n g l o g s . - n a m e : C o n f i g u r e s y s l o g s e r v e r h o s t s : a l l s u d o : y e s r o l e s : - c o m m o n - o r a c l e j d k - e l a s t i c s e a r c h - l o g s t a s h - k i b a n a - r o l e : n g i n x n g i n x _ s i t e s : - k i b a n a To run it with our overrides (to the port, which defaults to 80), we run a n s i b l e - p l a y b o o k - i m i t x s t a c k . i n i l o g _ s e r v e r . y m l - e K I B A N A _ N G I N X _ P O R T = 1 0 0 0 0 Once complete, verify by going to: http://192.168.33.10:10000 18 / 32
19. Where are the logs? So our kibana looks nice and all, but where are the logs? You get this one for free, to enable rsyslog forwarding of logs run: a n s i b l e a l l - i m i t x s t a c k . i n i - m s h e l l - a ' e c h o " * . * @ 1 2 7 . 0 . 0 . 1 " > \ / e t c / r s y s l o g . d / 9 9 - s y s l o g f o r w a r d . c o n f ' - s But you have to restart the rsyslog service. Get at: http://goo.gl/JGTjFA 19 / 32
20. Syslog Restart and Confirmation Restart with a module and command we have used already a n s i b l e a l l - i m i t x s t a c k . i n i - m s e r v i c e - a ' n a m e = r s y s l o g s t a t e = r e s t a r t e d ' - s You can generate a test message with: a n s i b l e a l l - i m i t x s t a c k . i n i - m s h e l l - a ' l o g g e r T h i s i s m y l o g m e s s a g e ' To start getting edx logs change s e r v e r - v a r s . y m l to set E D X A P P _ S Y S L O G _ S E R V E R to 'localhost' and re-run: a n s i b l e - p l a y b o o k e d x - e a s t / e d x a p p . y m l - i m i t x s t a c k . i n i - e @ s e r v e r - v a r s . y m l \ - - t a g s d e p l o y Get at http://goo.gl/LpO8dp 20 / 32
21. Extra Roles, gitreload, and secrets Let's install a role not in edx/configuration with gitreload Repository with role is at: https://github.mit.edu/mitx- devops/gitreload-role or tar balled at: http://public.mitx.mit.edu/dist/gitreload.tar.gz Hints Check e d x / c o n f i g u r a t i o n / p l a y b o o k s / a n s i b l e . c f g for where to put the role. Check out r u n _ r o l e . y m l for how to run this against mitxstack g r o u p _ v a r s / a l l defines secure_dir and you will need to create it for this role and add a key (even if it is fake) Verify at https://192.168.33.10:8095/queue 21 / 32
22. gitreload Solution Download role to e d x / c o n f i g u r a t i o n / . . / . . / a n s i b l e _ r o l e s Run: a n s i b l e - p l a y b o o k - i m i t x s t a c k . i n i r u n _ r o l e . y m l - e r o l e = g i t r e l o a d Have that fail at step i n s t a l l s s h k e y f o r t h e c o n t e n t r e p o s since you likely don't have . . / . . / o p s / e d x / c o n f i g u r a t i o n / p l a y b o o k s / p a t h / t o / s e c u r e _ e x a m p l e / k e y s / g i Create a folder somewhere (usually e d x / c o n f i g u r a t i o n . . / . . / s e c u r e _ d i r ) add a k e y s folder and either create a blank file or copy your private ssh key file to g i t r e l o a d inside that directory. Run a n s i b l e - p l a y b o o k - i m i t x s t a c k . i n i r u n _ r o l e . y m l - e r o l e = g i t r e l o a d \ - e s e c u r e _ d i r = ' . . / . . / s e c u r e _ d i r ' Get at: http://goo.gl/twGbWG 22 / 32
23. Skipping Steps and Tags Notice that we skipped a couple tasks in that play, well let's run those! Sysadmin Dashboard: http://192.168.33.10/sysadmin Delete demo course Tasks Run play such that the default course gets imported when running gitreload Run play such that only the course gets imported (no other tasks run) Hints c o u r s e _ c h e c k o u t tags 23 / 32
24. Skipping Steps and tags Solution To run gitreload, check out . . / . . / a n s i b l e _ r o l e s / g i t r e l o a d / t a s k s / m a i n . y m l and notice there is a step with w h e n : c o u r s e _ c h e c k o u t | b o o l and run: a n s i b l e - p l a y b o o k - i m i t x s t a c k . i n i r u n _ r o l e . y m l - e r o l e = g i t r e l o a d \ - e s e c u r e _ d i r = ' . . / . . / s e c u r e _ d i r ' \ - e c o u r s e _ c h e c k o u t = y e s Also notice the t a g s : c o u r s e _ p u l l statement, and run a n s i b l e - p l a y b o o k - i m i t x s t a c k . i n i r u n _ r o l e . y m l - e r o l e = g i t r e l o a d \ - e s e c u r e _ d i r = ' . . / . . / s e c u r e _ d i r ' \ - e c o u r s e _ c h e c k o u t = y e s - - t a g s c o u r s e _ p u l l to only run the course import. Checkout the the s y s a d m i n d a s h b o a r d to confirm the course is loaded and what sha1 it has to confirm it was git loaded Get at: http://goo.gl/61oSw1 class: title, center, middle 24 / 32
25. Production Scale Not really different than mitxstack Differences largely: Caring about availability Caring about data "common cluster" nginx templates HAProxy 25 / 32
26. Demo: OpenStack Horizon Networks Instances Naming Conventions 26 / 32
27. Demo: Zenoss Monitoring Infrastructure Events Network Map 27 / 32
28. Demo: Fix ShellShock Same as earlier, but swap the inventory a n s i b l e t a g _ e n v _ r p - o s - i n o v a . p y - m a p t \ - a ' n a m e = b a s h s t a t e = l a t e s t u p d a t e _ c a c h e = t r u e ' - s Get all demo commands at: http://goo.gl/u2hBqy 28 / 32
29. Demo: Upgrade to latest platform Running an "in-production" deploy Not much different than running e d x a p p . y m l from earlier Adds more advanced s e r i a l concept of removing app server, upgrading, adding it back. t i m e . . / s h e l l / a p p _ d e p l o y _ o s . s h - d - v d e v r p - o s d _ e d x a p p . y m l & \ t i m e . . / s h e l l / a p p _ d e p l o y _ o s . s h - d - v p r o d r p - o s p _ e d x a p p . y m l 29 / 32
30. Mongo Cluster Management Find the master node with ansible a n s i b l e a l l - i m o n g o 1 - a z 1 - r p . m i t x . m i t . e d u , - m s h e l l \ - a " e c h o ' r s . s t a t u s ( ) ' | m o n g o " | g r e p - B 3 P R I M A R Y Failing mongo master and recovery a n s i b l e ' t a g _ g r o u p _ a p p s : t a g _ g r o u p _ p a p p s : & t a g _ e n v _ r p - o s ' - i n o v a . p y \ - m s h e l l - a ' / e d x / b i n / s u p e r v i s o r c t l r e s t a r t e d x a p p : * ; \ / e d x / b i n / s u p e r v i s o r c t l r e s t a r t e d x a p p _ w o r k e r : * ; ' - s - f 1 30 / 32
31. RabbitMQ Cluster Management Show current queue status: a n s i b l e ' t a g _ g r o u p _ m o n g o : & t a g _ e n v _ r p - o s ' - i n o v a . p y \ - m s h e l l - a ' r a b b i t m q c t l c l u s t e r _ s t a t u s ' - s a n s i b l e ' t a g _ g r o u p _ m o n g o : & t a g _ e n v _ r p - o s ' - i n o v a . p y \ - m s h e l l - a ' r a b b i t m q c t l l i s t _ v h o s t s ' - s a n s i b l e ' t a g _ g r o u p _ m o n g o : & t a g _ e n v _ r p - o s ' - i n o v a . p y \ - m s h e l l - a ' r a b b i t m q c t l l i s t _ q u e u e s - p / d e v ' - s a n s i b l e ' t a g _ g r o u p _ m o n g o : & t a g _ e n v _ r p - o s ' - i n o v a . p y \ - m s h e l l - a ' r a b b i t m q c t l l i s t _ q u e u e s - p / p r o d ' - s 31 / 32
32. Production Kibana Login to https://log-rp-os.mitx.mit.edu Mess around as there should be many more logs than on mitxstack Open up OpenStack Residential Environment dashboard Try filtering by host Add pie chart (terms) for logs by host Find yourself in tracking logs from browsing https://prod-rp- os.mitx.mit.edu 32 / 32