Introduction to Server Operations and Automations

Transcript

1. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders. Server Operation & Automation SFC-RG Lecture 14 May 2015 Hirotaka Nakajima (@nunnun) 
2. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders. Server Operations  "Racks line" by Tristan Schmurr is licensed under CC BY 2.0
3. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Way to build a web server • Purchase a server • Installation (Physically) • Install an Operating System • Configure an Operating System • Install Applications • Configure Application settings • Build a Web application • Done!!
4. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Purchase • Design a specification • CPU • Architecture: x86_64? SPARC? ARM? • How many cores? CPUs? • Memory • ECC or Registered? • Disk requirement • Capacity • Reliability; RAID? which level? • Speed; SSD? SATA? SAS? • Network requirement • Interface • Budget and political issues
5. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Installation • Mount a server to Rack • Cabling • Power backup plan "new space" by emmma peel is licensed under CC BY 2.0 "Data Center" by Bob Mical is licensed under CC BY 2.0
6. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Operating System • Which Operating System? • Linux? Windows? • Which Linux? Debian? Ubuntu? RedHat? • Version? Latest? Stable version? • Configuration • Hostname • Date & Time • Network • Disk initialization • User Management • Installation takes 1-2 hours
7. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Application • Application • Which service; Web service? Mail service? • Which Application for Web service? • Apache? lighthttpd? nginx? • Configuration • Host • Which hostname is served with the service • Security • Using https? how about a server certificate? • Directory • Where web documents are located?
8. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Content • Server-side Dynamic? Static? • Which Language & Runtime? • PHP, Perl, Java, C#, JavaScript…. • Middle-ware? • WordPress etc… • Framework? • CakePHP, Symfony? • Client-side Dynamic? Static? • JavaScript Libraries • jQuery, Angular.js
9. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders. Done!!! 
10. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Am I all set and free to go? • NO!!!! • We need to do maintenance works • Maintenance work • OS update • Application, Middleware, Library update • Hardware failure • Security update
11. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Documentation • What if we need to setup 100 servers 
 with same configuration? • Make a setup manual • How about changes? • Make a changelog • Are we sure 
 if we can deploy service to
 a server like this? [May 1] Server is delivered. [May 2] Debian version x is installed. Installation log is here [May 3] Apache is installed. Configurations are here [Jun 1] DNS Cache setting changed… …. [Jun 15] PHP settings changed… [Aug 10] Apache Configuration changed..
12. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Undocumented Knowledge • Information not documented • Forget to be documented • Workaround not figured out why it work • They don’t know why but it works • “Secret Recipe” Issue / ൿ఻ͷλϨ໰୊ • “Don’t touch the system if it’s running”
13. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Security Update comes suddenly… • All software contains bugs • Some bugs may trigger a security incident • Vulnerability (੬ऑੑ) • We don’t know vulnerabilities beforehand • Once they publish, we need to handle • Apply a workaround • Apply a security update • What if we have 100++ servers? • Update all server by hand? LINE Icons are copyrights LINE corporation
14. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders. Virtualisation 
15. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  What is virtualisation? • “virtualization refers to the act of creating a virtual (rather than actual) version of something, including (but not limited to) a virtual computer hardware platform, operating system (OS), storage device, or computer network resources” from wikipedia • Hardware Virtualization • Operating System Virtualizaion
16. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Hardware Virtualization • Virtualize server architecture on a server • Virtual Machines are packaged in files. • We don’t need to bind a specific server • We can move a virtual machine • Resource Optimization Hardware Virtualisation image from http://download.parallels.com/doc/psbm/v5/rtm/Parallels_Server_Bare_Metal_Users_Guide/29765.htm
17. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  IaaS • Infrastructure as a Service • Provides computing infrastructure as an Internet service • Virtual Machine • Network • Storage • Amazon Web Services, Google Compute Engine • OpenStack, CloudStack
18. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  VM Image & Template • Customized virtual machine • Install specific softwares, configurations • Easy for massive deployment • Deployment can be done by RESTful API • Server deployment becomes programmable
19. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders. Automation 
20. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Infrastructure as Code • IaaS achieves 
 program/system can operate infrastructure • We don’t need to do all by hand. • How about a configuration? • We can automate a server configuration • Configuration Management Tools • CFEngine, LCFG, BCFG • Apply configuration • Test if the configuration is current
21. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Define what the server should be. • Manual deployment • 1. Install Apache, 2. Set the document root • What if somebody uninstalled a Apache? • Re-apply a entire installation step? • Define a state not a procedure • Apache should be installed • Document root should be “/var/www/htdocs” • Puppet, Chef
22. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Idempotence / ႈ౳ੑ • Idempotence is the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application. • ৘ใ޻ֶʹ͓͚Δႈ౳ͱ͸ɺ͋Δૢ࡞Λ1౓ ߦͬͯ΋ෳ਺ճߦͬͯ΋ಉ͡ޮՌͱͳΔ͜ͱΛ ݴ͏ɻಛʹɺԿճߦͬͯ΋Τϥʔ΍ෆ੔߹ͷঢ় ଶ͕มΘΒͳ͍ૢ࡞Λࢦ͢ɻ
23. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  What Puppet, Chef do? • Keep a machine to meet a defined state • Automated server configuration • Defined state • Completing an installation on abandoned server • Somebody abandoned an installation in a middle • Update a server configuration to current • Revert a temporary change to original configuration • Operating System independent configuration • With version control tools • Track changes on the infrastructure • Review the changes before applying to whole server
24. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Puppet repository on GitHub
25. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Security update with Automation • Automation tools can help deployment of security patches • Recipe example; • We don’t need to patch all servers by hand!! package { "openssl": ensure => "latest"; "libssl": ensure => "latest"; "unsafesoftware": ensure => "purged"; }
26. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Test driven Development • Puppet/Chef enables automated server configuration • How we assure the recipe applied to all servers? • SSH login and confirm? • In software development, we have test-driven development • Write a test case what function/program is expected to work • Write a function/program • Run a test if function/program works correctly • Let’s do a same thing!
27. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Test driven Infrastructure • Test the infrastructure if it meets the test case • Apache is installed • OpenSSL is up-to-date • Test-driven Infrastructure • Write a test case of infrastructure • Write a recipe of automation tool • Apply the recipe to servers • Run a test case • Serverspec
28. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Serverspec example require "spec_helper" describe 'nginx' do it { should be_installed } it { should be_enabled } it { should be_running } end describe 'port 443' do it { should be_listening } end describe '/etc/nginx/nginx.conf' do it { should be_file } it { should contain "server_name" blog.nunnun.jp } end
29. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Old-fashioned software development • Code a module • Then merge modules and test • Most of the case, it fails!! • Someone will fix the issue • Someone will implement new feature • Then merge again, of course it will fail! MS Project image from http://projectsprofiler.com/images/blog/msfsw_1.PNG
30. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Continuous Integration (CI) • Any changes must be tested with entire system. • If you find issues, contact other developer to fix the issue.
31. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  CI on Infrastructure • Unable to test a recipe before applying • Sometimes untested recipe breaks a production server • Want to test my recipe before applying
 production server • Test a recipe before applying production environment with test environment • Test environment is build from scratch in every test using container technologies (Docker) # puppet agent --test Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for ldap0.kamoike.net Info: Applying configuration version '1431407810' Error: /Stage[main]/Kamoikeldap::Server/ Openldap::Server::Dbindex[master_mdb_conf ig_entryCSN]/ Openldap_dbindex[master_mdb_config_entryC SN]/ensure: change from absent to present failed: LDIF content: dn: olcDatabase={1}mdb,cn=config add: olcDbIndex olcDbIndex: entryCSN eq Error message: Execution of '/usr/bin/ ldapmodify -Y EXTERNAL -H ldapi:/// -f / tmp/ openldap_dbindex20150514-9204-17e4z8y' returned 20: SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=ex ternal,cn=auth SASL SSF: 0 ldap_modify: Type or value exists (20) additional info: modify/add: olcDbIndex: value #0 already exists modifying entry "olcDatabase={1} mdb,cn=config" Notice: Finished catalog run in 2.08 seconds
32. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Immutable / Disposable Infrastructure • Immutable
 มΘΒͳ͍ɺมԽ͠ͳ͍ɺෆมͷɺෆқͷɺม ͑Δ͜ͱͷͰ͖ͳ͍ɺมߋෆՄೳͳ • Disposable
 ࢖͍ࣺͯͷɺ؆୯ʹॲ෼Ͱ͖Δɺ࢖͍ࣺͯͰ͖ Δ
33. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Amazon’s example • Deploy 1,000 times per hour Image from http://www.publickey1.jp/blog/12/amazon11000_aws_reinventday2_am.html
34. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Amazon’s example Image from http://www.publickey1.jp/blog/12/amazon11000_aws_reinventday2_am.html
35. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  So… • If your Mac/PC is not working good,
 we sometimes reinstall the OS and setup again. • Same thing • Build an instance based on recipe, • If it works correctly, use it • If not, just keep using old instance
36. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima. Keio University Logo is licensed by its copyright holders.  Conclusion • Focus on server operations and deployment • Old-fashioned deployment • Infrastructure as a Service • Infrastructure as Code • Test-driven Infrastructure • Continuous Integration on Infrastructure • Immutable / Disposable Infrastructure • Now we’re able to design and code an infrastructure • How about network? • It may be possible, but more difficult. • e.g. Route change continuously by external issues