Introduction to Server Operations and Automations

Transcript

1. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
   Keio University Logo is licensed by its copyright holders.
   Server Operation & Automation
   SFC-RG Lecture
   
   14 May 2015
   
   Hirotaka Nakajima (@nunnun)
   
   

   View Slide

2. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
   Keio University Logo is licensed by its copyright holders.
   Server Operations
   
   "Racks line" by Tristan Schmurr is licensed under CC BY 2.0
   

   View Slide

3. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
   Keio University Logo is licensed by its copyright holders.
   
   Way to build a web server
   • Purchase a server
   
   • Installation (Physically)
   
   • Install an Operating System
   
   • Configure an Operating System
   
   • Install Applications
   
   • Configure Application settings
   
   • Build a Web application
   
   • Done!!
   

   View Slide

4. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
   Keio University Logo is licensed by its copyright holders.
   
   Purchase
   • Design a specification
   
   • CPU
   
   • Architecture: x86_64? SPARC? ARM?
   
   • How many cores? CPUs?
   
   • Memory
   
   • ECC or Registered?
   
   • Disk requirement
   
   • Capacity
   
   • Reliability; RAID? which level?
   
   • Speed; SSD? SATA? SAS?
   
   • Network requirement
   
   • Interface
   
   • Budget and political issues
   

   View Slide

5. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
   Keio University Logo is licensed by its copyright holders.
   
   Installation
   • Mount a server to Rack
   
   • Cabling
   
   • Power backup plan
   "new space" by emmma peel is licensed under CC BY 2.0
   "Data Center" by Bob Mical is licensed under CC BY 2.0
   

   View Slide

6. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
   Keio University Logo is licensed by its copyright holders.
   
   Operating System
   • Which Operating System?
   
   • Linux? Windows?
   
   • Which Linux? Debian? Ubuntu? RedHat?
   
   • Version? Latest? Stable version?
   
   • Configuration
   
   • Hostname
   
   • Date & Time
   
   • Network
   
   • Disk initialization
   
   • User Management
   
   • Installation takes 1-2 hours
   

   View Slide

7. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
   Keio University Logo is licensed by its copyright holders.
   
   Application
   • Application
   
   • Which service; Web service? Mail service?
   
   • Which Application for Web service?
   
   • Apache? lighthttpd? nginx?
   
   • Configuration
   
   • Host
   
   • Which hostname is served with the service
   
   • Security
   
   • Using https? how about a server certificate?
   
   • Directory
   
   • Where web documents are located?
   

   View Slide

8. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
   Keio University Logo is licensed by its copyright holders.
   
   Content
   • Server-side Dynamic? Static?
   
   • Which Language & Runtime?
   
   • PHP, Perl, Java, C#, JavaScript….
   
   • Middle-ware?
   
   • WordPress etc…
   
   • Framework?
   
   • CakePHP, Symfony?
   
   • Client-side Dynamic? Static?
   
   • JavaScript Libraries
   
   • jQuery, Angular.js
   

   View Slide

9. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
   Keio University Logo is licensed by its copyright holders.
   Done!!!
   
   

   View Slide

10. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Am I all set and free to go?
    • NO!!!!
    
    • We need to do maintenance works
    
    • Maintenance work
    
    • OS update
    
    • Application, Middleware, Library update
    
    • Hardware failure
    
    • Security update
    

    View Slide

11. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Documentation
    • What if we need to setup 100 servers 

    with same configuration?
    
    • Make a setup manual
    
    • How about changes?
    
    • Make a changelog
    
    • Are we sure 

    if we can deploy service to

    a server like this?
    [May 1] Server is delivered.
    [May 2] Debian version x is installed.
    Installation log is here
    [May 3] Apache is installed.
    Configurations are here
    [Jun 1] DNS Cache setting
    changed…
    ….
    [Jun 15] PHP settings changed…
    [Aug 10] Apache Configuration
    changed..
    

    View Slide

12. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Undocumented Knowledge
    • Information not documented
    
    • Forget to be documented
    
    • Workaround not figured out why it work
    
    • They don’t know why but it works
    
    • “Secret Recipe” Issue / ൿ఻ͷλϨ໰୊
    
    • “Don’t touch the system if it’s running”
    

    View Slide

13. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Security Update comes suddenly…
    • All software contains bugs
    
    • Some bugs may trigger a security
    incident
    
    • Vulnerability (੬ऑੑ)
    
    • We don’t know vulnerabilities
    beforehand
    
    • Once they publish, we need to
    handle
    
    • Apply a workaround
    
    • Apply a security update
    
    • What if we have 100++ servers?
    
    • Update all server by hand?
    LINE Icons are copyrights LINE corporation
    

    View Slide

14. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    Virtualisation
    
    

    View Slide

15. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    What is virtualisation?
    • “virtualization refers to the act of creating a
    virtual (rather than actual) version of something,
    including (but not limited to) a virtual computer
    hardware platform, operating system (OS),
    storage device, or computer network
    resources” from wikipedia
    
    • Hardware Virtualization
    
    • Operating System Virtualizaion
    

    View Slide

16. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Hardware Virtualization
    • Virtualize server architecture on a server
    
    • Virtual Machines are packaged in files.
    
    • We don’t need to bind a specific server
    
    • We can move a virtual machine
    
    • Resource Optimization
    Hardware Virtualisation image from http://download.parallels.com/doc/psbm/v5/rtm/Parallels_Server_Bare_Metal_Users_Guide/29765.htm
    

    View Slide

17. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    IaaS
    • Infrastructure as a Service
    
    • Provides computing infrastructure as an
    Internet service
    
    • Virtual Machine
    
    • Network
    
    • Storage
    
    • Amazon Web Services, Google Compute
    Engine
    
    • OpenStack, CloudStack
    

    View Slide

18. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    VM Image & Template
    • Customized virtual machine
    
    • Install specific softwares, configurations
    
    • Easy for massive deployment
    
    • Deployment can be done by RESTful API
    
    • Server deployment becomes programmable
    

    View Slide

19. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    Automation
    
    

    View Slide

20. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Infrastructure as Code
    • IaaS achieves 

    program/system can operate infrastructure
    
    • We don’t need to do all by hand.
    
    • How about a configuration?
    
    • We can automate a server configuration
    
    • Configuration Management Tools
    
    • CFEngine, LCFG, BCFG
    
    • Apply configuration
    
    • Test if the configuration is current
    

    View Slide

21. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Define what the server should be.
    • Manual deployment
    
    • 1. Install Apache, 2. Set the document root
    
    • What if somebody uninstalled a Apache?
    
    • Re-apply a entire installation step?
    
    • Define a state not a procedure
    
    • Apache should be installed
    
    • Document root should be “/var/www/htdocs”
    
    • Puppet, Chef
    

    View Slide

22. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Idempotence / ႈ౳ੑ
    • Idempotence is the property of certain
    operations in mathematics and computer
    science, that can be applied multiple times
    without changing the result beyond the initial
    application.
    
    • ৘ใ޻ֶʹ͓͚Δႈ౳ͱ͸ɺ͋Δૢ࡞Λ1౓
    ߦͬͯ΋ෳ਺ճߦͬͯ΋ಉ͡ޮՌͱͳΔ͜ͱΛ
    ݴ͏ɻಛʹɺԿճߦͬͯ΋Τϥʔ΍ෆ੔߹ͷঢ়
    ଶ͕มΘΒͳ͍ૢ࡞Λࢦ͢ɻ
    

    View Slide

23. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    What Puppet, Chef do?
    • Keep a machine to meet a defined state
    
    • Automated server configuration
    
    • Defined state
    
    • Completing an installation on abandoned server
    
    • Somebody abandoned an installation in a middle
    
    • Update a server configuration to current
    
    • Revert a temporary change to original configuration
    
    • Operating System independent configuration
    
    • With version control tools
    
    • Track changes on the infrastructure
    
    • Review the changes before applying to whole server
    

    View Slide

24. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Puppet repository on GitHub
    

    View Slide

25. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Security update with Automation
    • Automation tools can help deployment of
    security patches
    
    • Recipe example;
    
    • We don’t need to patch all servers by hand!!
    package {
    "openssl":
    ensure => "latest";
    "libssl":
    ensure => "latest";
    "unsafesoftware":
    ensure => "purged";
    }
    

    View Slide

26. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Test driven Development
    • Puppet/Chef enables automated server configuration
    
    • How we assure the recipe applied to all servers?
    
    • SSH login and confirm?
    
    • In software development, we have test-driven
    development
    
    • Write a test case what function/program is expected to
    work
    
    • Write a function/program
    
    • Run a test if function/program works correctly
    
    • Let’s do a same thing!
    

    View Slide

27. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Test driven Infrastructure
    • Test the infrastructure if it meets the test case
    
    • Apache is installed
    
    • OpenSSL is up-to-date
    
    • Test-driven Infrastructure
    
    • Write a test case of infrastructure
    
    • Write a recipe of automation tool
    
    • Apply the recipe to servers
    
    • Run a test case
    
    • Serverspec
    

    View Slide

28. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Serverspec example
    require "spec_helper"
    describe 'nginx' do
    it { should be_installed }
    it { should be_enabled }
    it { should be_running }
    end
    describe 'port 443' do
    it { should be_listening }
    end
    describe '/etc/nginx/nginx.conf' do
    it { should be_file }
    it { should contain "server_name" blog.nunnun.jp }
    end
    

    View Slide

29. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Old-fashioned software development
    • Code a module
    
    • Then merge modules and test
    
    • Most of the case, it fails!!
    
    • Someone will fix the issue
    
    • Someone will implement new feature
    
    • Then merge again, of course it will fail!
    MS Project image from http://projectsprofiler.com/images/blog/msfsw_1.PNG
    

    View Slide

30. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Continuous Integration (CI)
    • Any changes must be tested with entire system.
    
    • If you find issues, contact other developer to fix
    the issue.
    

    View Slide

31. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    CI on Infrastructure
    • Unable to test a recipe before applying
    
    • Sometimes untested recipe breaks a
    production server
    
    • Want to test my recipe before applying

    production server
    
    • Test a recipe before applying production
    environment with test environment
    
    • Test environment is build from scratch in every
    test using container technologies (Docker)
    # puppet agent --test
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Loading facts
    Info: Caching catalog for
    ldap0.kamoike.net
    Info: Applying configuration version
    '1431407810'
    Error: /Stage[main]/Kamoikeldap::Server/
    Openldap::Server::Dbindex[master_mdb_conf
    ig_entryCSN]/
    Openldap_dbindex[master_mdb_config_entryC
    SN]/ensure: change from absent to present
    failed: LDIF content:
    dn: olcDatabase={1}mdb,cn=config
    add: olcDbIndex
    olcDbIndex: entryCSN eq
    Error message: Execution of '/usr/bin/
    ldapmodify -Y EXTERNAL -H ldapi:/// -f /
    tmp/
    openldap_dbindex20150514-9204-17e4z8y'
    returned 20: SASL/EXTERNAL authentication
    started
    SASL username:
    gidNumber=0+uidNumber=0,cn=peercred,cn=ex
    ternal,cn=auth
    SASL SSF: 0
    ldap_modify: Type or value exists (20)
    additional info: modify/add:
    olcDbIndex: value #0 already exists
    modifying entry "olcDatabase={1}
    mdb,cn=config"
    Notice: Finished catalog run in 2.08
    seconds
    

    View Slide

32. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Immutable / Disposable Infrastructure
    • Immutable

    มΘΒͳ͍ɺมԽ͠ͳ͍ɺෆมͷɺෆқͷɺม
    ͑Δ͜ͱͷͰ͖ͳ͍ɺมߋෆՄೳͳ
    
    • Disposable

    ࢖͍ࣺͯͷɺ؆୯ʹॲ෼Ͱ͖Δɺ࢖͍ࣺͯͰ͖
    Δ
    

    View Slide

33. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Amazon’s example
    • Deploy 1,000 times per hour
    Image from http://www.publickey1.jp/blog/12/amazon11000_aws_reinventday2_am.html
    

    View Slide

34. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Amazon’s example
    Image from http://www.publickey1.jp/blog/12/amazon11000_aws_reinventday2_am.html
    

    View Slide

35. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    So…
    • If your Mac/PC is not working good,

    we sometimes reinstall the OS and setup again.
    
    • Same thing
    
    • Build an instance based on recipe,
    
    • If it works correctly, use it
    
    • If not, just keep using old instance
    

    View Slide

36. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    
    Conclusion
    • Focus on server operations and deployment
    
    • Old-fashioned deployment
    
    • Infrastructure as a Service
    
    • Infrastructure as Code
    
    • Test-driven Infrastructure
    
    • Continuous Integration on Infrastructure
    
    • Immutable / Disposable Infrastructure
    
    • Now we’re able to design and code an infrastructure
    
    • How about network?
    
    • It may be possible, but more difficult.
    
    • e.g. Route change continuously by external issues
    

    View Slide