$30 off During Our Annual Pro Sale. View Details »

Introduction to Server Operations and Automations

Introduction to Server Operations and Automations

Hirotaka Nakajima

May 14, 2015
Tweet

More Decks by Hirotaka Nakajima

Other Decks in Technology

Transcript

  1. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    Server Operation & Automation
    SFC-RG Lecture

    14 May 2015

    Hirotaka Nakajima (@nunnun)

    View Slide

  2. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    Server Operations

    "Racks line" by Tristan Schmurr is licensed under CC BY 2.0

    View Slide

  3. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Way to build a web server
    • Purchase a server

    • Installation (Physically)

    • Install an Operating System

    • Configure an Operating System

    • Install Applications

    • Configure Application settings

    • Build a Web application

    • Done!!

    View Slide

  4. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Purchase
    • Design a specification

    • CPU

    • Architecture: x86_64? SPARC? ARM?

    • How many cores? CPUs?

    • Memory

    • ECC or Registered?

    • Disk requirement

    • Capacity

    • Reliability; RAID? which level?

    • Speed; SSD? SATA? SAS?

    • Network requirement

    • Interface

    • Budget and political issues

    View Slide

  5. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Installation
    • Mount a server to Rack

    • Cabling

    • Power backup plan
    "new space" by emmma peel is licensed under CC BY 2.0
    "Data Center" by Bob Mical is licensed under CC BY 2.0

    View Slide

  6. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Operating System
    • Which Operating System?

    • Linux? Windows?

    • Which Linux? Debian? Ubuntu? RedHat?

    • Version? Latest? Stable version?

    • Configuration

    • Hostname

    • Date & Time

    • Network

    • Disk initialization

    • User Management

    • Installation takes 1-2 hours

    View Slide

  7. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Application
    • Application

    • Which service; Web service? Mail service?

    • Which Application for Web service?

    • Apache? lighthttpd? nginx?

    • Configuration

    • Host

    • Which hostname is served with the service

    • Security

    • Using https? how about a server certificate?

    • Directory

    • Where web documents are located?

    View Slide

  8. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Content
    • Server-side Dynamic? Static?

    • Which Language & Runtime?

    • PHP, Perl, Java, C#, JavaScript….

    • Middle-ware?

    • WordPress etc…

    • Framework?

    • CakePHP, Symfony?

    • Client-side Dynamic? Static?

    • JavaScript Libraries

    • jQuery, Angular.js

    View Slide

  9. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    Done!!!

    View Slide

  10. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Am I all set and free to go?
    • NO!!!!

    • We need to do maintenance works

    • Maintenance work

    • OS update

    • Application, Middleware, Library update

    • Hardware failure

    • Security update

    View Slide

  11. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Documentation
    • What if we need to setup 100 servers 

    with same configuration?

    • Make a setup manual

    • How about changes?

    • Make a changelog

    • Are we sure 

    if we can deploy service to

    a server like this?
    [May 1] Server is delivered.
    [May 2] Debian version x is installed.
    Installation log is here
    [May 3] Apache is installed.
    Configurations are here
    [Jun 1] DNS Cache setting
    changed…
    ….
    [Jun 15] PHP settings changed…
    [Aug 10] Apache Configuration
    changed..

    View Slide

  12. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Undocumented Knowledge
    • Information not documented

    • Forget to be documented

    • Workaround not figured out why it work

    • They don’t know why but it works

    • “Secret Recipe” Issue / ൿ఻ͷλϨ໰୊

    • “Don’t touch the system if it’s running”

    View Slide

  13. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Security Update comes suddenly…
    • All software contains bugs

    • Some bugs may trigger a security
    incident

    • Vulnerability (੬ऑੑ)

    • We don’t know vulnerabilities
    beforehand

    • Once they publish, we need to
    handle

    • Apply a workaround

    • Apply a security update

    • What if we have 100++ servers?

    • Update all server by hand?
    LINE Icons are copyrights LINE corporation

    View Slide

  14. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    Virtualisation

    View Slide

  15. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    What is virtualisation?
    • “virtualization refers to the act of creating a
    virtual (rather than actual) version of something,
    including (but not limited to) a virtual computer
    hardware platform, operating system (OS),
    storage device, or computer network
    resources” from wikipedia

    • Hardware Virtualization

    • Operating System Virtualizaion

    View Slide

  16. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Hardware Virtualization
    • Virtualize server architecture on a server

    • Virtual Machines are packaged in files.

    • We don’t need to bind a specific server

    • We can move a virtual machine

    • Resource Optimization
    Hardware Virtualisation image from http://download.parallels.com/doc/psbm/v5/rtm/Parallels_Server_Bare_Metal_Users_Guide/29765.htm

    View Slide

  17. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    IaaS
    • Infrastructure as a Service

    • Provides computing infrastructure as an
    Internet service

    • Virtual Machine

    • Network

    • Storage

    • Amazon Web Services, Google Compute
    Engine

    • OpenStack, CloudStack

    View Slide

  18. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    VM Image & Template
    • Customized virtual machine

    • Install specific softwares, configurations

    • Easy for massive deployment

    • Deployment can be done by RESTful API

    • Server deployment becomes programmable

    View Slide

  19. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.
    Automation

    View Slide

  20. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Infrastructure as Code
    • IaaS achieves 

    program/system can operate infrastructure

    • We don’t need to do all by hand.

    • How about a configuration?

    • We can automate a server configuration

    • Configuration Management Tools

    • CFEngine, LCFG, BCFG

    • Apply configuration

    • Test if the configuration is current

    View Slide

  21. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Define what the server should be.
    • Manual deployment

    • 1. Install Apache, 2. Set the document root

    • What if somebody uninstalled a Apache?

    • Re-apply a entire installation step?

    • Define a state not a procedure

    • Apache should be installed

    • Document root should be “/var/www/htdocs”

    • Puppet, Chef

    View Slide

  22. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Idempotence / ႈ౳ੑ
    • Idempotence is the property of certain
    operations in mathematics and computer
    science, that can be applied multiple times
    without changing the result beyond the initial
    application.

    • ৘ใ޻ֶʹ͓͚Δႈ౳ͱ͸ɺ͋Δૢ࡞Λ1౓
    ߦͬͯ΋ෳ਺ճߦͬͯ΋ಉ͡ޮՌͱͳΔ͜ͱΛ
    ݴ͏ɻಛʹɺԿճߦͬͯ΋Τϥʔ΍ෆ੔߹ͷঢ়
    ଶ͕มΘΒͳ͍ૢ࡞Λࢦ͢ɻ

    View Slide

  23. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    What Puppet, Chef do?
    • Keep a machine to meet a defined state

    • Automated server configuration

    • Defined state

    • Completing an installation on abandoned server

    • Somebody abandoned an installation in a middle

    • Update a server configuration to current

    • Revert a temporary change to original configuration

    • Operating System independent configuration

    • With version control tools

    • Track changes on the infrastructure

    • Review the changes before applying to whole server

    View Slide

  24. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Puppet repository on GitHub

    View Slide

  25. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Security update with Automation
    • Automation tools can help deployment of
    security patches

    • Recipe example;

    • We don’t need to patch all servers by hand!!
    package {
    "openssl":
    ensure => "latest";
    "libssl":
    ensure => "latest";
    "unsafesoftware":
    ensure => "purged";
    }

    View Slide

  26. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Test driven Development
    • Puppet/Chef enables automated server configuration

    • How we assure the recipe applied to all servers?

    • SSH login and confirm?

    • In software development, we have test-driven
    development

    • Write a test case what function/program is expected to
    work

    • Write a function/program

    • Run a test if function/program works correctly

    • Let’s do a same thing!

    View Slide

  27. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Test driven Infrastructure
    • Test the infrastructure if it meets the test case

    • Apache is installed

    • OpenSSL is up-to-date

    • Test-driven Infrastructure

    • Write a test case of infrastructure

    • Write a recipe of automation tool

    • Apply the recipe to servers

    • Run a test case

    • Serverspec

    View Slide

  28. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Serverspec example
    require "spec_helper"
    describe 'nginx' do
    it { should be_installed }
    it { should be_enabled }
    it { should be_running }
    end
    describe 'port 443' do
    it { should be_listening }
    end
    describe '/etc/nginx/nginx.conf' do
    it { should be_file }
    it { should contain "server_name" blog.nunnun.jp }
    end

    View Slide

  29. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Old-fashioned software development
    • Code a module

    • Then merge modules and test

    • Most of the case, it fails!!

    • Someone will fix the issue

    • Someone will implement new feature

    • Then merge again, of course it will fail!
    MS Project image from http://projectsprofiler.com/images/blog/msfsw_1.PNG

    View Slide

  30. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Continuous Integration (CI)
    • Any changes must be tested with entire system.

    • If you find issues, contact other developer to fix
    the issue.

    View Slide

  31. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    CI on Infrastructure
    • Unable to test a recipe before applying

    • Sometimes untested recipe breaks a
    production server

    • Want to test my recipe before applying

    production server

    • Test a recipe before applying production
    environment with test environment

    • Test environment is build from scratch in every
    test using container technologies (Docker)
    # puppet agent --test
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Loading facts
    Info: Caching catalog for
    ldap0.kamoike.net
    Info: Applying configuration version
    '1431407810'
    Error: /Stage[main]/Kamoikeldap::Server/
    Openldap::Server::Dbindex[master_mdb_conf
    ig_entryCSN]/
    Openldap_dbindex[master_mdb_config_entryC
    SN]/ensure: change from absent to present
    failed: LDIF content:
    dn: olcDatabase={1}mdb,cn=config
    add: olcDbIndex
    olcDbIndex: entryCSN eq
    Error message: Execution of '/usr/bin/
    ldapmodify -Y EXTERNAL -H ldapi:/// -f /
    tmp/
    openldap_dbindex20150514-9204-17e4z8y'
    returned 20: SASL/EXTERNAL authentication
    started
    SASL username:
    gidNumber=0+uidNumber=0,cn=peercred,cn=ex
    ternal,cn=auth
    SASL SSF: 0
    ldap_modify: Type or value exists (20)
    additional info: modify/add:
    olcDbIndex: value #0 already exists
    modifying entry "olcDatabase={1}
    mdb,cn=config"
    Notice: Finished catalog run in 2.08
    seconds

    View Slide

  32. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Immutable / Disposable Infrastructure
    • Immutable

    มΘΒͳ͍ɺมԽ͠ͳ͍ɺෆมͷɺෆқͷɺม
    ͑Δ͜ͱͷͰ͖ͳ͍ɺมߋෆՄೳͳ

    • Disposable

    ࢖͍ࣺͯͷɺ؆୯ʹॲ෼Ͱ͖Δɺ࢖͍ࣺͯͰ͖
    Δ

    View Slide

  33. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Amazon’s example
    • Deploy 1,000 times per hour
    Image from http://www.publickey1.jp/blog/12/amazon11000_aws_reinventday2_am.html

    View Slide

  34. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Amazon’s example
    Image from http://www.publickey1.jp/blog/12/amazon11000_aws_reinventday2_am.html

    View Slide

  35. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    So…
    • If your Mac/PC is not working good,

    we sometimes reinstall the OS and setup again.

    • Same thing

    • Build an instance based on recipe,

    • If it works correctly, use it

    • If not, just keep using old instance

    View Slide

  36. Unless otherwise noted, the text of and illustrations in this slide are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License by Hirotaka Nakajima.
    Keio University Logo is licensed by its copyright holders.

    Conclusion
    • Focus on server operations and deployment

    • Old-fashioned deployment

    • Infrastructure as a Service

    • Infrastructure as Code

    • Test-driven Infrastructure

    • Continuous Integration on Infrastructure

    • Immutable / Disposable Infrastructure

    • Now we’re able to design and code an infrastructure

    • How about network?

    • It may be possible, but more difficult.

    • e.g. Route change continuously by external issues

    View Slide