Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Apps for Immutable Servers

Daniel Gomes
December 06, 2018

Building Apps for Immutable Servers

The deck title is the original one but I changed it to be more accurate with the content. So the new title is "Scaling Apps with Immutable Servers".

This talk was presented at SymfonyCon 2018 in Lisbon.

Daniel Gomes

December 06, 2018
Tweet

More Decks by Daniel Gomes

Other Decks in Programming

Transcript

  1. Building Apps
    for Immutable Servers
    Daniel Gomes
    1
    #SymfonyCon @_dcsg

    View Slide

  2. Building Apps
    for Immutable Servers
    Daniel Gomes
    2
    #SymfonyCon @_dcsg

    View Slide

  3. Scaling Apps
    with Immutable Servers
    Daniel Gomes
    3
    #SymfonyCon @_dcsg

    View Slide

  4. Disclaimer
    • No Code
    • Not going to build an App
    4
    #SymfonyCon @_dcsg

    View Slide

  5. About me
    Daniel Gomes
    Problem Solver @
    @phplx organizer
    Twitter: @_dcsg
    Blog: dcsg.me
    5
    #SymfonyCon @_dcsg

    View Slide

  6. 6
    • Different types of Scaling
    • Configuration Drift/Management
    • Different types of Servers
    • Build Process for your Web/Application Server
    Some Topics to Cover
    #SymfonyCon @_dcsg

    View Slide

  7. The Beginning
    7
    Photo by José Alejandro Cuffia on Unsplash

    View Slide

  8. Build the App
    8
    Photo by Charles Deluvio on Unsplash

    View Slide

  9. 1st Infrastructure Architecture
    9
    One Server
    to rule them all
    #SymfonyCon @_dcsg

    View Slide

  10. Server/Monolith
    10
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server
    #SymfonyCon @_dcsg

    View Slide

  11. Launch the App
    11
    Photo by NASA on Unsplash

    View Slide

  12. Users growth
    Users per hour
    0
    250
    500
    750
    1000
    April May June July August September
    12
    #SymfonyCon @_dcsg

    View Slide

  13. Server Unavailable
    https://http.cat/503
    13
    #SymfonyCon @_dcsg

    View Slide

  14. Server does not handle the load
    CPU Usage
    0
    50
    100
    150
    200
    10:00 10:10 10:20 10:30 10:40 10:50 11:00 11:10 11:20
    14
    #SymfonyCon @_dcsg

    View Slide

  15. We need to SCALE!!!
    15
    Photo by Linda Xu on Unsplash
    #SymfonyCon @_dcsg

    View Slide

  16. Easy!
    16
    Photo by Aziz Acharki on Unsplash

    View Slide

  17. Is that easy?
    https://martinfowler.com/bliki/PhoenixServer.html
    17

    View Slide

  18. Can we scale this?
    18
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server
    #SymfonyCon @_dcsg

    View Slide

  19. Can we scale this?
    19
    YES
    and
    NO
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server
    #SymfonyCon @_dcsg

    View Slide

  20. Vertical Scaling
    20
    #SymfonyCon @_dcsg

    View Slide

  21. More Memory
    21
    Vertical Scaling
    #SymfonyCon @_dcsg

    View Slide

  22. More CPU
    22
    Vertical Scaling
    #SymfonyCon @_dcsg

    View Slide

  23. More Disk
    23
    Vertical Scaling
    #SymfonyCon @_dcsg

    View Slide

  24. Fine tuning
    PHP | PHP-FPM
    NGINX | Apache
    24
    Vertical Scaling
    #SymfonyCon @_dcsg

    View Slide

  25. Horizontal Scaling
    25
    #SymfonyCon @_dcsg

    View Slide

  26. Horizontal Scaling
    26
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Instance
    4 Instances
    Current Goal
    #SymfonyCon @_dcsg

    View Slide

  27. Horizontal Scaling
    27
    Load balancer
    Cluster
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 1
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 2
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 3
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 4
    #SymfonyCon @_dcsg

    View Slide

  28. Horizontal Scaling
    28
    Load balancer
    Cluster
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 1
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 2
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 3
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 4
    Will this work?
    #SymfonyCon @_dcsg

    View Slide

  29. Horizontal Scaling
    29
    Load balancer
    Cluster
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 1
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 2
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 3
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 4
    Will this work?
    NO
    #SymfonyCon @_dcsg

    View Slide

  30. Why?
    30
    #SymfonyCon @_dcsg

    View Slide

  31. Persistence and
    State
    31
    Horizontal Scaling
    #SymfonyCon @_dcsg

    View Slide

  32. The “new” Architecture
    32
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server
    Load Balancer
    Application Cluster
    AMQP Cluster
    DB Cluster
    CDN/Assets
    #SymfonyCon @_dcsg

    View Slide

  33. Our new Application Cluster
    33
    Load Balancer
    Application Cluster
    AMQP Cluster
    DB Cluster
    CDN/Assets
    #SymfonyCon @_dcsg

    View Slide

  34. But… problems happen!
    https://http.cat/500
    34
    #SymfonyCon @_dcsg

    View Slide

  35. SSH’s into production machine and live fixes are done
    35
    Photo by Markus Spiske on Unsplash
    #SymfonyCon @_dcsg

    View Slide

  36. Problem
    Solved
    36
    Photo by Aziz Acharki on Unsplash

    View Slide

  37. Or not?
    37
    #SymfonyCon @_dcsg

    View Slide

  38. The new cluster state
    Fixed Instance
    Instances

    not in same state
    38
    #SymfonyCon @_dcsg

    View Slide

  39. Configuration Drift
    39
    #SymfonyCon @_dcsg

    View Slide

  40. Configuration Drift
    40
    #SymfonyCon @_dcsg

    View Slide

  41. Configuration Drift
    • Manual ad-hoc changes and updates to servers that
    are not recorded
    40
    #SymfonyCon @_dcsg

    View Slide

  42. Configuration Drift
    • Manual ad-hoc changes and updates to servers that
    are not recorded
    • Servers in your infrastructure became more and more
    different from each others
    40
    #SymfonyCon @_dcsg

    View Slide

  43. The cluster state
    after a few weeks
    State A
    State C
    State B
    State B
    41
    #SymfonyCon @_dcsg

    View Slide

  44. Snowflake Server
    42
    #SymfonyCon @_dcsg

    View Slide

  45. Long running Servers
    43
    Snowflake Servers
    #SymfonyCon @_dcsg

    View Slide

  46. No consistency
    between Servers
    44
    Snowflake Servers
    #SymfonyCon @_dcsg

    View Slide

  47. Hard to reproduce
    45
    Snowflake Servers
    #SymfonyCon @_dcsg

    View Slide

  48. Lack of Confidence
    46
    Snowflake Servers
    #SymfonyCon @_dcsg

    View Slide

  49. How can we solve this?
    47
    Snowflake Servers
    #SymfonyCon @_dcsg

    View Slide

  50. Configuration Management
    with Automation Tools
    48
    #SymfonyCon @_dcsg

    View Slide

  51. Configuration Management
    The process of systematically handling
    changes to a system in a way that it
    maintains integrity over time.
    49
    #SymfonyCon @_dcsg

    View Slide

  52. Configuration Management
    Spin up
    Base Image
    Run
    Config Management
    Server
    in desired state
    Changes
    Edit config file
    Upgrade/install package

    Running Server
    50
    #SymfonyCon @_dcsg

    View Slide

  53. Applying
    to the cluster state
    State A
    State C
    State B
    State B
    Config
    Management
    Apply
    State D
    51
    #SymfonyCon @_dcsg

    View Slide

  54. The cluster state
    after CM ran
    State D
    State D
    State D
    State D
    52
    #SymfonyCon @_dcsg

    View Slide

  55. Automation Tools for CM
    53
    #SymfonyCon @_dcsg

    View Slide

  56. Quick recap
    54
    #SymfonyCon @_dcsg

    View Slide

  57. Quick recap
    • Vertical & Horizontal Scaling
    54
    #SymfonyCon @_dcsg

    View Slide

  58. Quick recap
    • Vertical & Horizontal Scaling
    • Configuration Drift
    54
    #SymfonyCon @_dcsg

    View Slide

  59. Quick recap
    • Vertical & Horizontal Scaling
    • Configuration Drift
    • Problems of the Snowflake Servers
    54
    #SymfonyCon @_dcsg

    View Slide

  60. Quick recap
    • Vertical & Horizontal Scaling
    • Configuration Drift
    • Problems of the Snowflake Servers
    • Configuration Management
    54
    #SymfonyCon @_dcsg

    View Slide

  61. Phoenix Servers
    https://martinfowler.com/bliki/PhoenixServer.html
    55

    View Slide

  62. “A server should be like a
    phoenix, regularly rising
    from the ashes.”
    - Martin Fowler in PhoenixServer
    56
    #SymfonyCon @_dcsg

    View Slide

  63. Phoenix Servers
    • Avoid configuration drifts
    • Disposable servers
    • Servers can be built from scratch
    57
    #SymfonyCon @_dcsg

    View Slide

  64. The cluster
    Re-launch an instance
    State D
    State D State D
    State D
    Terminate 

    Instance
    58
    #SymfonyCon @_dcsg

    View Slide

  65. The cluster
    Re-launch an instance
    State D
    State D State D
    58
    #SymfonyCon @_dcsg

    View Slide

  66. The cluster
    Re-launch an instance
    State D
    State D State D
    Config
    Management
    58
    #SymfonyCon @_dcsg

    View Slide

  67. The cluster
    Re-launch an instance
    State D
    State D State D
    Config
    Management
    58
    #SymfonyCon @_dcsg

    View Slide

  68. The cluster
    Re-launch an instance
    State D
    State D State D
    Config
    Management
    Apply
    State D
    58
    #SymfonyCon @_dcsg

    View Slide

  69. The cluster
    Re-launch an instance
    State D
    State D State D
    Config
    Management
    Apply
    State D
    58
    #SymfonyCon @_dcsg

    View Slide

  70. The cluster
    Re-launch an instance
    State D
    State D State D
    Config
    Management
    Apply
    State D
    State D
    58
    #SymfonyCon @_dcsg

    View Slide

  71. Spinning up new servers
    is not a problem anymore!
    59
    #SymfonyCon @_dcsg

    View Slide

  72. But…
    60
    #SymfonyCon @_dcsg

    View Slide

  73. Is Idempotence guaranteed?
    61
    #SymfonyCon @_dcsg

    View Slide

  74. What if the packages
    repositories are down?
    62
    #SymfonyCon @_dcsg

    View Slide

  75. Built Process
    Spin up
    Base Image Run
    Config Management
    Server
    in desire state
    Install packages
    Create folders
    Create user
    Upload app
    etc
    Run
    Config Management
    63
    #SymfonyCon @_dcsg

    View Slide

  76. Built Process
    Spin up
    Base Image Run
    Config Management
    Server
    in desire state
    Install packages
    Create folders
    Create user
    Upload app
    etc
    Run
    Config Management
    Repositories
    unavailable
    63
    #SymfonyCon @_dcsg

    View Slide

  77. Built Process
    Spin up
    Base Image Run
    Config Management
    Server
    in desire state
    Install packages
    Create folders
    Create user
    Upload app
    etc
    Run
    Config Management
    Repositories
    unavailable
    63
    #SymfonyCon @_dcsg

    View Slide

  78. How can we fix this?
    64
    #SymfonyCon @_dcsg

    View Slide

  79. Immutable Servers
    https://martinfowler.com/bliki/ImmutableServer.html
    65

    View Slide

  80. "An Immutable Server is a
    server, that once deployed, is
    never modified, merely
    replaced with a new updated
    instance."
    - Kief Morris in ImmutableServer
    66
    #SymfonyCon @_dcsg

    View Slide

  81. Final image
    with everything baked in.
    67
    #SymfonyCon @_dcsg

    View Slide

  82. No changes after it’s built
    68
    #SymfonyCon @_dcsg

    View Slide

  83. Include scripts to start
    the Application on boot.
    69
    #SymfonyCon @_dcsg

    View Slide

  84. Easy to scale-out,
    deploy, and rollback
    70
    #SymfonyCon @_dcsg

    View Slide

  85. Therefore…
    71
    #SymfonyCon @_dcsg

    View Slide

  86. Confidence
    72
    #SymfonyCon @_dcsg

    View Slide

  87. Reliability
    73
    #SymfonyCon @_dcsg

    View Slide

  88. Trust
    74
    #SymfonyCon @_dcsg

    View Slide

  89. Sleep quietly
    75
    Photo by Lauren Kay on Unsplash

    View Slide

  90. And also…
    76
    #SymfonyCon @_dcsg

    View Slide

  91. The Twelve-Factor App
    77
    https://12factor.net/
    #SymfonyCon @_dcsg

    View Slide

  92. Reproducible Builds
    78
    https://reproducible-builds.org/docs/definition/
    #SymfonyCon @_dcsg

    View Slide

  93. How do we Build it?
    79
    #SymfonyCon @_dcsg

    View Slide

  94. CI/CD Pipelines
    80
    #SymfonyCon @_dcsg

    View Slide

  95. Build !== Run
    81
    #SymfonyCon @_dcsg

    View Slide

  96. Immutable Server Build Process
    Spin up
    Base Image
    82
    #SymfonyCon @_dcsg

    View Slide

  97. Immutable Server Build Process
    Spin up
    Base Image
    Run
    Config Management
    (puppet, chef, ansible)
    82
    #SymfonyCon @_dcsg

    View Slide

  98. Immutable Server Build Process
    Server
    in desire state
    configure application

    environment
    Spin up
    Base Image
    Run
    Config Management
    (puppet, chef, ansible)
    82
    #SymfonyCon @_dcsg

    View Slide

  99. Immutable Server Build Process
    Bake In
    the App
    Server
    in desire state
    configure application

    environment
    Spin up
    Base Image
    Run
    Config Management
    (puppet, chef, ansible)
    82
    #SymfonyCon @_dcsg

    View Slide

  100. Immutable Server Build Process
    Bake In
    the App
    App
    Final Image
    Server
    in desire state
    configure application

    environment
    Spin up
    Base Image
    Run
    Config Management
    (puppet, chef, ansible)
    82
    #SymfonyCon @_dcsg

    View Slide

  101. Build Image Stages Flow Example
    Base Image
    OS Hardening
    Common tools

    (vim, htop, etc)
    etc
    83

    View Slide

  102. Build Image Stages Flow Example
    Base Image
    OS Hardening
    Common tools

    (vim, htop, etc)
    etc
    Application
    Base Image
    Install necessary
    software to run the App
    Create user/folders
    83

    View Slide

  103. Build Image Stages Flow Example
    Base Image
    OS Hardening
    Common tools

    (vim, htop, etc)
    etc
    Application
    Base Image
    Install necessary
    software to run the App
    Create user/folders
    Application
    Final Image
    Upload App
    Script to run App at boot
    Composer Install
    Clear Cache
    83

    View Slide

  104. Build Image Stages Flow Example
    Base Image
    OS Hardening
    Common tools

    (vim, htop, etc)
    etc
    Application
    Base Image
    Install necessary
    software to run the App
    Create user/folders
    Application
    Final Image
    Upload App
    Script to run App at boot
    Composer Install
    Clear Cache
    System upgrades &
    Security updates
    83

    View Slide

  105. Build Image Stages Flow Example
    Base Image
    OS Hardening
    Common tools

    (vim, htop, etc)
    etc
    Application
    Base Image
    Install necessary
    software to run the App
    Create user/folders
    Application
    Final Image
    Upload App
    Script to run App at boot
    Composer Install
    Clear Cache
    System upgrades &
    Security updates
    Application security,
    package, configuration
    updates
    83

    View Slide

  106. Building Tips for a Symfony App
    • composer install --no-dev --optimize-autoloader
    • php bin/console cache:clear --env=prod —no-debug
    • Clearing your APC cache
    • Build or pushing Assets to CDN
    84
    #SymfonyCon @_dcsg

    View Slide

  107. Build Tools
    85

    View Slide

  108. Next Steps
    86
    #SymfonyCon @_dcsg

    View Slide

  109. High-Availability
    • Auto-scaling
    • Web Tier
    • Application Services Tier
    • Backing Services Tier (DB, AMQP, etc)
    • Chaos Monkey
    • Chaos Kong
    87
    #SymfonyCon @_dcsg

    View Slide

  110. Final Notes
    88
    #SymfonyCon @_dcsg

    View Slide

  111. Prepare your Application
    for Immutable Servers
    89
    #SymfonyCon @_dcsg

    View Slide

  112. But only use Immutable Servers
    when you REALLY NEED!
    90
    #SymfonyCon @_dcsg

    View Slide

  113. Resources
    • https://martinfowler.com - Martin Fowler
    • https://medium.com/netflix-techblog/chaos-
    engineering-upgraded-878d341f15fa
    • https://symfony.com/blog/new-in-symfony-
    reproducible-builds
    91
    #SymfonyCon @_dcsg

    View Slide

  114. Questions
    Twitter: @_dcsg

    Blog: dcsg.me
    ?
    we are hiring!
    92
    #SymfonyCon @_dcsg

    View Slide

  115. Thanks!
    93
    #SymfonyCon @_dcsg

    View Slide