Building Apps for Immutable Servers

The deck title is the original one but I changed it to be more accurate with the content. So the new title is "Scaling Apps with Immutable Servers".

This talk was presented at SymfonyCon 2018 in Lisbon.

Transcript

1. Building Apps
   for Immutable Servers
   Daniel Gomes
   1
   #SymfonyCon @_dcsg
   

   View Slide

2. Building Apps
   for Immutable Servers
   Daniel Gomes
   2
   #SymfonyCon @_dcsg
   

   View Slide

3. Scaling Apps
   with Immutable Servers
   Daniel Gomes
   3
   #SymfonyCon @_dcsg
   

   View Slide

4. Disclaimer
   • No Code
   • Not going to build an App
   4
   #SymfonyCon @_dcsg
   

   View Slide

5. About me
   Daniel Gomes
   Problem Solver @
   @phplx organizer
   Twitter: @_dcsg
   Blog: dcsg.me
   5
   #SymfonyCon @_dcsg
   

   View Slide

6. 6
   • Different types of Scaling
   • Configuration Drift/Management
   • Different types of Servers
   • Build Process for your Web/Application Server
   Some Topics to Cover
   #SymfonyCon @_dcsg
   

   View Slide

7. The Beginning
   7
   Photo by José Alejandro Cuffia on Unsplash
   

   View Slide

8. Build the App
   8
   Photo by Charles Deluvio on Unsplash
   

   View Slide

9. 1st Infrastructure Architecture
   9
   One Server
   to rule them all
   #SymfonyCon @_dcsg
   

   View Slide

10. Server/Monolith
    10
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server
    #SymfonyCon @_dcsg
    

    View Slide

11. Launch the App
    11
    Photo by NASA on Unsplash
    

    View Slide

12. Users growth
    Users per hour
    0
    250
    500
    750
    1000
    April May June July August September
    12
    #SymfonyCon @_dcsg
    

    View Slide

13. Server Unavailable
    https://http.cat/503
    13
    #SymfonyCon @_dcsg
    

    View Slide

14. Server does not handle the load
    CPU Usage
    0
    50
    100
    150
    200
    10:00 10:10 10:20 10:30 10:40 10:50 11:00 11:10 11:20
    14
    #SymfonyCon @_dcsg
    

    View Slide

15. We need to SCALE!!!
    15
    Photo by Linda Xu on Unsplash
    #SymfonyCon @_dcsg
    

    View Slide

16. Easy!
    16
    Photo by Aziz Acharki on Unsplash
    

    View Slide

17. Is that easy?
    https://martinfowler.com/bliki/PhoenixServer.html
    17
    

    View Slide

18. Can we scale this?
    18
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server
    #SymfonyCon @_dcsg
    

    View Slide

19. Can we scale this?
    19
    YES
    and
    NO
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server
    #SymfonyCon @_dcsg
    

    View Slide

20. Vertical Scaling
    20
    #SymfonyCon @_dcsg
    

    View Slide

21. More Memory
    21
    Vertical Scaling
    #SymfonyCon @_dcsg
    

    View Slide

22. More CPU
    22
    Vertical Scaling
    #SymfonyCon @_dcsg
    

    View Slide

23. More Disk
    23
    Vertical Scaling
    #SymfonyCon @_dcsg
    

    View Slide

24. Fine tuning
    PHP | PHP-FPM
    NGINX | Apache
    24
    Vertical Scaling
    #SymfonyCon @_dcsg
    

    View Slide

25. Horizontal Scaling
    25
    #SymfonyCon @_dcsg
    

    View Slide

26. Horizontal Scaling
    26
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Instance
    4 Instances
    Current Goal
    #SymfonyCon @_dcsg
    

    View Slide

27. Horizontal Scaling
    27
    Load balancer
    Cluster
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 1
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 2
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 3
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 4
    #SymfonyCon @_dcsg
    

    View Slide

28. Horizontal Scaling
    28
    Load balancer
    Cluster
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 1
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 2
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 3
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 4
    Will this work?
    #SymfonyCon @_dcsg
    

    View Slide

29. Horizontal Scaling
    29
    Load balancer
    Cluster
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 1
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 2
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 3
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server 4
    Will this work?
    NO
    #SymfonyCon @_dcsg
    

    View Slide

30. Why?
    30
    #SymfonyCon @_dcsg
    

    View Slide

31. Persistence and
    State
    31
    Horizontal Scaling
    #SymfonyCon @_dcsg
    

    View Slide

32. The “new” Architecture
    32
    Database
    Cron Jobs
    Queues
    Cache
    Application
    Server
    Load Balancer
    Application Cluster
    AMQP Cluster
    DB Cluster
    CDN/Assets
    #SymfonyCon @_dcsg
    

    View Slide

33. Our new Application Cluster
    33
    Load Balancer
    Application Cluster
    AMQP Cluster
    DB Cluster
    CDN/Assets
    #SymfonyCon @_dcsg
    

    View Slide

34. But… problems happen!
    https://http.cat/500
    34
    #SymfonyCon @_dcsg
    

    View Slide

35. SSH’s into production machine and live fixes are done
    35
    Photo by Markus Spiske on Unsplash
    #SymfonyCon @_dcsg
    

    View Slide

36. Problem
    Solved
    36
    Photo by Aziz Acharki on Unsplash
    

    View Slide

37. Or not?
    37
    #SymfonyCon @_dcsg
    

    View Slide

38. The new cluster state
    Fixed Instance
    Instances

    not in same state
    38
    #SymfonyCon @_dcsg
    

    View Slide

39. Configuration Drift
    39
    #SymfonyCon @_dcsg
    

    View Slide

40. Configuration Drift
    40
    #SymfonyCon @_dcsg
    

    View Slide

41. Configuration Drift
    • Manual ad-hoc changes and updates to servers that
    are not recorded
    40
    #SymfonyCon @_dcsg
    

    View Slide

42. Configuration Drift
    • Manual ad-hoc changes and updates to servers that
    are not recorded
    • Servers in your infrastructure became more and more
    different from each others
    40
    #SymfonyCon @_dcsg
    

    View Slide

43. The cluster state
    after a few weeks
    State A
    State C
    State B
    State B
    41
    #SymfonyCon @_dcsg
    

    View Slide

44. Snowflake Server
    42
    #SymfonyCon @_dcsg
    

    View Slide

45. Long running Servers
    43
    Snowflake Servers
    #SymfonyCon @_dcsg
    

    View Slide

46. No consistency
    between Servers
    44
    Snowflake Servers
    #SymfonyCon @_dcsg
    

    View Slide

47. Hard to reproduce
    45
    Snowflake Servers
    #SymfonyCon @_dcsg
    

    View Slide

48. Lack of Confidence
    46
    Snowflake Servers
    #SymfonyCon @_dcsg
    

    View Slide

49. How can we solve this?
    47
    Snowflake Servers
    #SymfonyCon @_dcsg
    

    View Slide

50. Configuration Management
    with Automation Tools
    48
    #SymfonyCon @_dcsg
    

    View Slide

51. Configuration Management
    The process of systematically handling
    changes to a system in a way that it
    maintains integrity over time.
    49
    #SymfonyCon @_dcsg
    

    View Slide

52. Configuration Management
    Spin up
    Base Image
    Run
    Config Management
    Server
    in desired state
    Changes
    Edit config file
    Upgrade/install package
    …
    Running Server
    50
    #SymfonyCon @_dcsg
    

    View Slide

53. Applying
    to the cluster state
    State A
    State C
    State B
    State B
    Config
    Management
    Apply
    State D
    51
    #SymfonyCon @_dcsg
    

    View Slide

54. The cluster state
    after CM ran
    State D
    State D
    State D
    State D
    52
    #SymfonyCon @_dcsg
    

    View Slide

55. Automation Tools for CM
    53
    #SymfonyCon @_dcsg
    

    View Slide

56. Quick recap
    54
    #SymfonyCon @_dcsg
    

    View Slide

57. Quick recap
    • Vertical & Horizontal Scaling
    54
    #SymfonyCon @_dcsg
    

    View Slide

58. Quick recap
    • Vertical & Horizontal Scaling
    • Configuration Drift
    54
    #SymfonyCon @_dcsg
    

    View Slide

59. Quick recap
    • Vertical & Horizontal Scaling
    • Configuration Drift
    • Problems of the Snowflake Servers
    54
    #SymfonyCon @_dcsg
    

    View Slide

60. Quick recap
    • Vertical & Horizontal Scaling
    • Configuration Drift
    • Problems of the Snowflake Servers
    • Configuration Management
    54
    #SymfonyCon @_dcsg
    

    View Slide

61. Phoenix Servers
    https://martinfowler.com/bliki/PhoenixServer.html
    55
    

    View Slide

62. “A server should be like a
    phoenix, regularly rising
    from the ashes.”
    - Martin Fowler in PhoenixServer
    56
    #SymfonyCon @_dcsg
    

    View Slide

63. Phoenix Servers
    • Avoid configuration drifts
    • Disposable servers
    • Servers can be built from scratch
    57
    #SymfonyCon @_dcsg
    

    View Slide

64. The cluster
    Re-launch an instance
    State D
    State D State D
    State D
    Terminate 

    Instance
    58
    #SymfonyCon @_dcsg
    

    View Slide

65. The cluster
    Re-launch an instance
    State D
    State D State D
    58
    #SymfonyCon @_dcsg
    

    View Slide

66. The cluster
    Re-launch an instance
    State D
    State D State D
    Config
    Management
    58
    #SymfonyCon @_dcsg
    

    View Slide

67. The cluster
    Re-launch an instance
    State D
    State D State D
    Config
    Management
    58
    #SymfonyCon @_dcsg
    

    View Slide

68. The cluster
    Re-launch an instance
    State D
    State D State D
    Config
    Management
    Apply
    State D
    58
    #SymfonyCon @_dcsg
    

    View Slide

69. The cluster
    Re-launch an instance
    State D
    State D State D
    Config
    Management
    Apply
    State D
    58
    #SymfonyCon @_dcsg
    

    View Slide

70. The cluster
    Re-launch an instance
    State D
    State D State D
    Config
    Management
    Apply
    State D
    State D
    58
    #SymfonyCon @_dcsg
    

    View Slide

71. Spinning up new servers
    is not a problem anymore!
    59
    #SymfonyCon @_dcsg
    

    View Slide

72. But…
    60
    #SymfonyCon @_dcsg
    

    View Slide

73. Is Idempotence guaranteed?
    61
    #SymfonyCon @_dcsg
    

    View Slide

74. What if the packages
    repositories are down?
    62
    #SymfonyCon @_dcsg
    

    View Slide

75. Built Process
    Spin up
    Base Image Run
    Config Management
    Server
    in desire state
    Install packages
    Create folders
    Create user
    Upload app
    etc
    Run
    Config Management
    63
    #SymfonyCon @_dcsg
    

    View Slide

76. Built Process
    Spin up
    Base Image Run
    Config Management
    Server
    in desire state
    Install packages
    Create folders
    Create user
    Upload app
    etc
    Run
    Config Management
    Repositories
    unavailable
    63
    #SymfonyCon @_dcsg
    

    View Slide

77. Built Process
    Spin up
    Base Image Run
    Config Management
    Server
    in desire state
    Install packages
    Create folders
    Create user
    Upload app
    etc
    Run
    Config Management
    Repositories
    unavailable
    63
    #SymfonyCon @_dcsg
    

    View Slide

78. How can we fix this?
    64
    #SymfonyCon @_dcsg
    

    View Slide

79. Immutable Servers
    https://martinfowler.com/bliki/ImmutableServer.html
    65
    

    View Slide

80. "An Immutable Server is a
    server, that once deployed, is
    never modified, merely
    replaced with a new updated
    instance."
    - Kief Morris in ImmutableServer
    66
    #SymfonyCon @_dcsg
    

    View Slide

81. Final image
    with everything baked in.
    67
    #SymfonyCon @_dcsg
    

    View Slide

82. No changes after it’s built
    68
    #SymfonyCon @_dcsg
    

    View Slide

83. Include scripts to start
    the Application on boot.
    69
    #SymfonyCon @_dcsg
    

    View Slide

84. Easy to scale-out,
    deploy, and rollback
    70
    #SymfonyCon @_dcsg
    

    View Slide

85. Therefore…
    71
    #SymfonyCon @_dcsg
    

    View Slide

86. Confidence
    72
    #SymfonyCon @_dcsg
    

    View Slide

87. Reliability
    73
    #SymfonyCon @_dcsg
    

    View Slide

88. Trust
    74
    #SymfonyCon @_dcsg
    

    View Slide

89. Sleep quietly
    75
    Photo by Lauren Kay on Unsplash
    

    View Slide

90. And also…
    76
    #SymfonyCon @_dcsg
    

    View Slide

91. The Twelve-Factor App
    77
    https://12factor.net/
    #SymfonyCon @_dcsg
    

    View Slide

92. Reproducible Builds
    78
    https://reproducible-builds.org/docs/definition/
    #SymfonyCon @_dcsg
    

    View Slide

93. How do we Build it?
    79
    #SymfonyCon @_dcsg
    

    View Slide

94. CI/CD Pipelines
    80
    #SymfonyCon @_dcsg
    

    View Slide

95. Build !== Run
    81
    #SymfonyCon @_dcsg
    

    View Slide

96. Immutable Server Build Process
    Spin up
    Base Image
    82
    #SymfonyCon @_dcsg
    

    View Slide

97. Immutable Server Build Process
    Spin up
    Base Image
    Run
    Config Management
    (puppet, chef, ansible)
    82
    #SymfonyCon @_dcsg
    

    View Slide

98. Immutable Server Build Process
    Server
    in desire state
    configure application

    environment
    Spin up
    Base Image
    Run
    Config Management
    (puppet, chef, ansible)
    82
    #SymfonyCon @_dcsg
    

    View Slide

99. Immutable Server Build Process
    Bake In
    the App
    Server
    in desire state
    configure application

    environment
    Spin up
    Base Image
    Run
    Config Management
    (puppet, chef, ansible)
    82
    #SymfonyCon @_dcsg
    

    View Slide

100. Immutable Server Build Process
     Bake In
     the App
     App
     Final Image
     Server
     in desire state
     configure application

     environment
     Spin up
     Base Image
     Run
     Config Management
     (puppet, chef, ansible)
     82
     #SymfonyCon @_dcsg
     

     View Slide

101. Build Image Stages Flow Example
     Base Image
     OS Hardening
     Common tools

     (vim, htop, etc)
     etc
     83
     

     View Slide

102. Build Image Stages Flow Example
     Base Image
     OS Hardening
     Common tools

     (vim, htop, etc)
     etc
     Application
     Base Image
     Install necessary
     software to run the App
     Create user/folders
     83
     

     View Slide

103. Build Image Stages Flow Example
     Base Image
     OS Hardening
     Common tools

     (vim, htop, etc)
     etc
     Application
     Base Image
     Install necessary
     software to run the App
     Create user/folders
     Application
     Final Image
     Upload App
     Script to run App at boot
     Composer Install
     Clear Cache
     83
     

     View Slide

104. Build Image Stages Flow Example
     Base Image
     OS Hardening
     Common tools

     (vim, htop, etc)
     etc
     Application
     Base Image
     Install necessary
     software to run the App
     Create user/folders
     Application
     Final Image
     Upload App
     Script to run App at boot
     Composer Install
     Clear Cache
     System upgrades &
     Security updates
     83
     

     View Slide

105. Build Image Stages Flow Example
     Base Image
     OS Hardening
     Common tools

     (vim, htop, etc)
     etc
     Application
     Base Image
     Install necessary
     software to run the App
     Create user/folders
     Application
     Final Image
     Upload App
     Script to run App at boot
     Composer Install
     Clear Cache
     System upgrades &
     Security updates
     Application security,
     package, configuration
     updates
     83
     

     View Slide

106. Building Tips for a Symfony App
     • composer install --no-dev --optimize-autoloader
     • php bin/console cache:clear --env=prod —no-debug
     • Clearing your APC cache
     • Build or pushing Assets to CDN
     84
     #SymfonyCon @_dcsg
     

     View Slide

107. Build Tools
     85
     

     View Slide

108. Next Steps
     86
     #SymfonyCon @_dcsg
     

     View Slide

109. High-Availability
     • Auto-scaling
     • Web Tier
     • Application Services Tier
     • Backing Services Tier (DB, AMQP, etc)
     • Chaos Monkey
     • Chaos Kong
     87
     #SymfonyCon @_dcsg
     

     View Slide

110. Final Notes
     88
     #SymfonyCon @_dcsg
     

     View Slide

111. Prepare your Application
     for Immutable Servers
     89
     #SymfonyCon @_dcsg
     

     View Slide

112. But only use Immutable Servers
     when you REALLY NEED!
     90
     #SymfonyCon @_dcsg
     

     View Slide

113. Resources
     • https://martinfowler.com - Martin Fowler
     • https://medium.com/netflix-techblog/chaos-
     engineering-upgraded-878d341f15fa
     • https://symfony.com/blog/new-in-symfony-
     reproducible-builds
     91
     #SymfonyCon @_dcsg
     

     View Slide

114. Questions
     Twitter: @_dcsg

     Blog: dcsg.me
     ?
     we are hiring!
     92
     #SymfonyCon @_dcsg
     

     View Slide

115. Thanks!
     93
     #SymfonyCon @_dcsg
     

     View Slide