Upgrade to Pro — share decks privately, control downloads, hide ads and more …

25+ million members in 22 countries, how to scale with Symfony2

25+ million members in 22 countries, how to scale with Symfony2

Talk given with Benjamin De Bernardi (@genes0r) at PHP Frameworks day in Kiev.

Olivier Dolbeau

September 03, 2016
Tweet

More Decks by Olivier Dolbeau

Other Decks in Programming

Transcript

  1. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    25+ million members in 22 countries;
    How to scale with Symfony2?
    Benjamin De Bernardi - Olivier Dolbeau

    View full-size slide

  2. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    @genes0r
    Lead Engineer
    Benjamin De Bernardi

    View full-size slide

  3. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Olivier Dolbeau
    @odolbeau
    Web Architect

    View full-size slide

  4. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    100€ 25€ 25€ 25€
    100€ 25€

    View full-size slide

  5. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    15M
    20M
    10M
    5M
    1M
    2015
    2014
    2013
    2012
    2011
    2010
    2009
    25M
    members worldwide
    Over 25 million members and growing
    1.5 million+
    joining each month

    View full-size slide

  6. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r

    View full-size slide

  7. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    1. The past
    2. Why symfony?
    3. How?
    ○ How to deal with 22 countries?
    ○ Why do we love the EventDispatcher?
    ○ Performance & asynchronous
    ○ Doctrine
    4. Traps
    5. Here & now
    Talk overview:

    View full-size slide

  8. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    The past
    1

    View full-size slide

  9. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    v1.0
    2005-2006

    View full-size slide

  10. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r

    View full-size slide

  11. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    v2.0
    2008-2011
    (2014)

    View full-size slide

  12. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r

    View full-size slide

  13. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    2010
    New countries
    2011 2012

    View full-size slide

  14. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r

    View full-size slide

  15. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    3 770 lines of code
    69 functions
    926 lines for the longest one.
    lib.trip.php

    View full-size slide

  16. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Homemade framework

    View full-size slide

  17. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    NO TESTS

    View full-size slide

  18. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    v3.0
    2011 - today

    View full-size slide

  19. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r

    View full-size slide

  20. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    2012 2013 2014 2015
    New countries

    View full-size slide

  21. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Why Symfony?
    2

    View full-size slide

  22. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Huge community.

    View full-size slide

  23. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r

    View full-size slide

  24. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Attract new people

    View full-size slide

  25. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    How?
    3

    View full-size slide

  26. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    22 countries, how can we manage
    specificities?

    View full-size slide

  27. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    ➔ One codebase for each country?
    ➔ Conditions everywhere in the code?
    ➔ Configuration and Context?
    22 countries: how can we manage specificities?

    View full-size slide

  28. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    ➔ One codebase for each country?
    ➔ Conditions everywhere in the code?
    ➔ Configuration and Context?
    22 countries: how can we manage specificities?

    View full-size slide

  29. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Web request Listener
    Symfony parse & build a
    Request object
    Build Context from
    configuration and Request
    Response
    22 countries: Configuration & Context

    View full-size slide

  30. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    1 file for each country
    100+ configuration keys
    Inheritance
    Toggling of features
    Metadata
    22 countries: Configuration & Context
    Configuration:

    View full-size slide

  31. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    22 countries: Configuration & Context

    View full-size slide

  32. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    22 countries: Configuration & Context
    Get config from a controller:

    View full-size slide

  33. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    22 countries: Configuration & Context
    Get config from a controller:

    View full-size slide

  34. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    22 countries: Configuration & Context
    Get config from the view:

    View full-size slide

  35. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    22 countries: Configuration & Context

    View full-size slide

  36. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Lot of code, how to keep events clean?

    View full-size slide

  37. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    ➔ Symfony built-in EventDispatcher
    Lot of code, how to keep events clean?

    View full-size slide

  38. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    EventDispatcher how it works:
    Lot of code, how to keep events clean?
    Event Dispatcher
    Event
    Listener
    Listener

    View full-size slide

  39. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    User A rates User B:
    Lot of code, how to keep events clean?
    Event Dispatcher
    New
    Rating

    View full-size slide

  40. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    User A rates User B:
    Lot of code, how to keep events clean?
    Event Dispatcher
    New
    Rating
    Grade
    Listener
    Recalculate the grade
    of both users

    View full-size slide

  41. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    User A rates User B:
    Lot of code, how to keep events clean?
    Event Dispatcher
    New
    Rating
    Grade
    Listener
    Moderation
    Listener
    Ask for content
    verification

    View full-size slide

  42. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Lot of code, how to keep events clean?
    Sending the event:
    ● Sent from anywhere

    View full-size slide

  43. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Lot of code, how to keep events clean?
    Registering a listener:
    ● Update a configuration file

    View full-size slide

  44. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Lot of code, how to keep events clean?
    Listening the event:
    ● Inside a registered listener

    View full-size slide

  45. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    + Perfect to decouple code inside the app.
    + Small listeners, easy to understand, to test.
    + Easy to dispatch events outside application.
    - Sometimes, hard to understand the flow.
    - Need tools/rules to keep a common logic.
    Dispatching events Pros/Cons

    View full-size slide

  46. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    ■ Avoid Entities
    ■ Keep listener small and dedicated to a single task
    ■ Limit dependencies
    Dispatching events one last thing:

    View full-size slide

  47. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    High traffic, how to improve
    performance?

    View full-size slide

  48. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Problem:
    The website is a bit slow

    View full-size slide

  49. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Solution:
    All side tasks can be done asynchronously

    View full-size slide

  50. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Solution:
    All side tasks can be done asynchronously
    Tasks:
    ● Find a tool to transmit messages
    ● Create workers
    ● Use the event dispatcher

    View full-size slide

  51. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Asynchronous jobs everywhere.
    ● Performance
    ● Job splitting

    View full-size slide

  52. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r

    View full-size slide

  53. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Dispatch Event with the EventDispatcher.
    ● Sent from anywhere
    ● Listened with RabbitMQ or internal Listeners

    View full-size slide

  54. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Check eventName &
    event interface
    Dispatch the event
    in the application
    Decorated EventDispatcher.
    Publish to
    RabbitMQ

    View full-size slide

  55. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Send an Email
    ● Sent from anywhere
    ● Built using the User context in a dedicated worker

    View full-size slide

  56. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    + Perfect to decouple code
    + Small workers, easy to understand
    + Easy to dispatch events outside application
    + Separate projects & deployment
    + Scalable!
    - Learning curve
    - Need tools/rules to keep a common logic
    - Hard to test a complete flow with mock/stubs
    - Mapping is time consuming
    Asynchronous event & process Pros/Cons

    View full-size slide

  57. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    129 workers
    Mail
    SMS
    Publication
    Push
    Indexation
    Image
    resizing
    Alerts
    Profile
    completion
    I love
    workers!
    Payment

    View full-size slide

  58. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Asynchronous workers:
    Infrastructure:
    ■ PHP
    ■ Supervisord
    Code:
    ■ Symfony command inside the main app
    ■ Standalone app + Symfony components

    View full-size slide

  59. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Asynchronous workers, Symfony:
    + Access to Entities
    + Every already defined services
    - Entire Symfony
    - Not dedicated
    - Deployed along the website
    - Not the main goal of Symfony
    - High memory usage

    View full-size slide

  60. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Asynchronous workers, Standalone:
    + Dedicated to a single task/type
    + Maintainable small code
    + Deployed independently
    + Easily created & tested
    - Component splitting from the codebase
    - More monitoring

    View full-size slide

  61. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    About Doctrine...

    View full-size slide

  62. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    It’s not a troll.

    View full-size slide

  63. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Code fast!

    View full-size slide

  64. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Code well?

    View full-size slide

  65. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r

    View full-size slide

  66. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Queries from
    the view!
    Hard to control
    cache strategy
    No integrity
    check.

    View full-size slide

  67. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Doctrine Pros/Cons
    + Quick to bootstrap a new project
    + Quick to code a small project
    - Hard to respect MVC
    - Hard to deal with cache
    - Hard to deal with integrity problems in DB
    - Not made to decouple code (relations).
    - Not made to scale.

    View full-size slide

  68. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    We fell in traps...
    4

    View full-size slide

  69. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Never pimp your kernel!

    View full-size slide

  70. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Before
    Environment
    Debug?

    View full-size slide

  71. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Before
    After
    Branding Media
    Environment
    Debug?

    View full-size slide

  72. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Put everything in
    the same place can be
    dangerous!

    View full-size slide

  73. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r

    View full-size slide

  74. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    This is our
    application.
    This is our
    business code.

    View full-size slide

  75. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Don’t mess with
    your tools!

    View full-size slide

  76. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r

    View full-size slide

  77. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r

    View full-size slide

  78. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    We fell in traps...

    View full-size slide

  79. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    We had parachutes!

    View full-size slide

  80. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Problems solved?
    + Use symfony only as a tool
    + Work a lot on onboarding
    + Try to decouple everything with components
    - We still have one big application
    - As everyone, we lack of documentation

    View full-size slide

  81. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    5

    View full-size slide

  82. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    40 269
    Commits
    There are “only” 18 952 commits for symfony/symfony

    View full-size slide

  83. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    561 851
    NCLOC
    There are “only” 278 909 NCLOC for symfony/symfony

    View full-size slide

  84. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    ~ 20
    minutes to run all the
    unit/functional tests

    View full-size slide

  85. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    ~ 30
    minutes to deploy the application

    View full-size slide

  86. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    Too big!

    View full-size slide

  87. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    We’re working
    on it! ;)

    View full-size slide

  88. PHP Frameworks Day 2016 - 25+ million members in 22 countries; How to scale with Symfony2 @odolbeau & @genes0r
    We’re hiring!
    In Paris & Warsaw!
    http://blbl.cr/dreamjobs

    View full-size slide