Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What I learned trying to make Symfony and API Platform 50% faster

What I learned trying to make Symfony and API Platform 50% faster

Working on a big symfony project with a quite common stack (API with APIP, lots of doctrine entities and all config in YAML files), we experienced performances issues in both dev and prod mode.

I took my headband lamp (hi blackfire) and dug. I learned A LOT and discovered quite a few issues that results to a lot of pull requests that got merged.

This talk will be a feedback of the process used and the lessons learned. It will be both practical and theoric.

87137598c23e3e90755e5d7476d21405?s=128

Bastien Jaillot

December 04, 2020
Tweet

Transcript

  1. What I learned trying to make Symfony and API Platform

    50% faster SymfonyWorld Online 2020 – Bastien Jaillot – @bastnic Architect @JoliCode
  2. Such a loss of performance that the Developer eXperience (DX)

    has become very unpleasant.
  3. Normal! (?) - Docker on Mac - API Platform -

    a lot of Doctrine entities - autowire / autoconfigure
  4. Stop blaming yourself and REALLY analyze what's going on ...

  5. Performance nice again! - Use of Symfony binary on mac

    envs - APIs mocks - There's more than just our Symfony application, itself well optimized on paper. So we're going to try to optimize what's under the hood : Symfony, API Platform, Doctrine, bundles... 15 Pull Requests merged and 2 blog posts published… (and 2 talks ) .
  6. None
  7. Methodology

  8. Fabien Potencier, https://symfony.com/book

  9. Gives a complete and detailed view of your application and

    what actually happens while running: - views "call graph" and "timeline" - perfect and easy integration with docker compose - profiling from CLI and FPM - a "player" for lazy people to automate everything Disclaimer: for the purpose of the demo of this talk, Blackfire gave me a temporary entreprise licence
  10. Other tools - echo - var_dump - dump - dd

    - xdebug - … tracing tools (tideways, etc.)
  11. None
  12. Baseline prod BLACKFIRE_EXTERNAL_ID=init-prod \ php blackfire-player.phar run .blackfire.bkf \ --blackfire-env="bjaillot-laptop"

    Only one command run: - env prod, warmup done - multiple routes tested - direct access to blackfire profiles /!\ project less critical than the one that started this
  13. Futures analysis BLACKFIRE_EXTERNAL_ID=<nom du test> \ BLACKFIRE_EXTERNAL_PARENT_ID= init-prod \ php

    blackfire-player.phar run .blackfire.bkf \ --blackfire-env="bjaillot-laptop" Give directly the comparison profiles for each of my routes against the baseline.
  14. But really? - update everything - profile - read the

    code in vendors/ - if we have an idea for a patch, we modify the files in vendors - otherwise, you write a ticket to talk about your concern. - profile again - is performance improving or not? If not, back to step 3 - pull the project, patch, submit a PR
  15. Performance? What is costly? 1. useless executions, the fastest code

    is the one that is not written, or at least does not run. Hints: a. abnormal cardinalities b. call to unused services c. IOs, even with optimized PHP at best, a network call will always be slow 2. The execution itself
  16. How is a query broken down in the PROD environment?

  17. (without warmup) (with warmup)

  18. None
  19. None
  20. Make normalizers “cacheable” https://symfony.com/blog/new-in-symfony-4-1-faster-serializer

  21. None
  22. Symfony#35252 Use of isset on a nullable value. isset always

    returns false in that case, causes the computation. Remplacement par array_key_exists et voilà! What makes you think that? The cardinality of the "heavy" code called, I didn't have thousands of names to convert, so the cache wasn't efficient.
  23. Symfony#35079 Optimized cached generated by warmup was not memoized, so

    the call to cache backend happened all the time. On a large volume of data, it's a 10% gain free of charge... What makes me think that? Cardinality + the number of calls to apcu_fetch.
  24. ApiPlatform#3317 Another cardinality that seems off limit. It reveals that

    something must be cached. // <heavy code with a lot of is_a>
  25. Main path dependencies: LAZY Some classes on the critical path

    deserve your attention, at the very least: - EventListener / Subscriber - Normalizers Because they depend on the content of the request, and not something we can build upfront: they need to be booted to know if they are needed for the current req. If they have dependencies, they will be loaded. This slows down ALL requests.
  26. Prod: results * on an existing application chosen for its

    total lack of major interest. API entities list API single entity Baseline 161ms 121ms Final 119ms 98ms Variations -26% -19%
  27. None
  28. /!\ REMINDER /!\ in prod, do not forget to PRELOAD

    (Symfony provides the preload script to include) *PHP 7.4 compatible only
  29. How is a query broken down in the DEV environment?

  30. None
  31. Conditions that trigger a rebuild - any modification on Kernel.php,

    composer dependencies, config files, or CompilerPass classes - modification on class signature, phpdoc, public / protected properties or method signature - add / remove file in a folder monitored by autoconfig / autowire - and many other specificities (routes, translations, templates, etc.)
  32. Dev: where are we? * on an existing application chosen

    for its total lack of major interest. API entities list API single entity Baseline 838ms 839ms Final optim prod 822ms 826ms Variations almost nothing :( almost nothing :(
  33. None
  34. None
  35. None
  36. None
  37. doctrine/annotations#301 Doctrine has its own file watcher, based on filemtime.

    But for each file, it will look for all its interfaces, parent classes, traits, and retrieve their last modification date. I use a lot of traits and interfaces, which makes it costly. We add a static cache.
  38. symfony/recipes#405 The Kernel generated by default at the time was

    very suboptimal. It indicated a lot of possible folders / files, multiplying the paths to check at each request. Check your Kernel.php
  39. The default Kernel used to browse too many folders, in

    GLOB and on several extensions (php,xml,yaml,yml), now it only looks for a few files, and only in YAML. As a result : use of highly optimized code, and on less files / folders.
  40. Symfony#35009

  41. https://github.com/symfony/symfony/commit/d7a06790119b04e4f3e96f36dbf68e70ec5964ae

  42. https://github.com/symfony/symfony/commit/d7a06790119b04e4f3e96f36dbf68e70ec5964ae ¯\_(ツ)_/¯

  43. None
  44. Symfony#35109 In dev, the Yaml/XML validator and serialization are read

    at runtime. While if they are modified, a build is forced anyway. A warmer cache phase is added. This is a tip seen on Api Platform and backported in Symfony. Thanks Teoh
  45. Doctrine-bundle#1115 Doctrine Bundle validates the schema of your entities at

    each request. My schema has been frozen for month and there is a CI task to check it anyway. I added an option to disable it, thus avoiding the loading of unnecessary metadata.
  46. Dev: results WITHOUT the Alpine modification, which alone accounts for

    60%, which would have made the baseline 2s API entities list API single entity Baseline 838ms 839ms Post-prod 822ms 826ms Final optim dev 482ms 427ms Variations -42% -49%
  47. Watchers are expensive But they allow autowire / autoconfigure and

    simplified configuration that I'm not willing to sacrifice!
  48. So you can have a much more pleasant performance in

    dev while having a top notch Dx! (optimized) (optimized, and can be disabled) (optimized) (optimized)
  49. Strangebuzz/cache-warmer https://www.strangebuzz.com/en/blog/introducing-cw-a-cache-watcher-for-symfony Principle: watcher in go which relaunches the rebuild

    of the kernel in the background. Double gains: - you are not waiting for the cache rebuild when calling the API - we could remove the watchers since they are no longer useful. PS : also achievable from PHPStorm, gulp and co.
  50. So you can have production performance in dev while having

    a top notch Dx! (optimized) (optimized, and can be disabled) (optimized)
  51. Takeaways

  52. We no longer hesitate to read code in vendor/ (the

    blackfire trace can help you know where to look)
  53. Rubber ducks

  54. None
  55. Ticket / Pull request / Patience - Having a merged

    PR in large projects is complicated and requires a significant investment. - you have to be patient: your case is not the only one, and it has to be suitable for everyone, many of which you don't even know. - quality++ / test++ / documentation: nothing should be left to chance - performance is a feature, so could be released in a long time
  56. Performance is addictive • the time saved by my team

    on this project thanks to the performance gains in the development environment is equivalent to several hundred years of blackfire license. • on an open source project, everyone benefits from everyone's gains • these advices can be applied to ALL projects (I would do a pass on Sylius, Drupal...) • I can help you, ping me by Mail / Github / Slack / Twitter / Postcard / Carrier Pigeon
  57. Just Do It!

  58. Keep your stack up to date!

  59. Thanks. SymfonyWorld Online 2020 – Bastien Jaillot – @bastnic