What I learned trying to make Symfony and API Platform 50% faster

Working on a big symfony project with a quite common stack (API with APIP, lots of doctrine entities and all config in YAML files), we experienced performances issues in both dev and prod mode.

I took my headband lamp (hi blackfire) and dug. I learned A LOT and discovered quite a few issues that results to a lot of pull requests that got merged.

This talk will be a feedback of the process used and the lessons learned. It will be both practical and theoric.

Transcript

1. What I learned trying to make Symfony and API Platform 50% faster SymfonyWorld Online 2020 – Bastien Jaillot – @bastnic Architect @JoliCode
2. Such a loss of performance that the Developer eXperience (DX) has become very unpleasant.
3. Normal! (?) - Docker on Mac - API Platform - a lot of Doctrine entities - autowire / autoconfigure
4. Stop blaming yourself and REALLY analyze what's going on ...
5. Performance nice again! - Use of Symfony binary on mac envs - APIs mocks - There's more than just our Symfony application, itself well optimized on paper. So we're going to try to optimize what's under the hood : Symfony, API Platform, Doctrine, bundles... 15 Pull Requests merged and 2 blog posts published… (and 2 talks ) .
7. Methodology
8. Fabien Potencier, https://symfony.com/book
9. Gives a complete and detailed view of your application and what actually happens while running: - views "call graph" and "timeline" - perfect and easy integration with docker compose - profiling from CLI and FPM - a "player" for lazy people to automate everything Disclaimer: for the purpose of the demo of this talk, Blackfire gave me a temporary entreprise licence
10. Other tools - echo - var_dump - dump - dd - xdebug - … tracing tools (tideways, etc.)
12. Baseline prod BLACKFIRE_EXTERNAL_ID=init-prod \ php blackfire-player.phar run .blackfire.bkf \ --blackfire-env="bjaillot-laptop" Only one command run: - env prod, warmup done - multiple routes tested - direct access to blackfire profiles /!\ project less critical than the one that started this
13. Futures analysis BLACKFIRE_EXTERNAL_ID=<nom du test> \ BLACKFIRE_EXTERNAL_PARENT_ID= init-prod \ php blackfire-player.phar run .blackfire.bkf \ --blackfire-env="bjaillot-laptop" Give directly the comparison profiles for each of my routes against the baseline.
14. But really? - update everything - profile - read the code in vendors/ - if we have an idea for a patch, we modify the files in vendors - otherwise, you write a ticket to talk about your concern. - profile again - is performance improving or not? If not, back to step 3 - pull the project, patch, submit a PR
15. Performance? What is costly? 1. useless executions, the fastest code is the one that is not written, or at least does not run. Hints: a. abnormal cardinalities b. call to unused services c. IOs, even with optimized PHP at best, a network call will always be slow 2. The execution itself
16. How is a query broken down in the PROD environment?
17. (without warmup) (with warmup)
20. Make normalizers “cacheable” https://symfony.com/blog/new-in-symfony-4-1-faster-serializer
22. Symfony#35252 Use of isset on a nullable value. isset always returns false in that case, causes the computation. Remplacement par array_key_exists et voilà! What makes you think that? The cardinality of the "heavy" code called, I didn't have thousands of names to convert, so the cache wasn't efficient.
23. Symfony#35079 Optimized cached generated by warmup was not memoized, so the call to cache backend happened all the time. On a large volume of data, it's a 10% gain free of charge... What makes me think that? Cardinality + the number of calls to apcu_fetch.
24. ApiPlatform#3317 Another cardinality that seems off limit. It reveals that something must be cached. // <heavy code with a lot of is_a>
25. Main path dependencies: LAZY Some classes on the critical path deserve your attention, at the very least: - EventListener / Subscriber - Normalizers Because they depend on the content of the request, and not something we can build upfront: they need to be booted to know if they are needed for the current req. If they have dependencies, they will be loaded. This slows down ALL requests.
26. Prod: results * on an existing application chosen for its total lack of major interest. API entities list API single entity Baseline 161ms 121ms Final 119ms 98ms Variations -26% -19%
28. /!\ REMINDER /!\ in prod, do not forget to PRELOAD (Symfony provides the preload script to include) *PHP 7.4 compatible only
29. How is a query broken down in the DEV environment?
31. Conditions that trigger a rebuild - any modification on Kernel.php, composer dependencies, config files, or CompilerPass classes - modification on class signature, phpdoc, public / protected properties or method signature - add / remove file in a folder monitored by autoconfig / autowire - and many other specificities (routes, translations, templates, etc.)
32. Dev: where are we? * on an existing application chosen for its total lack of major interest. API entities list API single entity Baseline 838ms 839ms Final optim prod 822ms 826ms Variations almost nothing :( almost nothing :(
37. doctrine/annotations#301 Doctrine has its own file watcher, based on filemtime. But for each file, it will look for all its interfaces, parent classes, traits, and retrieve their last modification date. I use a lot of traits and interfaces, which makes it costly. We add a static cache.
38. symfony/recipes#405 The Kernel generated by default at the time was very suboptimal. It indicated a lot of possible folders / files, multiplying the paths to check at each request. Check your Kernel.php
39. The default Kernel used to browse too many folders, in GLOB and on several extensions (php,xml,yaml,yml), now it only looks for a few files, and only in YAML. As a result : use of highly optimized code, and on less files / folders.
40. Symfony#35009
41. https://github.com/symfony/symfony/commit/d7a06790119b04e4f3e96f36dbf68e70ec5964ae
42. https://github.com/symfony/symfony/commit/d7a06790119b04e4f3e96f36dbf68e70ec5964ae ¯\_(ツ)_/¯
44. Symfony#35109 In dev, the Yaml/XML validator and serialization are read at runtime. While if they are modified, a build is forced anyway. A warmer cache phase is added. This is a tip seen on Api Platform and backported in Symfony. Thanks Teoh
45. Doctrine-bundle#1115 Doctrine Bundle validates the schema of your entities at each request. My schema has been frozen for month and there is a CI task to check it anyway. I added an option to disable it, thus avoiding the loading of unnecessary metadata.
46. Dev: results WITHOUT the Alpine modification, which alone accounts for 60%, which would have made the baseline 2s API entities list API single entity Baseline 838ms 839ms Post-prod 822ms 826ms Final optim dev 482ms 427ms Variations -42% -49%
47. Watchers are expensive But they allow autowire / autoconfigure and simplified configuration that I'm not willing to sacrifice!
48. So you can have a much more pleasant performance in dev while having a top notch Dx! (optimized) (optimized, and can be disabled) (optimized) (optimized)
49. Strangebuzz/cache-warmer https://www.strangebuzz.com/en/blog/introducing-cw-a-cache-watcher-for-symfony Principle: watcher in go which relaunches the rebuild of the kernel in the background. Double gains: - you are not waiting for the cache rebuild when calling the API - we could remove the watchers since they are no longer useful. PS : also achievable from PHPStorm, gulp and co.
50. So you can have production performance in dev while having a top notch Dx! (optimized) (optimized, and can be disabled) (optimized)
51. Takeaways
52. We no longer hesitate to read code in vendor/ (the blackfire trace can help you know where to look)
53. Rubber ducks
55. Ticket / Pull request / Patience - Having a merged PR in large projects is complicated and requires a significant investment. - you have to be patient: your case is not the only one, and it has to be suitable for everyone, many of which you don't even know. - quality++ / test++ / documentation: nothing should be left to chance - performance is a feature, so could be released in a long time
56. Performance is addictive • the time saved by my team on this project thanks to the performance gains in the development environment is equivalent to several hundred years of blackfire license. • on an open source project, everyone benefits from everyone's gains • these advices can be applied to ALL projects (I would do a pass on Sylius, Drupal...) • I can help you, ping me by Mail / Github / Slack / Twitter / Postcard / Carrier Pigeon
57. Just Do It!
58. Keep your stack up to date!
59. Thanks. SymfonyWorld Online 2020 – Bastien Jaillot – @bastnic