What I learned trying to make Symfony and API Platform 50% faster

Slide 1

Slide 1 text

What I learned trying to make Symfony and API Platform 50% faster SymfonyWorld Online 2020 – Bastien Jaillot – @bastnic Architect @JoliCode

Slide 2

Slide 2 text

Such a loss of performance that the Developer eXperience (DX) has become very unpleasant.

Slide 3

Slide 3 text

Normal! (?) - Docker on Mac - API Platform - a lot of Doctrine entities - autowire / autoconﬁgure

Slide 4

Slide 4 text

Stop blaming yourself and REALLY analyze what's going on ...

Slide 5

Slide 5 text

Performance nice again! - Use of Symfony binary on mac envs - APIs mocks - There's more than just our Symfony application, itself well optimized on paper. So we're going to try to optimize what's under the hood : Symfony, API Platform, Doctrine, bundles... 15 Pull Requests merged and 2 blog posts published… (and 2 talks ) .

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Methodology

Slide 8

Slide 8 text

Fabien Potencier, https://symfony.com/book

Slide 9

Slide 9 text

Gives a complete and detailed view of your application and what actually happens while running: - views "call graph" and "timeline" - perfect and easy integration with docker compose - proﬁling from CLI and FPM - a "player" for lazy people to automate everything Disclaimer: for the purpose of the demo of this talk, Blackﬁre gave me a temporary entreprise licence

Slide 10

Slide 10 text

Other tools - echo - var_dump - dump - dd - xdebug - … tracing tools (tideways, etc.)

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Baseline prod BLACKFIRE_EXTERNAL_ID=init-prod \ php blackfire-player.phar run .blackfire.bkf \ --blackfire-env="bjaillot-laptop" Only one command run: - env prod, warmup done - multiple routes tested - direct access to blackﬁre proﬁles /!\ project less critical than the one that started this

Slide 13

Slide 13 text

Futures analysis BLACKFIRE_EXTERNAL_ID= \ BLACKFIRE_EXTERNAL_PARENT_ID= init-prod \ php blackfire-player.phar run .blackfire.bkf \ --blackfire-env="bjaillot-laptop" Give directly the comparison proﬁles for each of my routes against the baseline.

Slide 14

Slide 14 text

But really? - update everything - profile - read the code in vendors/ - if we have an idea for a patch, we modify the files in vendors - otherwise, you write a ticket to talk about your concern. - profile again - is performance improving or not? If not, back to step 3 - pull the project, patch, submit a PR

Slide 15

Slide 15 text

Performance? What is costly? 1. useless executions, the fastest code is the one that is not written, or at least does not run. Hints: a. abnormal cardinalities b. call to unused services c. IOs, even with optimized PHP at best, a network call will always be slow 2. The execution itself

Slide 16

Slide 16 text

How is a query broken down in the PROD environment?

Slide 17

Slide 17 text

(without warmup) (with warmup)

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Make normalizers “cacheable” https://symfony.com/blog/new-in-symfony-4-1-faster-serializer

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

Symfony#35252 Use of isset on a nullable value. isset always returns false in that case, causes the computation. Remplacement par array_key_exists et voilà! What makes you think that? The cardinality of the "heavy" code called, I didn't have thousands of names to convert, so the cache wasn't efﬁcient.

Slide 23

Slide 23 text

Symfony#35079 Optimized cached generated by warmup was not memoized, so the call to cache backend happened all the time. On a large volume of data, it's a 10% gain free of charge... What makes me think that? Cardinality + the number of calls to apcu_fetch.

Slide 24

Slide 24 text

ApiPlatform#3317 Another cardinality that seems off limit. It reveals that something must be cached. //

Slide 25

Slide 25 text

Main path dependencies: LAZY Some classes on the critical path deserve your attention, at the very least: - EventListener / Subscriber - Normalizers Because they depend on the content of the request, and not something we can build upfront: they need to be booted to know if they are needed for the current req. If they have dependencies, they will be loaded. This slows down ALL requests.

Slide 26

Slide 26 text

Prod: results * on an existing application chosen for its total lack of major interest. API entities list API single entity Baseline 161ms 121ms Final 119ms 98ms Variations -26% -19%

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

/!\ REMINDER /!\ in prod, do not forget to PRELOAD (Symfony provides the preload script to include) *PHP 7.4 compatible only

Slide 29

Slide 29 text

How is a query broken down in the DEV environment?

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Conditions that trigger a rebuild - any modification on Kernel.php, composer dependencies, config files, or CompilerPass classes - modification on class signature, phpdoc, public / protected properties or method signature - add / remove file in a folder monitored by autoconfig / autowire - and many other specificities (routes, translations, templates, etc.)

Slide 32

Slide 32 text

Dev: where are we? * on an existing application chosen for its total lack of major interest. API entities list API single entity Baseline 838ms 839ms Final optim prod 822ms 826ms Variations almost nothing :( almost nothing :(

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

doctrine/annotations#301 Doctrine has its own file watcher, based on filemtime. But for each file, it will look for all its interfaces, parent classes, traits, and retrieve their last modification date. I use a lot of traits and interfaces, which makes it costly. We add a static cache.

Slide 38

Slide 38 text

symfony/recipes#405 The Kernel generated by default at the time was very suboptimal. It indicated a lot of possible folders / ﬁles, multiplying the paths to check at each request. Check your Kernel.php

Slide 39

Slide 39 text

The default Kernel used to browse too many folders, in GLOB and on several extensions (php,xml,yaml,yml), now it only looks for a few ﬁles, and only in YAML. As a result : use of highly optimized code, and on less ﬁles / folders.

Slide 40

Slide 40 text

Symfony#35009

Slide 41

Slide 41 text

https://github.com/symfony/symfony/commit/d7a06790119b04e4f3e96f36dbf68e70ec5964ae

Slide 42

Slide 42 text

https://github.com/symfony/symfony/commit/d7a06790119b04e4f3e96f36dbf68e70ec5964ae ¯\_(ツ)_/¯

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

Symfony#35109 In dev, the Yaml/XML validator and serialization are read at runtime. While if they are modiﬁed, a build is forced anyway. A warmer cache phase is added. This is a tip seen on Api Platform and backported in Symfony. Thanks Teoh

Slide 45

Slide 45 text

Doctrine-bundle#1115 Doctrine Bundle validates the schema of your entities at each request. My schema has been frozen for month and there is a CI task to check it anyway. I added an option to disable it, thus avoiding the loading of unnecessary metadata.

Slide 46

Slide 46 text

Dev: results WITHOUT the Alpine modiﬁcation, which alone accounts for 60%, which would have made the baseline 2s API entities list API single entity Baseline 838ms 839ms Post-prod 822ms 826ms Final optim dev 482ms 427ms Variations -42% -49%

Slide 47

Slide 47 text

Watchers are expensive But they allow autowire / autoconfigure and simplified configuration that I'm not willing to sacrifice!

Slide 48

Slide 48 text

So you can have a much more pleasant performance in dev while having a top notch Dx! (optimized) (optimized, and can be disabled) (optimized) (optimized)

Slide 49

Slide 49 text

Strangebuzz/cache-warmer https://www.strangebuzz.com/en/blog/introducing-cw-a-cache-watcher-for-symfony Principle: watcher in go which relaunches the rebuild of the kernel in the background. Double gains: - you are not waiting for the cache rebuild when calling the API - we could remove the watchers since they are no longer useful. PS : also achievable from PHPStorm, gulp and co.

Slide 50

Slide 50 text

So you can have production performance in dev while having a top notch Dx! (optimized) (optimized, and can be disabled) (optimized)

Slide 51

Slide 51 text

Takeaways

Slide 52

Slide 52 text

We no longer hesitate to read code in vendor/ (the blackﬁre trace can help you know where to look)

Slide 53

Slide 53 text

Rubber ducks

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

Ticket / Pull request / Patience - Having a merged PR in large projects is complicated and requires a signiﬁcant investment. - you have to be patient: your case is not the only one, and it has to be suitable for everyone, many of which you don't even know. - quality++ / test++ / documentation: nothing should be left to chance - performance is a feature, so could be released in a long time

Slide 56

Slide 56 text

Performance is addictive ● the time saved by my team on this project thanks to the performance gains in the development environment is equivalent to several hundred years of blackﬁre license. ● on an open source project, everyone beneﬁts from everyone's gains ● these advices can be applied to ALL projects (I would do a pass on Sylius, Drupal...) ● I can help you, ping me by Mail / Github / Slack / Twitter / Postcard / Carrier Pigeon