PHP OPCache, Realpath Cache and Preloading (PHPUK Conference 2020)

Slide 1

Slide 1 text

@coudenysj - jachim.be PHP OPCache, Realpath Cache and Preloading How to tune your PHP installation

Slide 2

Slide 2 text

@coudenysj - jachim.be New PHP versions go faster, but we can make them go even faster!

Slide 3

Slide 3 text

@coudenysj - jachim.be Some groundwork

Slide 4

Slide 4 text

@coudenysj - jachim.be PHP ● Scripting Language ● Fire and forget ● No "manual" compilation

Slide 5

Slide 5 text

@coudenysj - jachim.be PHP-FPM ● Only "relevant" SAPI nowadays :) ● Master process ● Dynamic worker child processes ● Shared memory in the master jachimbe 24546 0.0 0.0 301644 10152 ? Ss 2019 1:04 php-fpm: master process (/conf/jachimbe/php/fpm.conf) jachimbe 21457 3.6 0.1 304680 35732 ? S 10:57 0:00 \_ php-fpm: pool jachimbe jachimbe 21458 3.3 0.1 304680 35672 ? S 10:57 0:00 \_ php-fpm: pool jachimbe jachimbe 21459 3.3 0.1 304680 35672 ? S 10:57 0:00 \_ php-fpm: pool jachimbe jachimbe 21460 4.0 0.1 304680 35672 ? S 10:57 0:00 \_ php-fpm: pool jachimbe

Slide 6

Slide 6 text

@coudenysj - jachim.be Shared memory??? In computer hardware, shared memory refers to a (typically large) block of random access memory (RAM) that can be accessed by several different central processing units (CPUs) in a multiprocessor computer system.

Slide 7

Slide 7 text

@coudenysj - jachim.be All set?

Slide 8

Slide 8 text

@coudenysj - jachim.be Jachim Coudenys

Slide 9

Slide 9 text

@coudenysj - jachim.be So this talk is about performance, right?

Slide 10

Slide 10 text

@coudenysj - jachim.be Right!

Slide 11

Slide 11 text

@coudenysj - jachim.be Why this presentation?

Slide 12

Slide 12 text

@coudenysj - jachim.be 80% of performance issues have nothing to do with your server! https://www.combell.com/en/performance-team

Slide 13

Slide 13 text

@coudenysj - jachim.be Latency Comparison Numbers (~2012) L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns 14x L1 cache Main memory reference 100 ns 20x L2 cache, 200x L1 Send 1K bytes over 1 Gbps network 10,000 ns 10 us Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD Read 1 MB sequentially from memory 250,000 ns 250 us Round trip within same datacenter 500,000 ns 500 us Read 1 MB sequentially from SSD* 1,000,000 ns 1,000 us 1 ms ~1GB/sec SSD, 4X memory Disk seek 10,000,000 ns 10,000 us 10 ms 20x datacenter roundtrip Read 1 MB sequentially from disk 20,000,000 ns 20,000 us 20 ms 80x memory, 20X SSD Send packet CA->Netherlands->CA 150,000,000 ns 150,000 us 150 ms

Slide 14

Slide 14 text

@coudenysj - jachim.be https://gist.github.com/jboner/2841832

Slide 15

Slide 15 text

@coudenysj - jachim.be How can we ﬁx it?

Slide 16

Slide 16 text

@coudenysj - jachim.be Disk IO

Slide 17

Slide 17 text

@coudenysj - jachim.be

Slide 18

Slide 18 text

@coudenysj - jachim.be

Slide 19

Slide 19 text

@coudenysj - jachim.be

Slide 20

Slide 20 text

@coudenysj - jachim.be Network File System (NFS)

Slide 21

Slide 21 text

@coudenysj - jachim.be

Slide 22

Slide 22 text

@coudenysj - jachim.be Tackle these challenges in PHP Realpath Cache OPCache Preloading

Slide 23

Slide 23 text

@coudenysj - jachim.be Realpath Cache

Slide 24

Slide 24 text

@coudenysj - jachim.be Realpath Cache ● Is used to reduce IO ● Caches all possible paths to dest path ● explode('/') ● No shared memory! Mem*Workers ● 4M ○ Prior to PHP 7.0.16 and 7.1.2, the default was "16K" ● Helps a lot with NFS

Slide 25

Slide 25 text

@coudenysj - jachim.be php.net/manual/en/ref.ﬁlesystem.php ● realpath_cache_get() ● realpath_cache_size() ● realpath()

Slide 26

Slide 26 text

@coudenysj - jachim.be demo php realpath.php

Slide 27

Slide 27 text

@coudenysj - jachim.be

Slide 28

Slide 28 text

@coudenysj - jachim.be [/home/jachim/demo] => Array ( [key] => 1.6354972010384E+19 [is_dir] => 1 [realpath] => /home/jachim/demo [expires] => 1579859105 ) [/home] => Array ( [key] => 4353355791257440477 [is_dir] => 1 [realpath] => /home [expires] => 1579859105 ) [/home/jachim] => Array ( [key] => 5522554812971572568 [is_dir] => 1 [realpath] => /home/jachim [expires] => 1579859105 )

Slide 29

Slide 29 text

@coudenysj - jachim.be [/home/jachim/demo/../ffi.php] => Array ( [key] => 1.6164035761241E+19 [is_dir] => [realpath] => /home/jachim/ffi.php [expires] => 1579859105 ) [/home/jachim/ffi.php] => Array ( [key] => 5100116734180765326 [is_dir] => [realpath] => /home/jachim/ffi.php [expires] => 1579859105 ) [/home/jachim/demo/realpath.php] => Array ( [key] => 1.8190176096283E+19 [is_dir] => [realpath] => /home/jachim/demo/realpath.php [expires] => 1579859105 )

Slide 30

Slide 30 text

@coudenysj - jachim.be Realpath Cache: what can we tune?

Slide 31

Slide 31 text

@coudenysj - jachim.be php.net/manual/en/ini.core.php ● realpath_cache_size ● realpath_cache_ttl

Slide 32

Slide 32 text

@coudenysj - jachim.be OPCache OPcache improves PHP performance by storing precompiled script bytecode in shared memory, thereby removing the need for PHP to load and parse scripts on each request.

Slide 33

Slide 33 text

@coudenysj - jachim.be https://engineering.facile.it/blog/eng/realpath-cache-is-it- all-php-opcache-s-fault/

Slide 34

Slide 34 text

@coudenysj - jachim.be Throwaway language ● Always performs same steps ○ Read PHP code ○ Lexing + Parsing: Tokens ○ Compiling: Opcodes ○ Executing ● All information is discarded ● Request per request

Slide 35

Slide 35 text

@coudenysj - jachim.be https://engineering.facile.it/blog/eng/realpath-cache-is-it- all-php-opcache-s-fault/

Slide 36

Slide 36 text

@coudenysj - jachim.be opcodes? In computing, an opcode (abbreviated from operation code) is the portion of a machine language instruction that specifies the operation to be performed.

Slide 37

Slide 37 text

@coudenysj - jachim.be Vulcan Logic Dumper https://3v4l.org/A1B4H/vld

Slide 38

Slide 38 text

@coudenysj - jachim.be This can get big quite quickly ● Libraries ● Frameworks ● etc...

Slide 39

Slide 39 text

@coudenysj - jachim.be OPCodes (usually) don't change ● Let’s add in some cache!

Slide 40

Slide 40 text

@coudenysj - jachim.be OPCache ● Bunch of opcode cachers ○ APC ○ Turck MMCache ○ Zend Optimizer ○ etc... ● Zend Optimizer "donated" by Zend ● Since PHP 5.5

Slide 41

Slide 41 text

@coudenysj - jachim.be https://engineering.facile.it/blog/eng/realpath-cache-is-it- all-php-opcache-s-fault/

Slide 42

Slide 42 text

@coudenysj - jachim.be Shared memory ● Stored in the master process of FPM

Slide 43

Slide 43 text

@coudenysj - jachim.be OPCache Optimizer ● Gets better every release ● Optimizes: ○ Branches ○ Dead code ○ ++$a vs $a++

Slide 44

Slide 44 text

@coudenysj - jachim.be Optimized opcodes (example) https://3v4l.org/A1B4H/vld

Slide 45

Slide 45 text

@coudenysj - jachim.be php.net/manual/en/ref.opcache.php ● Information ○ opcache_get_configuration() ○ opcache_get_status() ○ opcache_is_script_cached() ● Actions ○ opcache_compile_file() ○ opcache_invalidate() ○ opcache_reset()

Slide 46

Slide 46 text

@coudenysj - jachim.be demo php opcache.php

Slide 47

Slide 47 text

@coudenysj - jachim.be

Slide 48

Slide 48 text

@coudenysj - jachim.be [opcache_enabled] => 1 [cache_full] => [restart_pending] => [restart_in_progress] => [memory_usage] => Array ( [used_memory] => 9168648 [free_memory] => 125049080 [wasted_memory] => 0 [current_wasted_percentage] => 0 ) [interned_strings_usage] => Array ( [buffer_size] => 6291008 [used_memory] => 371880 [free_memory] => 5919128 [number_of_strings] => 7869 ) ...

Slide 49

Slide 49 text

@coudenysj - jachim.be [opcache_statistics] => Array ( [num_cached_scripts] => 1 [num_cached_keys] => 2 [max_cached_keys] => 16229 [hits] => 0 [start_time] => 1579859174 [last_restart_time] => 0 [oom_restarts] => 0 [hash_restarts] => 0 [manual_restarts] => 0 [misses] => 1 [blacklist_misses] => 0 [blacklist_miss_ratio] => 0 [opcache_hit_rate] => 0 ) ...

Slide 50

Slide 50 text

@coudenysj - jachim.be [scripts] => Array ( [/home/jachim/demo/opcache.php] => Array ( [full_path] => /home/jachim/demo/opcache.php [hits] => 0 [memory_consumption] => 880 [last_used] => Fri Jan 24 10:46:14 2020 [last_used_timestamp] => 1579859174 [timestamp] => 1579858691 ) ) ...

Slide 51

Slide 51 text

@coudenysj - jachim.be Interned strings ● Used in a lot of languages ● Since PHP 5.4 ● "Compression" for source code ● In shared memory (FPM master process)

Slide 52

Slide 52 text

@coudenysj - jachim.be Keys ● Full path to file ● Relative paths to files

Slide 53

Slide 53 text

@coudenysj - jachim.be Wasted memory ● Opcache doesn't do "defragmentation" ● File changes cause recompilation

Slide 54

Slide 54 text

@coudenysj - jachim.be Opcache restarts ● OOM Restarts ○ Memory ● Hash Restarts ○ Keys ● Manual Restarts ○ opcache_reset()

Slide 55

Slide 55 text

@coudenysj - jachim.be NEVER have a full cache!

Slide 56

Slide 56 text

@coudenysj - jachim.be Opcache File Cache ● Read opcodes from disk ● Can help busy sites ● Supports CLI (used to be "fire and forget") ● Cache to "user directory" opcache.file_cache=/var/tmp/php/opcache opcache.file_cache_only=1 # Useful for CLI opcache.file_cache_consistency_checks=1 # Adler checksum

Slide 57

Slide 57 text

@coudenysj - jachim.be demo php -d opcache.enable_cli=On -d opcache.file_cache="/tmp/opcache" opcache.php

Slide 58

Slide 58 text

@coudenysj - jachim.be

Slide 59

Slide 59 text

@coudenysj - jachim.be └╼ tree cache cache └── 26f7903bdd40555497c24242ea455a66 └── home └── jachim └── demo └── opcache.php.bin 4 directories, 1 file

Slide 60

Slide 60 text

@coudenysj - jachim.be PHPArch Opcache article https://www.phparch.com/article/diving-in-the -opcache/ ● Explores the option of using opcache file caching as pre-compiled PHP programs.

Slide 61

Slide 61 text

@coudenysj - jachim.be OPCache: what can we tune?

Slide 62

Slide 62 text

@coudenysj - jachim.be “Disaster Recovery” Scenarios ● Memory Full? ● Interned Strings Full? ● Key store full?

Slide 63

Slide 63 text

@coudenysj - jachim.be php.net/manual/en/opcache.conﬁguration.php ● Memory ○ opcache.memory_consumption ○ opcache.interned_strings_buffer ○ opcache.max_accelerated_files (keys) ● Invalidation ○ opcache.validate_timestamps ○ opcache.revalidate_freq ● Thresholds ○ opcache.max_wasted_percentage

Slide 64

Slide 64 text

@coudenysj - jachim.be Getting stats ● opcache_get_status()

Slide 65

Slide 65 text

@coudenysj - jachim.be https://github.com/rlerdorf/opcache-status

Slide 66

Slide 66 text

@coudenysj - jachim.be https://gist.github.com/ck-on/

Slide 67

Slide 67 text

@coudenysj - jachim.be Opcache Priming / Deployment Strategies ● opcache_compile_file() ● FPM Pools ● or…

Slide 68

Slide 68 text

@coudenysj - jachim.be Preloading Preload PHP functions and classes once and use them in the context of any future request without overhead.

Slide 69

Slide 69 text

@coudenysj - jachim.be Preload PHP functions and classes once and use them in the context of any future request without overhead.

Slide 70

Slide 70 text

@coudenysj - jachim.be OPCache on steroids

Slide 71

Slide 71 text

@coudenysj - jachim.be Preloading ● PHP 7.4 ● Part of opcache ● Starts in master process before anything else ● Loads code in memory "permanently" ● No need to copy from shared memory to process memory

Slide 72

Slide 72 text

@coudenysj - jachim.be Preloading ● Opcache only works per file ● Preloading helps with class libraries ● Code will perform as internal entities (e.g. strlen, etc...) ● Simple file with autoloading magic

Slide 73

Slide 73 text

@coudenysj - jachim.be php.net/manual/en/opcache.conﬁguration.php ● opcache.preload=/path/to/preload.php ● opcache_compile_file()

Slide 74

Slide 74 text

@coudenysj - jachim.be Unlinked classes aren’t preloaded

Slide 75

Slide 75 text

@coudenysj - jachim.be Preloading in the wild ● github.com/brendt/laravel-preload/blob/master/preload.php ● symfony.com/blog/new-in-symfony-4-4-preloading-symfony-applications-in-php -7-4 // var/cache/dev/srcApp_KernelDevDebugContainer.preload.php // This file has been auto-generated by the Symfony Dependency Injection Component // You can reference it in the "opcache.preload" php.ini setting on PHP >= 7.4 when preloading is desired use Symfony\Component\DependencyInjection\Dumper\Preloader; require dirname(__DIR__, 3).'/vendor/autoload.php'; require __DIR__.'/ContainerZxvZ783/srcApp_KernelDevDebugContainer.php'; $classes = []; $classes[] = 'App\Kernel'; Preloader::preload($classes);

Slide 76

Slide 76 text

@coudenysj - jachim.be Caveats ● “Full server” (fpm) ○ php.ini setting ● Server restart

Slide 77

Slide 77 text

@coudenysj - jachim.be Composer ● Sounds like a job for composer, right? ● [RFC] Preloading support (https://github.com/composer/composer/issue s/7777)

Slide 78

Slide 78 text

@coudenysj - jachim.be Composer benchmarks https://github.com/composer/composer/issues/7 777#issuecomment-440268416 ● No preloading ● Preloading only "hot" classes ○ $files = opcache_get_status(true)['scripts']; ● Preloading all the classes ○ $files = require 'vendor/composer/autoload_classmap.php';

Slide 79

Slide 79 text

@coudenysj - jachim.be Composer benchmarks https://github.com/composer/composer/issues/7777#issuecomment-440268416

Slide 80

Slide 80 text

@coudenysj - jachim.be Composer at the moment https://github.com/composer/composer/issues/7777#issuecomment-559725760

Slide 81

Slide 81 text

@coudenysj - jachim.be Preloading issues ● There were some problems with using require() instead of opcache_compile_file() ○ Should be fixed in latest release ● Some issues in bugtracker, but not that bad ○ https://bugs.php.net/search.php?cmd=display&search_for =preload

Slide 82

Slide 82 text

@coudenysj - jachim.be Performance ● Is it worth the effort? ● Depends on the situation ● We mostly see an improvement ● Stick with the hot classes ● Currently looking at how we can apply it at scale for our customers at Combell ● Some reports: ○ https://stitcher.io/blog/php-preload-benchmarks ○ https://ezplatform.com/blog/php-7.4-opcache-preloading-benchmark ○ https://developer.happyr.com/php-74-preload

Slide 83

Slide 83 text

@coudenysj - jachim.be Preload resources ● https://externals.io/message/103333 ● https://wiki.php.net/rfc/preload ● https://github.com/php/php-src/pull/3538

Slide 84

Slide 84 text

@coudenysj - jachim.be In conclusion

Slide 85

Slide 85 text

@coudenysj - jachim.be PHP Performance ● Know enough of PHP internals ● Know your application (use information functions) ● Finetune ● Repeat

Slide 86

Slide 86 text

@coudenysj - jachim.be Elastic Stack @ Combell ● Gather realpath/opcache/etc... data from all accounts ● Act upon that data (which is changing constantly) ● Autotuning & autoconfigure preload

Slide 87

Slide 87 text

@coudenysj - jachim.be Thank you! @coudenysj - jachim.be https://joind.in/talk/d1cfa