Slide 1

Slide 1 text

HUNTING DOWN HUNTING DOWN HUNTING DOWN MEMORY LEAKS WITH MEMORY LEAKS WITH MEMORY LEAKS WITH PHP MEMINFO PHP MEMINFO PHP MEMINFO Benoit Jacquemont @bjacquemont

Slide 2

Slide 2 text

What Is A Memory Leak? A memory leak occurs when memory which is no longer needed is not released. en.wikipedia.org/wiki/Memory_leak

Slide 3

Slide 3 text

Memory Leaks: Should You Care? Usually, PHP process life time = HTTP exchange lifetime PHP process memory released a er each HTTP exchange Yes but... more and more capable-but-complex stacks long running background jobs

Slide 4

Slide 4 text

Why I Care About Memory Leaks E-COMMERCE MOBILE APPLICATION PRINT CATALOG POINTS OF SALE ERP MEDIA SERVER SUPPLIERS PURCHASING DPT MARKETING DPT CSV FTP XML XLS SUPPLIERS PORTAL ENRIC H TRAN SLATE CO N T RO L Import Jobs Export Jobs Mass Edition Jobs Rules Engine Jobs ...

Slide 5

Slide 5 text

The Effects Of A Memory Leak Less memory available for other processes (surprise !) The more memory your program uses, the slower it becomes

Slide 6

Slide 6 text

But, How Is Memory Released In PHP?

Slide 7

Slide 7 text

The Refcounter Basic and efficient. Frees memory as soon as it isn't used anymore.

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

1st object will be released only when 2nd object is released 2nd object will be released only when 1st object is released Neither of them can be released by the refcounter... The Refcounter Cannot Handle Circular References

Slide 27

Slide 27 text

"Yeah sure, but circular references don't exist in the real world..." Tree structure: parent → children → parent Doctrine bidirectional mapping: entity1 → entity2 → entity1 ...

Slide 28

Slide 28 text

Circular Reference Collector To The Rescue! garbage collector exclusively dedicated to collect circular references

Slide 29

Slide 29 text

But my memory usage keeps increasing! function buildObjects() { $objA = new StdClass(); $objB = new StdClass(); $objA->attr = $objB; $objB->attr = $objA; } for ($i = 0; $i < 2000; $i++) { buildObjects(); }

Slide 30

Slide 30 text

Keep Calm And Wait For The Collect! There's a delay before the Collector cleans up memory

Slide 31

Slide 31 text

Circular Reference Collection How It Works Each time an object or array has its refcount decremented, it gets added to the collector buffer.

Slide 32

Slide 32 text

Circular Reference Collection How It Works

Slide 33

Slide 33 text

Circular Reference Collection How It Works

Slide 34

Slide 34 text

Circular Reference Collection How It Works

Slide 35

Slide 35 text

Circular Reference Collection How It Works

Slide 36

Slide 36 text

Circular Reference Collection How It Works

Slide 37

Slide 37 text

Circular Reference Collection How It Works

Slide 38

Slide 38 text

Circular Reference Collection How It Works

Slide 39

Slide 39 text

Circular Reference Collection How It Works

Slide 40

Slide 40 text

Circular Reference Collection How It Works

Slide 41

Slide 41 text

Circular Reference Collection How It Works

Slide 42

Slide 42 text

Circular Reference Collection How It Works

Slide 43

Slide 43 text

Circular Reference Collection How It Works

Slide 44

Slide 44 text

Circular Reference Collection How It Works

Slide 45

Slide 45 text

Refcounter + Cycles Collector = Proper Cleanup Of Items That Are Not Referenced Anymore

Slide 46

Slide 46 text

Memory Leak = Items Non Needed Anymore But Still Referenced, Directly Or Indirectly, By A Variable Still Alive

Slide 47

Slide 47 text

Let's Leak ! function buildObjects() { $objA = new StdClass(); $objB = new StdClass(); $objA->attr = $objB; $objB->attr = $objA; return $objA; } $leakHolder = []; for ($i = 0; $i < 200000; $i++) { $object = buildObjects(); if ($i % 10 === 0) { $leakHolder[] = $object; } }

Slide 48

Slide 48 text

GC + Memory Leak Perf Impacts

Slide 49

Slide 49 text

Why The Slow Down When The GC Buffer Is Full Of Referenced Items?

Slide 50

Slide 50 text

Why The Slow Down When The GC Buffer Is Full Of Referenced Items?

Slide 51

Slide 51 text

Why The Slow Down When The GC Buffer Is Full Of Referenced Items?

Slide 52

Slide 52 text

Why The Slow Down When The GC Buffer Is Full Of Referenced Items? Each time a refcount is decremented, your program is interrupted to rescan the buffer!

Slide 53

Slide 53 text

Hunting Down Memory Leaks from bird's eye view to ant's eye view

Slide 54

Slide 54 text

Hunting Memory Leaks With Profilers only provides information on when (at function level) memory is taken or released not on what or why

Slide 55

Slide 55 text

Hunting Memory Leaks With PhpMeminfo provides information on what so we can get to the why faster

Slide 56

Slide 56 text

PHP Meminfo MIT License Two parts: the extension itself, to dump memory content analyzers to work with memory dump files github.com/BitOne/php-meminfo

Slide 57

Slide 57 text

Import Example

Slide 58

Slide 58 text

Import Example - Data File durgan.esta,[email protected],"Ms. Juliana Friesen" gmorar,[email protected],"Tessie Haley" norma11,[email protected],"Mr. Zack Sawayn"

Slide 59

Slide 59 text

Import Example foreach ($fileContent as $data) { $counter++; $user = new User(); $user->setLogin($data[0]); $user->setEmail($data[1]); $user->setFullName($data[2]); $this->getEntityManager()->persist($user); if (0 === $counter % 1000) { $this->getEntityManager()->flush(); } } $this->getEntityManager()->flush();

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

Dumping Memory Content ... $this->getEntityManager()->persist($user); if (0 === $counter % 1000) { $this->getEntityManager()->flush(); meminfo_dump(fopen("mem$counter.json", 'w')); } ...

Slide 62

Slide 62 text

Memory Content Summary $ bin/analyzer summary mem1000.json +-----------------------+-----------------+---------------------+ | Type | Instances Count | Cumulated Self Size | +-----------------------+-----------------+---------------------+ | string | 7641 | 303612 | | integer | 4418 | 70688 | | array | 2691 | 193752 | | AppBundle\Entity\User | 1001 | 72072 | | boolean | 463 | 7408 | | null | 413 | 6608 | ... $ bin/analyzer summary mem5000.json +-----------------------+-----------------+---------------------+ | Type | Instances Count | Cumulated Self Size | +-----------------------+-----------------+---------------------+ | string | 31638 | 1072852 | | integer | 20418 | 326688 | | array | 10690 | 769680 | | AppBundle\Entity\User | 5001 | 360072 | | boolean | 463 | 7408 |

Slide 63

Slide 63 text

Finding A Good Leak Candidate $ bin/analyzer query -f "class~User" -v mem5000.json +-----------+------------------------------+---------------------+ | Item ids | Item data | Children | +-----------+------------------------------+---------------------+ | 0x2b46aa8 | Type: object | id: 0x2b3fa48 | | | Class: AppBundle\Entity\User | login: 0x2af8440 | | | Object Handle: 374 | fullName: 0x2584c48 | | | Size: 56 B | email: 0x2592580 | | | Is root: No | | +-----------+------------------------------+---------------------+ | 0x2b489f0 | Type: object | id: 0x2b47ae8 | | | Class: AppBundle\Entity\User | login: 0x2b45738 | | | Object Handle: 376 | fullName: 0x2a25448 | | | Size: 56 B | email: 0x2884590 | | | Is root: No | | +-----------+------------------------------+---------------------+ | 0x2b49250 | Type: object | id: 0x2acb7a0 | | | Class: AppBundle\Entity\User | login: 0x2b45df8 | | | Object Handle: 377 | fullName: 0x2883170 | | | Size: 56 B | email: 0x2abd7f0 | | | Is root: No | |

Slide 64

Slide 64 text

Why Is This Object Still In Memory? $ bin/analyzer ref-path 0x2b46aa8 mem5000.json Found 1 path(s) Path to 0x2b46aa8 ()$GLOBALS["kernel"] ->container ->services["doctrine.orm.default_entity_manager"] ->unitOfWork ->identityMap["AppBundle\Entity\User"]["9"]

Slide 65

Slide 65 text

Fixing The Leak! .... $this->getEntityManager()->persist($user); if (0 === $counter % 1000) { $this->getEntityManager()->flush(); $this->getEntityManager()->clear('AppBundle\\Entity\\User'); } ...

Slide 66

Slide 66 text

Performances Before And A er Fix

Slide 67

Slide 67 text

Memory Leak - Good Practices monitor your long running processes speed and memory usage avoid (or at least be aware of) stateful services use a reasonable memory limit (not -1 !) keep the number of items in memory at a low level

Slide 68

Slide 68 text

Thank You! Questions? @bjacquemont joind.in/talk/4492c https://github.com/BitOne/php-meminfo www.akeneo.com