Hunting Down Memory Leaks with php Meminfo

To better understand how a memory leak can occurs in PHP, this presentation intr
oduces how memory freeing is managed in PHP. Then it shows how the php Meminfo e
xtension can be used to debug and to help fix memory leaks.

Transcript

1. HUNTING DOWN HUNTING DOWN HUNTING DOWN MEMORY LEAKS WITH MEMORY LEAKS WITH MEMORY LEAKS WITH PHP MEMINFO PHP MEMINFO PHP MEMINFO Benoit Jacquemont @bjacquemont
2. What Is A Memory Leak? A memory leak occurs when memory which is no longer needed is not released. en.wikipedia.org/wiki/Memory_leak
3. Memory Leaks: Should You Care? Usually, PHP process life time = HTTP exchange lifetime PHP process memory released a er each HTTP exchange Yes but... more and more capable-but-complex stacks long running background jobs
4. Why I Care About Memory Leaks E-COMMERCE MOBILE APPLICATION PRINT CATALOG POINTS OF SALE ERP MEDIA SERVER SUPPLIERS PURCHASING DPT MARKETING DPT CSV FTP XML XLS SUPPLIERS PORTAL ENRIC H TRAN SLATE CO N T RO L Import Jobs Export Jobs Mass Edition Jobs Rules Engine Jobs ...
5. The Effects Of A Memory Leak Less memory available for other processes (surprise !) The more memory your program uses, the slower it becomes
6. But, How Is Memory Released In PHP?
7. The Refcounter Basic and efficient. Frees memory as soon as it isn't used anymore.
26. 1st object will be released only when 2nd object is released 2nd object will be released only when 1st object is released Neither of them can be released by the refcounter... The Refcounter Cannot Handle Circular References
27. "Yeah sure, but circular references don't exist in the real world..." Tree structure: parent → children → parent Doctrine bidirectional mapping: entity1 → entity2 → entity1 ...
28. Circular Reference Collector To The Rescue! garbage collector exclusively dedicated to collect circular references
29. But my memory usage keeps increasing! function buildObjects() { $objA = new StdClass(); $objB = new StdClass(); $objA->attr = $objB; $objB->attr = $objA; } for ($i = 0; $i < 2000; $i++) { buildObjects(); }
30. Keep Calm And Wait For The Collect! There's a delay before the Collector cleans up memory
31. Circular Reference Collection How It Works Each time an object or array has its refcount decremented, it gets added to the collector buffer.
32. Circular Reference Collection How It Works
33. Circular Reference Collection How It Works
34. Circular Reference Collection How It Works
35. Circular Reference Collection How It Works
36. Circular Reference Collection How It Works
37. Circular Reference Collection How It Works
38. Circular Reference Collection How It Works
39. Circular Reference Collection How It Works
40. Circular Reference Collection How It Works
41. Circular Reference Collection How It Works
42. Circular Reference Collection How It Works
43. Circular Reference Collection How It Works
44. Circular Reference Collection How It Works
45. Refcounter + Cycles Collector = Proper Cleanup Of Items That Are Not Referenced Anymore
46. Memory Leak = Items Non Needed Anymore But Still Referenced, Directly Or Indirectly, By A Variable Still Alive
47. Let's Leak ! function buildObjects() { $objA = new StdClass(); $objB = new StdClass(); $objA->attr = $objB; $objB->attr = $objA; return $objA; } $leakHolder = []; for ($i = 0; $i < 200000; $i++) { $object = buildObjects(); if ($i % 10 === 0) { $leakHolder[] = $object; } }
48. GC + Memory Leak Perf Impacts
49. Why The Slow Down When The GC Buffer Is Full Of Referenced Items?
50. Why The Slow Down When The GC Buffer Is Full Of Referenced Items?
51. Why The Slow Down When The GC Buffer Is Full Of Referenced Items?
52. Why The Slow Down When The GC Buffer Is Full Of Referenced Items? Each time a refcount is decremented, your program is interrupted to rescan the buffer!
53. Hunting Down Memory Leaks from bird's eye view to ant's eye view
54. Hunting Memory Leaks With Profilers only provides information on when (at function level) memory is taken or released not on what or why
55. Hunting Memory Leaks With PhpMeminfo provides information on what so we can get to the why faster
56. PHP Meminfo MIT License Two parts: the extension itself, to dump memory content analyzers to work with memory dump files github.com/BitOne/php-meminfo
57. Import Example
58. Import Example - Data File durgan.esta,maryjane82@blick.com,"Ms. Juliana Friesen" gmorar,hessel.jarrod@gmail.com,"Tessie Haley" norma11,maida55@hotmail.com,"Mr. Zack Sawayn"
59. Import Example foreach ($fileContent as $data) { $counter++; $user = new User(); $user->setLogin($data[0]); $user->setEmail($data[1]); $user->setFullName($data[2]); $this->getEntityManager()->persist($user); if (0 === $counter % 1000) { $this->getEntityManager()->flush(); } } $this->getEntityManager()->flush();
61. Dumping Memory Content ... $this->getEntityManager()->persist($user); if (0 === $counter % 1000) { $this->getEntityManager()->flush(); meminfo_dump(fopen("mem$counter.json", 'w')); } ...
62. Memory Content Summary $ bin/analyzer summary mem1000.json +-----------------------+-----------------+---------------------+ | Type | Instances Count | Cumulated Self Size | +-----------------------+-----------------+---------------------+ | string | 7641 | 303612 | | integer | 4418 | 70688 | | array | 2691 | 193752 | | AppBundle\Entity\User | 1001 | 72072 | | boolean | 463 | 7408 | | null | 413 | 6608 | ... $ bin/analyzer summary mem5000.json +-----------------------+-----------------+---------------------+ | Type | Instances Count | Cumulated Self Size | +-----------------------+-----------------+---------------------+ | string | 31638 | 1072852 | | integer | 20418 | 326688 | | array | 10690 | 769680 | | AppBundle\Entity\User | 5001 | 360072 | | boolean | 463 | 7408 |
63. Finding A Good Leak Candidate $ bin/analyzer query -f "class~User" -v mem5000.json +-----------+------------------------------+---------------------+ | Item ids | Item data | Children | +-----------+------------------------------+---------------------+ | 0x2b46aa8 | Type: object | id: 0x2b3fa48 | | | Class: AppBundle\Entity\User | login: 0x2af8440 | | | Object Handle: 374 | fullName: 0x2584c48 | | | Size: 56 B | email: 0x2592580 | | | Is root: No | | +-----------+------------------------------+---------------------+ | 0x2b489f0 | Type: object | id: 0x2b47ae8 | | | Class: AppBundle\Entity\User | login: 0x2b45738 | | | Object Handle: 376 | fullName: 0x2a25448 | | | Size: 56 B | email: 0x2884590 | | | Is root: No | | +-----------+------------------------------+---------------------+ | 0x2b49250 | Type: object | id: 0x2acb7a0 | | | Class: AppBundle\Entity\User | login: 0x2b45df8 | | | Object Handle: 377 | fullName: 0x2883170 | | | Size: 56 B | email: 0x2abd7f0 | | | Is root: No | |
64. Why Is This Object Still In Memory? $ bin/analyzer ref-path 0x2b46aa8 mem5000.json Found 1 path(s) Path to 0x2b46aa8 (<GLOBAL>)$GLOBALS["kernel"] ->container ->services["doctrine.orm.default_entity_manager"] ->unitOfWork ->identityMap["AppBundle\Entity\User"]["9"]
65. Fixing The Leak! .... $this->getEntityManager()->persist($user); if (0 === $counter % 1000) { $this->getEntityManager()->flush(); $this->getEntityManager()->clear('AppBundle\\Entity\\User'); } ...
66. Performances Before And A er Fix
67. Memory Leak - Good Practices monitor your long running processes speed and memory usage avoid (or at least be aware of) stateful services use a reasonable memory limit (not -1 !) keep the number of items in memory at a low level
68. Thank You! Questions? @bjacquemont joind.in/talk/4492c https://github.com/BitOne/php-meminfo www.akeneo.com