Hunting Down Memory Leaks with php Meminfo

Hunting Down Memory Leaks with php Meminfo

To better understand how a memory leak can occurs in PHP, this presentation intr
oduces how memory freeing is managed in PHP. Then it shows how the php Meminfo e
xtension can be used to debug and to help fix memory leaks.

6648bd4390fba79c9baa6045e58fa337?s=128

Benoit Jacquemont

May 10, 2019
Tweet

Transcript

  1. HUNTING DOWN HUNTING DOWN HUNTING DOWN MEMORY LEAKS WITH MEMORY

    LEAKS WITH MEMORY LEAKS WITH PHP MEMINFO PHP MEMINFO PHP MEMINFO Benoit Jacquemont @bjacquemont
  2. What Is A Memory Leak? A memory leak occurs when

    memory which is no longer needed is not released. en.wikipedia.org/wiki/Memory_leak
  3. Memory Leaks: Should You Care? Usually, PHP process life time

    = HTTP exchange lifetime PHP process memory released a er each HTTP exchange Yes but... more and more capable-but-complex stacks long running background jobs
  4. Why I Care About Memory Leaks E-COMMERCE MOBILE APPLICATION PRINT

    CATALOG POINTS OF SALE ERP MEDIA SERVER SUPPLIERS PURCHASING DPT MARKETING DPT CSV FTP XML XLS SUPPLIERS PORTAL ENRIC H TRAN SLATE CO N T RO L Import Jobs Export Jobs Mass Edition Jobs Rules Engine Jobs ...
  5. The Effects Of A Memory Leak Less memory available for

    other processes (surprise !) The more memory your program uses, the slower it becomes
  6. But, How Is Memory Released In PHP?

  7. The Refcounter Basic and efficient. Frees memory as soon as

    it isn't used anymore.
  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. None
  15. None
  16. None
  17. None
  18. None
  19. None
  20. None
  21. None
  22. None
  23. None
  24. None
  25. None
  26. 1st object will be released only when 2nd object is

    released 2nd object will be released only when 1st object is released Neither of them can be released by the refcounter... The Refcounter Cannot Handle Circular References
  27. "Yeah sure, but circular references don't exist in the real

    world..." Tree structure: parent → children → parent Doctrine bidirectional mapping: entity1 → entity2 → entity1 ...
  28. Circular Reference Collector To The Rescue! garbage collector exclusively dedicated

    to collect circular references
  29. But my memory usage keeps increasing! function buildObjects() { $objA

    = new StdClass(); $objB = new StdClass(); $objA->attr = $objB; $objB->attr = $objA; } for ($i = 0; $i < 2000; $i++) { buildObjects(); }
  30. Keep Calm And Wait For The Collect! There's a delay

    before the Collector cleans up memory
  31. Circular Reference Collection How It Works Each time an object

    or array has its refcount decremented, it gets added to the collector buffer.
  32. Circular Reference Collection How It Works

  33. Circular Reference Collection How It Works

  34. Circular Reference Collection How It Works

  35. Circular Reference Collection How It Works

  36. Circular Reference Collection How It Works

  37. Circular Reference Collection How It Works

  38. Circular Reference Collection How It Works

  39. Circular Reference Collection How It Works

  40. Circular Reference Collection How It Works

  41. Circular Reference Collection How It Works

  42. Circular Reference Collection How It Works

  43. Circular Reference Collection How It Works

  44. Circular Reference Collection How It Works

  45. Refcounter + Cycles Collector = Proper Cleanup Of Items That

    Are Not Referenced Anymore
  46. Memory Leak = Items Non Needed Anymore But Still Referenced,

    Directly Or Indirectly, By A Variable Still Alive
  47. Let's Leak ! function buildObjects() { $objA = new StdClass();

    $objB = new StdClass(); $objA->attr = $objB; $objB->attr = $objA; return $objA; } $leakHolder = []; for ($i = 0; $i < 200000; $i++) { $object = buildObjects(); if ($i % 10 === 0) { $leakHolder[] = $object; } }
  48. GC + Memory Leak Perf Impacts

  49. Why The Slow Down When The GC Buffer Is Full

    Of Referenced Items?
  50. Why The Slow Down When The GC Buffer Is Full

    Of Referenced Items?
  51. Why The Slow Down When The GC Buffer Is Full

    Of Referenced Items?
  52. Why The Slow Down When The GC Buffer Is Full

    Of Referenced Items? Each time a refcount is decremented, your program is interrupted to rescan the buffer!
  53. Hunting Down Memory Leaks from bird's eye view to ant's

    eye view
  54. Hunting Memory Leaks With Profilers only provides information on when

    (at function level) memory is taken or released not on what or why
  55. Hunting Memory Leaks With PhpMeminfo provides information on what so

    we can get to the why faster
  56. PHP Meminfo MIT License Two parts: the extension itself, to

    dump memory content analyzers to work with memory dump files github.com/BitOne/php-meminfo
  57. Import Example

  58. Import Example - Data File durgan.esta,maryjane82@blick.com,"Ms. Juliana Friesen" gmorar,hessel.jarrod@gmail.com,"Tessie Haley"

    norma11,maida55@hotmail.com,"Mr. Zack Sawayn"
  59. Import Example foreach ($fileContent as $data) { $counter++; $user =

    new User(); $user->setLogin($data[0]); $user->setEmail($data[1]); $user->setFullName($data[2]); $this->getEntityManager()->persist($user); if (0 === $counter % 1000) { $this->getEntityManager()->flush(); } } $this->getEntityManager()->flush();
  60. None
  61. Dumping Memory Content ... $this->getEntityManager()->persist($user); if (0 === $counter %

    1000) { $this->getEntityManager()->flush(); meminfo_dump(fopen("mem$counter.json", 'w')); } ...
  62. Memory Content Summary $ bin/analyzer summary mem1000.json +-----------------------+-----------------+---------------------+ | Type

    | Instances Count | Cumulated Self Size | +-----------------------+-----------------+---------------------+ | string | 7641 | 303612 | | integer | 4418 | 70688 | | array | 2691 | 193752 | | AppBundle\Entity\User | 1001 | 72072 | | boolean | 463 | 7408 | | null | 413 | 6608 | ... $ bin/analyzer summary mem5000.json +-----------------------+-----------------+---------------------+ | Type | Instances Count | Cumulated Self Size | +-----------------------+-----------------+---------------------+ | string | 31638 | 1072852 | | integer | 20418 | 326688 | | array | 10690 | 769680 | | AppBundle\Entity\User | 5001 | 360072 | | boolean | 463 | 7408 |
  63. Finding A Good Leak Candidate $ bin/analyzer query -f "class~User"

    -v mem5000.json +-----------+------------------------------+---------------------+ | Item ids | Item data | Children | +-----------+------------------------------+---------------------+ | 0x2b46aa8 | Type: object | id: 0x2b3fa48 | | | Class: AppBundle\Entity\User | login: 0x2af8440 | | | Object Handle: 374 | fullName: 0x2584c48 | | | Size: 56 B | email: 0x2592580 | | | Is root: No | | +-----------+------------------------------+---------------------+ | 0x2b489f0 | Type: object | id: 0x2b47ae8 | | | Class: AppBundle\Entity\User | login: 0x2b45738 | | | Object Handle: 376 | fullName: 0x2a25448 | | | Size: 56 B | email: 0x2884590 | | | Is root: No | | +-----------+------------------------------+---------------------+ | 0x2b49250 | Type: object | id: 0x2acb7a0 | | | Class: AppBundle\Entity\User | login: 0x2b45df8 | | | Object Handle: 377 | fullName: 0x2883170 | | | Size: 56 B | email: 0x2abd7f0 | | | Is root: No | |
  64. Why Is This Object Still In Memory? $ bin/analyzer ref-path

    0x2b46aa8 mem5000.json Found 1 path(s) Path to 0x2b46aa8 (<GLOBAL>)$GLOBALS["kernel"] ->container ->services["doctrine.orm.default_entity_manager"] ->unitOfWork ->identityMap["AppBundle\Entity\User"]["9"]
  65. Fixing The Leak! .... $this->getEntityManager()->persist($user); if (0 === $counter %

    1000) { $this->getEntityManager()->flush(); $this->getEntityManager()->clear('AppBundle\\Entity\\User'); } ...
  66. Performances Before And A er Fix

  67. Memory Leak - Good Practices monitor your long running processes

    speed and memory usage avoid (or at least be aware of) stateful services use a reasonable memory limit (not -1 !) keep the number of items in memory at a low level
  68. Thank You! Questions? @bjacquemont joind.in/talk/4492c https://github.com/BitOne/php-meminfo www.akeneo.com