Hunting Down Memory Leaks with php Meminfo

To better understand how a memory leak can occurs in PHP, this presentation intr
oduces how memory freeing is managed in PHP. Then it shows how the php Meminfo e
xtension can be used to debug and to help fix memory leaks.

Transcript

1. HUNTING DOWN
   HUNTING DOWN
   HUNTING DOWN
   MEMORY LEAKS WITH
   MEMORY LEAKS WITH
   MEMORY LEAKS WITH
   PHP MEMINFO
   PHP MEMINFO
   PHP MEMINFO
   Benoit Jacquemont
   @bjacquemont
   

   View Slide

2. What Is A Memory Leak?
   A memory leak occurs when memory which
   is no longer needed is not released.
   en.wikipedia.org/wiki/Memory_leak
   

   View Slide

3. Memory Leaks: Should You Care?
   Usually, PHP process life time = HTTP exchange lifetime
   PHP process memory released a er each HTTP exchange
   Yes but...
   more and more capable-but-complex stacks
   long running background jobs
   

   View Slide

4. Why I Care About Memory Leaks
   E-COMMERCE
   MOBILE APPLICATION
   PRINT CATALOG
   POINTS OF SALE
   ERP
   MEDIA SERVER
   SUPPLIERS
   PURCHASING DPT
   MARKETING DPT
   CSV
   FTP
   XML
   XLS
   SUPPLIERS
   PORTAL
   ENRIC
   H
   TRAN
   SLATE
   CO N T RO L
   Import Jobs Export Jobs
   Mass Edition Jobs
   Rules Engine Jobs
   ...
   

   View Slide

5. The Effects Of A Memory Leak
   Less memory available for other processes (surprise !)
   The more memory your program uses, the slower it
   becomes
   

   View Slide

6. But,
   How Is Memory Released In PHP?
   

   View Slide

7. The Refcounter
   Basic and efficient.
   Frees memory as soon as it isn't used anymore.
   

   View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. 1st object will be released only when 2nd object is released
    2nd object will be released only when 1st object is released
    Neither of them can be released by the refcounter...
    The Refcounter Cannot Handle
    Circular References
    
    

    View Slide

27. "Yeah sure, but circular references don't
    exist in the real world..."
    Tree structure:
    parent → children → parent
    Doctrine bidirectional mapping:
    entity1 → entity2 → entity1
    ...
    

    View Slide

28. Circular Reference Collector
    To The Rescue!
    garbage collector exclusively dedicated to collect circular
    references
    

    View Slide

29. But my memory usage keeps increasing!
    function buildObjects()
    {
    $objA = new StdClass();
    $objB = new StdClass();
    $objA->attr = $objB;
    $objB->attr = $objA;
    }
    for ($i = 0; $i < 2000; $i++) { buildObjects(); }
    

    View Slide

30. Keep Calm And Wait For The Collect!
    There's a delay before the Collector cleans up memory
    

    View Slide

31. Circular Reference Collection
    How It Works
    Each time an object or array has its refcount decremented,
    it gets added to the collector buffer.
    

    View Slide

32. Circular Reference Collection
    How It Works
    

    View Slide

33. Circular Reference Collection
    How It Works
    

    View Slide

34. Circular Reference Collection
    How It Works
    

    View Slide

35. Circular Reference Collection
    How It Works
    

    View Slide

36. Circular Reference Collection
    How It Works
    

    View Slide

37. Circular Reference Collection
    How It Works
    

    View Slide

38. Circular Reference Collection
    How It Works
    

    View Slide

39. Circular Reference Collection
    How It Works
    

    View Slide

40. Circular Reference Collection
    How It Works
    

    View Slide

41. Circular Reference Collection
    How It Works
    

    View Slide

42. Circular Reference Collection
    How It Works
    

    View Slide

43. Circular Reference Collection
    How It Works
    

    View Slide

44. Circular Reference Collection
    How It Works
    

    View Slide

45. Refcounter +
    Cycles Collector
    =
    Proper Cleanup Of Items
    That Are Not Referenced
    Anymore
    

    View Slide

46. Memory Leak
    =
    Items Non Needed Anymore
    But Still Referenced, Directly
    Or Indirectly,
    By A Variable Still Alive
    

    View Slide

47. Let's Leak !
    function buildObjects() {
    $objA = new StdClass();
    $objB = new StdClass();
    $objA->attr = $objB;
    $objB->attr = $objA;
    return $objA;
    }
    $leakHolder = [];
    for ($i = 0; $i < 200000; $i++) {
    $object = buildObjects();
    if ($i % 10 === 0) {
    $leakHolder[] = $object;
    }
    }
    

    View Slide

48. GC + Memory Leak Perf Impacts
    

    View Slide

49. Why The Slow Down When The GC
    Buffer Is Full Of Referenced Items?
    

    View Slide

50. Why The Slow Down When The GC
    Buffer Is Full Of Referenced Items?
    

    View Slide

51. Why The Slow Down When The GC
    Buffer Is Full Of Referenced Items?
    

    View Slide

52. Why The Slow Down When The GC
    Buffer Is Full Of Referenced Items?
    Each time a refcount is decremented,
    your program is interrupted to rescan
    the buffer!
    

    View Slide

53. Hunting Down Memory Leaks
    from bird's eye view to ant's eye view
    

    View Slide

54. Hunting Memory Leaks
    With Profilers
    only provides information on when (at function level)
    memory is taken or released
    not on what or why
    

    View Slide

55. Hunting Memory Leaks
    With PhpMeminfo
    provides information on what so we can get to the why
    faster
    

    View Slide

56. PHP Meminfo
    MIT License
    Two parts:
    the extension itself, to dump memory content
    analyzers to work with memory dump files
    github.com/BitOne/php-meminfo
    

    View Slide

57. Import Example
    

    View Slide

58. Import Example - Data File
    durgan.esta,maryjane82@blick.com,"Ms. Juliana Friesen"
    gmorar,hessel.jarrod@gmail.com,"Tessie Haley"
    norma11,maida55@hotmail.com,"Mr. Zack Sawayn"
    

    View Slide

59. Import Example
    foreach ($fileContent as $data) {
    $counter++;
    $user = new User();
    $user->setLogin($data[0]);
    $user->setEmail($data[1]);
    $user->setFullName($data[2]);
    $this->getEntityManager()->persist($user);
    if (0 === $counter % 1000) {
    $this->getEntityManager()->flush();
    }
    }
    $this->getEntityManager()->flush();
    

    View Slide

60. View Slide

61. Dumping Memory Content
    ...
    $this->getEntityManager()->persist($user);
    if (0 === $counter % 1000) {
    $this->getEntityManager()->flush();
    meminfo_dump(fopen("mem$counter.json", 'w'));
    }
    ...
    

    View Slide

62. Memory Content Summary
    $ bin/analyzer summary mem1000.json
    +-----------------------+-----------------+---------------------+
    | Type | Instances Count | Cumulated Self Size |
    +-----------------------+-----------------+---------------------+
    | string | 7641 | 303612 |
    | integer | 4418 | 70688 |
    | array | 2691 | 193752 |
    | AppBundle\Entity\User | 1001 | 72072 |
    | boolean | 463 | 7408 |
    | null | 413 | 6608 |
    ...
    $ bin/analyzer summary mem5000.json
    +-----------------------+-----------------+---------------------+
    | Type | Instances Count | Cumulated Self Size |
    +-----------------------+-----------------+---------------------+
    | string | 31638 | 1072852 |
    | integer | 20418 | 326688 |
    | array | 10690 | 769680 |
    | AppBundle\Entity\User | 5001 | 360072 |
    | boolean | 463 | 7408 |
    

    View Slide

63. Finding A Good Leak Candidate
    $ bin/analyzer query -f "class~User" -v mem5000.json
    +-----------+------------------------------+---------------------+
    | Item ids | Item data | Children |
    +-----------+------------------------------+---------------------+
    | 0x2b46aa8 | Type: object | id: 0x2b3fa48 |
    | | Class: AppBundle\Entity\User | login: 0x2af8440 |
    | | Object Handle: 374 | fullName: 0x2584c48 |
    | | Size: 56 B | email: 0x2592580 |
    | | Is root: No | |
    +-----------+------------------------------+---------------------+
    | 0x2b489f0 | Type: object | id: 0x2b47ae8 |
    | | Class: AppBundle\Entity\User | login: 0x2b45738 |
    | | Object Handle: 376 | fullName: 0x2a25448 |
    | | Size: 56 B | email: 0x2884590 |
    | | Is root: No | |
    +-----------+------------------------------+---------------------+
    | 0x2b49250 | Type: object | id: 0x2acb7a0 |
    | | Class: AppBundle\Entity\User | login: 0x2b45df8 |
    | | Object Handle: 377 | fullName: 0x2883170 |
    | | Size: 56 B | email: 0x2abd7f0 |
    | | Is root: No | |
    

    View Slide

64. Why Is This Object Still In Memory?
    $ bin/analyzer ref-path 0x2b46aa8 mem5000.json
    Found 1 path(s)
    Path to 0x2b46aa8
    ()$GLOBALS["kernel"]
    ->container
    ->services["doctrine.orm.default_entity_manager"]
    ->unitOfWork
    ->identityMap["AppBundle\Entity\User"]["9"]
    

    View Slide

65. Fixing The Leak!
    ....
    $this->getEntityManager()->persist($user);
    if (0 === $counter % 1000) {
    $this->getEntityManager()->flush();
    $this->getEntityManager()->clear('AppBundle\\Entity\\User');
    }
    ...
    

    View Slide

66. Performances Before And A er Fix
    

    View Slide

67. Memory Leak - Good Practices
    monitor your long running processes speed and memory
    usage
    avoid (or at least be aware of) stateful services
    use a reasonable memory limit (not -1 !)
    keep the number of items in memory at a low level
    

    View Slide

68. Thank You!
    Questions?
    @bjacquemont
    joind.in/talk/4492c
    https://github.com/BitOne/php-meminfo
    www.akeneo.com
    

    View Slide