Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hunting Down Memory Leaks with php Meminfo

Hunting Down Memory Leaks with php Meminfo

To better understand how a memory leak can occurs in PHP, this presentation intr
oduces how memory freeing is managed in PHP. Then it shows how the php Meminfo e
xtension can be used to debug and to help fix memory leaks.

Benoit Jacquemont

May 10, 2019
Tweet

More Decks by Benoit Jacquemont

Other Decks in Programming

Transcript

  1. HUNTING DOWN
    HUNTING DOWN
    HUNTING DOWN
    MEMORY LEAKS WITH
    MEMORY LEAKS WITH
    MEMORY LEAKS WITH
    PHP MEMINFO
    PHP MEMINFO
    PHP MEMINFO
    Benoit Jacquemont
    @bjacquemont

    View Slide

  2. What Is A Memory Leak?
    A memory leak occurs when memory which
    is no longer needed is not released.
    en.wikipedia.org/wiki/Memory_leak

    View Slide

  3. Memory Leaks: Should You Care?
    Usually, PHP process life time = HTTP exchange lifetime
    PHP process memory released a er each HTTP exchange
    Yes but...
    more and more capable-but-complex stacks
    long running background jobs

    View Slide

  4. Why I Care About Memory Leaks
    E-COMMERCE
    MOBILE APPLICATION
    PRINT CATALOG
    POINTS OF SALE
    ERP
    MEDIA SERVER
    SUPPLIERS
    PURCHASING DPT
    MARKETING DPT
    CSV
    FTP
    XML
    XLS
    SUPPLIERS
    PORTAL
    ENRIC
    H
    TRAN
    SLATE
    CO N T RO L
    Import Jobs Export Jobs
    Mass Edition Jobs
    Rules Engine Jobs
    ...

    View Slide

  5. The Effects Of A Memory Leak
    Less memory available for other processes (surprise !)
    The more memory your program uses, the slower it
    becomes

    View Slide

  6. But,
    How Is Memory Released In PHP?

    View Slide

  7. The Refcounter
    Basic and efficient.
    Frees memory as soon as it isn't used anymore.

    View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. View Slide

  16. View Slide

  17. View Slide

  18. View Slide

  19. View Slide

  20. View Slide

  21. View Slide

  22. View Slide

  23. View Slide

  24. View Slide

  25. View Slide

  26. 1st object will be released only when 2nd object is released
    2nd object will be released only when 1st object is released
    Neither of them can be released by the refcounter...
    The Refcounter Cannot Handle
    Circular References

    View Slide

  27. "Yeah sure, but circular references don't
    exist in the real world..."
    Tree structure:
    parent → children → parent
    Doctrine bidirectional mapping:
    entity1 → entity2 → entity1
    ...

    View Slide

  28. Circular Reference Collector
    To The Rescue!
    garbage collector exclusively dedicated to collect circular
    references

    View Slide

  29. But my memory usage keeps increasing!
    function buildObjects()
    {
    $objA = new StdClass();
    $objB = new StdClass();
    $objA->attr = $objB;
    $objB->attr = $objA;
    }
    for ($i = 0; $i < 2000; $i++) { buildObjects(); }

    View Slide

  30. Keep Calm And Wait For The Collect!
    There's a delay before the Collector cleans up memory

    View Slide

  31. Circular Reference Collection
    How It Works
    Each time an object or array has its refcount decremented,
    it gets added to the collector buffer.

    View Slide

  32. Circular Reference Collection
    How It Works

    View Slide

  33. Circular Reference Collection
    How It Works

    View Slide

  34. Circular Reference Collection
    How It Works

    View Slide

  35. Circular Reference Collection
    How It Works

    View Slide

  36. Circular Reference Collection
    How It Works

    View Slide

  37. Circular Reference Collection
    How It Works

    View Slide

  38. Circular Reference Collection
    How It Works

    View Slide

  39. Circular Reference Collection
    How It Works

    View Slide

  40. Circular Reference Collection
    How It Works

    View Slide

  41. Circular Reference Collection
    How It Works

    View Slide

  42. Circular Reference Collection
    How It Works

    View Slide

  43. Circular Reference Collection
    How It Works

    View Slide

  44. Circular Reference Collection
    How It Works

    View Slide

  45. Refcounter +
    Cycles Collector
    =
    Proper Cleanup Of Items
    That Are Not Referenced
    Anymore

    View Slide

  46. Memory Leak
    =
    Items Non Needed Anymore
    But Still Referenced, Directly
    Or Indirectly,
    By A Variable Still Alive

    View Slide

  47. Let's Leak !
    function buildObjects() {
    $objA = new StdClass();
    $objB = new StdClass();
    $objA->attr = $objB;
    $objB->attr = $objA;
    return $objA;
    }
    $leakHolder = [];
    for ($i = 0; $i < 200000; $i++) {
    $object = buildObjects();
    if ($i % 10 === 0) {
    $leakHolder[] = $object;
    }
    }

    View Slide

  48. GC + Memory Leak Perf Impacts

    View Slide

  49. Why The Slow Down When The GC
    Buffer Is Full Of Referenced Items?

    View Slide

  50. Why The Slow Down When The GC
    Buffer Is Full Of Referenced Items?

    View Slide

  51. Why The Slow Down When The GC
    Buffer Is Full Of Referenced Items?

    View Slide

  52. Why The Slow Down When The GC
    Buffer Is Full Of Referenced Items?
    Each time a refcount is decremented,
    your program is interrupted to rescan
    the buffer!

    View Slide

  53. Hunting Down Memory Leaks
    from bird's eye view to ant's eye view

    View Slide

  54. Hunting Memory Leaks
    With Profilers
    only provides information on when (at function level)
    memory is taken or released
    not on what or why

    View Slide

  55. Hunting Memory Leaks
    With PhpMeminfo
    provides information on what so we can get to the why
    faster

    View Slide

  56. PHP Meminfo
    MIT License
    Two parts:
    the extension itself, to dump memory content
    analyzers to work with memory dump files
    github.com/BitOne/php-meminfo

    View Slide

  57. Import Example

    View Slide

  58. Import Example - Data File
    durgan.esta,[email protected],"Ms. Juliana Friesen"
    gmorar,[email protected],"Tessie Haley"
    norma11,[email protected],"Mr. Zack Sawayn"

    View Slide

  59. Import Example
    foreach ($fileContent as $data) {
    $counter++;
    $user = new User();
    $user->setLogin($data[0]);
    $user->setEmail($data[1]);
    $user->setFullName($data[2]);
    $this->getEntityManager()->persist($user);
    if (0 === $counter % 1000) {
    $this->getEntityManager()->flush();
    }
    }
    $this->getEntityManager()->flush();

    View Slide

  60. View Slide

  61. Dumping Memory Content
    ...
    $this->getEntityManager()->persist($user);
    if (0 === $counter % 1000) {
    $this->getEntityManager()->flush();
    meminfo_dump(fopen("mem$counter.json", 'w'));
    }
    ...

    View Slide

  62. Memory Content Summary
    $ bin/analyzer summary mem1000.json
    +-----------------------+-----------------+---------------------+
    | Type | Instances Count | Cumulated Self Size |
    +-----------------------+-----------------+---------------------+
    | string | 7641 | 303612 |
    | integer | 4418 | 70688 |
    | array | 2691 | 193752 |
    | AppBundle\Entity\User | 1001 | 72072 |
    | boolean | 463 | 7408 |
    | null | 413 | 6608 |
    ...
    $ bin/analyzer summary mem5000.json
    +-----------------------+-----------------+---------------------+
    | Type | Instances Count | Cumulated Self Size |
    +-----------------------+-----------------+---------------------+
    | string | 31638 | 1072852 |
    | integer | 20418 | 326688 |
    | array | 10690 | 769680 |
    | AppBundle\Entity\User | 5001 | 360072 |
    | boolean | 463 | 7408 |

    View Slide

  63. Finding A Good Leak Candidate
    $ bin/analyzer query -f "class~User" -v mem5000.json
    +-----------+------------------------------+---------------------+
    | Item ids | Item data | Children |
    +-----------+------------------------------+---------------------+
    | 0x2b46aa8 | Type: object | id: 0x2b3fa48 |
    | | Class: AppBundle\Entity\User | login: 0x2af8440 |
    | | Object Handle: 374 | fullName: 0x2584c48 |
    | | Size: 56 B | email: 0x2592580 |
    | | Is root: No | |
    +-----------+------------------------------+---------------------+
    | 0x2b489f0 | Type: object | id: 0x2b47ae8 |
    | | Class: AppBundle\Entity\User | login: 0x2b45738 |
    | | Object Handle: 376 | fullName: 0x2a25448 |
    | | Size: 56 B | email: 0x2884590 |
    | | Is root: No | |
    +-----------+------------------------------+---------------------+
    | 0x2b49250 | Type: object | id: 0x2acb7a0 |
    | | Class: AppBundle\Entity\User | login: 0x2b45df8 |
    | | Object Handle: 377 | fullName: 0x2883170 |
    | | Size: 56 B | email: 0x2abd7f0 |
    | | Is root: No | |

    View Slide

  64. Why Is This Object Still In Memory?
    $ bin/analyzer ref-path 0x2b46aa8 mem5000.json
    Found 1 path(s)
    Path to 0x2b46aa8
    ()$GLOBALS["kernel"]
    ->container
    ->services["doctrine.orm.default_entity_manager"]
    ->unitOfWork
    ->identityMap["AppBundle\Entity\User"]["9"]

    View Slide

  65. Fixing The Leak!
    ....
    $this->getEntityManager()->persist($user);
    if (0 === $counter % 1000) {
    $this->getEntityManager()->flush();
    $this->getEntityManager()->clear('AppBundle\\Entity\\User');
    }
    ...

    View Slide

  66. Performances Before And A er Fix

    View Slide

  67. Memory Leak - Good Practices
    monitor your long running processes speed and memory
    usage
    avoid (or at least be aware of) stateful services
    use a reasonable memory limit (not -1 !)
    keep the number of items in memory at a low level

    View Slide

  68. Thank You!
    Questions?
    @bjacquemont
    joind.in/talk/4492c
    https://github.com/BitOne/php-meminfo
    www.akeneo.com

    View Slide