How Doctrine caching can skyrocket your application (SymfonyCon Amsterdam 2019)

When people talk about Doctrine (or any ORM for that matter), the performance issue always comes up fairly quickly. Besides the fact that Doctrine will help you develop faster, so a little overhead doesn't really matter, there are numerous options to increase the performance of the application.

By understanding how the system works in the first place, a lot of issues can be avoided right away.

When you have done everything to avoid these pitfalls, you can bring in the big guns: caching. Doctrine has several caching mechanism and since Doctrine 2.5 "Second Level Cache" was added to our toolbox. After this talk, you should know what the impact is of every cache and how to use them.

Transcript

1. HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   Jachim Coudenys
   Jachim Coudenys
   Jachim Coudenys
   Jachim Coudenys
   Jachim Coudenys
   Jachim Coudenys
   Jachim Coudenys
   Jachim Coudenys
   @coudenysj - jachim.be
   @coudenysj - jachim.be
   @coudenysj - jachim.be
   @coudenysj - jachim.be
   @coudenysj - jachim.be
   @coudenysj - jachim.be
   @coudenysj - jachim.be
   @coudenysj - jachim.be
   jachim.coudenys@combellgroup.com
   jachim.coudenys@combellgroup.com
   jachim.coudenys@combellgroup.com
   jachim.coudenys@combellgroup.com
   jachim.coudenys@combellgroup.com
   jachim.coudenys@combellgroup.com
   jachim.coudenys@combellgroup.com
   jachim.coudenys@combellgroup.com
   

   View Slide

2. Jachim Coudenys
   Jachim Coudenys
   

   View Slide

3. View Slide

4. View Slide

5. HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   

   View Slide

6. HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   HOW DOCTRINE
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   CACHING CAN
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   SKYROCKET YOUR
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   APPLICATION
   If you're using Doctrine ORM.
   If you're using Doctrine ORM.
   

   View Slide

7. WHAT IS DOCTRINE
   

   View Slide

8. OBJECT-RELATIONAL MAPPER
   ... is a programming technique for
   converting data between incompatible
   type systems using object-oriented
   programming languages.
   — https://en.wikipedia.org/wiki/Object-relational_mapping
   

   View Slide

9. TERMINOLOGY
   Entity
   Mapping
   Repository
   EntityManager
   

   View Slide

10. A Bug has a description, creation date, status, reporter and
    engineer
    A Bug can occur on different Products (platforms)
    A Product has a name.
    Bug reporters and engineers are both Users of the system.
    A User can create new Bugs.
    The assigned engineer can close a Bug.
    A User can see all his reported or assigned Bugs.
    Bugs can be paginated through a list-view.
    AN EXAMPLE MODEL: BUG
    TRACKER
    

    View Slide

11. https://github.com/coudenysj/doctrine2-orm-tutorial
    https://github.com/coudenysj/doctrine2-orm-tutorial
    git checkout query-logging
    ls src/*
    ./tutorial.sh
    DOCTRINE2-ORM-TUTORIAL
    

    View Slide

12. SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS
    

    View Slide

13. Application Repository DBAL --> EM
    

    View Slide

14. ENTITYMANAGER
    Central access point
    API used to query and manage object
    

    View Slide

15. --> Unit of Work
    

    View Slide

16. UNIT OF WORK
    Maintains a list of objects affected by a
    business transaction and coordinates
    the writing out of changes and the
    resolution of concurrency problems.
    — https://martinfowler.com/eaaCatalog/unitOfWork.html
    

    View Slide

17. UNIT OF WORK
    Unit of Work pattern
    keep track of things to do with next flush
    no direct interaction
    

    View Slide

18. View Slide

19. TRANSACTIONAL WRITE-
    BEHIND
    UoW concept
    Delays execution of queries (optimized in short
    transaction)
    

    View Slide

20. --> Identity Map
    

    View Slide

21. IDENTITY MAP
    Ensures that each object gets loaded
    only once by keeping every loaded
    object in a map. Looks up objects using
    the map when referring to them.
    — https://martinfowler.com/eaaCatalog/identityMap.html
    

    View Slide

22. IDENTITY MAP
    A man with two watches never knows what time it
    is
    Load data in the same in-memory object
    Sync to database correctly
    

    View Slide

23. --> States
    

    View Slide

24. ENTITY STATES
    NEW
    MANAGED
    DETACHED
    REMOVED
    

    View Slide

25. --> Hydration
    

    View Slide

26. HYDRATION
    

    View Slide

27. https://symfony.com/doc/current/components/serializer.html
    https://symfony.com/doc/current/components/serializer.html
    

    View Slide

28. --> Proxies
    

    View Slide

29. PROXIES (LAZY LOADING)
    An object that doesn't contain all of
    the data you need but knows how to
    get it.
    — https://martinfowler.com/eaaCatalog/lazyLoad.html
    

    View Slide

30. Complete picture
    

    View Slide

31. https://github.com/coudenysj/doctrine2-orm-tutorial
    https://github.com/coudenysj/doctrine2-orm-tutorial
    git checkout internals
    SQL_LOGGER=1 ./tutorial.sh
    SQL_LOGGER=1 ./internals.sh
    DOCTRINE2-ORM-TUTORIAL
    

    View Slide

32. TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES
    

    View Slide

33. CREATING
    Create new object (NEW)
    Persist the object (MANAGED)
    Flush the EM
    UoW calculates changes => INSERT SQL
    Transaction
    

    View Slide

34. GETTING
    Request object
    Identity Map lookup
    Database query
    Object hydration
    Save hydrated object to IM (MANAGED)
    

    View Slide

35. UPDATING
    Get the object (MANAGED)
    Change a property
    Persist the object (no-op)
    Flush the EM
    UoW calculates changes => UPDATE SQL
    Transaction
    

    View Slide

36. DELETING
    Get the object (MANAGED)
    Ask for removal (REMOVED)
    Flush the EM
    UoW calculates changes => DELETE SQL
    Transaction
    

    View Slide

37. CHANGE TRACKING POLICIES
    

    View Slide

38. DEFERRED IMPLICIT
    Default
    Property-by-property comparison
    Persistence by reachability
    Slowest
    

    View Slide

39. DEFERRED EXPLICIT
    Property-by-property comparison
    EntityManager#persist(entity)
    No "dirty" checking
    

    View Slide

40. DEFERRED EXPLICIT
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#deferred-explicit
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#deferred-explicit
    /**
    * @Entity
    * @ChangeTrackingPolicy("DEFERRED_EXPLICIT")
    */
    class User
    {
    // ...
    }
    

    View Slide

41. NOTIFY
    Notify interested listeners of changes
    NotifyPropertyChanged
    

    View Slide

42. NOTIFY
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify
    use Doctrine\Common\NotifyPropertyChanged,
    Doctrine\Common\PropertyChangedListener;
    /**
    * @Entity
    * @ChangeTrackingPolicy("NOTIFY")
    */
    class MyEntity implements NotifyPropertyChanged
    {
    private $listeners = array();
    public function addPropertyChangedListener(PropertyChangedListener $listener)
    {
    $this->listeners[] = $listener;
    }
    }
    

    View Slide

43. NOTIFY
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify
    class MyEntity implements NotifyPropertyChanged
    {
    protected function onPropertyChanged($propName, $oldValue, $newValue)
    {
    if ($this->listeners) {
    foreach ($this->listeners as $listener) {
    $listener->propertyChanged($this, $propName, $oldValue, $newValue);
    }
    }
    }
    public function setData($data)
    {
    if ($data != $this->data) {
    $this->onPropertyChanged('data', $this->data, $data);
    $this->data = $data;
    }
    }
    }
    

    View Slide

44. FURTHER
    IMPROVEMENTS
    

    View Slide

45. ALTERNATIVE QUERY RESULT
    FORMATS
    Read only data
    Scalar values
    Nested array graph
    

    View Slide

46. ALTERNATIVE QUERY RESULT
    FORMATS
    $dql = "SELECT b, e, r, p FROM Bug b JOIN b.engineer e ".
    "JOIN b.reporter r JOIN b.products p ORDER BY b.created DESC";
    $query = $entityManager->createQuery($dql);
    $bugs = $query->getArrayResult();
    foreach ($bugs as $bug) {
    echo $bug['description'] . " - " . $bug['created']->format('d.m.Y')."\n";
    echo " Reported by: ".$bug['reporter']['name']."\n";
    echo " Assigned to: ".$bug['engineer']['name']."\n";
    foreach ($bug['products'] as $product) {
    echo " Platform: ".$product['name']."\n";
    }
    echo "\n";
    }
    

    View Slide

47. READ ONLY ENTITIES
    /**
    * @Entity(readOnly=true)
    * @Table(name="products")
    */
    class Product
    {
    }
    

    View Slide

48. https://github.com/coudenysj/doctrine2-orm-tutorial
    https://github.com/coudenysj/doctrine2-orm-tutorial
    git checkout read-only
    SQL_LOGGER=1 ./read_only.sh
    DOCTRINE2-ORM-TUTORIAL
    

    View Slide

49. EXTRA-LAZY COLLECTIONS
    LAZY (default)
    EAGER (join)
    EXTRA_LAZY
    unloaded collection: contains, count, etc...
    postpone database calls
    

    View Slide

50. EXTRA-LAZY COLLECTIONS
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/tutorials/extra-lazy-associations.html#enabling-extra-lazy-associations
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/tutorials/extra-lazy-associations.html#enabling-extra-lazy-associations
    namespace Doctrine\Tests\Models\CMS;
    use Doctrine\ORM\Annotation as ORM;
    /**
    * @ORM\Entity
    */
    class CmsGroup
    {
    /**
    * @ORM\ManyToMany(
    * targetEntity="CmsUser", mappedBy="groups", fetch="EXTRA_LAZY"
    * )
    */
    public $users;
    }
    

    View Slide

51. DOCTRINE\CACHE
    

    View Slide

52. DOCTRINE\COMMON\
    CACHE\CACHE
    INTERFACE
    fetch($id)
    contains($id)
    save($id, $data, $lifeTime = false)
    delete($id)
    

    View Slide

53. CACHE DRIVERS
    APCu
    Memcached
    Xcache
    Redis
    File
    Array*
    ...
    

    View Slide

54. CACHING IN DOCTRINE
    

    View Slide

55. METADATA
    Removes parsing overhead (Annotations, YML,
    XML)
    

    View Slide

56. QUERY
    DQL => SQL
    

    View Slide

57. RESULT
    Store SQL data result to cache
    Still a need to hydrate objects
    Joins will be stored as-is*
    

    View Slide

58. CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING
    

    View Slide

59. HYDRATED RESULT
    

    View Slide

60. REPOSITORY DECORATOR
    

    View Slide

61. GENERALLY A BAD IDEA
    Serializing entities?
    Serializing collections?
    Detached entities
    https://github.com/doctrine/doctrine2/pull/172/files
    

    View Slide

62. View Slide

63. https://cowburn.info/public/files/php7_logo.svg
    https://cowburn.info/public/files/php7_logo.svg
    

    View Slide

64. SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    CACHING
    CACHING
    CACHING
    CACHING
    CACHING
    CACHING
    CACHING
    CACHING
    

    View Slide

65. View Slide

66. SECOND LEVEL CACHING
    No entity instances
    Only entity identifier and values
    Best suited for read-only data
    

    View Slide

67. TYPES
    Entity data: id + values
    Collection data: ownerId id + list of ids
    Query data: list of ids
    

    View Slide

68. ENTITY CACHE
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#entity-cache-definition
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#entity-cache-definition
    /**
    * @Entity
    * @Cache()
    */
    class Country
    {
    // ...
    }
    

    View Slide

69. ASSOCIATION CACHE
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#association-cache-definition
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#association-cache-definition
    /**
    * @Entity
    * @Cache()
    */
    class State
    {
    //...
    /**
    * @Cache()
    * @ManyToOne(targetEntity="Country")
    * @JoinColumn(name="country_id", referencedColumnName="id")
    */
    protected $country;
    }
    

    View Slide

70. REGIONS
    Every type has own region
    Can be configured
    Can be used for invalidation (specific entities)
    Own lifetime
    

    View Slide

71. MODES
    READ_ONLY
    NONSTRICT_READ_WRITE (no locks)
    READ_WRITE (locks)
    

    View Slide

72. QUERY CACHE
    

    View Slide

73. QUERY CACHE
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#using-the-query-cache
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#using-the-query-cache
    $result1 = $em->createQuery('SELECT c FROM Country c ORDER BY c.name')
    ->setCacheable(true)
    ->getResult();
    

    View Slide

74. QUERY CACHE: MODES
    Cache::MODE_GET
    Cache::MODE_PUT
    Cache::MODE_NORMAL (default)
    Cache::MODE_REFRESH
    

    View Slide

75. QUERY CACHE: DELETE /
    UPDATE QUERIES
    Directly into a database
    Bypass the second-level cache
    Entities that are already cached will NOT be
    invalidated
    Query hint
    

    View Slide

76. QUERY CACHE: DELETE /
    UPDATE QUERIES
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    // Execute and invalidate
    $this->em->createQuery(
    "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1"
    )
    ->setHint(Query::HINT_CACHE_EVICT, true)
    ->execute();
    

    View Slide

77. QUERY CACHE: DELETE /
    UPDATE QUERIES
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    // Execute
    $this->em->createQuery(
    "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1"
    )
    ->execute();
    // Invoke Cache API
    $em->getCache()->evictEntityRegion('Entity\Country');
    

    View Slide

78. QUERY CACHE: DELETE /
    UPDATE QUERIES
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    // Execute
    $this->em->createQuery(
    "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1"
    )
    ->execute();
    // Invoke Cache API
    $em->getCache()->evictEntity('Entity\Country', 1);
    

    View Slide

79. LIMITATIONS
    Single application
    Single Primary key column
    

    View Slide

80. https://github.com/coudenysj/doctrine2-orm-tutorial
    https://github.com/coudenysj/doctrine2-orm-tutorial
    git checkout second-level-cache
    SQL_LOGGER=1 ./second_level_cache.sh
    DOCTRINE2-ORM-TUTORIAL
    

    View Slide

81. CONCLUSION
    Keep Identity Map / internals in mind
    Using the basic caching (query + mapping) + opcode
    caching is a must
    Cache heavy queries with result cache
    Give Second Level Caching a try (even if only for
    entities)
    Query Cache !== Second Level Query Cache
    

    View Slide

82. http://doctrine-project.org/
    https://www.flickr.com/photos/spacex/40126461851
    https://github.com/coudenysj?tab=repositories&q=doctrine
    Jachim Coudenys
    @coudenysj - jachim.be
    jachim.coudenys@combellgroup.com
    THANK YOU
    

    View Slide