Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Doctrine caching can skyrocket your application (SymfonyCon Amsterdam 2019)

How Doctrine caching can skyrocket your application (SymfonyCon Amsterdam 2019)

When people talk about Doctrine (or any ORM for that matter), the performance issue always comes up fairly quickly. Besides the fact that Doctrine will help you develop faster, so a little overhead doesn't really matter, there are numerous options to increase the performance of the application.

By understanding how the system works in the first place, a lot of issues can be avoided right away.

When you have done everything to avoid these pitfalls, you can bring in the big guns: caching. Doctrine has several caching mechanism and since Doctrine 2.5 "Second Level Cache" was added to our toolbox. After this talk, you should know what the impact is of every cache and how to use them.

Jachim Coudenys

November 21, 2019
Tweet

More Decks by Jachim Coudenys

Other Decks in Programming

Transcript

  1. HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    Jachim Coudenys
    Jachim Coudenys
    Jachim Coudenys
    Jachim Coudenys
    Jachim Coudenys
    Jachim Coudenys
    Jachim Coudenys
    Jachim Coudenys
    @coudenysj - jachim.be
    @coudenysj - jachim.be
    @coudenysj - jachim.be
    @coudenysj - jachim.be
    @coudenysj - jachim.be
    @coudenysj - jachim.be
    @coudenysj - jachim.be
    @coudenysj - jachim.be
    [email protected]
    [email protected]
    [email protected]
    [email protected]
    [email protected]
    [email protected]
    [email protected]
    [email protected]

    View full-size slide

  2. Jachim Coudenys
    Jachim Coudenys

    View full-size slide

  3. HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION

    View full-size slide

  4. HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    HOW DOCTRINE
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    CACHING CAN
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    SKYROCKET YOUR
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    APPLICATION
    If you're using Doctrine ORM.
    If you're using Doctrine ORM.

    View full-size slide

  5. WHAT IS DOCTRINE

    View full-size slide

  6. OBJECT-RELATIONAL MAPPER
    ... is a programming technique for
    converting data between incompatible
    type systems using object-oriented
    programming languages.
    — https://en.wikipedia.org/wiki/Object-relational_mapping

    View full-size slide

  7. TERMINOLOGY
    Entity
    Mapping
    Repository
    EntityManager

    View full-size slide

  8. A Bug has a description, creation date, status, reporter and
    engineer
    A Bug can occur on different Products (platforms)
    A Product has a name.
    Bug reporters and engineers are both Users of the system.
    A User can create new Bugs.
    The assigned engineer can close a Bug.
    A User can see all his reported or assigned Bugs.
    Bugs can be paginated through a list-view.
    AN EXAMPLE MODEL: BUG
    TRACKER

    View full-size slide

  9. https://github.com/coudenysj/doctrine2-orm-tutorial
    https://github.com/coudenysj/doctrine2-orm-tutorial
    git checkout query-logging
    ls src/*
    ./tutorial.sh
    DOCTRINE2-ORM-TUTORIAL

    View full-size slide

  10. SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS
    SOME INTERNALS

    View full-size slide

  11. Application Repository DBAL --> EM

    View full-size slide

  12. ENTITYMANAGER
    Central access point
    API used to query and manage object

    View full-size slide

  13. --> Unit of Work

    View full-size slide

  14. UNIT OF WORK
    Maintains a list of objects affected by a
    business transaction and coordinates
    the writing out of changes and the
    resolution of concurrency problems.
    — https://martinfowler.com/eaaCatalog/unitOfWork.html

    View full-size slide

  15. UNIT OF WORK
    Unit of Work pattern
    keep track of things to do with next flush
    no direct interaction

    View full-size slide

  16. TRANSACTIONAL WRITE-
    BEHIND
    UoW concept
    Delays execution of queries (optimized in short
    transaction)

    View full-size slide

  17. --> Identity Map

    View full-size slide

  18. IDENTITY MAP
    Ensures that each object gets loaded
    only once by keeping every loaded
    object in a map. Looks up objects using
    the map when referring to them.
    — https://martinfowler.com/eaaCatalog/identityMap.html

    View full-size slide

  19. IDENTITY MAP
    A man with two watches never knows what time it
    is
    Load data in the same in-memory object
    Sync to database correctly

    View full-size slide

  20. ENTITY STATES
    NEW
    MANAGED
    DETACHED
    REMOVED

    View full-size slide

  21. --> Hydration

    View full-size slide

  22. https://symfony.com/doc/current/components/serializer.html
    https://symfony.com/doc/current/components/serializer.html

    View full-size slide

  23. PROXIES (LAZY LOADING)
    An object that doesn't contain all of
    the data you need but knows how to
    get it.
    — https://martinfowler.com/eaaCatalog/lazyLoad.html

    View full-size slide

  24. Complete picture

    View full-size slide

  25. https://github.com/coudenysj/doctrine2-orm-tutorial
    https://github.com/coudenysj/doctrine2-orm-tutorial
    git checkout internals
    SQL_LOGGER=1 ./tutorial.sh
    SQL_LOGGER=1 ./internals.sh
    DOCTRINE2-ORM-TUTORIAL

    View full-size slide

  26. TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES
    TRACKING CHANGES

    View full-size slide

  27. CREATING
    Create new object (NEW)
    Persist the object (MANAGED)
    Flush the EM
    UoW calculates changes => INSERT SQL
    Transaction

    View full-size slide

  28. GETTING
    Request object
    Identity Map lookup
    Database query
    Object hydration
    Save hydrated object to IM (MANAGED)

    View full-size slide

  29. UPDATING
    Get the object (MANAGED)
    Change a property
    Persist the object (no-op)
    Flush the EM
    UoW calculates changes => UPDATE SQL
    Transaction

    View full-size slide

  30. DELETING
    Get the object (MANAGED)
    Ask for removal (REMOVED)
    Flush the EM
    UoW calculates changes => DELETE SQL
    Transaction

    View full-size slide

  31. CHANGE TRACKING POLICIES

    View full-size slide

  32. DEFERRED IMPLICIT
    Default
    Property-by-property comparison
    Persistence by reachability
    Slowest

    View full-size slide

  33. DEFERRED EXPLICIT
    Property-by-property comparison
    EntityManager#persist(entity)
    No "dirty" checking

    View full-size slide

  34. DEFERRED EXPLICIT
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#deferred-explicit
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#deferred-explicit
    /**
    * @Entity
    * @ChangeTrackingPolicy("DEFERRED_EXPLICIT")
    */
    class User
    {
    // ...
    }

    View full-size slide

  35. NOTIFY
    Notify interested listeners of changes
    NotifyPropertyChanged

    View full-size slide

  36. NOTIFY
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify
    use Doctrine\Common\NotifyPropertyChanged,
    Doctrine\Common\PropertyChangedListener;
    /**
    * @Entity
    * @ChangeTrackingPolicy("NOTIFY")
    */
    class MyEntity implements NotifyPropertyChanged
    {
    private $listeners = array();
    public function addPropertyChangedListener(PropertyChangedListener $listener)
    {
    $this->listeners[] = $listener;
    }
    }

    View full-size slide

  37. NOTIFY
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify
    class MyEntity implements NotifyPropertyChanged
    {
    protected function onPropertyChanged($propName, $oldValue, $newValue)
    {
    if ($this->listeners) {
    foreach ($this->listeners as $listener) {
    $listener->propertyChanged($this, $propName, $oldValue, $newValue);
    }
    }
    }
    public function setData($data)
    {
    if ($data != $this->data) {
    $this->onPropertyChanged('data', $this->data, $data);
    $this->data = $data;
    }
    }
    }

    View full-size slide

  38. FURTHER
    IMPROVEMENTS

    View full-size slide

  39. ALTERNATIVE QUERY RESULT
    FORMATS
    Read only data
    Scalar values
    Nested array graph

    View full-size slide

  40. ALTERNATIVE QUERY RESULT
    FORMATS
    $dql = "SELECT b, e, r, p FROM Bug b JOIN b.engineer e ".
    "JOIN b.reporter r JOIN b.products p ORDER BY b.created DESC";
    $query = $entityManager->createQuery($dql);
    $bugs = $query->getArrayResult();
    foreach ($bugs as $bug) {
    echo $bug['description'] . " - " . $bug['created']->format('d.m.Y')."\n";
    echo " Reported by: ".$bug['reporter']['name']."\n";
    echo " Assigned to: ".$bug['engineer']['name']."\n";
    foreach ($bug['products'] as $product) {
    echo " Platform: ".$product['name']."\n";
    }
    echo "\n";
    }

    View full-size slide

  41. READ ONLY ENTITIES
    /**
    * @Entity(readOnly=true)
    * @Table(name="products")
    */
    class Product
    {
    }

    View full-size slide

  42. https://github.com/coudenysj/doctrine2-orm-tutorial
    https://github.com/coudenysj/doctrine2-orm-tutorial
    git checkout read-only
    SQL_LOGGER=1 ./read_only.sh
    DOCTRINE2-ORM-TUTORIAL

    View full-size slide

  43. EXTRA-LAZY COLLECTIONS
    LAZY (default)
    EAGER (join)
    EXTRA_LAZY
    unloaded collection: contains, count, etc...
    postpone database calls

    View full-size slide

  44. EXTRA-LAZY COLLECTIONS
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/tutorials/extra-lazy-associations.html#enabling-extra-lazy-associations
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/tutorials/extra-lazy-associations.html#enabling-extra-lazy-associations
    namespace Doctrine\Tests\Models\CMS;
    use Doctrine\ORM\Annotation as ORM;
    /**
    * @ORM\Entity
    */
    class CmsGroup
    {
    /**
    * @ORM\ManyToMany(
    * targetEntity="CmsUser", mappedBy="groups", fetch="EXTRA_LAZY"
    * )
    */
    public $users;
    }

    View full-size slide

  45. DOCTRINE\CACHE

    View full-size slide

  46. DOCTRINE\COMMON\
    CACHE\CACHE
    INTERFACE
    fetch($id)
    contains($id)
    save($id, $data, $lifeTime = false)
    delete($id)

    View full-size slide

  47. CACHE DRIVERS
    APCu
    Memcached
    Xcache
    Redis
    File
    Array*
    ...

    View full-size slide

  48. CACHING IN DOCTRINE

    View full-size slide

  49. METADATA
    Removes parsing overhead (Annotations, YML,
    XML)

    View full-size slide

  50. QUERY
    DQL => SQL

    View full-size slide

  51. RESULT
    Store SQL data result to cache
    Still a need to hydrate objects
    Joins will be stored as-is*

    View full-size slide

  52. CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING
    CUSTOM CACHING

    View full-size slide

  53. HYDRATED RESULT

    View full-size slide

  54. REPOSITORY DECORATOR

    View full-size slide

  55. GENERALLY A BAD IDEA
    Serializing entities?
    Serializing collections?
    Detached entities
    https://github.com/doctrine/doctrine2/pull/172/files

    View full-size slide

  56. https://cowburn.info/public/files/php7_logo.svg
    https://cowburn.info/public/files/php7_logo.svg

    View full-size slide

  57. SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    SECOND LEVEL
    CACHING
    CACHING
    CACHING
    CACHING
    CACHING
    CACHING
    CACHING
    CACHING

    View full-size slide

  58. SECOND LEVEL CACHING
    No entity instances
    Only entity identifier and values
    Best suited for read-only data

    View full-size slide

  59. TYPES
    Entity data: id + values
    Collection data: ownerId id + list of ids
    Query data: list of ids

    View full-size slide

  60. ENTITY CACHE
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#entity-cache-definition
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#entity-cache-definition
    /**
    * @Entity
    * @Cache()
    */
    class Country
    {
    // ...
    }

    View full-size slide

  61. ASSOCIATION CACHE
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#association-cache-definition
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#association-cache-definition
    /**
    * @Entity
    * @Cache()
    */
    class State
    {
    //...
    /**
    * @Cache()
    * @ManyToOne(targetEntity="Country")
    * @JoinColumn(name="country_id", referencedColumnName="id")
    */
    protected $country;
    }

    View full-size slide

  62. REGIONS
    Every type has own region
    Can be configured
    Can be used for invalidation (specific entities)
    Own lifetime

    View full-size slide

  63. MODES
    READ_ONLY
    NONSTRICT_READ_WRITE (no locks)
    READ_WRITE (locks)

    View full-size slide

  64. QUERY CACHE
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#using-the-query-cache
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#using-the-query-cache
    $result1 = $em->createQuery('SELECT c FROM Country c ORDER BY c.name')
    ->setCacheable(true)
    ->getResult();

    View full-size slide

  65. QUERY CACHE: MODES
    Cache::MODE_GET
    Cache::MODE_PUT
    Cache::MODE_NORMAL (default)
    Cache::MODE_REFRESH

    View full-size slide

  66. QUERY CACHE: DELETE /
    UPDATE QUERIES
    Directly into a database
    Bypass the second-level cache
    Entities that are already cached will NOT be
    invalidated
    Query hint

    View full-size slide

  67. QUERY CACHE: DELETE /
    UPDATE QUERIES
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    // Execute and invalidate
    $this->em->createQuery(
    "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1"
    )
    ->setHint(Query::HINT_CACHE_EVICT, true)
    ->execute();

    View full-size slide

  68. QUERY CACHE: DELETE /
    UPDATE QUERIES
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    // Execute
    $this->em->createQuery(
    "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1"
    )
    ->execute();
    // Invoke Cache API
    $em->getCache()->evictEntityRegion('Entity\Country');

    View full-size slide

  69. QUERY CACHE: DELETE /
    UPDATE QUERIES
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries
    // Execute
    $this->em->createQuery(
    "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1"
    )
    ->execute();
    // Invoke Cache API
    $em->getCache()->evictEntity('Entity\Country', 1);

    View full-size slide

  70. LIMITATIONS
    Single application
    Single Primary key column

    View full-size slide

  71. https://github.com/coudenysj/doctrine2-orm-tutorial
    https://github.com/coudenysj/doctrine2-orm-tutorial
    git checkout second-level-cache
    SQL_LOGGER=1 ./second_level_cache.sh
    DOCTRINE2-ORM-TUTORIAL

    View full-size slide

  72. CONCLUSION
    Keep Identity Map / internals in mind
    Using the basic caching (query + mapping) + opcode
    caching is a must
    Cache heavy queries with result cache
    Give Second Level Caching a try (even if only for
    entities)
    Query Cache !== Second Level Query Cache

    View full-size slide

  73. http://doctrine-project.org/
    https://www.flickr.com/photos/spacex/40126461851
    https://github.com/coudenysj?tab=repositories&q=doctrine
    Jachim Coudenys
    @coudenysj - jachim.be
    [email protected]
    THANK YOU

    View full-size slide