How Doctrine caching can skyrocket your application (SymfonyCon Amsterdam 2019)

How Doctrine caching can skyrocket your application (SymfonyCon Amsterdam 2019)

When people talk about Doctrine (or any ORM for that matter), the performance issue always comes up fairly quickly. Besides the fact that Doctrine will help you develop faster, so a little overhead doesn't really matter, there are numerous options to increase the performance of the application.

By understanding how the system works in the first place, a lot of issues can be avoided right away.

When you have done everything to avoid these pitfalls, you can bring in the big guns: caching. Doctrine has several caching mechanism and since Doctrine 2.5 "Second Level Cache" was added to our toolbox. After this talk, you should know what the impact is of every cache and how to use them.

4ce755c7f3ddf4e0c92e1aeaeea7677b?s=128

Jachim Coudenys

November 21, 2019
Tweet

Transcript

  1. HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE

    HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION Jachim Coudenys Jachim Coudenys Jachim Coudenys Jachim Coudenys Jachim Coudenys Jachim Coudenys Jachim Coudenys Jachim Coudenys @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com
  2. Jachim Coudenys Jachim Coudenys

  3. None
  4. None
  5. HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE

    HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION
  6. HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE

    HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION If you're using Doctrine ORM. If you're using Doctrine ORM.
  7. WHAT IS DOCTRINE

  8. OBJECT-RELATIONAL MAPPER ... is a programming technique for converting data

    between incompatible type systems using object-oriented programming languages. — https://en.wikipedia.org/wiki/Object-relational_mapping
  9. TERMINOLOGY Entity Mapping Repository EntityManager

  10. A Bug has a description, creation date, status, reporter and

    engineer A Bug can occur on different Products (platforms) A Product has a name. Bug reporters and engineers are both Users of the system. A User can create new Bugs. The assigned engineer can close a Bug. A User can see all his reported or assigned Bugs. Bugs can be paginated through a list-view. AN EXAMPLE MODEL: BUG TRACKER
  11. https://github.com/coudenysj/doctrine2-orm-tutorial https://github.com/coudenysj/doctrine2-orm-tutorial git checkout query-logging ls src/* ./tutorial.sh DOCTRINE2-ORM-TUTORIAL

  12. SOME INTERNALS SOME INTERNALS SOME INTERNALS SOME INTERNALS SOME INTERNALS

    SOME INTERNALS SOME INTERNALS SOME INTERNALS
  13. Application Repository DBAL --> EM

  14. ENTITYMANAGER Central access point API used to query and manage

    object
  15. --> Unit of Work

  16. UNIT OF WORK Maintains a list of objects affected by

    a business transaction and coordinates the writing out of changes and the resolution of concurrency problems. — https://martinfowler.com/eaaCatalog/unitOfWork.html
  17. UNIT OF WORK Unit of Work pattern keep track of

    things to do with next flush no direct interaction
  18. None
  19. TRANSACTIONAL WRITE- BEHIND UoW concept Delays execution of queries (optimized

    in short transaction)
  20. --> Identity Map

  21. IDENTITY MAP Ensures that each object gets loaded only once

    by keeping every loaded object in a map. Looks up objects using the map when referring to them. — https://martinfowler.com/eaaCatalog/identityMap.html
  22. IDENTITY MAP A man with two watches never knows what

    time it is Load data in the same in-memory object Sync to database correctly
  23. --> States

  24. ENTITY STATES NEW MANAGED DETACHED REMOVED

  25. --> Hydration

  26. HYDRATION

  27. https://symfony.com/doc/current/components/serializer.html https://symfony.com/doc/current/components/serializer.html

  28. --> Proxies

  29. PROXIES (LAZY LOADING) An object that doesn't contain all of

    the data you need but knows how to get it. — https://martinfowler.com/eaaCatalog/lazyLoad.html
  30. Complete picture

  31. https://github.com/coudenysj/doctrine2-orm-tutorial https://github.com/coudenysj/doctrine2-orm-tutorial git checkout internals SQL_LOGGER=1 ./tutorial.sh SQL_LOGGER=1 ./internals.sh DOCTRINE2-ORM-TUTORIAL

  32. TRACKING CHANGES TRACKING CHANGES TRACKING CHANGES TRACKING CHANGES TRACKING CHANGES

    TRACKING CHANGES TRACKING CHANGES TRACKING CHANGES
  33. CREATING Create new object (NEW) Persist the object (MANAGED) Flush

    the EM UoW calculates changes => INSERT SQL Transaction
  34. GETTING Request object Identity Map lookup Database query Object hydration

    Save hydrated object to IM (MANAGED)
  35. UPDATING Get the object (MANAGED) Change a property Persist the

    object (no-op) Flush the EM UoW calculates changes => UPDATE SQL Transaction
  36. DELETING Get the object (MANAGED) Ask for removal (REMOVED) Flush

    the EM UoW calculates changes => DELETE SQL Transaction
  37. CHANGE TRACKING POLICIES

  38. DEFERRED IMPLICIT Default Property-by-property comparison Persistence by reachability Slowest

  39. DEFERRED EXPLICIT Property-by-property comparison EntityManager#persist(entity) No "dirty" checking

  40. DEFERRED EXPLICIT http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#deferred-explicit http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#deferred-explicit /** * @Entity * @ChangeTrackingPolicy("DEFERRED_EXPLICIT") */

    class User { // ... }
  41. NOTIFY Notify interested listeners of changes NotifyPropertyChanged

  42. NOTIFY http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify use Doctrine\Common\NotifyPropertyChanged, Doctrine\Common\PropertyChangedListener; /** * @Entity *

    @ChangeTrackingPolicy("NOTIFY") */ class MyEntity implements NotifyPropertyChanged { private $listeners = array(); public function addPropertyChangedListener(PropertyChangedListener $listener) { $this->listeners[] = $listener; } }
  43. NOTIFY http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify class MyEntity implements NotifyPropertyChanged { protected function

    onPropertyChanged($propName, $oldValue, $newValue) { if ($this->listeners) { foreach ($this->listeners as $listener) { $listener->propertyChanged($this, $propName, $oldValue, $newValue); } } } public function setData($data) { if ($data != $this->data) { $this->onPropertyChanged('data', $this->data, $data); $this->data = $data; } } }
  44. FURTHER IMPROVEMENTS

  45. ALTERNATIVE QUERY RESULT FORMATS Read only data Scalar values Nested

    array graph
  46. ALTERNATIVE QUERY RESULT FORMATS $dql = "SELECT b, e, r,

    p FROM Bug b JOIN b.engineer e ". "JOIN b.reporter r JOIN b.products p ORDER BY b.created DESC"; $query = $entityManager->createQuery($dql); $bugs = $query->getArrayResult(); foreach ($bugs as $bug) { echo $bug['description'] . " - " . $bug['created']->format('d.m.Y')."\n"; echo " Reported by: ".$bug['reporter']['name']."\n"; echo " Assigned to: ".$bug['engineer']['name']."\n"; foreach ($bug['products'] as $product) { echo " Platform: ".$product['name']."\n"; } echo "\n"; }
  47. READ ONLY ENTITIES /** * @Entity(readOnly=true) * @Table(name="products") */ class

    Product { }
  48. https://github.com/coudenysj/doctrine2-orm-tutorial https://github.com/coudenysj/doctrine2-orm-tutorial git checkout read-only SQL_LOGGER=1 ./read_only.sh DOCTRINE2-ORM-TUTORIAL

  49. EXTRA-LAZY COLLECTIONS LAZY (default) EAGER (join) EXTRA_LAZY unloaded collection: contains,

    count, etc... postpone database calls
  50. EXTRA-LAZY COLLECTIONS http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/tutorials/extra-lazy-associations.html#enabling-extra-lazy-associations http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/tutorials/extra-lazy-associations.html#enabling-extra-lazy-associations namespace Doctrine\Tests\Models\CMS; use Doctrine\ORM\Annotation as ORM;

    /** * @ORM\Entity */ class CmsGroup { /** * @ORM\ManyToMany( * targetEntity="CmsUser", mappedBy="groups", fetch="EXTRA_LAZY" * ) */ public $users; }
  51. DOCTRINE\CACHE

  52. DOCTRINE\COMMON\ CACHE\CACHE INTERFACE fetch($id) contains($id) save($id, $data, $lifeTime = false)

    delete($id)
  53. CACHE DRIVERS APCu Memcached Xcache Redis File Array* ...

  54. CACHING IN DOCTRINE

  55. METADATA Removes parsing overhead (Annotations, YML, XML)

  56. QUERY DQL => SQL

  57. RESULT Store SQL data result to cache Still a need

    to hydrate objects Joins will be stored as-is*
  58. CUSTOM CACHING CUSTOM CACHING CUSTOM CACHING CUSTOM CACHING CUSTOM CACHING

    CUSTOM CACHING CUSTOM CACHING CUSTOM CACHING
  59. HYDRATED RESULT

  60. REPOSITORY DECORATOR

  61. GENERALLY A BAD IDEA Serializing entities? Serializing collections? Detached entities

    https://github.com/doctrine/doctrine2/pull/172/files
  62. None
  63. https://cowburn.info/public/files/php7_logo.svg https://cowburn.info/public/files/php7_logo.svg

  64. SECOND LEVEL SECOND LEVEL SECOND LEVEL SECOND LEVEL SECOND LEVEL

    SECOND LEVEL SECOND LEVEL SECOND LEVEL CACHING CACHING CACHING CACHING CACHING CACHING CACHING CACHING
  65. None
  66. SECOND LEVEL CACHING No entity instances Only entity identifier and

    values Best suited for read-only data
  67. TYPES Entity data: id + values Collection data: ownerId id

    + list of ids Query data: list of ids
  68. ENTITY CACHE http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#entity-cache-definition http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#entity-cache-definition /** * @Entity * @Cache() */

    class Country { // ... }
  69. ASSOCIATION CACHE http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#association-cache-definition http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#association-cache-definition /** * @Entity * @Cache() */

    class State { //... /** * @Cache() * @ManyToOne(targetEntity="Country") * @JoinColumn(name="country_id", referencedColumnName="id") */ protected $country; }
  70. REGIONS Every type has own region Can be configured Can

    be used for invalidation (specific entities) Own lifetime
  71. MODES READ_ONLY NONSTRICT_READ_WRITE (no locks) READ_WRITE (locks)

  72. QUERY CACHE

  73. QUERY CACHE http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#using-the-query-cache http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#using-the-query-cache $result1 = $em->createQuery('SELECT c FROM Country

    c ORDER BY c.name') ->setCacheable(true) ->getResult();
  74. QUERY CACHE: MODES Cache::MODE_GET Cache::MODE_PUT Cache::MODE_NORMAL (default) Cache::MODE_REFRESH

  75. QUERY CACHE: DELETE / UPDATE QUERIES Directly into a database

    Bypass the second-level cache Entities that are already cached will NOT be invalidated Query hint
  76. QUERY CACHE: DELETE / UPDATE QUERIES http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries // Execute

    and invalidate $this->em->createQuery( "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1" ) ->setHint(Query::HINT_CACHE_EVICT, true) ->execute();
  77. QUERY CACHE: DELETE / UPDATE QUERIES http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries // Execute

    $this->em->createQuery( "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1" ) ->execute(); // Invoke Cache API $em->getCache()->evictEntityRegion('Entity\Country');
  78. QUERY CACHE: DELETE / UPDATE QUERIES http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries // Execute

    $this->em->createQuery( "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1" ) ->execute(); // Invoke Cache API $em->getCache()->evictEntity('Entity\Country', 1);
  79. LIMITATIONS Single application Single Primary key column

  80. https://github.com/coudenysj/doctrine2-orm-tutorial https://github.com/coudenysj/doctrine2-orm-tutorial git checkout second-level-cache SQL_LOGGER=1 ./second_level_cache.sh DOCTRINE2-ORM-TUTORIAL

  81. CONCLUSION Keep Identity Map / internals in mind Using the

    basic caching (query + mapping) + opcode caching is a must Cache heavy queries with result cache Give Second Level Caching a try (even if only for entities) Query Cache !== Second Level Query Cache
  82. http://doctrine-project.org/ https://www.flickr.com/photos/spacex/40126461851 https://github.com/coudenysj?tab=repositories&q=doctrine Jachim Coudenys @coudenysj - jachim.be jachim.coudenys@combellgroup.com THANK

    YOU