Doctrine effizient verwenden

6a1345d8e6dd15b2c78eff0c331963b1?s=47 Denis Brumann
September 25, 2019

Doctrine effizient verwenden

Doctrine ORM ist ein elementarer Bestandteil in vielen Symfony-Anwendungen, egal
ob Full-Stack, API oder Command Line-Tool. Seit dem Release von Version 2.6 Ende
Dezember 2017 wird an ORM Next (3.0) gearbeitet und der Branch enthält bereits
einige Änderungen, die man in aktuellen Projekten berücksichtigen kann, um ein
späteres Upgrade einfacher zu gestalten.

In diesem Workshop besprechen wir aktuelle Best Practices beim Einsatz von Doctrine in einer modernen Symfony-Anwendung mit Fokus auf die Vermeidung von Speicher- und Performanceproblemen und einem Blick auf zukünftige Änderungen und ihre Auswirkungen auf Projekte, die Doctrine ORM einsetzen. Der Workshop richtet sich an Entwickler*innen, die bereits Erfahrung mit Doctrine haben, aber ihr Verständnis vertiefen möchten.

Die Praxisbeispiele bauen auf einer aktuellen Symfony 4-Anwendung auf und Features, die zum Einsatz kommen, wie das Autowiring, werden als bekannt vorausgesetzt.

6a1345d8e6dd15b2c78eff0c331963b1?s=128

Denis Brumann

September 25, 2019
Tweet

Transcript

  1. Doctrine effizient verwenden

  2. Schedule Zeiten 11:00 – 11:30 Kaffeepause 13:00 – 14:00 Mittagspause

    15:30 – 16:00 Kaffeepause 17:30 Feierabend
  3. Organisatorisches https://joind.in/talk/e6388

  4. Intro Denis Brumann denis.brumann@sensiolabs.de @dbrumann

  5. Architektur Common DBAL ORM ODM

  6. Architektur / Common Libs •Annotations •Cache •Collections •Event Manager •Persistence

    •Proxy Manager •Lexer •Inflector
  7. Architektur / DBAL The Doctrine database abstraction & access layer

    (DBAL) offers a lightweight and thin runtime layer around a PDO-like API and a lot of additional, horizontal features like database schema introspection and manipulation through an OO API.
  8. Architektur / DBAL •Database Driver pdo_mysql, pdo_pgsql, pdo_sqlsrv, … •Platform

    MySQL 5.7, MySQL 8.0, MariaDB10 •Connection (carries driver & executes queries) •Statement (represents a request, executed or being prepared)
  9. Architektur / DBAL / Konfiguration # config/packages/doctrine.yaml doctrine: dbal: server_version:

    5.7 url: '%env(resolve:DATABASE_URL)%' # .env ###> doctrine/doctrine-bundle ### #... DATABASE_URL=mysql://db_user:db_password@127.0.0.1:3306/db_name ###< doctrine/doctrine-bundle ###
  10. Architektur / DBAL / Connection function query($sql); function prepare($sql); function

    beginTransaction(); function commit(); function rollback(); function lastInsertId();
  11. Architektur / DBAL / Connection function insert($expression, $data); function update($expression,

    $data, $id); function delete($expression, $id); function transactional(Closure $func);
  12. Architektur / ORM •Object Manager main public interface •Object Repository

    retrieve instances of your mapped objects •ClassMetadata
  13. Architektur / ORM / Object Manager

  14. Architektur / ORM / Object Manager

  15. Architektur / ORM / Object Repository

  16. Architektur / ORM / Mapping

  17. Architektur / ORM / Entities Klasse darf nicht final sein

    Klasse darf keine finalen Methoden enthalten persistente Properties sollten private oder protected sein __clone darf nicht implementiert werden* __wakeup darf nicht implementiert werden* func_get_args() statt definierter Argumente, darf nicht verwendet werden
  18. Architektur / ORM / Unit Of Work Enthält eine Liste

    der (gemanagten) Entities Daten werden im Arbeitsspeicher vorgehalten Kontrolliert: • entity state • commit order • what to write and how to write (update/insert)
  19. Architektur Common DBAL ORM ODM

  20. Setup / Challenge https://github.com/dbrumann/live2019-doctrine Projekt klonen und einrichten (siehe README.md)

    php –S localhost:8000 –t public
  21. Setup / Challenge version: '3' services: database: image: mysql:latest ports:

    - '3306:3306' environment: MYSQL_ALLOW_EMPTY_PASSWORD: '1' MYSQL_DATABASE: workshop_app MYSQL_USER: root volumes: - 'app_data:/var/lib/mysql' command: - 'mysqld' - '--character-set-server=utf8mb4' - '--collation-server=utf8mb4_unicode_ci' - '--default-authentication-plugin=mysql_native_password' volumes: app_data: ~
  22. Mapping / Field Types • array • simple_array • json_array

    • json • bigint • boolean • datetimetz • date • time • decimal • integer • object • smallint • string • text • binary • blob • float • guid • dateinterval • datetime
  23. Mapping / Generator Strategies AUTO SEQUENCE IDENTITY UUID CUSTOM NONE

    TABLE
  24. Mapping / Assoziationen @OneToOne @OneToMany @ManyToOne @ManyToMany

  25. Mapping / Embeddable @Embeddable @Embedded

  26. Mapping / Embeddable • Embeddables sind keine Entities und dürfen

    keine Assoziationen enthalten • Felder werden wie bei Entities mit @Column annotiert • Nullable Columns können nicht in der Entity überschrieben werden
  27. Mapping / Embeddable /** @ORM\Embeddable */ class Address { /**

    @ORM\Column */ private $street; /** @ORM\Column */ private $postalCode; /** @ORM\Column */ private $city; /** @ORM\Column */ private $country; }
  28. Mapping / Embeddable /** @ORM\Entity */ class Order { /**

    @ORM\Embedded(class="Address") */ private $shippingAddress; /** @ORM\Embedded(class="Address") */ private $billingAddress; }
  29. Mapping / Embeddable /** @ORM\Entity */ class Profile { /**

    @ORM\Embedded(class="Address", columnPrefix=false) */ private $address; }
  30. Mapping / Inheritance @MappedSuperclass

  31. Mapping / MappedSuperclass / Restrictions • MappedSuperclass ist keine eigene

    Entity • nur uni-direktionale Assoziationen mit owning side möglich • Keine Queries über MappedSuperclass möglich • Overrides für Assoziationen und Felder sind möglich
  32. Mapping / Inheritance / STI /** * @Entity * @InheritanceType("SINGLE_TABLE")

    * @DiscriminatorColumn(name="discr", type="string") * @DiscriminatorMap({"person" = "Person", "employee" = "Employee"}) */ class Person { // ... } /** * @Entity */ class Employee extends Person { // ... }
  33. Mapping / STI • DiscriminatorMap kann automatisch generiert werden •

    nicht verwenden! • Daten befinden sich in einer Tabelle • keine Joins notwendig • Subtypen können einfach mit WHERE identifiziert werden
  34. Mapping / Inheritance / CTI /** * @Entity * @InheritanceType("JOINED")

    * @DiscriminatorColumn(name="discr", type="string") * @DiscriminatorMap({"person" = "Person", "employee" = "Employee"}) */ class Person { // ... } /** * @Entity */ class Employee extends Person { // ... }
  35. Mapping / CTI • DiscriminatorMap kann automatisch generiert werden •

    nicht verwenden! • Basis-Tabelle mit Join-Tabelle für Subtypen • Subtypen werden per Join angehangen • Abfrage über alle Suchtypen ist entsprechend ineffizient
  36. Querying DQL QueryBuilder NativeQuery

  37. Querying / DQL / SELECT $dql = <<<DQL SELECT task

    FROM App\Entity\TaskItem task WHERE task.list = :list_id AND task.done = true DQL; return $em->createQuery($dql) ->setParameter('list_id', 1) ->getResult();
  38. Querying / DQL / UPDATE $dql = <<<DQL UPDATE App\Entity\TaskItem

    task SET done = true WHERE task.list = :list_id DQL;
  39. Querying / DQL / DELETE $dql = <<<DQL DELETE App\Entity\TaskItem

    task WHERE task.list = :list_id DQL;
  40. Querying / DQL / Expressions •DQL Functions CONCAT(), SIZE(), LENGTH(),

    … •Arithmetic Operators +, /, * •Aggregate Functions AVG(), COUNT(), MIN(), MAX(), SUM() •Other Expressions BETWEEN, IN, LIKE
  41. Querying / DQL / Custom Expressions •Create custom FunctionNodes Doctrine\ORM\Query\AST\Functions\FunctionNode

    •Libraries Benjamin Eberlei DoctrineExtensions – https://github.com/beberlei/DoctrineExtensions
  42. Querying / QueryBuilder •Creates DQL-queries programmatically •Combine expressions in complex

    manner
  43. Querying / QueryBuilder / SELECT $builder->select('task') ->from(TaskItem::class, 'task') ->where($builder->expr()->andX( $builder->expr()->eq('task.list',

    ':list_id'), $builder->expr()->eq('task.done', true), )) ->setParameter('list_id', 1) ->getResult();
  44. Querying / NativeQuery / ResultSetMapping $rsm = new ResultSetMappingBuilder(); $rsm->addRootEntityFromClassMetadata(TaskItem::class,

    'task');
  45. Querying / NativeQuery / SQL $dql = <<<SQL SELECT task

    FROM app_task_item AS task WHERE task.list_id = :list_id AND task.done = 1 SQL; return $em->createNativeQuery($dql, $rsm) ->setParameter('list_id', 1) ->getSingleScalarResult();
  46. Querying / Challenge TaskListRepository: - findTaskListsContributedBy(User $user) - findTaskListsOwnedBy(User $owner)

    - findArchivedTaskLists - findUnarchivedTaskLists
  47. Querying / DQL / PARTIAL RESULTS $dql = <<<DQL SELECT

    partial task.{name,done} FROM App\Entity\TaskItem task WHERE task.list = :list_id AND task.done = true DQL; return $em->createQuery($dql) ->setParameter('list_id', 1) ->getResult();
  48. Querying / Lazy Associations /** * @Entity */ class CmsGroup

    { /** * @ManyToMany( * targetEntity="CmsUser", * mappedBy="groups", * fetch="EXTRA_LAZY" * ) */ public $users; }
  49. Querying / Hydration $query ->getResult(); ->getSingleResult(); ->getOneOrNullResult(); ->getArrayResult(); ->getScalarResult(); ->getSingleScalarResult();

    ->iterate();
  50. Querying / Hydration /** * Hydrates an object graph. This

    is the default behavior. */ const HYDRATE_OBJECT = 1; /** * Hydrates an array graph. */ const HYDRATE_ARRAY = 2; /** * Hydrates a flat, rectangular result set with scalar values. */ const HYDRATE_SCALAR = 3; /** * Hydrates a single scalar value. */ const HYDRATE_SINGLE_SCALAR = 4; /** * Very simple object hydrator (optimized for performance). */ const HYDRATE_SIMPLEOBJECT = 5;
  51. Querying / Hydration • ObjectHydrator • ArrayHydrator • ScalarHydrator •

    SimpleObjectHydrator • SingleScalarHydrator • GeneratedHydrator – https://github.com/Ocramius/GeneratedHydrator
  52. Querying / Challenge Custom Object Hydration

  53. Querying / Challenge Custom Object Hydration

  54. Querying / Challenge

  55. Querying / Challenge

  56. Querying / Challenge Custom Object Hydration 1. Objekt für TaskListSummary

    erstellen alle relevanten Argumente im Constructor und nur Getter 2. Neue Methode im TaskListRepository SELECT NEW TaskListSummary(…) FROM TaskList::class …
  57. Locking Schützt vor konkurrierenden Zugriffen funktioniert über mehrere Prozesse

  58. Locking Modes Optimistic Locking Pessimistic Locking

  59. Locking / Optimistic Locking Locking erfolgt auf Object-Level benötigt einen

    zusätzlichen Flag auf dem Objekt
  60. Locking / Optimistic Locking class User { // ... /**

    * @Version * @Column(type="integer") */ private $version; // ... } alternativ: datetime
  61. Locking / Optimistic Locking use Doctrine\DBAL\LockMode; use Doctrine\ORM\OptimisticLockException; $id =

    1; $version = 184; try { $entity = $em->find(User::class, $id, LockMode::OPTIMISTIC, $version); // do the work $em->flush(); } catch(OptimisticLockException $e) { echo "Change detected! Someone already wrote a newer version"; }
  62. Locking / Pessimistic Locking Locking erfolgt auf Datanbenk-Ebene keine Entity-Anpassungen

    notwendig, aber Anpassungen an DB-Interaktion disable Auto-Commit mode Transaktionen PESSIMISTIC_WRITE, PESSIMISTIC_READ
  63. Locking / Optimistic Locking Using EntityManager:: find($className, $id, \Doctrine\DBAL\LockMode::PESSIMISTIC_WRITE) or

    find($className, $id, \Doctrine\DBAL\LockMode::PESSIMISTIC_READ) Using EntityManager:: lock($entity, \Doctrine\DBAL\LockMode::PESSIMISTIC_WRITE) or lock($entity, \Doctrine\DBAL\LockMode::PESSIMISTIC_READ) Using Query:: setLockMode(\Doctrine\DBAL\LockMode::PESSIMISTIC_WRITE) or setLockMode(\Doctrine\DBAL\LockMode::PESSIMISTIC_READ)
  64. Locking / Challenge TaskList Optimistic Locking implementieren Beim Archivieren der

    Liste prüfen, ob es Änderungen gab
  65. Locking / Challenge TaskList - neues Attribut (+ Getter) für

    Version hinzufügen - Neue Aktion im Controller, um Liste zu archivieren - show-Template anpassen (form action hinzufügen)
  66. Transaktionen $em->getConnection()->beginTransaction(); // suspend auto-commit try { //... do some

    work $user = new User(); $user->setName('George'); $em->persist($user); $em->flush(); $em->getConnection()->commit(); } catch (Exception $e) { $em->getConnection()->rollBack(); throw $e; }
  67. Transaktionen $em->transactional(function($em) { //... do some work $user = new

    User(); $user->setName('George'); $em->persist($user); });
  68. Lifecycle Events postLoad prePersist postPersist preUpdate postUpdate preRemove postRemove

  69. Lifecycle Events / Callbacks /** * @ORM\Entity * @ORM\HasLifecycleCallbacks */

    class BlogPost { // ... /** * @ORM\PreUpdate */ public function setUpdatedAt() { $this->updatedAt = new \DateTimeImmutable('now'); } }
  70. Lifecycle Events / Event Listeners services: # ... App\EventListener\SearchIndexer: tags:

    - name: 'doctrine.event_listener' event: 'postPersist' priority: 500 # connection: 'default'
  71. Lifecycle Events / Entity Listeners services: # ... App\EventListener\UserChangedNotifier: tags:

    - name: 'doctrine.orm.entity_listener' event: 'postUpdate' entity: 'App\Entity\User' lazy: true entity_manager: 'custom' method: 'checkUserChanges'
  72. Cascade Options persist remove merge detach refresh all

  73. Cascade Options /** * @ORM\OneToMany( * targetEntity="App\Entity\TaskItem", * mappedBy="list", *

    cascade={"persist"} * ) */ private $items;
  74. Cascade Options / Mini-Challenge cascade in TaskList-Entity entfernen

  75. Change Tracking Policies Deferred implicit Deferred explicit Notify

  76. Change Tracking Policies /** * @Entity * @ChangeTrackingPolicy("DEFERRED_EXPLICIT") */ class

    User { // ... }
  77. Change Tracking Policies /** * @Entity * @ChangeTrackingPolicy("NOTIFY") */ class

    User implements NotifyPropertyChanged { // ... }
  78. Change Tracking Policies use Doctrine\Common\NotifyPropertyChanged; use Doctrine\Common\PropertyChangedListener; /** * @Entity

    * @ChangeTrackingPolicy("NOTIFY") */ class MyEntity implements NotifyPropertyChanged { // ... private $_listeners = []; public function addPropertyChangedListener(PropertyChangedListener $listener) { $this->_listeners[] = $listener; } } protected function _onPropertyChanged($propName, $oldValue, $newValue) { if ($this->_listeners) { foreach ($this->_listeners as $listener) { $listener->propertyChanged( $this, $propName, $oldValue, $newValue ); } } }
  79. Change Tracking Policies / Challenge TaskItem Add NOTIFY-Policy

  80. Caching Verfügbar Cache Driver: • APCu • Xcache • Memcache/Memcached

    • Redis • ArrayCache • Filesystem
  81. Caching / Symfony Cache cache.app cache.system

  82. Caching / PROD config/packages/prod/doctrine.yaml

  83. Caching / Metadata Cache Annotation, XML bzw. YAML-Mapping muss in

    ClassMetadata umgewandelt werden à Cache verhindert Parsing bei jedem Request.
  84. Caching / Query Cache DQL muss in SQL transformiert werden

    à Cache verhindert SQL-Parsing, wenn DQL-Query nicht geändert wurde
  85. Caching / Result Cache Ergebnis der SQL-Abfrage kann unter Umständen

    gecacht werden, um weniger oft auf die Datenbank zugreifen zu müssen. Trade Off zwischen Kosten für DB-Verbindung vs. Cache- Zugriff. à Ergebnis einer Query und der resultierenden Hydration wird gespeichert
  86. Caching / Using Result Cache $query = $em->createQuery( 'select u

    from \Entities\User u' ); $query ->useResultCache(true) ->setResultCacheLifetime(3600) ->setResultCacheId('my_id') // ->useResultCache(true, 3600, 'my_id') ;
  87. Caching / Challenge ResultCache TaskList-Queries mit dem ResultCache cachen (Am

    besten Cache-Konfiguration für dev anlegen)
  88. Second Level Cache Neuer Cache Layer überhalb der bestehenden Caches

    Cache schreibt in verschieden Regions. Jede Region hat einen eigenen Namespace und Lifetime. Caching kann in CacheFactory angepasst werden.
  89. Second Level Cache /** * @Entity * @Cache(usage="READ_ONLY", region="my_region") */

    class Product { }
  90. Second Level Cache / Cache Mode READ_ONLY Can do reads,

    inserts and deletes, but no updates or locking NONSTRICT_READ_WRITE Does not employ locks, but can do reads, inserts, updates and deletes READ_WRITE Employs locks before update/delete
  91. Second Level Cache / Query Cache $query = $em->createQuery( 'select

    u from \Entities\User u' ); $query->setCacheable(true);
  92. Second Level Cache / Query Cache / Cache Mode Cache::MODE_GET

    May read items from the cache, but will not add items. Cache::MODE_PUT Will never read items from the cache, but will add items to the cache as it reads them from the database. Cache::MODE_NORMAL May read items from the cache, and add items to the cache. Cache::MODE_REFRESH The query will never read items from the cache, but will refresh items to the cache as it reads them from the database.
  93. Second Level Cache / Cache Eviction $this->_em->createQuery( 'UPDATE Entity\Country u

    SET u.name = 'unknown' WHERE u.id = 1' ) ->setHint(Query::HINT_CACHE_EVICT, true) ->execute(); // $em->getCache()->evictEntityRegion('Entity\Country');
  94. Indizes /** * @Entity * @Table( * name="ecommerce_products", * indexes={@Index(

    * name="search_idx", * columns={"name", "email"} * )} * ) */ class Product { }
  95. Indizes / Show & Tell Indizes auf Datenbankebene EXPLAIN

  96. Batch-Processing An ORM is not primarily well-suited for mass inserts,

    updates or deletions.
  97. Batch-Processing / Bulk Insert $batchSize = 20; for ($i =

    1; $i <= 10000; ++$i) { $user = new CmsUser(); $user->setStatus('user'); $user->setUsername('user' . $i); $user->setName('Mr.Smith-' . $i); $em->persist($user); if (($i % $batchSize) === 0) { $em->flush(); $em->clear(); // Detaches all objects from Doctrine! } } //Persist objects that did not make up an entire batch $em->flush(); $em->clear();
  98. Batch-Processing / Iterating Results $batchSize = 20; $i = 0;

    $q = $em->createQuery('select u from MyProject\Model\User u'); $iterableResult = $q->iterate(); while (($row = $iterableResult->next()) !== false) { $em->remove($row[0]); if (($i % $batchSize) === 0) { $em->flush(); $em->clear(); } ++$i; } $em->flush();
  99. Batch-Processing Second Level Cache

  100. Migrations composer require doctrine/doctrine-migrations- bundle

  101. Migrations use Doctrine\DBAL\Schema\Schema; use Doctrine\Migrations\AbstractMigration; /** * Auto-generated Migration: Please

    modify to your needs! */ final class Version20180601193057 extends AbstractMigration { public function getDescription() : string { return ''; } public function up(Schema $schema) : void { … } public function down(Schema $schema) : void { … } }
  102. Migrations / Additional Behavior public function isTransactional() : bool {

    return false; } public function preUp(Schema $schema) : void { } public function postUp(Schema $schema) : void { } public function preDown(Schema $schema) : void { } public function postDown(Schema $schema) : void { }
  103. Migrations / Special Conditions public function up(Schema $schema) : void

    { // ... $this->warnIf(true, 'Something might be going wrong'); $this->abortIf(true, 'Something went wrong. Aborting.'); $this->skipIf(true, 'Skipping this migration.'); // ... }
  104. Migrations / Unrecoverable Migrations public function down(Schema $schema) : void

    { $this->throwIrreversibleMigrationException(); }
  105. Migrations

  106. Fixtures composer require --dev doctrine/doctrine-fixtures-bunde

  107. Fixtures namespace App\DataFixtures; use App\Entity\Product; use Doctrine\Bundle\FixturesBundle\Fixture; use Doctrine\Common\Persistence\ObjectManager; class

    AppFixtures extends Fixture { public function load(ObjectManager $manager) { $product = new Product(); //… $manager->persist($product); $manager->flush(); } }
  108. Fixtures php bin/console doctrine:fixtures:load

  109. Fixtures composer require --dev hautelook/alice-bundle

  110. Fixtures # fixtures/dummy.yaml App\Entity\Dummy: dummy_{1..10}: name: <name()> related_dummy: '@related_dummy*' App\Entity\RelatedDummy:

    related_dummy_{1..10}: name: <name()>
  111. Fixtures composer require --dev nelmio/alice fzaninotto/faker

  112. Testing Doctrine Bundle TestHelper

  113. Testing use Symfony\Bridge\Doctrine\Test\DoctrineTestHelper; class TaskListRepositoryTest extends TestCase { public function

    testSomething() { $config = DoctrineTestHelper::createTestConfiguration(); $entityManager = DoctrineTestHelper::createTestEntityManager($config); } }
  114. Testing composer require --dev dama/doctrine-test-bundle

  115. Testing <!-- phpunit.xml.dist --> <phpunit> <!-- ... --> <!-- Add

    this in PHPUnit 8 or higher --> <extensions> <extension class="DAMA\DoctrineTestBundle\PHPUnit\PHPUnitExtension"/> </extensions> <!-- Add this in PHPUnit 7 or lower --> <listeners> <listener class="\DAMA\DoctrineTestBundle\PHPUnit\PHPUnitListener"/> </listeners> </phpunit>