Tweet
Tweet

Slide 1

Slide 1 text

Teaching Doctrine to be Lazy by Kevin Bond Image Credit: Sébastien Lavalaye

Slide 2

Slide 2 text

Me? From Ontario, Canada Husband, father of three Symfony user since 1.0 Symfony Core Team @kbond on GitHub/Slack @zenstruck on Twitter

Slide 3

Slide 3 text

zenstruck? A GitHub organization where my open source packages live zenstruck/foundry zenstruck/browser zenstruck/messenger-test zenstruck/filesystem (wip) zenstruck/schedule-bundle (for <6.3) ... Many now co-maintained by Nicolas PHILIPPE ( @nikophil )

Slide 4

Slide 4 text

What we'll cover Hydration considerations Lazy batch iterating (readonly) Lazy batch processing Updating/Deleting/Persisting Lazy relationships Future ideas Teaching Doctrine to be Lazy Kevin Bond • @zenstruck • github.com/kbond 4

Slide 5

Slide 5 text

Sample App +----------+ +------------+ | PRODUCT | | PURCHASE | |----------| |------------| | id |---+ | id | | sku | +--<| product_id | | stock | | date | | category | | amount | +----------+ +------------+ 1,000+ products, 100,000+ purchases Products may have 1,000's of purchases Teaching Doctrine to be Lazy Kevin Bond • @zenstruck • github.com/kbond 5

Slide 6

Slide 6 text

Mongo? With some tweaks, the demonstrated techniques should/could apply to any doctrine/persistence implementation I'm using doctrine/orm for the examples in this talk Teaching Doctrine to be Lazy Kevin Bond • @zenstruck • github.com/kbond 6

Slide 7

Slide 7 text

Part 1: Hydration Considerations Hydration is expensive Some rules Only hydrate what you need Only hydrate when you need it Cleanup after yourself Teaching Doctrine to be Lazy Kevin Bond • @zenstruck • github.com/kbond 7

Slide 8

Slide 8 text

Profiling Hydrations Web Profiler? debesha/doctrine-hydration-profiler-bundle DoctrineBundle? Needs a hook in doctrine/orm Blackfire.io metrics.doctrine.entities.hydrated Teaching Doctrine to be Lazy - Part 1: Hydration Considerations Kevin Bond • @zenstruck • github.com/kbond 8

Slide 9

Slide 9 text

Part 2: Batch Iterating Read-only Use SQL? purchase:report command Generates a report for all purchases Teaching Doctrine to be Lazy Kevin Bond • @zenstruck • github.com/kbond 9

Slide 10

Slide 10 text

$repo->findAll() 100000/100000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% 1 sec/1 sec 166.0 MiB // Time: 2 secs, Queries: 1 Only hydrate what you need Only hydrate when you need it Cleanup after yourself Teaching Doctrine to be Lazy - Part 2: Batch Iterating Kevin Bond • @zenstruck • github.com/kbond 10

Slide 11

Slide 11 text

$repo->matching(new Criteria()) 100000/100000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% 1 sec/1 sec 168.0 MiB // Time: 1 sec, Queries: 2 Only hydrate what you need Only hydrate when you need it Cleanup after yourself Teaching Doctrine to be Lazy - Part 2: Batch Iterating Kevin Bond • @zenstruck • github.com/kbond 11

Slide 12

Slide 12 text

Doctrine\ORM\Query::toIterable() 100000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 2 secs 166.0 MiB // Time: 2 secs, Queries: 1 Only hydrate what you need Only hydrate when you need it Cleanup after yourself Teaching Doctrine to be Lazy - Part 2: Batch Iterating Kevin Bond • @zenstruck • github.com/kbond 12

Slide 13

Slide 13 text

Batch Utilities - Iterator ocramius/doctrine-batch-utils Takes an ORM Query object and iterates over the result set in batches Clear the ObjectManager after each batch to free memory Enhanced: Accepts any iterable and any ObjectManager instance Teaching Doctrine to be Lazy - Part 2: Batch Iterating Kevin Bond • @zenstruck • github.com/kbond 13

Slide 14

Slide 14 text

Use BatchIterator $iterator = new BatchIterator($query->toIterable(), $this->em); 100000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% 2 secs 20.0 MiB // Time: 2 secs, Queries: 1 Only hydrate what you need Only hydrate when you need it Cleanup after yourself Teaching Doctrine to be Lazy - Part 2: Batch Iterating Kevin Bond • @zenstruck • github.com/kbond 14

Slide 15

Slide 15 text

Memory Stays Constant, Time Increases 200,000 purchases? 200000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% 4 secs 20.0 MiB // Time: 4 secs, Queries: 1 1,000,000 purchases? 1000000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 19 secs 22.0 MiB // Time: 19 secs, Queries: 1 Teaching Doctrine to be Lazy - Part 2: Batch Iterating Kevin Bond • @zenstruck • github.com/kbond 15

Slide 16

Slide 16 text

1,000,000 Purchases Using $repo->findAll() ? 1000000/1000000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% 2 secs/2 secs 1.5 GiB // Time: 16 secs, Queries: 1 1.5 GiB of memory? Teaching Doctrine to be Lazy - Part 2: Batch Iterating Kevin Bond • @zenstruck • github.com/kbond 16

Slide 17

Slide 17 text

Part 3: Batch Processing Teaching Doctrine to be Lazy Kevin Bond • @zenstruck • github.com/kbond 17

Slide 18

Slide 18 text

Batch Updating product:stock-update Command Loop through all products Update stock level from a source (ie. CSV files, API, etc) Teaching Doctrine to be Lazy - Part 3: Batch Processing (Update) Kevin Bond • @zenstruck • github.com/kbond 18

Slide 19

Slide 19 text

$repo->findAll() foreach ($repo->findAll() as $product) { /** @var Product $product */ $product->setStock($this->currentStockFor($product)); $this->em->flush(); } Teaching Doctrine to be Lazy - Part 3: Batch Processing (Update) Kevin Bond • @zenstruck • github.com/kbond 19

Slide 20

Slide 20 text

$repo->findAll() foreach ($repo->findAll() as $product) { /** @var Product $product */ $product->setStock($this->currentStockFor($product)); $this->em->flush(); } 1000/1000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% 8 secs/8 secs 16.0 MiB // Time: 8 secs, Queries: 988 Teaching Doctrine to be Lazy - Part 3: Batch Processing (Update) Kevin Bond • @zenstruck • github.com/kbond 20

Slide 21

Slide 21 text

$repo->findAll() , Delay Flush foreach ($repo->findAll() as $product) { /** @var Product $product */ $product->setStock($this->currentStockFor($product)); } $this->em->flush(); Teaching Doctrine to be Lazy - Part 3: Batch Processing (Update) Kevin Bond • @zenstruck • github.com/kbond 21

Slide 22

Slide 22 text

$repo->findAll() , Delay Flush foreach ($repo->findAll() as $product) { /** @var Product $product */ $product->setStock($this->currentStockFor($product)); } $this->em->flush(); 1000/1000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% < 1 sec/< 1 sec 16.0 MiB // Time: < 1 sec, Queries: 2 Teaching Doctrine to be Lazy - Part 3: Batch Processing (Update) Kevin Bond • @zenstruck • github.com/kbond 22

Slide 23

Slide 23 text

$repo->findAll() , Delay Flush 100,000 products? 100000/100000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% < 1 sec/< 1 sec 186.0 MiB // Time: 12 secs, Queries: 2 Teaching Doctrine to be Lazy - Part 3: Batch Processing (Update) Kevin Bond • @zenstruck • github.com/kbond 23

Slide 24

Slide 24 text

Batch Utilities - Processor ocramius/doctrine-batch-utils Takes an ORM Query object and iterates over the result set in batches Flush and clear the ObjectManager after each batch to free memory and save changes Wrap everything in a transaction Enhanced: Accepts any iterable and any ObjectManager instance Teaching Doctrine to be Lazy - Part 3: Batch Processing (Update) Kevin Bond • @zenstruck • github.com/kbond 24

Slide 25

Slide 25 text

Using BatchProcessor $processor = new BatchProcessor($query->toIterable(), $this->em); foreach ($processor as $product) { /** @var Product $product */ $product->setStock($this->currentStockFor($product)); } // no need for "flush" 1000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] < 1 sec 16.0 MiB // Time: < 1 sec, Queries: 1 Teaching Doctrine to be Lazy - Part 3: Batch Processing (Update) Kevin Bond • @zenstruck • github.com/kbond 25

Slide 26

Slide 26 text

Using BatchProcessor - 100,000 Products 100000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 11 secs 22.0 MiB // Time: 11 secs, Queries: 2 Teaching Doctrine to be Lazy - Part 3: Batch Processing (Update) Kevin Bond • @zenstruck • github.com/kbond 26

Slide 27

Slide 27 text

Batch Deleting DQL DELETE statement? PreRemove / PostRemove events? purchase:purge Command Delete all purchases older than X days Imagine a PostRemove event that archives the purged purchases Teaching Doctrine to be Lazy - Part 3: Batch Processing (Delete) Kevin Bond • @zenstruck • github.com/kbond 27

Slide 28

Slide 28 text

Using BatchProcessor $processor = new BatchProcessor($query->toIterable(), $this->em); foreach ($processor as $purchase) { /** @var Purchase $purchase */ $this->em->remove($purchase); // no need for "flush" } Teaching Doctrine to be Lazy - Part 3: Batch Processing (Delete) Kevin Bond • @zenstruck • github.com/kbond 28

Slide 29

Slide 29 text

Using BatchProcessor - 100,000 Purchases 75237 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% 9 secs 18.0 MiB // Time: 9 secs, Queries: 1 Teaching Doctrine to be Lazy - Part 3: Batch Processing (Delete) Kevin Bond • @zenstruck • github.com/kbond 29

Slide 30

Slide 30 text

Using BatchProcessor - 1,000,000 Purchases 753854 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% 1 min 18.0 MiB // Time: 1 min, Queries: 1 Teaching Doctrine to be Lazy - Part 3: Batch Processing (Delete) Kevin Bond • @zenstruck • github.com/kbond 30

Slide 31

Slide 31 text

Batch Persisting product:import Command Imports products from a source (ie. CSV files, API, etc) We'll use a Generator to yield Product instances from our source Requires enhanced BatchProcessor Accepts any iterable Teaching Doctrine to be Lazy - Part 3: Batch Processing (Persist) Kevin Bond • @zenstruck • github.com/kbond 31

Slide 32

Slide 32 text

Using BatchProcessor $processor = new BatchProcessor( $this->products(), // Product[] - our "source" $this->em, ); foreach ($processor as $product) { /** @var Product $product */ $this->em->persist($product); // no need for "flush" } Teaching Doctrine to be Lazy - Part 3: Batch Processing (Persist) Kevin Bond • @zenstruck • github.com/kbond 32

Slide 33

Slide 33 text

Using BatchProcessor - Import 1,000 1000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] < 1 sec 16.0 MiB // Time: < 1 sec, Queries: 1 Teaching Doctrine to be Lazy - Part 3: Batch Processing (Persist) Kevin Bond • @zenstruck • github.com/kbond 33

Slide 34

Slide 34 text

Using BatchProcessor - Import 100,000 100000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 12 secs 16.0 MiB // Time: 12 secs, Queries: 1 Teaching Doctrine to be Lazy - Part 3: Batch Processing (Persist) Kevin Bond • @zenstruck • github.com/kbond 34

Slide 35

Slide 35 text

Part 4: Lazy Relationships product:report Command Loop over all products (using our BatchIterator ) For each product Fetch details on the most recent purchase Fetch number of purchases in the last 30 days Some products have 10,000+ purchases Teaching Doctrine to be Lazy Kevin Bond • @zenstruck • github.com/kbond 35

Slide 36

Slide 36 text

Command Code foreach ($products as $product) { /** @var Product $product */ /** @var Collection&Selectable $purchases */ $purchases = $product->getPurchases(); $last30Days = Criteria::create()->where( Criteria::expr()->gte('date', new \DateTimeImmutable('-30 days')) ); $this->addToReport( $product->getSku(), $purchases->first() ?: null, // most recent purchase $purchases->matching($last30Days)->count(), ); } Teaching Doctrine to be Lazy - Part 4: Lazy Relationships Kevin Bond • @zenstruck • github.com/kbond 36

Slide 37

Slide 37 text

Standard One-to-Many Relationship #[ORM\Entity] class Product { #[ORM\OneToMany(mappedBy: 'product', targetEntity: Purchase::class)] #[ORM\OrderBy(['date' => 'DESC'])] private Collection $purchases; public function getPurchases(): Collection { return $this->purchases; } } Teaching Doctrine to be Lazy - Part 4: Lazy Relationships Kevin Bond • @zenstruck • github.com/kbond 37

Slide 38

Slide 38 text

Standard One-to-Many Relationship $purchases = $product->getPurchases(); $purchases->count(); // initializes entire collection $purchases->first(); // initializes entire collection $purchases->slice(0, 10); // initializes entire collection foreach ($purchases as $purchase) { // initializes entire collection } Teaching Doctrine to be Lazy - Part 4: Lazy Relationships Kevin Bond • @zenstruck • github.com/kbond 38

Slide 39

Slide 39 text

Standard One-to-Many Relationship 1000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 6 secs 128.0 MiB // Time: 6 secs, Queries: 1001 Teaching Doctrine to be Lazy - Part 4: Lazy Relationships Kevin Bond • @zenstruck • github.com/kbond 39

Slide 40

Slide 40 text

Extra Lazy One-to-Many Relationship #[ORM\Entity] class Product { #[ORM\OneToMany( mappedBy: 'product', targetEntity: Purchase::class, fetch: 'EXTRA_LAZY', // !!! )] #[ORM\OrderBy(['date' => 'DESC'])] private Collection $purchases; } Teaching Doctrine to be Lazy - Part 4: Lazy Relationships Kevin Bond • @zenstruck • github.com/kbond 40

Slide 41

Slide 41 text

Extra Lazy One-to-Many Relationship Assuming the collection hasn't been previously initialized, Certain methods create new queries: $purchases = $product->getPurchases(); $purchases->count(); // creates an additional "count" query $purchases->first(); // initializes entire collection !! $purchases->slice(0, 10); // creates an additional "slice" query foreach ($purchases as $purchase) { // initializes entire collection } Teaching Doctrine to be Lazy - Part 4: Lazy Relationships Kevin Bond • @zenstruck • github.com/kbond 41

Slide 42

Slide 42 text

Extra Lazy One-to-Many Relationship More efficient first() : $purchases = $product->getPurchases(); $purchases->slice(0, 1)[0] ?? null; Teaching Doctrine to be Lazy - Part 4: Lazy Relationships Kevin Bond • @zenstruck • github.com/kbond 42

Slide 43

Slide 43 text

Updated Command Code foreach ($products as $product) { // ... $this->addToReport( $product->getSku(), $purchases->slice(0, 1)[0] ?? null, // most recent purchase $purchases->matching($last30Days)->count(), ); } Teaching Doctrine to be Lazy - Part 4: Lazy Relationships Kevin Bond • @zenstruck • github.com/kbond 43

Slide 44

Slide 44 text

Extra Lazy One-to-Many Relationship 1000 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 1 sec 18.0 MiB // Time: 1 sec, Queries: 2001 Teaching Doctrine to be Lazy - Part 4: Lazy Relationships Kevin Bond • @zenstruck • github.com/kbond 44

Slide 45

Slide 45 text

n+x Problem? ...it depends... Saving the number of queries at all costs is not always the best solution If the collection has many items, hydration will be more expensive than the extra queries Evaluate your models and use cases Teaching Doctrine to be Lazy - Part 4: Lazy Relationships Kevin Bond • @zenstruck • github.com/kbond 45

Slide 46

Slide 46 text

Batch Summary Hydration is expensive The BatchIterator / Processor can keep the expense down to time only When you have a large or unknown amount of data to process, it's better to move the processing to background tasks Teaching Doctrine to be Lazy Kevin Bond • @zenstruck • github.com/kbond 46

Slide 47

Slide 47 text

Part 5: Future Ideas Exploring some ideas in zenstruck/collection . Teaching Doctrine to be Lazy Kevin Bond • @zenstruck • github.com/kbond 47

Slide 48

Slide 48 text

Alternate Lazy by Default ObjectRepository Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 48

Slide 49

Slide 49 text

New ObjectRepository Interface /** * @template T of object * @extends \IteratorAggregate */ interface ObjectRepository extends \IteratorAggregate, \Countable { /** * @param mixed|Criteria $specification * * @return Result */ public function filter(mixed $specification): Result; } Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 49

Slide 50

Slide 50 text

The Result Interface /** * @template T of object * @extends \IteratorAggregate */ interface Result extends \IteratorAggregate, \Countable { public function first(): T|null; public function take(int $limit, int $offset = 0): self; public function process(int $chunkSize = 100): BatchProcessor public function toArray(): array; // ... } Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 50

Slide 51

Slide 51 text

ORM ObjectRepository::filter() $specification can be: array : works like findBy() Criteria : works like matching() callable(QueryBuilder, string): void : custom query Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 51

Slide 52

Slide 52 text

Using the $specification callable $purchases = $repo->filter( function(QueryBuilder $qb, string $root) use ($newerThan) { $qb->where("{$root}.date > :newerThan") ->setParameter('newerThan', $newerThan) ; } ); Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 52

Slide 53

Slide 53 text

Specification Objects You could extend this ObjectRepository to add your methods, but, because filter() accepts callable(QueryBuilder) , you can create invokable specification objects instead. Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 53

Slide 54

Slide 54 text

Between Specification final class Between { public function __invoke(QueryBuilder $qb, string $root): void { if ($this->from) { $qb->andWhere("{$root}.date >= :from") ->setParameter('from', $this->from) ; } // "to" logic... } } Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 54

Slide 55

Slide 55 text

Inject as a Service (Symfony 6.3+) /** * @param ObjectRepository $repo */ public function someAction( // extends "Autowire" (creates repo from factory service) #[ForClass(Purchase::class)] ObjectRepository $repo, ) { $products = $repo->filter(new Between('2021-01-01', '2021-12-31')); // ... } Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 55

Slide 56

Slide 56 text

Thank You! @kbond on GitHub/Slack @zenstruck on Twitter Sample Code: github.com/kbond/lazy-doctrine Slides: speakerdeck.com/kbond zenstruck/collection Teaching Doctrine to be Lazy Kevin Bond • @zenstruck • github.com/kbond 56

Slide 57

Slide 57 text

Paginating the Result class ResultPagerfantaAdapter implements AdapterInterface { public function getNbResults(): int { return $this->result->count(); } public function getSlice(int $offset, int $length): array { return $this->result->take($length, $offset)->toArray(); } } Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 57

Slide 58

Slide 58 text

Lazier Doctrine Collection $purchase = $purchases->first(); // use slice(0, 1)[0] ?? null internally foreach ($purchases as $purchase) { // lazily iterate "chunks" if large count } Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 58

Slide 59

Slide 59 text

Generic Specification System $specification = Spec::andX( new Between(from: new \DateTimeImmutable('-1 year')), // in last year Spec::greaterThan('amount', 100.00), // amount > $100.00 Spec::sortDesc('date'), // sort by date ); Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 59

Slide 60

Slide 60 text

Generic Specification System Use the same specification object in multiple places: // use with ORM $purchases = $ormPurchaseRepository->filter($specification); // use with Mongo $purchases = $mongoPurchaseRepository->filter($specification); // use with Collection $purchases = $product->getPurchases()->filter($specification); Teaching Doctrine to be Lazy - Part 5: Future Ideas Kevin Bond • @zenstruck • github.com/kbond 60