Magento 2 Indexer

- What's an index?
- What's an indexer?
- What's indexing?
A brief introduction about most of the indexing stuff in Magento 2.
What's an mview?
When will data be processed?
Naming: reatime, update on schedule, update on save, ...
- Lock Management
- Parallel processing
- Dependencies
- Trigger
- Full reindex
- Cronjobs

Transcript

1. Magento 2 Indexer 1
2. Indexing? Indexer? Index? What? netz98 a valantic company | Christian Münch @cmuench 2
3. Indexing is how Adobe Commerce and Magento Open Source transform data such as products and categories, to improve the performance of your storefront. As data changes, the transformed data must be updated or reindexed. The application has a very sophisticated architecture that stores lots of merchant data (including catalog data, prices, users, and stores) in many database tables. To optimize storefront performance, the application accumulates data into special tables using indexers. https://developer.adobe.com/commerce/php/development/components/indexing/ Definition by Adobe/Magento netz98 a valantic company | Christian Münch @cmuench 3
4. Dictionary: Original data entered to the system. Dictionaries are organized in normal form to facilitate maintenance (updating the data). Index: Representation of the original data for optimized reading and searching. Indexes can contain results of aggregations and various calculations. Index data can be always re-created from a dictionary using a certain algorithm. Indexer : Object that creates an index. https://developer.adobe.com/commerce/php/development/components/indexing/#indexi ng-terminology Terminology netz98 a valantic company | Christian Münch @cmuench 4
5. php bin/magento indexer:info customer_grid Customer Grid design_config_grid Design Config Grid catalog_category_product Category Products catalog_product_category Product Categories catalogrule_rule Catalog Rule Product catalog_product_attribute Product EAV cataloginventory_stock Stock catalog_product_price Product Price catalogrule_product Catalog Product Rule catalogsearch_fulltext Catalog Search targetrule_product_rule Product/Target Rule targetrule_rule_product Target Rule/Product salesrule_rule Sales Rule elasticsuite_categories_fulltext ElasticSuite Category Indexing elasticsuite_cms_page_fulltext Elasticsuite Cms Page Indexing elasticsuite_thesaurus ElasticSuite Thesaurus Indexing Available Indexers netz98 a valantic company | Christian Münch @cmuench 5
6. php bin/magento indexer:reindex php bin/magento indexer:reindex catalog_category_product catalogrule_product Full Reindex netz98 a valantic company | Christian Münch @cmuench 6
7. php bin/magento indexer:show-mode Customer Grid: Update by Schedule Design Config Grid: Update by Schedule Category Products: Update by Schedule Product Categories: Update by Schedule Catalog Rule Product: Update by Schedule Product EAV: Update by Schedule Stock: Update by Schedule Product Price: Update by Schedule Catalog Product Rule: Update by Schedule Catalog Search: Update by Schedule Product/Target Rule: Update by Schedule Target Rule/Product: Update by Schedule Sales Rule: Update by Schedule ElasticSuite Category Indexing: Update on Save Elasticsuite Cms Page Indexing: Update on Save ElasticSuite Thesaurus Indexing: Update on Save Indexer Mode netz98 a valantic company | Christian Münch @cmuench 7
8. Update "on save" Synchronously aka "realtime" After every model save. Uses PHP logic. Minimal delay Cause database locks on concurrent writes Update "on scheduled" Asynchronously Sequentially processed by cron job (Group "indexer"). Cronjob creates database triggers. Detects database changes. Also without PHP logic. Delay (depends on cronjob runtime and frequency) It is scaling! No database locks. Indexer Mode netz98 a valantic company | Christian Münch @cmuench 8
9. php bin/magento indexer:set-mode <realtime|schedule> Index mode for Indexer Customer Grid has not been changed Index mode for Indexer Design Config Grid has not been changed Index mode for Indexer Category Products has not been changed Index mode for Indexer Product Categories has not been changed Index mode for Indexer Catalog Rule Product has not been changed Index mode for Indexer Product EAV has not been changed Index mode for Indexer Stock has not been changed Index mode for Indexer Product Price has not been changed Index mode for Indexer Catalog Product Rule has not been changed Index mode for Indexer Catalog Search has not been changed Index mode for Indexer Product/Target Rule has not been changed Index mode for Indexer Target Rule/Product has not been changed Index mode for Indexer Sales Rule has not been changed Index mode for Indexer ElasticSuite Category Indexing was changed from 'Update on Save' to 'Update by Schedule' Index mode for Indexer Elasticsuite Cms Page Indexing was changed from 'Update on Save' to 'Update by Schedule' Index mode for Indexer ElasticSuite Thesaurus Indexing was changed from 'Update on Save' to 'Update by Schedule' Change Mode netz98 a valantic company | Christian Münch @cmuench 9
10. php bin/magento indexer:status Is everything up-to-date? - Indexer Status netz98 a valantic company | Christian Münch @cmuench 10
11. php bin/magento indexer:reset php bin/magento indexer:reset catalog_product_price Reset after a interrupted index:reindex netz98 a valantic company | Christian Münch @cmuench 11
12. Supported is currently only the price indexer. bin/magento indexer:show-dimensions-mode bin/magento indexer:set-dimensions-mode catalog_product_price website Mode can be "website, customer_group,website_and_customer_group" Indexer Mode netz98 a valantic company | Christian Münch @cmuench 12
13. MAGE_INDEXER_THREADS_COUNT=3 php -f bin/magento indexer:reindex catalog_product_price Uses pcntl_fork Parallel netz98 a valantic company | Christian Münch @cmuench 13
14. Indexer Implementation netz98 a valantic company | Christian Münch @cmuench 14
15. via config in a module . └── MyModule └── etc └── indexer.xml <?xml version="1.0" encoding="UTF-8"?> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Indexer/etc/indexer.xsd"> <indexer id="inventory" view_id="inventory" class="Magento\InventoryIndexer\Indexer\InventoryIndexer"> <title translate="true">Inventory</title> <description translate="true">Inventory index (MSI)</description> </indexer> <indexer id="catalog_product_price"> <dependencies> <indexer id="inventory"/> </dependencies> </indexer> </config> Indexer Registration netz98 a valantic company | Christian Münch @cmuench 15
16. class \Magento\Indexer\Model\Indexer\DependencyDecorator implements \Magento\Framework\Indexer\IndexerInterface { // ... public function reindexRow($id) { $this->cacheCleaner->start(); $this->indexer->reindexRow($id); $dependentIndexerIds = $this->dependencyInfoProvider->getIndexerIdsToRunAfter($this->indexer->getId()); foreach ($dependentIndexerIds as $indexerId) { $dependentIndexer = $this->indexerRegistry->get($indexerId); if (!$dependentIndexer->isScheduled()) { $dependentIndexer->reindexRow($id); } } $this->cacheCleaner->flush(); } } Default implementation defined in di.xml of Magento_Indexer module. Dependency "update on save" netz98 a valantic company | Christian Münch @cmuench 16
17. \Magento\Indexer\Console\Command\IndexerReindexCommand::getIndexers protected function getIndexers(InputInterface $input) { $indexers = parent::getIndexers($input); $allIndexers = $this->getAllIndexers(); if (!array_diff_key($allIndexers, $indexers)) { return $indexers; } $relatedIndexers = []; $dependentIndexers = []; foreach ($indexers as $indexer) { $relatedIndexers[] = $this->getRelatedIndexerIds($indexer->getId()); $dependentIndexers[] = $this->getDependentIndexerIds($indexer->getId()); } $relatedIndexers = array_unique(array_merge([], ...$relatedIndexers)); $dependentIndexers = array_merge([], ...$dependentIndexers); $invalidRelatedIndexers = []; foreach ($relatedIndexers as $relatedIndexer) { if ($allIndexers[$relatedIndexer]->isInvalid()) { $invalidRelatedIndexers[] = $relatedIndexer; } } return array_intersect_key( $allIndexers, array_flip( array_unique( array_merge( array_keys($indexers), $invalidRelatedIndexers, $dependentIndexers ) ) ) ); } Dependency index:reindex command (Hip hip array) 17
18. class MyIndexer implements \Magento\Framework\Indexer\ActionInterface, \Magento\Framework\Mview\ActionInterface { // ... } Logical separation: \Magento\Framework\Indexer\ActionInterface : Update on save / adhoc \Magento\Framework\Mview\ActionInterface : Update on schedule Indexer Class netz98 a valantic company | Christian Münch @cmuench 18
19. interface ActionInterface { /** * Execute full indexation * * @return void */ public function executeFull(); /** * Execute partial indexation by ID list * * @param int[] $ids * @return void */ public function executeList(array $ids); /** * Execute partial indexation by ID * * @param int $id * @return void */ public function executeRow($id); } \Magento\Framework\Indexer\ActionInterface netz98 a valantic company | Christian Münch @cmuench 19
20. Called by scheduled indexer job (via cronjob) interface ActionInterface { /** * Execute materialization on ids entities * * @param int[] $ids * @return void * @api */ public function execute($ids); } \Magento\Framework\Mview\ActionInterface netz98 a valantic company | Christian Münch @cmuench 20
21. How update on schedule works netz98 a valantic company | Christian Münch @cmuench 21
22. In computing, a materialized view is a database object that contains the results of a query. For example, it may be a local copy of data located remotely, or may be a subset of the rows and/or columns of a table or join result, or may be a summary using an aggregate function. -- Wikipedia MView Definition netz98 a valantic company | Christian Münch @cmuench 22
23. © https://oracle-base.com/articles/misc/materialized-views Materialized Views in a Oracle Database netz98 a valantic company | Christian Münch @cmuench 23
24. . └── MyModule └── etc └── mview.xml <?xml version="1.0" encoding="UTF-8"?> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Mview/etc/mview.xsd"> <view id="catalog_product_price" class="Magento\Catalog\Model\Indexer\Product\Price" group="indexer"> <subscriptions> <table name="catalog_product_entity" entity_column="entity_id" /> <table name="catalog_product_entity_datetime" entity_column="entity_id" /> <table name="catalog_product_entity_decimal" entity_column="entity_id" /> <table name="catalog_product_entity_int" entity_column="entity_id" /> <table name="catalog_product_entity_tier_price" entity_column="entity_id" /> </subscriptions> </view> </config> MView Definition -> Trigger Config netz98 a valantic company | Christian Münch @cmuench 24
25. Status: working, valid, invalid Indexer State in the database netz98 a valantic company | Christian Münch @cmuench 25
26. When will triggers be created? netz98 a valantic company | Christian Münch @cmuench 26
27. Insert: BEGIN INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (NEW.`entity_id`); END Update: BEGIN IF (NEW.`entity_id` <=> OLD.`entity_id` OR NEW.`attribute_set_id` <=> OLD.`attribute_set_id` OR NEW.`parent_id` <=> OLD.`parent_id` OR NEW.`created_at` <=> OLD.`created_at` OR NEW.`path` <=> OLD.`path` OR NEW.`position` <=> OLD.`position` OR NEW.`level` <=> OLD.`level` OR NEW.`children_count` <=> OLD.`children_count`) THEN INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (NEW.`entity_id`); END IF; END Delete: BEGIN INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (OLD.`entity_id`); END How triggers look like netz98 a valantic company | Christian Münch @cmuench 27
28. Changelog Tables 28
29. Version-ID (last id in changelog table) status -> idle, working, suspended Up-to-date: version_id == last version_id in _CL table MView Status netz98 a valantic company | Christian Münch @cmuench 29
30. php bin/magento cron:run # only indexer jobs php bin/magento cron:run --group index Indexer Cron netz98 a valantic company | Christian Münch @cmuench 30
31. Job Description Frequency indexer_clean_all_changelogs Cleanup old entries in _CL tables. daily indexer_reindex_all_invalid Start full reindex of broken indexers every minute indexer_update_all_views Executes "update on schedule" every minute Indexer Cronjobs netz98 a valantic company | Christian Münch @cmuench 31
32. Update on scheduled processing netz98 a valantic company | Christian Münch @cmuench 32
33. Indexer batching (Finetuning, depending on data) Configure batch size via ENV variables or in env.php Indexer table switching (for full reindex) https://developer.adobe.com/commerce/php/development/components/indexing/optimiz ation/ Optimizations netz98 a valantic company | Christian Münch @cmuench 33
34. The status values in the indexer_state or mview_state database tables may not be the same as what is observed, because they sometimes do not get updated when an indexer fails. https://developer.adobe.com/commerce/php/development/components/indexing/#using-application-lock-mode-for- reindex-processes more accurate status by using a \Magento\Framework\Lock\LockManagerInterface implementation (DB, Zookeeper, FileLock, ...). cronjob will retry to index without a reset of the indexer env.php 'indexer' => [ 'use_application_lock' => true ] Don't trust the rabbit netz98 a valantic company | Christian Münch @cmuench 34
35. https://twitter.com/ospadano/status/1570859136379924482? t=R9Gp9EkBz1eYiYw05fgFFQ Can Indexer Cronjobs conflict? netz98 a valantic company | Christian Münch @cmuench 35
36. Lock Manager is configured correct IMHO no ... if netz98 a valantic company | Christian Münch @cmuench 36
37. <type name="Magento\Framework\Mview\View\Subscription"> <arguments> <argument name="ignoredUpdateColumnsBySubscription" xsi:type="array"> <item name="my_custom_view" xsi:type="array"> <item name="salesrule" xsi:type="array"> <item name="times_used" xsi:type="string">times_used</item> </item> </item> </argument> </arguments> </type> https://github.com/magento/magento2/issues/30243 Ignore columns for mview to be specified at the subscription level netz98 a valantic company | Christian Münch @cmuench 37
38. If triggers are missing (can be possible if DB dump was exported without triggers) ... n98-magerun2.phar index:trigger:recreate Force trigger recreation netz98 a valantic company | Christian Münch @cmuench 38
39. Update On Save netz98 a valantic company | Christian Münch @cmuench 39
40. Must be handled manually create, update, delete Different kind of implementations in Magento Commit Callbacks in Resource Models (own implementation) Plugins (modify an entity which is not fully controlled by your module) Events/Observer (process/entity dependency -> Trigger indexer of another entity) Most the time we extend indexer and do not require to handle the re-indexing. Test if indexer processes the data after a modification (e.g mview.xml) Test in Magento Admin if there is a UI Test changes via webapi Update On Save netz98 a valantic company | Christian Münch @cmuench 40
41. \Magento\Catalog\Model\Product::afterSave public function afterSave() { $this->getLinkInstance()->saveProductRelations($this); $this->getTypeInstance()->save($this); if ($this->getStockData()) { $this->setForceReindexEavRequired(true); } $this->_getResource()->addCommitCallback([$this, 'priceReindexCallback']); $this->_getResource()->addCommitCallback([$this, 'eavReindexCallback']); $result = parent::afterSave(); $this->_getResource()->addCommitCallback([$this, 'reindex']); $this->reloadPriceInfo(); return $result; } /** * Reindex callback for EAV indexer * * @return void */ public function eavReindexCallback() { if ($this->isObjectNew() || $this->isDataChanged()) { $this->_productEavIndexerProcessor->reindexRow($this->getEntityId()); } } Example: Commit Callback / Resource Model netz98 a valantic company | Christian Münch @cmuench 41
42. Magento_CatalogRule module di.xml <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd"> <type name="Magento\Catalog\Model\Product"> <plugin name="apply_catalog_rules_after_product_save_and_reindex" type="Magento\CatalogRule\Plugin\Indexer\Product\Save\ApplyRulesAfterReindex"/> </type> </config> class ApplyRulesAfterReindex { /** * @param ProductRuleProcessor $productRuleProcessor */ public function __construct(ProductRuleProcessor $productRuleProcessor) { $this->productRuleProcessor = $productRuleProcessor; } // ... public function afterReindex(Product $subject) { $this->productRuleProcessor->reindexRow($subject->getId()); } } Example: Plugin netz98 a valantic company | Christian Münch @cmuench 42
43. vendor/magento/module-catalog-inventory/etc/events.xml <event name="sales_model_service_quote_submit_success"> <observer name="inventory" instance="Magento\CatalogInventory\Observer\ReindexQuoteInventoryObserver"/> </event> Example 3: Event/Observer netz98 a valantic company | Christian Münch @cmuench 43
44. public function execute(EventObserver $observer) { // Reindex quote ids $quote = $observer->getEvent()->getQuote(); $productIds = []; foreach ($quote->getAllItems() as $item) { $productIds[$item->getProductId()] = $item->getProductId(); $children = $item->getChildrenItems(); if ($children) { foreach ($children as $childItem) { $productIds[$childItem->getProductId()] = $childItem->getProductId(); } } } if ($productIds) { $this->stockIndexerProcessor->reindexList($productIds); // <-- CALL INDEXER 1 } // Reindex previously remembered items $productIds = []; foreach ($this->itemsForReindex->getItems() as $item) { $item->save(); $productIds[] = $item->getProductId(); } if (!empty($productIds)) { $this->priceIndexer->reindexList($productIds); // <-- CALL INDEXER 2 } $this->itemsForReindex->clear(); // Clear list of remembered items - we don't need it anymore } netz98 a valantic company | Christian Münch @cmuench 44
45. Cache Cleaning netz98 a valantic company | Christian Münch @cmuench 45
46. vendor/magento/module-indexer/etc/di.xml <type name="Magento\Framework\Indexer\ActionInterface"> <plugin name="cache_cleaner_after_reindex" type="Magento\Indexer\Model\Indexer\CacheCleaner" /> </type> <type name="Magento\Framework\Indexer\CacheContext"> <plugin name="defer_cache_cleaning" type="Magento\Indexer\Model\Indexer\DeferCacheCleaning" /> </type> Plugins in Magento_Indexer module netz98 a valantic company | Christian Münch @cmuench 46
47. class CacheCleaner { /** * Defer cache cleaning until after execute full * * @param ActionInterface $subject * @return void * @SuppressWarnings(PHPMD.UnusedFormalParameter) */ public function beforeExecuteFull(ActionInterface $subject) { $this->cacheCleaner->start(); } /** * Clean cache after full reindex full * * @param ActionInterface $subject * @return void * @SuppressWarnings(PHPMD.UnusedFormalParameter) */ public function afterExecuteFull(ActionInterface $subject) { $this->cacheCleaner->flush(); } /** * Defer cache cleaning until after execute list * * @param ActionInterface $subject * @return void * @SuppressWarnings(PHPMD.UnusedFormalParameter) */ public function beforeExecuteList(ActionInterface $subject) { $this->cacheCleaner->start(); } /** * Clean cache after reindexed list. * * @param ActionInterface $subject * @return void * @SuppressWarnings(PHPMD.UnusedFormalParameter) */ public function afterExecuteList(ActionInterface $subject) { $this->cacheCleaner->flush(); } /** * Defer cache cleaning until after execute row * * @param ActionInterface $subject * @return void * @SuppressWarnings(PHPMD.UnusedFormalParameter) */ public function beforeExecuteRow(ActionInterface $subject) { $this->cacheCleaner->start(); } /** * Clean cache after reindexed row. * * @param ActionInterface $subject * @return void * @SuppressWarnings(PHPMD.UnusedFormalParameter) */ public function afterExecuteRow(ActionInterface $subject) { $this->cacheCleaner->flush(); } } netz98 a valantic company | Christian Münch @cmuench 47
48. class DeferredCacheCleaner { // ... /** * @param EventManager $eventManager * @param CacheInterface $appCache * @param DeferredCacheContext $deferredCacheContext * @param CacheContext $cacheContext */ public function __construct( EventManager $eventManager, CacheInterface $appCache, DeferredCacheContext $deferredCacheContext, CacheContext $cacheContext ) { $this->eventManager = $eventManager; $this->deferredCacheContext = $deferredCacheContext; $this->appCache = $appCache; $this->cacheContext = $cacheContext; } /** * Defer cache cleaning until flush() is called * * @see flush() */ public function start(): void { $this->deferredCacheContext->start(); } /** * Flush cache */ public function flush(): void { $this->deferredCacheContext->commit(); $this->eventManager->dispatch('clean_cache_by_tags', ['object' => $this->cacheContext]); $identities = $this->cacheContext->getIdentities(); if (!empty($identities)) { $this->appCache->clean($identities); $this->cacheContext->flush(); } } } netz98 a valantic company | Christian Münch @cmuench 48
49. vendor/magento/module-cache-invalidate/etc/events.xml <event name="clean_cache_by_tags"> <observer name="invalidate_varnish" instance="Magento\CacheInvalidate\Observer\InvalidateVarnishObserver"/> </event> vendor/magento/module-page-cache/etc/events.xml <event name="clean_cache_by_tags"> <observer name="invalidate_builtin" instance="Magento\PageCache\Observer\FlushCacheByTags" /> </event> clean_cache_by_tags -> FPC Invalidation netz98 a valantic company | Christian Münch @cmuench 49
50. vendor/magento/module-customer/etc/indexer.xml <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Indexer/etc/indexer.xsd"> <indexer id="customer_grid" class="Magento\Framework\Indexer\Action\Entity" primary="customer"> <!-- ... --> <fieldset name="customer" source="Magento\Customer\Model\Indexer\Source" provider="Magento\Customer\Model\Indexer\AttributeProvider"> <field name="name" xsi:type="searchable" dataType="text" handler="CustomerNameHandler"/> <field name="email" xsi:type="searchable" dataType="varchar"/> <field name="group_id" xsi:type="filterable" dataType="int"/> <field name="created_at" xsi:type="filterable" dataType="timestamp"/> <!-- ... --> </fieldset> <fieldset name="shipping" source="Magento\Customer\Model\ResourceModel\Address\Collection"> <reference fieldset="customer" from="entity_id" to="default_shipping"/> <field name="full" xsi:type="searchable" dataType="text" handler="ShippingAddressHandler"/> </fieldset> <fieldset name="billing"> <!-- ... --> </fieldset> <saveHandler class="Magento\Framework\Indexer\SaveHandler\Grid"/> <structure class="Magento\Framework\Indexer\GridStructure"/> </indexer> </config> "Structure" Handling / Save Handler netz98 a valantic company | Christian Münch @cmuench 50
51. https://experienceleague.adobe.com/docs/commerce-operations/configuration- guide/cli/manage-indexers.html MView Implementation: vendor/magento/framework/Mview https://amasty.com/blog/comprehensive-guide-to-magento-2-indexing/ https://developer.adobe.com/commerce/php/development/components/indexing/ Performance: https://developer.adobe.com/commerce/php/development/components/indexing/op timization/ Resources netz98 a valantic company | Christian Münch @cmuench 51
52. netz98 a valantic company | Christian Münch @cmuench 52
53. Photos provided by Canva Tobias Fischer netz98 a valantic company | Christian Münch @cmuench 53