Magento 2 Indexer

Magento 2 Indexer
1

View Slide

Indexing? Indexer? Index? What?
netz98 a valantic company | Christian Münch @cmuench
2

View Slide

Indexing is how Adobe Commerce and Magento Open Source transform data
such as products and categories, to improve the performance of your storefront.
As data changes, the transformed data must be updated or reindexed. The
application has a very sophisticated architecture that stores lots of merchant data
(including catalog data, prices, users, and stores) in many database tables. To
optimize storefront performance, the application accumulates data into special
tables using indexers.

https://developer.adobe.com/commerce/php/development/components/indexing/
Definition by Adobe/Magento
3

View Slide

Dictionary: Original data entered to the system. Dictionaries are organized in
normal form to facilitate maintenance (updating the data).
Index: Representation of the original data for optimized reading and searching.
Indexes can contain results of aggregations and various calculations. Index data
can be always re-created from a dictionary using a certain algorithm.
Indexer : Object that creates an index.
https://developer.adobe.com/commerce/php/development/components/indexing/#indexi
ng-terminology
Terminology
4

View Slide

php bin/magento indexer:info

customer_grid Customer Grid

design_config_grid Design Config Grid

catalog_category_product Category Products

catalog_product_category Product Categories

catalogrule_rule Catalog Rule Product

catalog_product_attribute Product EAV

cataloginventory_stock Stock

catalog_product_price Product Price

catalogrule_product Catalog Product Rule

catalogsearch_fulltext Catalog Search

targetrule_product_rule Product/Target Rule

targetrule_rule_product Target Rule/Product

salesrule_rule Sales Rule

elasticsuite_categories_fulltext ElasticSuite Category Indexing

elasticsuite_cms_page_fulltext Elasticsuite Cms Page Indexing

elasticsuite_thesaurus ElasticSuite Thesaurus Indexing

Available Indexers
5

View Slide

php bin/magento indexer:reindex

php bin/magento indexer:reindex catalog_category_product catalogrule_product

Full Reindex
6

View Slide

php bin/magento indexer:show-mode

Customer Grid: Update by Schedule

Design Config Grid: Update by Schedule

Category Products: Update by Schedule

Product Categories: Update by Schedule

Catalog Rule Product: Update by Schedule

Product EAV: Update by Schedule

Stock: Update by Schedule

Product Price: Update by Schedule

Catalog Product Rule: Update by Schedule

Catalog Search: Update by Schedule

Product/Target Rule: Update by Schedule

Target Rule/Product: Update by Schedule

Sales Rule: Update by Schedule

ElasticSuite Category Indexing: Update on Save

Elasticsuite Cms Page Indexing: Update on Save

ElasticSuite Thesaurus Indexing: Update on Save

Indexer Mode
7

View Slide

Update "on save"
Synchronously aka "realtime"
After every model save. Uses PHP logic.
Minimal delay
Cause database locks on concurrent writes
Update "on scheduled"
Asynchronously
Sequentially processed by cron job (Group "indexer").
Cronjob creates database triggers. Detects database changes. Also without
PHP logic.
Delay (depends on cronjob runtime and frequency)
It is scaling! No database locks.
Indexer Mode
8

View Slide

php bin/magento indexer:set-mode

Index mode for Indexer Customer Grid has not been changed

Index mode for Indexer Design Config Grid has not been changed

Index mode for Indexer Category Products has not been changed

Index mode for Indexer Product Categories has not been changed

Index mode for Indexer Catalog Rule Product has not been changed

Index mode for Indexer Product EAV has not been changed

Index mode for Indexer Stock has not been changed

Index mode for Indexer Product Price has not been changed

Index mode for Indexer Catalog Product Rule has not been changed

Index mode for Indexer Catalog Search has not been changed

Index mode for Indexer Product/Target Rule has not been changed

Index mode for Indexer Target Rule/Product has not been changed

Index mode for Indexer Sales Rule has not been changed

Index mode for Indexer ElasticSuite Category Indexing was changed from 'Update on Save' to 'Update by Schedule'

Index mode for Indexer Elasticsuite Cms Page Indexing was changed from 'Update on Save' to 'Update by Schedule'

Index mode for Indexer ElasticSuite Thesaurus Indexing was changed from 'Update on Save' to 'Update by Schedule'

Change Mode
9

View Slide

php bin/magento indexer:status

Is everything up-to-date? - Indexer Status
10

View Slide

php bin/magento indexer:reset

php bin/magento indexer:reset catalog_product_price

Reset after a interrupted index:reindex
11

View Slide

Supported is currently only the price indexer.
bin/magento indexer:show-dimensions-mode

bin/magento indexer:set-dimensions-mode catalog_product_price website

Mode can be "website, customer_group,website_and_customer_group"
Indexer Mode
12

View Slide

MAGE_INDEXER_THREADS_COUNT=3 php -f bin/magento indexer:reindex catalog_product_price

Uses pcntl_fork
Parallel
13

View Slide

Indexer Implementation
14

View Slide

via config in a module
.

└── MyModule

└── etc

└── indexer.xml

Inventory

Inventory index (MSI)

Indexer Registration
15

View Slide

class \Magento\Indexer\Model\Indexer\DependencyDecorator implements \Magento\Framework\Indexer\IndexerInterface

{

// ...

public function reindexRow($id)

{

$this->cacheCleaner->start();

$this->indexer->reindexRow($id);

$dependentIndexerIds = $this->dependencyInfoProvider->getIndexerIdsToRunAfter($this->indexer->getId());

foreach ($dependentIndexerIds as $indexerId) {

$dependentIndexer = $this->indexerRegistry->get($indexerId);

if (!$dependentIndexer->isScheduled()) {

$dependentIndexer->reindexRow($id);

}

}

$this->cacheCleaner->flush();

}

}

Default implementation defined in di.xml of Magento_Indexer module.
Dependency "update on save"
16

View Slide

\Magento\Indexer\Console\Command\IndexerReindexCommand::getIndexers
protected function getIndexers(InputInterface $input)
{
$indexers = parent::getIndexers($input);
$allIndexers = $this->getAllIndexers();
if (!array_diff_key($allIndexers, $indexers)) {
return $indexers;
}
$relatedIndexers = [];

$dependentIndexers = [];

foreach ($indexers as $indexer) {
$relatedIndexers[] = $this->getRelatedIndexerIds($indexer->getId());

$dependentIndexers[] = $this->getDependentIndexerIds($indexer->getId());

}
$relatedIndexers = array_unique(array_merge([], ...$relatedIndexers));

$dependentIndexers = array_merge([], ...$dependentIndexers);

$invalidRelatedIndexers = [];

foreach ($relatedIndexers as $relatedIndexer) {

if ($allIndexers[$relatedIndexer]->isInvalid()) {
$invalidRelatedIndexers[] = $relatedIndexer;

}
}
return array_intersect_key(

$allIndexers,
array_flip(
array_unique(
array_merge(

array_keys($indexers),

$invalidRelatedIndexers,

$dependentIndexers

)

)
)
);
}
Dependency index:reindex command (Hip hip array)
17

View Slide

class MyIndexer implements

\Magento\Framework\Indexer\ActionInterface,

\Magento\Framework\Mview\ActionInterface

{

// ...
}

Logical separation:
\Magento\Framework\Indexer\ActionInterface : Update on save / adhoc

\Magento\Framework\Mview\ActionInterface : Update on schedule
Indexer Class
18

View Slide

interface ActionInterface

{

/**

* Execute full indexation

*

* @return void

*/

public function executeFull();

/**

* Execute partial indexation by ID list

*

* @param int[] $ids

* @return void

*/

public function executeList(array $ids);

/**

* Execute partial indexation by ID

*

* @param int $id

* @return void

*/

public function executeRow($id);

}

\Magento\Framework\Indexer\ActionInterface
19

View Slide

Called by scheduled indexer job (via cronjob)
interface ActionInterface

{

/**

* Execute materialization on ids entities

*

* @param int[] $ids

* @return void

* @api

*/

public function execute($ids);

}

\Magento\Framework\Mview\ActionInterface
20

View Slide

How update on schedule works
21

View Slide

In computing, a materialized view is a database object that contains the results of
a query. For example, it may be a local copy of data located remotely, or may be a
subset of the rows and/or columns of a table or join result, or may be a summary
using an aggregate function.
-- Wikipedia
MView Definition
22

View Slide

© https://oracle-base.com/articles/misc/materialized-views
Materialized Views in a Oracle Database
23

View Slide

.

└── MyModule

└── etc

└── mview.xml

xsi:noNamespaceSchemaLocation="urn:magento:framework:Mview/etc/mview.xsd">

id="catalog_product_price"

class="Magento\Catalog\Model\Indexer\Product\Price"

group="indexer">

MView Definition -> Trigger Config
24

View Slide

Status: working, valid, invalid
Indexer State in the database
25

View Slide

When will triggers be created?
26

View Slide

Insert:
BEGIN
INSERT IGNORE INTO `catalog_category_product_cl` (èntity_id`) VALUES (NEW.èntity_id`);

END
Update:
BEGIN
IF (NEW.èntity_id` <=> OLD.èntity_id`

OR NEW.àttribute_set_id` <=> OLD.àttribute_set_id`

OR NEW.`parent_id` <=> OLD.`parent_id`

OR NEW.`created_at` <=> OLD.`created_at`

OR NEW.`path` <=> OLD.`path`
OR NEW.`position` <=> OLD.`position`

OR NEW.`level` <=> OLD.`level`
OR NEW.`children_count` <=> OLD.`children_count`)

THEN INSERT IGNORE INTO `catalog_category_product_cl` (èntity_id`) VALUES (NEW.èntity_id`);

END IF;
END
Delete:
BEGIN
INSERT IGNORE INTO `catalog_category_product_cl` (èntity_id`) VALUES (OLD.èntity_id`);

END
How triggers look like
27

View Slide

Changelog Tables
28

View Slide

Version-ID (last id in changelog table)
status -> idle, working, suspended
Up-to-date: version_id == last version_id in _CL table
MView Status
29

View Slide

php bin/magento cron:run

# only indexer jobs

php bin/magento cron:run --group index

Indexer Cron
30

View Slide

Job Description Frequency
indexer_clean_all_changelogs Cleanup old entries in _CL tables. daily
indexer_reindex_all_invalid Start full reindex of broken indexers every minute
indexer_update_all_views Executes "update on schedule" every minute
Indexer Cronjobs
31

View Slide

Update on scheduled processing
32

View Slide

Indexer batching (Finetuning, depending on data)
Configure batch size via ENV variables or in env.php
Indexer table switching (for full reindex)
https://developer.adobe.com/commerce/php/development/components/indexing/optimiz
ation/
Optimizations
33

View Slide

The status values in the indexer_state or mview_state database tables may not be the same as what is observed,
because they sometimes do not get updated when an indexer fails.

https://developer.adobe.com/commerce/php/development/components/indexing/#using-application-lock-mode-for-
reindex-processes
more accurate status by using a
\Magento\Framework\Lock\LockManagerInterface implementation

(DB, Zookeeper, FileLock, ...).
cronjob will retry to index without a reset of the indexer
env.php
'indexer' => [
'use_application_lock' => true

]

Don't trust the rabbit
34

View Slide

https://twitter.com/ospadano/status/1570859136379924482?
t=R9Gp9EkBz1eYiYw05fgFFQ
Can Indexer Cronjobs conflict?
35

View Slide

Lock

Manager is configured correct
IMHO no ... if
36

View Slide

times_used

https://github.com/magento/magento2/issues/30243
Ignore columns for mview to be specified at the
subscription level
37

View Slide

If triggers are missing (can be possible if DB dump was exported without triggers) ...
n98-magerun2.phar index:trigger:recreate

Force trigger recreation
38

View Slide

Update On Save
39

View Slide

Must be handled manually
create, update, delete
Different kind of implementations in Magento
Commit Callbacks in Resource Models (own implementation)
Plugins (modify an entity which is not fully controlled by your module)
Events/Observer (process/entity dependency -> Trigger indexer of another
entity)
Most the time we extend indexer and do not require to handle the re-indexing.
Test if indexer processes the data after a modification (e.g mview.xml)
Test in Magento Admin if there is a UI
Test changes via webapi
Update On Save
40

View Slide

\Magento\Catalog\Model\Product::afterSave
public function afterSave()
{
$this->getLinkInstance()->saveProductRelations($this);

$this->getTypeInstance()->save($this);
if ($this->getStockData()) {
$this->setForceReindexEavRequired(true);

}
$this->_getResource()->addCommitCallback([$this, 'priceReindexCallback']);

$this->_getResource()->addCommitCallback([$this, 'eavReindexCallback']);

$result = parent::afterSave();
$this->_getResource()->addCommitCallback([$this, 'reindex']);

$this->reloadPriceInfo();
return $result;
}
/**
* Reindex callback for EAV indexer

*
* @return void
*/
public function eavReindexCallback()

{
if ($this->isObjectNew() || $this->isDataChanged()) {
$this->_productEavIndexerProcessor->reindexRow($this->getEntityId());

}
}
Example: Commit Callback / Resource Model
41

View Slide

Magento_CatalogRule module di.xml

name="apply_catalog_rules_after_product_save_and_reindex" type="Magento\CatalogRule\Plugin\Indexer\Product\Save\ApplyRulesAfterReindex"/>

class ApplyRulesAfterReindex

{
/**
* @param ProductRuleProcessor $productRuleProcessor

*/
public function __construct(ProductRuleProcessor $productRuleProcessor)

{
$this->productRuleProcessor = $productRuleProcessor;

}
// ...
public function afterReindex(Product $subject)
{
$this->productRuleProcessor->reindexRow($subject->getId());

}
}
Example: Plugin
42

View Slide

vendor/magento/module-catalog-inventory/etc/events.xml

Example 3: Event/Observer
43

View Slide

public function execute(EventObserver $observer)

{

// Reindex quote ids

$quote = $observer->getEvent()->getQuote();

$productIds = [];

foreach ($quote->getAllItems() as $item) {

$productIds[$item->getProductId()] = $item->getProductId();

$children = $item->getChildrenItems();

if ($children) {

foreach ($children as $childItem) {

$productIds[$childItem->getProductId()] = $childItem->getProductId();

}

}

}

if ($productIds) {

$this->stockIndexerProcessor->reindexList($productIds); // <-- CALL INDEXER 1

}

// Reindex previously remembered items

$productIds = [];

foreach ($this->itemsForReindex->getItems() as $item) {

$item->save();

$productIds[] = $item->getProductId();

}

if (!empty($productIds)) {

$this->priceIndexer->reindexList($productIds); // <-- CALL INDEXER 2

}

$this->itemsForReindex->clear();

// Clear list of remembered items - we don't need it anymore

}

44

View Slide

Cache Cleaning
45

View Slide

vendor/magento/module-indexer/etc/di.xml

Plugins in Magento_Indexer module
46

View Slide

class CacheCleaner

{

/**

* Defer cache cleaning until after execute full

*

* @param ActionInterface $subject

* @return void

* @SuppressWarnings(PHPMD.UnusedFormalParameter)

*/

public function beforeExecuteFull(ActionInterface $subject)

{


}

/**

* Clean cache after full reindex full

*


* @return void


*/

public function afterExecuteFull(ActionInterface $subject)

{


}

/**

* Defer cache cleaning until after execute list

*


* @return void


*/

public function beforeExecuteList(ActionInterface $subject)

{


}

/**

* Clean cache after reindexed list.

*


* @return void


*/

public function afterExecuteList(ActionInterface $subject)

{


}

/**

* Defer cache cleaning until after execute row

*


* @return void


*/

public function beforeExecuteRow(ActionInterface $subject)

{


}

/**

* Clean cache after reindexed row.

*


* @return void


*/

public function afterExecuteRow(ActionInterface $subject)

{


}

}

47

View Slide

class DeferredCacheCleaner

{

// ...

/**

* @param EventManager $eventManager

* @param CacheInterface $appCache

* @param DeferredCacheContext $deferredCacheContext

* @param CacheContext $cacheContext

*/

public function __construct(

EventManager $eventManager,

CacheInterface $appCache,

DeferredCacheContext $deferredCacheContext,

CacheContext $cacheContext

) {

$this->eventManager = $eventManager;

$this->deferredCacheContext = $deferredCacheContext;

$this->appCache = $appCache;

$this->cacheContext = $cacheContext;

}

/**

* Defer cache cleaning until flush() is called

*

* @see flush()

*/

public function start(): void

{

$this->deferredCacheContext->start();

}

/**

* Flush cache

*/

public function flush(): void

{

$this->deferredCacheContext->commit();

$this->eventManager->dispatch('clean_cache_by_tags', ['object' => $this->cacheContext]);

$identities = $this->cacheContext->getIdentities();

if (!empty($identities)) {

$this->appCache->clean($identities);

$this->cacheContext->flush();

}

}

}

48

View Slide

vendor/magento/module-cache-invalidate/etc/events.xml

vendor/magento/module-page-cache/etc/events.xml

clean_cache_by_tags -> FPC Invalidation
49

View Slide

vendor/magento/module-customer/etc/indexer.xml

xsi:noNamespaceSchemaLocation="urn:magento:framework:Indexer/etc/indexer.xsd">

provider="Magento\Customer\Model\Indexer\AttributeProvider">

"Structure" Handling / Save Handler
50

View Slide

https://experienceleague.adobe.com/docs/commerce-operations/configuration-
guide/cli/manage-indexers.html
MView Implementation: vendor/magento/framework/Mview
https://amasty.com/blog/comprehensive-guide-to-magento-2-indexing/
https://developer.adobe.com/commerce/php/development/components/indexing/
Performance:
https://developer.adobe.com/commerce/php/development/components/indexing/op
timization/
Resources
51

View Slide

52

View Slide

Photos provided by
Canva
Tobias Fischer
53

View Slide

Magento 2 Indexer

Magento 2 Indexer

Christian Münch

More Decks by Christian Münch

Other Decks in Technology

Featured

Transcript