Solr et recherche dans un site ecommerce @AdrienBrault

Recherche dans un site ecommerce

No content

No content

No content

Suggestions

Pertinence

Filtres

No content

Standalone enterprise search server • Projet Apache (Comme Hadoop/CouchDB/Subversion) • Fonctions de recherche text avancées • Optimisé pour sites à fort traffic • API “REST” au format XML/JSON/CSV • Administration web • Replication et Sharding (SolrCloud) • “Near Real-time indexing”

Pourquoi pas MySQL/ PostgreSQL ? • Performances • Facets • Fonctionnalités FULL Text • Support entreprise pour de la recherche

Solr vs ElasticSearch • Tous les deux basés sur Lucene • ElasticSearch est plus récent et mieux pensé / plus “developer friendly” • Solr peut se resumer a “une interface HTTP simple a Lucene” • Avec Solr tout ne fonctionne pas avec le Sharding • solr-vs-elasticsearch.com

Pourquoi pas ElasticSearch ?

PRODUCT SKU SKU MySQL/ PostgreSQL Solr/ ElasticSearch 1 1..n Indexation/ Dénormalisation

SI pas de grouping = Mauvaise experience 46 42 42 44 46 40

Grouping = Client heureux

ES parent/child ? • Certaines requêtes impossibles • Mais probablement plus performant qu’un simple grouping

Hautelook et Solr … qui guidera en partie la suite

Type événement avec MySQL

Type catalog … avec Solr

Main Inventory Popularity CRON MySQL Triggers Déclencheur Workers/ Indexation Solr DOC DOC DOC

Différents workers • Inventory • Beaucoup de message … doit être rapide • Popularity • Donnée calculée, donc chargée périodiquement • Main • Il faut bien quelque chose pour le reste!

La suite n’est pas un tutoriel … Mais ce qui vous ferait gagner du temps

Installation

# Get latest “Download” link from # https://lucene.apache.org/solr 
 wget http://mir2.ovh.net/ ftp.apache.org/dist/lucene/solr/ 4.7.1/solr-4.7.1.tgz 
 tar -xf solr-4.7.1.tgz
 cd solr-4.7.1/example 
 java -jar start.jar

No content

No content

$JETTY_HOME

Exemples à supprimer

Solr … est dans le solr.war

$SOLR_HOME Core Ce que vous versionnerez avec git

Fichiers de config important


 
 
 
 
 
 
 
 
 
 
 
 
 sku
 
 schema.xml


 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 schema.xml


 
 ${solr.autoCommit.maxTime:15000}
 ${solr.autoCommit.maxDocs:100000}
 true
 
 
 
 ${solr.autoSoftCommit.maxTime:-1}
 
 
 
 
 
 ${enable.master:false}
 commit
 startup
 solrconfig.xml,schema.xml
 
 
 ${enable.slave:false}
 ${solr.slave.masterUrl:}
 00:00:30
 
 solrconfig.xml


 
 10
 
 edismax
 name t_category^50 t_color^40 t_size
 sum(sqrt(popularity),if(isFeatured,5,0))
 0.1
 100%
 
 true
 ...
 10
 
 
 
 spellcheck
 
 solrconfig.xml

Packages PHP pour solr

solarium/solarium • API complete pour interagir avec solr • QueryBuilder • Batch Updates • http://wiki.solarium-project.org/index.php/ V3:Basic_usage

internations/solr-query-component use InterNations\Component\Solr\Expression \ExpressionBuilder;
 
 $eb = new ExpressionBuilder();
 echo $eb->field(
 'name',
 $eb->boost(
 $eb->eq('John Doe'),
 100 )
 );
 
 // name:"John Doe"^100 • https://github.com/InterNations/SolrQueryComponent

nelmio/solarium-bundle

hautelook/solarium-cache $client = ...;
 $cache = new RedisCache(); // or any doctrine cache adapter
 
 $plugin = new CachePlugin();
 $plugin->setCache($cache);
 $client->registerPlugin('cache', $plugin);
 
 $query = $client->createSelect(array(
 'cache_lifetime' => 60,
 ));
 $result = $client->execute($query); • Bundle inclus

floriansemm/solr-bundle • Principe similaire a friendsofsymfony/elastica- bundle • Trop simple pour notre cas cependant • synchrone • indexation trop simple

Implementation dans Symfony2

Objectifs de l’architecture • IndexerBundle • Asynchrone • ETL facilement testable • ClientBundle • Utilisation web et api possible • API “publique” découplée de Solr/Solarium

Acme\Search\IndexerBundle

No content


 namespace Acme\Search\IndexerBundle\ETL\Inventory;
 
 class Extractor
 {
 public function getSkuQuantity($sku)
 {
 $qb = $this->createQB();
 
 $qb
 ->andWhere('sku.sku = :sku')
 ->setParameter('sku', $sku)
 ;
 
 return $qb->execute()->fetch();
 }
 
 public function getProductSkusQuantities($productId)
 {
 $qb = $this->createQB();
 
 $qb
 ->andWhere('sku.product = :product_id')
 ->setParameter('product_id', $productId)
 ;
 
 return $qb->execute()->fetchAll();
 }
 
 private function createQB() {}
 }

namespace Acme\Search\IndexerBundle\ETL\Inventory;
 
 class Transformer
 {
 public function updateDocument(
 Document $document,
 $sku,
 $quantity
 ) {
 $document->setSku($sku); // PK
 $document->setQuantity($quantity);
 }
 }

namespace Acme\Search\IndexerBundle\ETL;
 
 class Loader
 {
 public function updateSkuQuantity($sku)
 {
 return $this->updateQuantities(
 array($this->inventoryExtractor->getSkuQuantity($sku))
 );
 }
 
 public function updateProductSkusQuantities($productId)
 {
 return $this->updateQuantities(
 $this->inventoryExtractor->getProductSkusQuantities($productId)
 );
 }
 
 private function updateQuantities(array $extractorResults)
 {
 $documents = array();
 foreach ($extractorResults as $extractorResult) {
 $documents[] = $document = new Document();
 $sku = $extractorResult['sku'];
 $quantity = $extractorResult['quantity'];
 
 $this->itemTransformer->updateDocument($document, $sku, $quantity);
 }
 
 return $this->loaderClient->sendDocuments($documents);
 }
 }


 namespace Acme\Search\IndexerBundle\ETL;
 
 class LoaderClient
 {
 private $solarium;
 private $commitOnUpdate;
 
 public function sendDocuments(array $documents)
 {
 $update = $this->solarium->createUpdate();
 $update->addDocuments($documents);
 
 return $this->sendUpdate($update);
 }
 
 private function sendUpdate(Query $update)
 {
 if ($this->commitOnUpdate) {
 $update->addCommit();
 }
 
 $result = $this->solarium->update($update);
 
 if ($result->getStatus() !== 0) {
 throw new UpdateErrorException($result);
 }
 
 return $result;
 }
 }

Vendor\Search\ClientBundle

Filtre namespace Acme\Search\ClientBundle\Filter; 
 class Filter
 {
 private $query;
 private $colors;
 private $sizes;
 private $page;
 private $limit;
 
 public function setQuery($query = null) {}
 public function getQuery() {}
 
 public function setColors(array $colors) {}
 public function getColors() {}
 
 // ...
 }

Abstraire le filtre namespace Acme\Search\ClientBundle\Filter;
 
 class Filter
 {
 // ...
 
 const SORT_RELEVANCY = 'relevancy';
 const SORT_MOST_POPULAR = 'most_popular';
 const SORT_PRICE_DESC = 'price_desc';
 
 private $sort = self::SORT_MOST_POPULAR;
 
 public function getSort() {}
 public function setSort($sort) {}
 }

Client pour le Catalogue namespace Acme\Search\ClientBundle\Catalog; " class Client
 {
 /**
 * @param Filter $filter
 * @return Result
 */
 public function search(Filter $filter)
 {
 $query = $this->queryFactory->createQuery($filter);
 $result = $this->client->execute($query);
 
 return $this->resultBuilder->create($result);
 }
 }

QueryFactory namespace Acme\Search\ClientBundle\Solarium\Select;
 
 class QueryFactory
 {
 public function createCatalogQuery(Filter $filter)
 {
 $query = new CatalogQuery();
 $query->setQuery((new ExpressionBuilder())->all());
 
 if ($filter->hasFullTextQuery()) {
 $query->setQuery($filter->getQuery());
 }
 
 $query->addFilterColorFilterQuery($filter);
 $query->addFilterSizeFilterQuery($filter);
 
 return $query;
 }
 }

Query namespace Acme\Search\ClientBundle\Solarium\Select;
 
 class CatalogQuery extends Query
 {
 protected function init()
 {
 parent::init();
 
 $this
 ->createFilterQuery(Document::FIELD_QUANTITY)
 ->setQuery(
 $this->expressionBuilder->field(
 Document::FIELD_QUANTITY,
 $this->getExpressionBuilder()->btwnRange(0, null)
 )
 )
 ;
 }
 }

namespace Acme\Search\ClientBundle\Solarium\Select;
 
 class CatalogQuery extends Query
 {
 public function addFilterColorFilterQuery(Filter $filter)
 {
 $this->addFieldTermsFilterQuery(
 Document::FIELD_COLOR,
 $filter->getColors()
 );
 }
 
 public function addFieldTermsFilterQuery($field, $terms)
 {
 if (count($terms) === 0) {
 return;
 }
 
 $this
 ->createFilterQuery($field)
 ->setQuery(
 $this->getExpressionBuilder()->field(
 $field,
 $this->getExpressionBuilder()->grp($terms)
 )
 )
 ;
 }
 }

Resultat 
 namespace Acme\Search\ClientBundle\Catalog;
 
 class Result implements \IteratorAggregate
 {
 /** @var Pagerfanta */
 private $pager;
 
 /** @var Product[] */
 private $products;
 
 /** @var FieldCount[] */
 private $colors;
 
 /** @var FieldCount[] */
 private $sizes;
 
 /** @var null|string */
 private $suggestedSpelling;
 }

Utilisation web + api

public function indexAction(Request $request)
 {
 $filter = new Filter();
 
 $bindingForm = $this->get('form.factory')->createNamed(
 '',
 'filter_binding',
 $filter
 );
 $bindingForm->submit($request->query->all(), false);
 
 $catalogClient = $this->get('acme_search_client.catalog.client');
 $result = $catalogClient->search($filter);
 
 $viewForm = $this->get('form.factory')->createNamed(
 '',
 'filter',
 $filter,
 [
 'method' => 'GET',
 'catalog_result' => $result,
 ]
 );
 
 return array(
 'filter' => $filter,
 'form' => $viewForm->createView(),
 'result' => $result,
 );
 }

/**
 * @Method("GET")
 * @Route("/search")
 */
 public function searchAction(Request $request)
 {
 $filter = new Filter();
 
 $bindingForm = $this->get('form.factory')->createNamed(
 '',
 'filter_binding',
 $filter
 );
 $bindingForm->submit($request->query->all(), false);
 
 $catalogClient = $this->get('acme_search_client.catalog.client');
 $result = $catalogClient->search($filter);
 
 // Hateoas goodness!
 $catalogRepresentation = new CatalogRepresentation($result, $filter);
 
 return View::create($catalogRepresentation);
 }

namespace Acme\AppBundle\Form\Type;
 
 class FilterBindingType extends AbstractType
 {
 public function buildForm(FormBuilderInterface $builder, array $options)
 {
 $builder
 ->add('query', 'hidden')
 ->add('sizes', 'collection', ['allow_add' => true])
 ->add('colors', 'collection', ['allow_add' => true])
 ->add('sort', 'hidden')
 ->add('page', 'hidden')
 ->add('limit', 'hidden')
 ;
 
 $builder->addEventListener(
 FormEvents::PRE_SUBMIT,
 function (FormEvent $event) {
 $submittedData = $event->getData();
 
 if (isset($submittedData['query'])
 && !isset($submittedData['sort'])
 ) {
 $submittedData['sort'] = Filter::SORT_RELEVANCY;
 }
 
 $event->setData($submittedData);
 }
 );
 }
 }

namespace Acme\AppBundle\Form\Type;
 
 class FilterType extends AbstractType
 {
 public function buildForm(
 FormBuilderInterface $builder,
 array $options
 ) {
 
 }
 
 public function setDefaultOptions(OptionsResolverInterface $resolver)
 {
 $resolver->setDefaults([
 'data_class' => Filter::CLASS,
 'csrf_protection' => false,
 ]);
 $resolver->setRequired([
 'catalog_result'
 ]);
 $resolver->setAllowedTypes([
 'catalog_result' => Result::CLASS,
 ]);
 }
 }

public function buildForm(
 FormBuilderInterface $builder,
 array $options
 ) {
 $builder
 ->add(
 'sizes',
 'choice',
 [
 'choice_list' => new FieldCountChoiceList(
 $options['catalog_result']->getSizes()
 ),
 'required' => false,
 ]
 )
 ->add(
 'colors',
 'choice',
 [
 'choice_list' => new FieldCountChoiceList(
 $options['catalog_result']->getColors()
 ),
 'required' => false,
 ]
 )
 ->add(
 'sort',
 'choice',
 [
 'choices' => self::getSortChoices(),
 ]
 )
 ;
 }

Travailler sur la pertinence

use Solarium\QueryType\Select\Query\Component\EdisMax; " class EDisMaxType extends AbstractType
 {
 public function buildForm(FormBuilderInterface $builder, array $options)
 {
 $options = ['required' => false];
 $builder
 ->add('queryFields', 'textarea', $options)
 ->add('queryAlternative', 'text', $options)
 ->add('minimumMatch', 'text', $options)
 ->add('phraseFields', 'text', $options)
 ->add('phraseSlop', 'text', $options)
 ->add('queryPhraseSlop', 'text', $options)
 ->add('tie', 'text', $options)
 ->add('boostQuery', 'text', $options)
 ->add('boostFunctions', 'text', $options)
 ->add('boostFunctionsMult', 'text', $options)
 ->add('phraseBigramFields', 'text', $options)
 ->add('phraseBigramSlop', 'text', $options)
 ->add('phraseTrigramFields', 'text', $options)
 ->add('phraseTrigramSlop', 'text', $options)
 ->add('userFields', 'text', $options)
 ;
 }
 
 public function setDefaultOptions(OptionsResolverInterface $resolver)
 {
 $resolver->setDefaults([
 'data_class' => EDisMax::CLASS,
 ]);
 }
 }

DebugController 
 $query1 = $catalogClient->createQuery($filter);
 $query2 = $catalogClient->createQuery($filter);
 
 $debugForm = $this->get('form.factory')
 ->createBuilder(
 'form',
 [
 'left' => $query1->getEDisMax(),
 'right' => $query2->getEDisMax(),
 'filter' => $filter,
 ],
 ['method' => 'GET']
 )
 ->add('query', 'text', ['property_path' => '[filter].query'])
 ->add('left', 'edismax')
 ->add('right', 'edismax')
 ->getForm()
 ;
 $debugForm->submit($request->query->all(), false);
 
 $catalogResult1 = $catalogClient->createSearchResults($filter, $query1);
 $catalogResult2 = $catalogClient->createSearchResults($filter, $query2);

No content

Retours d’experience

Suggestions • Suggester suggèste des noms … et non des recherches • Nous envisageons de créer un second core pour les suggestions

Performances

No content

Performance “killers” • group=true / group.ngroups=true / group.facet=true • Form choice avec 2000 choix … -> AJAX • AutoCommit trop frequent • Trop de slaves • Ne pas utiliser solr cloud -> hard commit obligatoire pour la replication

Questions ? • Merci! • speakerdeck.com/adrienbrault/solr-et-recherche- dans-un-site-ecommerce • @AdrienBrault sur Twitter/Github