Solr et recherche dans un site ecommerce

Transcript

1. Solr et recherche dans un site ecommerce @AdrienBrault
2. Recherche dans un site ecommerce
6. Suggestions
7. Pertinence
8. Filtres
10. Standalone enterprise search server • Projet Apache (Comme Hadoop/CouchDB/Subversion) • Fonctions de recherche text avancées • Optimisé pour sites à fort traffic • API “REST” au format XML/JSON/CSV • Administration web • Replication et Sharding (SolrCloud) • “Near Real-time indexing”
11. Pourquoi pas MySQL/ PostgreSQL ? • Performances • Facets • Fonctionnalités FULL Text • Support entreprise pour de la recherche
12. Solr vs ElasticSearch • Tous les deux basés sur Lucene • ElasticSearch est plus récent et mieux pensé / plus “developer friendly” • Solr peut se resumer a “une interface HTTP simple a Lucene” • Avec Solr tout ne fonctionne pas avec le Sharding • solr-vs-elasticsearch.com
13. Pourquoi pas ElasticSearch ?
14. PRODUCT SKU SKU MySQL/ PostgreSQL Solr/ ElasticSearch 1 1..n Indexation/ Dénormalisation
15. SI pas de grouping = Mauvaise experience 46 42 42 44 46 40
16. Grouping = Client heureux
17. ES parent/child ? • Certaines requêtes impossibles • Mais probablement plus performant qu’un simple grouping
18. Hautelook et Solr … qui guidera en partie la suite
19. Type événement avec MySQL
20. Type catalog … avec Solr
21. Main Inventory Popularity CRON MySQL Triggers Déclencheur Workers/ Indexation Solr DOC DOC DOC
22. Différents workers • Inventory • Beaucoup de message … doit être rapide • Popularity • Donnée calculée, donc chargée périodiquement • Main • Il faut bien quelque chose pour le reste!
23. La suite n’est pas un tutoriel … Mais ce qui vous ferait gagner du temps
24. Installation
25. # Get latest “Download” link from # https://lucene.apache.org/solr 
 wget http://mir2.ovh.net/ ftp.apache.org/dist/lucene/solr/ 4.7.1/solr-4.7.1.tgz 
 tar -xf solr-4.7.1.tgz
 cd solr-4.7.1/example 
 java -jar start.jar
28. $JETTY_HOME
29. Exemples à supprimer
30. Solr … est dans le solr.war
31. $SOLR_HOME Core Ce que vous versionnerez avec git
32. Fichiers de config important
33. <fields>
 <field name="sku" type="text_en_splitting_tight" indexed="true" stored="true" omitNorms="true"/>
 
 <field name="name" type="text_general" indexed="true" stored="true"/>
 <field name="color" type="string" indexed="true" stored="true"/>
 
 <field name="price" type="float" indexed="true" stored="true"/>
 <field name="popularity" type="int" indexed="true" stored="true"/>
 <field name="inStock" type="boolean" indexed="true" stored=“true"/> <field name="isFeatured" type="boolean" indexed="true" stored="true"/>
 
 <dynamicField name=“t_*” type="text_general" indexed="true" stored="false"/>
 </fields>
 
 <uniqueKey>sku</uniqueKey>
 
 <copyField source="color" dest="t_color"/> schema.xml
34. <types>
 <fieldType name="string" class="solr.StrField" sortMissingLast="true"/>
 <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true"/>
 
 <fieldType name="text_general" 
 lass="solr.TextField"
 positionIncrementGap="100">
 <analyzer type="index">
 <tokenizer class="solr.StandardTokenizerFactory"/>
 <filter class="solr.StopFilterFactory"
 ignoreCase="true"
 words="stopwords.txt"/>
 <filter class="solr.LowerCaseFilterFactory"/>
 </analyzer> 
 <analyzer type="query">
 <tokenizer class="solr.StandardTokenizerFactory"/>
 <filter class="solr.StopFilterFactory"
 ignoreCase="true"
 words="stopwords.txt"/>
 <filter class="solr.SynonymFilterFactory"
 synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
 <filter class="solr.LowerCaseFilterFactory"/>
 </analyzer>
 </fieldType>
 </types> schema.xml
35. <updateHandler class="solr.DirectUpdateHandler2">
 <autoCommit>
 <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
 <maxDocs>${solr.autoCommit.maxDocs:100000}</maxDocs>
 <openSearcher>true</openSearcher>
 </autoCommit>
 
 <autoSoftCommit>
 <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
 </autoSoftCommit>
 </updateHandler>
 
 <requestHandler name="/replication" class="solr.ReplicationHandler">
 <lst name="master">
 <str name="enable">${enable.master:false}</str>
 <str name="replicateAfter">commit</str>
 <str name="replicateAfter">startup</str>
 <str name="confFiles">solrconfig.xml,schema.xml</str>
 </lst>
 <lst name="slave">
 <str name="enable">${enable.slave:false}</str>
 <str name="masterUrl">${solr.slave.masterUrl:}</str>
 <str name="pollInterval">00:00:30</str>
 </lst>
 </requestHandler> solrconfig.xml
36. <requestHandler name="/select" class="solr.SearchHandler">
 <lst name="defaults">
 <int name="rows">10</int>
 
 <str name="defType">edismax</str>
 <str name="qf">name t_category^50 t_color^40 t_size</str>
 <str name="bf">sum(sqrt(popularity),if(isFeatured,5,0))</str>
 <str name="tie">0.1</str>
 <str name="mm">100%</str>
 
 <str name="spellcheck">true</str>
 ...
 <str name="spellcheck.count">10</str>
 </lst>
 
 <arr name="last-components">
 <str>spellcheck</str>
 </arr>
 </requestHandler> solrconfig.xml
37. Packages PHP pour solr
38. solarium/solarium • API complete pour interagir avec solr • QueryBuilder • Batch Updates • http://wiki.solarium-project.org/index.php/ V3:Basic_usage
39. internations/solr-query-component use InterNations\Component\Solr\Expression \ExpressionBuilder;
 
 $eb = new ExpressionBuilder();
 echo $eb->field(
 'name',
 $eb->boost(
 $eb->eq('John Doe'),
 100 )
 );
 
 // name:"John Doe"^100 • https://github.com/InterNations/SolrQueryComponent
40. nelmio/solarium-bundle
41. hautelook/solarium-cache $client = ...;
 $cache = new RedisCache(); // or any doctrine cache adapter
 
 $plugin = new CachePlugin();
 $plugin->setCache($cache);
 $client->registerPlugin('cache', $plugin);
 
 $query = $client->createSelect(array(
 'cache_lifetime' => 60,
 ));
 $result = $client->execute($query); • Bundle inclus
42. floriansemm/solr-bundle • Principe similaire a friendsofsymfony/elastica- bundle • Trop simple pour notre cas cependant • synchrone • indexation trop simple
43. Implementation dans Symfony2
44. Objectifs de l’architecture • IndexerBundle • Asynchrone • ETL facilement testable • ClientBundle • Utilisation web et api possible • API “publique” découplée de Solr/Solarium
45. Acme\Search\IndexerBundle
47. 
 namespace Acme\Search\IndexerBundle\ETL\Inventory;
 
 class Extractor
 {
 public function getSkuQuantity($sku)
 {
 $qb = $this->createQB();
 
 $qb
 ->andWhere('sku.sku = :sku')
 ->setParameter('sku', $sku)
 ;
 
 return $qb->execute()->fetch();
 }
 
 public function getProductSkusQuantities($productId)
 {
 $qb = $this->createQB();
 
 $qb
 ->andWhere('sku.product = :product_id')
 ->setParameter('product_id', $productId)
 ;
 
 return $qb->execute()->fetchAll();
 }
 
 private function createQB() {}
 }
48. namespace Acme\Search\IndexerBundle\ETL\Inventory;
 
 class Transformer
 {
 public function updateDocument(
 Document $document,
 $sku,
 $quantity
 ) {
 $document->setSku($sku); // PK
 $document->setQuantity($quantity);
 }
 }
49. namespace Acme\Search\IndexerBundle\ETL;
 
 class Loader
 {
 public function updateSkuQuantity($sku)
 {
 return $this->updateQuantities(
 array($this->inventoryExtractor->getSkuQuantity($sku))
 );
 }
 
 public function updateProductSkusQuantities($productId)
 {
 return $this->updateQuantities(
 $this->inventoryExtractor->getProductSkusQuantities($productId)
 );
 }
 
 private function updateQuantities(array $extractorResults)
 {
 $documents = array();
 foreach ($extractorResults as $extractorResult) {
 $documents[] = $document = new Document();
 $sku = $extractorResult['sku'];
 $quantity = $extractorResult['quantity'];
 
 $this->itemTransformer->updateDocument($document, $sku, $quantity);
 }
 
 return $this->loaderClient->sendDocuments($documents);
 }
 }
50. 
 namespace Acme\Search\IndexerBundle\ETL;
 
 class LoaderClient
 {
 private $solarium;
 private $commitOnUpdate;
 
 public function sendDocuments(array $documents)
 {
 $update = $this->solarium->createUpdate();
 $update->addDocuments($documents);
 
 return $this->sendUpdate($update);
 }
 
 private function sendUpdate(Query $update)
 {
 if ($this->commitOnUpdate) {
 $update->addCommit();
 }
 
 $result = $this->solarium->update($update);
 
 if ($result->getStatus() !== 0) {
 throw new UpdateErrorException($result);
 }
 
 return $result;
 }
 }
51. Vendor\Search\ClientBundle
52. Filtre namespace Acme\Search\ClientBundle\Filter; 
 class Filter
 {
 private $query;
 private $colors;
 private $sizes;
 private $page;
 private $limit;
 
 public function setQuery($query = null) {}
 public function getQuery() {}
 
 public function setColors(array $colors) {}
 public function getColors() {}
 
 // ...
 }
53. Abstraire le filtre namespace Acme\Search\ClientBundle\Filter;
 
 class Filter
 {
 // ...
 
 const SORT_RELEVANCY = 'relevancy';
 const SORT_MOST_POPULAR = 'most_popular';
 const SORT_PRICE_DESC = 'price_desc';
 
 private $sort = self::SORT_MOST_POPULAR;
 
 public function getSort() {}
 public function setSort($sort) {}
 }
54. Client pour le Catalogue namespace Acme\Search\ClientBundle\Catalog; " class Client
 {
 /**
 * @param Filter $filter
 * @return Result
 */
 public function search(Filter $filter)
 {
 $query = $this->queryFactory->createQuery($filter);
 $result = $this->client->execute($query);
 
 return $this->resultBuilder->create($result);
 }
 }
55. QueryFactory namespace Acme\Search\ClientBundle\Solarium\Select;
 
 class QueryFactory
 {
 public function createCatalogQuery(Filter $filter)
 {
 $query = new CatalogQuery();
 $query->setQuery((new ExpressionBuilder())->all());
 
 if ($filter->hasFullTextQuery()) {
 $query->setQuery($filter->getQuery());
 }
 
 $query->addFilterColorFilterQuery($filter);
 $query->addFilterSizeFilterQuery($filter);
 
 return $query;
 }
 }
56. Query namespace Acme\Search\ClientBundle\Solarium\Select;
 
 class CatalogQuery extends Query
 {
 protected function init()
 {
 parent::init();
 
 $this
 ->createFilterQuery(Document::FIELD_QUANTITY)
 ->setQuery(
 $this->expressionBuilder->field(
 Document::FIELD_QUANTITY,
 $this->getExpressionBuilder()->btwnRange(0, null)
 )
 )
 ;
 }
 }
57. namespace Acme\Search\ClientBundle\Solarium\Select;
 
 class CatalogQuery extends Query
 {
 public function addFilterColorFilterQuery(Filter $filter)
 {
 $this->addFieldTermsFilterQuery(
 Document::FIELD_COLOR,
 $filter->getColors()
 );
 }
 
 public function addFieldTermsFilterQuery($field, $terms)
 {
 if (count($terms) === 0) {
 return;
 }
 
 $this
 ->createFilterQuery($field)
 ->setQuery(
 $this->getExpressionBuilder()->field(
 $field,
 $this->getExpressionBuilder()->grp($terms)
 )
 )
 ;
 }
 }
58. Resultat 
 namespace Acme\Search\ClientBundle\Catalog;
 
 class Result implements \IteratorAggregate
 {
 /** @var Pagerfanta */
 private $pager;
 
 /** @var Product[] */
 private $products;
 
 /** @var FieldCount[] */
 private $colors;
 
 /** @var FieldCount[] */
 private $sizes;
 
 /** @var null|string */
 private $suggestedSpelling;
 }
59. Utilisation web + api
60. public function indexAction(Request $request)
 {
 $filter = new Filter();
 
 $bindingForm = $this->get('form.factory')->createNamed(
 '',
 'filter_binding',
 $filter
 );
 $bindingForm->submit($request->query->all(), false);
 
 $catalogClient = $this->get('acme_search_client.catalog.client');
 $result = $catalogClient->search($filter);
 
 $viewForm = $this->get('form.factory')->createNamed(
 '',
 'filter',
 $filter,
 [
 'method' => 'GET',
 'catalog_result' => $result,
 ]
 );
 
 return array(
 'filter' => $filter,
 'form' => $viewForm->createView(),
 'result' => $result,
 );
 }
61. /**
 * @Method("GET")
 * @Route("/search")
 */
 public function searchAction(Request $request)
 {
 $filter = new Filter();
 
 $bindingForm = $this->get('form.factory')->createNamed(
 '',
 'filter_binding',
 $filter
 );
 $bindingForm->submit($request->query->all(), false);
 
 $catalogClient = $this->get('acme_search_client.catalog.client');
 $result = $catalogClient->search($filter);
 
 // Hateoas goodness!
 $catalogRepresentation = new CatalogRepresentation($result, $filter);
 
 return View::create($catalogRepresentation);
 }
62. namespace Acme\AppBundle\Form\Type;
 
 class FilterBindingType extends AbstractType
 {
 public function buildForm(FormBuilderInterface $builder, array $options)
 {
 $builder
 ->add('query', 'hidden')
 ->add('sizes', 'collection', ['allow_add' => true])
 ->add('colors', 'collection', ['allow_add' => true])
 ->add('sort', 'hidden')
 ->add('page', 'hidden')
 ->add('limit', 'hidden')
 ;
 
 $builder->addEventListener(
 FormEvents::PRE_SUBMIT,
 function (FormEvent $event) {
 $submittedData = $event->getData();
 
 if (isset($submittedData['query'])
 && !isset($submittedData['sort'])
 ) {
 $submittedData['sort'] = Filter::SORT_RELEVANCY;
 }
 
 $event->setData($submittedData);
 }
 );
 }
 }
63. namespace Acme\AppBundle\Form\Type;
 
 class FilterType extends AbstractType
 {
 public function buildForm(
 FormBuilderInterface $builder,
 array $options
 ) {
 
 }
 
 public function setDefaultOptions(OptionsResolverInterface $resolver)
 {
 $resolver->setDefaults([
 'data_class' => Filter::CLASS,
 'csrf_protection' => false,
 ]);
 $resolver->setRequired([
 'catalog_result'
 ]);
 $resolver->setAllowedTypes([
 'catalog_result' => Result::CLASS,
 ]);
 }
 }
64. public function buildForm(
 FormBuilderInterface $builder,
 array $options
 ) {
 $builder
 ->add(
 'sizes',
 'choice',
 [
 'choice_list' => new FieldCountChoiceList(
 $options['catalog_result']->getSizes()
 ),
 'required' => false,
 ]
 )
 ->add(
 'colors',
 'choice',
 [
 'choice_list' => new FieldCountChoiceList(
 $options['catalog_result']->getColors()
 ),
 'required' => false,
 ]
 )
 ->add(
 'sort',
 'choice',
 [
 'choices' => self::getSortChoices(),
 ]
 )
 ;
 }
65. Travailler sur la pertinence
66. use Solarium\QueryType\Select\Query\Component\EdisMax; " class EDisMaxType extends AbstractType
 {
 public function buildForm(FormBuilderInterface $builder, array $options)
 {
 $options = ['required' => false];
 $builder
 ->add('queryFields', 'textarea', $options)
 ->add('queryAlternative', 'text', $options)
 ->add('minimumMatch', 'text', $options)
 ->add('phraseFields', 'text', $options)
 ->add('phraseSlop', 'text', $options)
 ->add('queryPhraseSlop', 'text', $options)
 ->add('tie', 'text', $options)
 ->add('boostQuery', 'text', $options)
 ->add('boostFunctions', 'text', $options)
 ->add('boostFunctionsMult', 'text', $options)
 ->add('phraseBigramFields', 'text', $options)
 ->add('phraseBigramSlop', 'text', $options)
 ->add('phraseTrigramFields', 'text', $options)
 ->add('phraseTrigramSlop', 'text', $options)
 ->add('userFields', 'text', $options)
 ;
 }
 
 public function setDefaultOptions(OptionsResolverInterface $resolver)
 {
 $resolver->setDefaults([
 'data_class' => EDisMax::CLASS,
 ]);
 }
 }
67. DebugController 
 $query1 = $catalogClient->createQuery($filter);
 $query2 = $catalogClient->createQuery($filter);
 
 $debugForm = $this->get('form.factory')
 ->createBuilder(
 'form',
 [
 'left' => $query1->getEDisMax(),
 'right' => $query2->getEDisMax(),
 'filter' => $filter,
 ],
 ['method' => 'GET']
 )
 ->add('query', 'text', ['property_path' => '[filter].query'])
 ->add('left', 'edismax')
 ->add('right', 'edismax')
 ->getForm()
 ;
 $debugForm->submit($request->query->all(), false);
 
 $catalogResult1 = $catalogClient->createSearchResults($filter, $query1);
 $catalogResult2 = $catalogClient->createSearchResults($filter, $query2);
69. Retours d’experience
70. Suggestions • Suggester suggèste des noms … et non des recherches • Nous envisageons de créer un second core pour les suggestions
71. Performances
73. Performance “killers” • group=true / group.ngroups=true / group.facet=true • Form choice avec 2000 choix … -> AJAX • AutoCommit trop frequent • Trop de slaves • Ne pas utiliser solr cloud -> hard commit obligatoire pour la replication
74. Questions ? • Merci! • speakerdeck.com/adrienbrault/solr-et-recherche- dans-un-site-ecommerce • @AdrienBrault sur Twitter/Github