Part II - Search Engine

Part II of MeetUp @ DigitecGalaxus

Transcript

1. Renato Bosshart 07. Feb. 2018
2. Table of Content 07.02.18 • An overview of our search-framework • Structured documents • Analyzers • and a • Clean Search Template • with a good • Testing Framework • produces great • Results • and makes your life a lot easier  Digitec Galaxus search engine 2
3. Architecture 07.02.18 Digitec Galaxus search engine 3 Datasource Updaters Elastic- Cluster API Other Components Shop Write Access Read-Only Keep the data accurate Pre-processing the request Post-processing the response API Nuget client
4. Roadmap 07.02.18 • Highly flexible and automated index mapping • Automated response parsing and mapping • Template and Index versioning • Unit-test on elastic cluster • Integration test on the Api-endpoints • Integration-tests on the nuget client Digitec Galaxus search engine 4 • Trying to interpret to query before sending it to elastic • Improve user guidance • Word decompose • Ironman bluray What we reached so far Future ideas
5. Application Lifecycle 07.02.18 Digitec Galaxus search engine 5 Elastic Cluster API Updaters Dev User Create indices & Templates Deploy Creates/updates docs Changes data Creates/updates docs Search request Template query User response Full update
6. Typed Documents 07.02.18 • The base-class contains all generic properties • Id, Uri, Completionfields, Filter options, Rank, Type, … • Interfaces for other general properties • IModelWithMultilanguageName • IModelWithValidity for data with an date to validate • Individual properties in the class itself • Price, Manufacturer-key • All the properties have annotations to comply with C# and elastic naming guidelines • The name of the class is automatic mapped to a legal and helpful elastic index-name Digitec Galaxus search engine 6
7. Mapping 07.02.18 1. Compose the index-settings with all type mappings 2. Iterating trough all the types 3. Adding our similarities. (They are not type specific yet) 4. Analyzing the required languages for this type 5. Adding all the required analyzers, tokenizers and char filters for this type. 6. Create the property mapping 1. The properties from the base class are added 2. Specific properties are added 7. Upload the settings to elastic 8. Thanks to the strict index and property creation options you will get exceptions instead of voodoo magic Digitec Galaxus search engine 7
8. 07.02.18 Digitec Galaxus search engine 8 GetBaseProperties() { var properties = new Properties(); properties.Add("id", new NumberProperty(NumberType.Integer)); … return properties; } GetTypeProperties(IProperties properties) { properties.Add("firstname", new TextProperty { Analyzer = CustomAnalyzerLibrary.GetLocalizedAnalyzerName("name", ""), Fields = new Properties { {"phonetic_rs", new TextProperty { Analyzer = AnalyzerLibrary.GetAnalyzerName("ref_soundex_phonetic"), Boost = 0.3D } }, … } }); … return properties; }
9. Mapping 07.02.18 1. Compose the index-settings with all type mappings 2. Iterating trough all the types 3. Adding our similarities. (They are not type specific yet) 4. Analyzing the required languages for this type 5. Adding all the required analyzers, tokenizers and char filters for this type. 6. Create the property mapping 1. The properties from the base class are added 2. Specific properties are added 7. Upload the settings to elastic 8. Thanks to the strict index and property creation options you will get exceptions instead of voodoo magic ;-) Digitec Galaxus search engine 9
10. Unleash the power of analyzers but don’t let them run wild 07.02.18 • Analyzers are very powerful • ISBN13 to Gtin conversion • Remove leading zeros • https://www.digitec.ch/de/Search?q=crosshair+6 • https://www.digitec.ch/de/Search?q=mac+book • Don’t trust the documentation, check the functionality • Invest enough time to evaluate the best analyzer • If the description matches your use-case, you still may have problems • https://www.galaxus.ch/de/Search?q=Dachs • Shoes are phonetic equal to HTC • RX Vega != R10 5ega Digitec Galaxus search engine 10
11. Search-templates: smaller requests, cleaner code 07.02.18 • Moustache and Json doesn’t work well together. Try it anyway. (And notify me if you found a good editor ) • Use boosting on index level, not in the template • You can search across multiple Languages • https://pastebin.com/2hEKhmYQ • Use a script to calculate a function score • Generate aggregations using JSON lists • Toggle blocks from parameters Digitec Galaxus search engine 11
12. Unit Testing Framework 07.02.18 Digitec Galaxus search engine 12 Assembly Base Class Type Base Test Create indices and upload templates Create type document insert type document Execute search request Verify result Clear documents from indices Delete Indices
13. Some sample unit-test 07.02.18 Digitec Galaxus search engine 13 var book = InsertAProductWithName("Spongebob Schwammkopf Buch"); var film = InsertADeletedProductWithName("Spongebob Schwammkopf Film"); var search = EnterSearchInGalaxusShopFor("Spongebob"); search.ContainsTheseItems(book); search.DoesNotContainTheseItems(film); var mpGerman = InsertAMarketingPageWithTitleInAllLanguages(titleGerman: "Kindergarten"); var mpEnglish = InsertAMarketingPageWithTitleInAllLanguages(titleGerman: null, titleEnglish: "Kindergarten Kindergarten"); var search = EnterSearchInGalaxusShopFor("Kindergarten"); search.ContainsTheseItems(mpGerman, mpEnglish); search.ThisIsFirst(mpGerman); var book = InsertAProductWithNameAndGtins(productName: "Der Schwarm", gtins: ReadOnlyList.Create(9783596164530)); var car = InsertAProductWithNameAndGtins(productName: "VW Bus", gtins: ReadOnlyList.Create(1111111111111)); var search = IncrementalSearchInGalaxusShopFor("978-3-596-16453-0"); search.ContainsTheseItems(book); search.DoesNotContainTheseItems(car); var brand = InsertABrandWithName("Sony"); var productType = InsertAProductTypeWithName("Sofa"); var tag = InsertATagWithName("Samy"); var search = EnterSearchInGalaxusShopFor("Somy"); search.SuggestionsHasString(brand.Name); Test search filters Test suggestions Test multilingual features Test analyzers Test document updater logic GivenAnIndex(new ProductCacheModel{Id = productId1, ShowInShopSearch = true}); GivenAProductNameChangedEventHandler(); WhenProductNameChangedEventIsReceived(productId: productId1); ThenProductCacheModelHasName(productId: productId1, name: "Sports");
14. Some problems and how we solved them 07.02.18 • Smartphone-covers contains the Smartphones name more often then the smartphone itself. • We used a function score with our bestseller ranking Digitec Galaxus search engine 14 Function score
15. Some problems and how we solved them 07.02.18 • Strange products appeared in the top results • We boosted the products brand and type more then the name itself • We removed stemming from the main analyzer and added a less boosted stemmed field Digitec Galaxus search engine 15 Wecker:
16. Some problems and how we solved them 07.02.18 • We had results where only part of the searchterm matched because we had multiple query blocks • We added a query-string block as filter to remove the part-matching products. You can use the full searchterm but escaping is important Digitec Galaxus search engine 16
17. Questions? 07.02.18 Digitec Galaxus search engine 17 digitec Showroom Zürich Or contact me directly: renato.bosshart@digitecgalaxus.ch Looking for a Job? https://www.galaxus.ch/de/JobOffer