Elastic @ General Mills: The Journey Continues

The journey of Elastic at General Mills began as they replaced the public-facing, Endeca-based search and service-oriented architecture on their various core website properties, including Pillsbury.com, BettyCrocker.com, BoxTops for Education, Tablespoon.com, and the bilingual QueRicaVida.com, with Elasticsearch.

The adventure continues now with using those successes and lessons learned to expand their Elastic use cases in order to move away from a relational database model to a flexible, document-based model across their data.

This talk will cover all the tasty details behind that delicious journey.

Doug Nelson l Application Architect l General Mills, Inc.

Transcript

1. General Mills 02/27/2017 doug.nelson@genmills.com What’s next for Elastic @ General Mills Doug Nelson, Application Architect
2. • Application Architect • Major emphasis on search and content delivery for public facing web sites ๏ bettycrocker.com ๏ pillsbury.com ๏ tablespoon.com • General Mills in 50 Seconds • Passionate about motorcycle riding and downhill skiing About Me
3. • Content Delivery System • Endeca to Elastic transition • Elastic{ON} - Chicago • Production Cluster is small • 5 Data Nodes, 2 Client Nodes, 3 Master Nodes • Performance was excellent during our busiest season ( Thanksgiving -> Christmas ) • Maxed out at about 3500 shard searches per second while ingesting 350 records per second • Max node utilization was around 40% Elastic - Powering our websites 3 Index Content Delivery
4. Simple Search – Indexed Content Delivery Overview Elastic Cluster SimpleSearch Searcher SearchNx Simple Search DTOs Searcher implementation = a test Content List Presentation on sites Consumer search page
5. • Things are more similar than they are different • Recipes, Articles, Videos, Products, Coupons, etc…. • Summary and Detail representations • SearchResultBase • ItemId - <Guid> • Title – Most important when searching for relevant results • Metadata – Classifications used to find similar results with strong context • SimpleSearch is successful because it is evolutionary, changing as needed. Lessons learned from SimpleSearch
6. • Label (Category) • Value (Subcategory) • Groups – Organize Label-Values in a logical group • Parent - Child Groups – Provides hierarchal structure for groups Lessons learned from Metadata
7. • Elastic is an excellent document store. • Elastic performance exceeds traditional distributive cache performance. • Titles and classifications (metadata) work best when searching for relevant results. Lessons Learned from Search
8. • Re-indexing is divine forgiveness! • Index Aliases are your best friends ever • Add, Remove in a single action is atomic. • Index Alias -> IndexAliasName.<Timestamp (yyyyMMdd.hhmmss) > • Allows rebuilding new index in background • Rebuild from the current index • Provides for restore points Lessons learned from Elastic
9. SYSTEMS ARE BETTER WHEN YOU EMPOWER YOUR USERS!
10. • Developed a application to support insurance claims. Case workers needed to be able to capture and organize ever changing information. • Added a notepad functionality to a base form, all forms used this base. • Added the ability to classify types of notes. Enabled admin to add additional classifications. • Added reporting to allow notes to be printed and added to case file. • Net effect – users were able to make the system into their own without the constant need for development support. It was a system they loved!! Tale of a simple notepad
11. • Guids have a most interesting property • Guids are guaranteed to be unique independent of type. • What does this resemble in the real world? Guids (or UUID) – Globally Unique Identifier "MetadataIds": [ "bc6f9836-75ba-54fc-86ba-74f8d8b7aa25", "43287d78-f6ad-45d3-8160-c38d4de16e23", "a2412e88-729b-5f6f-b1d8-934aae849589", "8906e453-1a60-464e-a0d1-3b1011fc1d01", "8d9dc531-656a-56c1-9e2e-a8cfb718d4c5", "98a9b22f-7dcd-4c60-9366-365ad0da0731", "bd7868a9-6fea-5d07-ad96-f02f6628a34d", "180b7609-b259-4410-bdf6-e33b1fc41392", "0131d3a8-7cb7-4753-8bf8-191e8078f9ed", "a6e64f24-ca66-46d9-95ab-e00fc64641c2", "add6bd34-ccb1-45b0-8485-da156a58e77c", "a327d056-9ffe-4980-8229-ff0103a8cd2e", "46dbab0b-bf6f-4d54-a9de-26fccc90117f" ]
12. • Each individual guid -> DNA Fragment • DNA Fragments -> Classification or an Instruction • Each fragment has a specific context avoiding ambiguity. Providing the ability to match by ids instead of phrase matching. DNA
13. • InformationBase • ItemId (Guid) • Title (string) • Values (List<Guid>) • Attachments (List<Guid>) From Documents to Information GET gbs.info.metadata/_search { "query": {"multi_match": { "query": "9c60f30c-edc4-4a30-b583-9bc9b9b82eda", "fields": ["ItemId^10", "Attachments"] }} }
14. internal interface IAttachmentProcessor { void Attach(InformationBase info, InformationBase item); void Detach(InformationBase info, InformationBase item); } For the given info item, attach the passed item by it’s Id. Attachment Processor
15. internal interface IInformationRepository { List<InformationBase> GetItemWithAttachments(Guid itemId, IEnumerable<Type> types ); void Save(InformationBase item); void Delete(InformationBase item); } Information Repository
16. internal class Note:InformationBase { public string NoteBody { get; set; } } Note
17. Off to the note demo
18. internal class Label : InformationBase {} internal class Value : InformationBase {} Metadata – Label and Values
19. internal interface IMetadataRepository { List<LabelValue> GetLabelValues(); List<Label> GetLabels(); List<Value> GetValues(); } Metadata Repository
20. • Show metadata label value creation Demo time – Sense and Code
21. internal class Group:InformationBase { public string GroupOwner { get; set; } public string GroupFunction { get; set; } } Groups allow for individual metadata labels and values to be organized for easier reference. Metadata Group – Metadata gets better!
22. internal class GroupMember:InformationBase { public bool ExclusionFlag { get; set; } public double Weight { get; set; } = 1.0; public double Minimum { get; set; } = 0.0; public double Maximum { get; set; } = 1.0; } Group Members add a bit of additional information for the selected metadata value. Metadata Group Member
23. • When we configure a site search index, we need three pieces of information. • Elastic Cluster Host • Index Language • Index Color ( Red/Blue a version pattern we use) • Let’s define a metadata group to organize this information Building out a metadata group
24. Metadata group definition demo
25. Site and a Search Index internal class SearchIndex:InformationBase{ public string IndexAlias { get; set; } } internal class Site:InformationBase { } Enough with all this setup, let’s build something
26. Demo site search index creation
27. 27 More Questions? Visit us at the AMA
28. www.elastic.co
29. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 29 Please attribute Elastic with a link to elastic.co