Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elastic @ General Mills: The Journey Continues

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
March 07, 2017

Elastic @ General Mills: The Journey Continues

The journey of Elastic at General Mills began as they replaced the public-facing, Endeca-based search and service-oriented architecture on their various core website properties, including Pillsbury.com, BettyCrocker.com, BoxTops for Education, Tablespoon.com, and the bilingual QueRicaVida.com, with Elasticsearch.

The adventure continues now with using those successes and lessons learned to expand their Elastic use cases in order to move away from a relational database model to a flexible, document-based model across their data.

This talk will cover all the tasty details behind that delicious journey.

Doug Nelson l Application Architect l General Mills, Inc.


Elastic Co

March 07, 2017


  1. General Mills 02/27/2017 doug.nelson@genmills.com What’s next for Elastic @ General

    Mills Doug Nelson, Application Architect
  2. • Application Architect • Major emphasis on search and content

    delivery for public facing web sites ๏ bettycrocker.com ๏ pillsbury.com ๏ tablespoon.com • General Mills in 50 Seconds • Passionate about motorcycle riding and downhill skiing About Me
  3. • Content Delivery System • Endeca to Elastic transition •

    Elastic{ON} - Chicago • Production Cluster is small • 5 Data Nodes, 2 Client Nodes, 3 Master Nodes • Performance was excellent during our busiest season ( Thanksgiving -> Christmas ) • Maxed out at about 3500 shard searches per second while ingesting 350 records per second • Max node utilization was around 40% Elastic - Powering our websites 3 Index Content Delivery
  4. Simple Search – Indexed Content Delivery Overview Elastic Cluster SimpleSearch

    Searcher SearchNx Simple Search DTOs Searcher implementation = a test Content List Presentation on sites Consumer search page
  5. • Things are more similar than they are different •

    Recipes, Articles, Videos, Products, Coupons, etc…. • Summary and Detail representations • SearchResultBase • ItemId - <Guid> • Title – Most important when searching for relevant results • Metadata – Classifications used to find similar results with strong context • SimpleSearch is successful because it is evolutionary, changing as needed. Lessons learned from SimpleSearch
  6. • Label (Category) • Value (Subcategory) • Groups – Organize

    Label-Values in a logical group • Parent - Child Groups – Provides hierarchal structure for groups Lessons learned from Metadata
  7. • Elastic is an excellent document store. • Elastic performance

    exceeds traditional distributive cache performance. • Titles and classifications (metadata) work best when searching for relevant results. Lessons Learned from Search
  8. • Re-indexing is divine forgiveness! • Index Aliases are your

    best friends ever • Add, Remove in a single action is atomic. • Index Alias -> IndexAliasName.<Timestamp (yyyyMMdd.hhmmss) > • Allows rebuilding new index in background • Rebuild from the current index • Provides for restore points Lessons learned from Elastic

  10. • Developed a application to support insurance claims. Case workers

    needed to be able to capture and organize ever changing information. • Added a notepad functionality to a base form, all forms used this base. • Added the ability to classify types of notes. Enabled admin to add additional classifications. • Added reporting to allow notes to be printed and added to case file. • Net effect – users were able to make the system into their own without the constant need for development support. It was a system they loved!! Tale of a simple notepad
  11. • Guids have a most interesting property • Guids are

    guaranteed to be unique independent of type. • What does this resemble in the real world? Guids (or UUID) – Globally Unique Identifier "MetadataIds": [ "bc6f9836-75ba-54fc-86ba-74f8d8b7aa25", "43287d78-f6ad-45d3-8160-c38d4de16e23", "a2412e88-729b-5f6f-b1d8-934aae849589", "8906e453-1a60-464e-a0d1-3b1011fc1d01", "8d9dc531-656a-56c1-9e2e-a8cfb718d4c5", "98a9b22f-7dcd-4c60-9366-365ad0da0731", "bd7868a9-6fea-5d07-ad96-f02f6628a34d", "180b7609-b259-4410-bdf6-e33b1fc41392", "0131d3a8-7cb7-4753-8bf8-191e8078f9ed", "a6e64f24-ca66-46d9-95ab-e00fc64641c2", "add6bd34-ccb1-45b0-8485-da156a58e77c", "a327d056-9ffe-4980-8229-ff0103a8cd2e", "46dbab0b-bf6f-4d54-a9de-26fccc90117f" ]
  12. • Each individual guid -> DNA Fragment • DNA Fragments

    -> Classification or an Instruction • Each fragment has a specific context avoiding ambiguity. Providing the ability to match by ids instead of phrase matching. DNA
  13. • InformationBase • ItemId (Guid) • Title (string) • Values

    (List<Guid>) • Attachments (List<Guid>) From Documents to Information GET gbs.info.metadata/_search { "query": {"multi_match": { "query": "9c60f30c-edc4-4a30-b583-9bc9b9b82eda", "fields": ["ItemId^10", "Attachments"] }} }
  14. internal interface IAttachmentProcessor { void Attach(InformationBase info, InformationBase item); void

    Detach(InformationBase info, InformationBase item); } For the given info item, attach the passed item by it’s Id. Attachment Processor
  15. internal interface IInformationRepository { List<InformationBase> GetItemWithAttachments(Guid itemId, IEnumerable<Type> types );

    void Save(InformationBase item); void Delete(InformationBase item); } Information Repository
  16. internal class Note:InformationBase { public string NoteBody { get; set;

    } } Note
  17. Off to the note demo

  18. internal class Label : InformationBase {} internal class Value :

    InformationBase {} Metadata – Label and Values
  19. internal interface IMetadataRepository { List<LabelValue> GetLabelValues(); List<Label> GetLabels(); List<Value> GetValues();

    } Metadata Repository
  20. • Show metadata label value creation Demo time – Sense

    and Code
  21. internal class Group:InformationBase { public string GroupOwner { get; set;

    } public string GroupFunction { get; set; } } Groups allow for individual metadata labels and values to be organized for easier reference. Metadata Group – Metadata gets better!
  22. internal class GroupMember:InformationBase { public bool ExclusionFlag { get; set;

    } public double Weight { get; set; } = 1.0; public double Minimum { get; set; } = 0.0; public double Maximum { get; set; } = 1.0; } Group Members add a bit of additional information for the selected metadata value. Metadata Group Member
  23. • When we configure a site search index, we need

    three pieces of information. • Elastic Cluster Host • Index Language • Index Color ( Red/Blue a version pattern we use) • Let’s define a metadata group to organize this information Building out a metadata group
  24. Metadata group definition demo

  25. Site and a Search Index internal class SearchIndex:InformationBase{ public string

    IndexAlias { get; set; } } internal class Site:InformationBase { } Enough with all this setup, let’s build something
  26. Demo site search index creation

  27. 27 More Questions? Visit us at the AMA

  28. www.elastic.co

  29. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/

    Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 29 Please attribute Elastic with a link to elastic.co