Elastic @ General Mills: The Journey Continues

The journey of Elastic at General Mills began as they replaced the public-facing, Endeca-based search and service-oriented architecture on their various core website properties, including Pillsbury.com, BettyCrocker.com, BoxTops for Education, Tablespoon.com, and the bilingual QueRicaVida.com, with Elasticsearch.

The adventure continues now with using those successes and lessons learned to expand their Elastic use cases in order to move away from a relational database model to a flexible, document-based model across their data.

This talk will cover all the tasty details behind that delicious journey.

Doug Nelson l Application Architect l General Mills, Inc.

Transcript

1. General Mills
   02/27/2017
   doug.nelson@genmills.com
   What’s next for Elastic @
   General Mills
   Doug Nelson, Application Architect
   

   View Slide

2. • Application Architect
   • Major emphasis on search and content delivery for public facing web sites
   ๏
   bettycrocker.com
   ๏
   pillsbury.com
   ๏
   tablespoon.com
   • General Mills in 50 Seconds
   • Passionate about motorcycle riding and downhill skiing
   About Me
   

   View Slide

3. • Content Delivery System
   • Endeca to Elastic transition
   • Elastic{ON} - Chicago
   • Production Cluster is small
   • 5 Data Nodes, 2 Client Nodes, 3 Master Nodes
   • Performance was excellent during our busiest season ( Thanksgiving -> Christmas )
   • Maxed out at about 3500 shard searches per second while ingesting 350 records per second
   • Max node utilization was around 40%
   Elastic - Powering our websites
   3
   Index Content Delivery
   

   View Slide

4. Simple Search – Indexed Content Delivery Overview
   Elastic Cluster
   SimpleSearch
   Searcher
   SearchNx
   Simple Search DTOs
   Searcher implementation
   = a test
   Content List Presentation on sites
   Consumer search page
   

   View Slide

5. • Things are more similar than they are different
   • Recipes, Articles, Videos, Products, Coupons, etc….
   • Summary and Detail representations
   • SearchResultBase
   • ItemId -
   • Title – Most important when searching for relevant results
   • Metadata – Classifications used to find similar results with strong context
   • SimpleSearch is successful because it is evolutionary, changing as needed.
   Lessons learned from SimpleSearch
   

   View Slide

6. • Label (Category)
   • Value (Subcategory)
   • Groups – Organize Label-Values in a logical group
   • Parent - Child Groups – Provides hierarchal structure for groups
   Lessons learned from Metadata
   

   View Slide

7. • Elastic is an excellent document store.
   • Elastic performance exceeds traditional distributive cache performance.
   • Titles and classifications (metadata) work best when searching for relevant results.
   Lessons Learned from Search
   

   View Slide

8. • Re-indexing is divine forgiveness!
   • Index Aliases are your best friends ever
   • Add, Remove in a single action is atomic.
   • Index Alias -> IndexAliasName.
   • Allows rebuilding new index in background
   • Rebuild from the current index
   • Provides for restore points
   Lessons learned from Elastic
   

   View Slide

9. SYSTEMS ARE BETTER
   WHEN YOU EMPOWER
   YOUR USERS!
   

   View Slide

10. • Developed a application to support insurance claims. Case workers needed to be
    able to capture and organize ever changing information.
    • Added a notepad functionality to a base form, all forms used this base.
    • Added the ability to classify types of notes. Enabled admin to add additional
    classifications.
    • Added reporting to allow notes to be printed and added to case file.
    • Net effect – users were able to make the system into their own without the constant
    need for development support. It was a system they loved!!
    Tale of a simple notepad
    

    View Slide

11. • Guids have a most interesting property
    • Guids are guaranteed to be unique independent of type.
    • What does this resemble in the real world?
    Guids (or UUID) – Globally Unique Identifier
    "MetadataIds": [
    "bc6f9836-75ba-54fc-86ba-74f8d8b7aa25",
    "43287d78-f6ad-45d3-8160-c38d4de16e23",
    "a2412e88-729b-5f6f-b1d8-934aae849589",
    "8906e453-1a60-464e-a0d1-3b1011fc1d01",
    "8d9dc531-656a-56c1-9e2e-a8cfb718d4c5",
    "98a9b22f-7dcd-4c60-9366-365ad0da0731",
    "bd7868a9-6fea-5d07-ad96-f02f6628a34d",
    "180b7609-b259-4410-bdf6-e33b1fc41392",
    "0131d3a8-7cb7-4753-8bf8-191e8078f9ed",
    "a6e64f24-ca66-46d9-95ab-e00fc64641c2",
    "add6bd34-ccb1-45b0-8485-da156a58e77c",
    "a327d056-9ffe-4980-8229-ff0103a8cd2e",
    "46dbab0b-bf6f-4d54-a9de-26fccc90117f" ]
    

    View Slide

12. • Each individual guid -> DNA Fragment
    • DNA Fragments -> Classification or an Instruction
    • Each fragment has a specific context avoiding ambiguity. Providing the ability to
    match by ids instead of phrase matching.
    DNA
    

    View Slide

13. • InformationBase
    • ItemId (Guid)
    • Title (string)
    • Values (List)
    • Attachments (List)
    From Documents to Information
    GET gbs.info.metadata/_search
    {
    "query": {"multi_match": {
    "query": "9c60f30c-edc4-4a30-b583-9bc9b9b82eda",
    "fields": ["ItemId^10", "Attachments"]
    }}
    }
    

    View Slide

14. internal interface IAttachmentProcessor {
    void Attach(InformationBase info, InformationBase item);
    void Detach(InformationBase info, InformationBase item);
    }
    For the given info item, attach the passed item by it’s Id.
    Attachment Processor
    

    View Slide

15. internal interface IInformationRepository
    {
    List GetItemWithAttachments(Guid itemId,
    IEnumerable types );
    void Save(InformationBase item);
    void Delete(InformationBase item);
    }
    Information Repository
    

    View Slide

16. internal class Note:InformationBase
    {
    public string NoteBody { get; set; }
    }
    Note
    

    View Slide

17. Off to the note demo
    

    View Slide

18. internal class Label : InformationBase {}
    internal class Value : InformationBase {}
    Metadata – Label and Values
    

    View Slide

19. internal interface IMetadataRepository {
    List GetLabelValues();
    List GetLabels();
    List GetValues();
    }
    Metadata Repository
    

    View Slide

20. • Show metadata label value creation
    Demo time – Sense and Code
    

    View Slide

21. internal class Group:InformationBase
    {
    public string GroupOwner { get; set; }
    public string GroupFunction { get; set; }
    }
    Groups allow for individual metadata labels and values to be organized for easier
    reference.
    Metadata Group – Metadata gets better!
    

    View Slide

22. internal class GroupMember:InformationBase {
    public bool ExclusionFlag { get; set; }
    public double Weight { get; set; } = 1.0;
    public double Minimum { get; set; } = 0.0;
    public double Maximum { get; set; } = 1.0;
    }
    Group Members add a bit of additional information for the selected metadata value.
    Metadata Group Member
    

    View Slide

23. • When we configure a site search index, we need three pieces of information.
    • Elastic Cluster Host
    • Index Language
    • Index Color ( Red/Blue a version pattern we use)
    • Let’s define a metadata group to organize this information
    Building out a metadata group
    

    View Slide

24. Metadata group definition demo
    

    View Slide

25. Site and a Search Index
    internal class SearchIndex:InformationBase{
    public string IndexAlias { get; set; } }
    internal class Site:InformationBase { }
    Enough with all this setup, let’s build something
    

    View Slide

26. Demo site search index creation
    

    View Slide

27. 27
    More Questions?
    Visit us at the AMA
    

    View Slide

28. www.elastic.co
    

    View Slide

29. Except where otherwise noted, this work is licensed under
    http://creativecommons.org/licenses/by-nd/4.0/
    Creative Commons and the double C in a circle are
    registered trademarks of Creative Commons in the United States and other countries.
    Third party marks and brands are the property of their respective holders.
    29
    Please attribute Elastic with a link to elastic.co
    

    View Slide