Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Making the Content Conduit Faster: Search @ Emerson

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
November 19, 2015

Making the Content Conduit Faster: Search @ Emerson

For Emerson Process Management, Elasticsearch became the cornerstone technology and changed the game for indexing, searching, and retrieving critical documentation of customer quotes and orders. Indexing changed from months to hours, searches that took 40-50 seconds are now done in sub-seconds, and unhappy users are now ecstatic when retrieving critical customer order documentation.

Mark Heindselman | Elastic{ON} Tour Chicago | November 19, 2015


Elastic Co

November 19, 2015

More Decks by Elastic Co

Other Decks in Technology


  1. 1

  2. Distributed Elasticsearch at Emerson Process Management
 Mark Heindselman, Director of

    Knowledge Network and Information Services | Emerson Process Management 2
  3. Customer Integration Solution & Technology
 With Problem Implementation Partner Solution

  4. Agenda • What is a GAME CHANGER! • Who is

    Emerson Process Management • Who is Redstone Content Solutions • Distributed Elastic Search Implementation Overview • The Emerson Process Management Story o Environment, Problem & Impact o Solution & Demo o Value o Future • Wrap Up 4
  5. What is a GAME CHANGER! A newly introduced element or

    factor that changes an existing situation or activity in a significant way 5
  6. Emerson Process Management • A part of Emerson Electric •

    35,000 Employees • Global Manufacturing locations • $10B Revenue • Leading Supplier of Products & Services for the Process Automation Industry ▪ Process Control Systems & Software ▪ Measurement & Analytical Instruments o Analytical, Flow, Level, Pressure, Temperature ▪ Control Valves & Regulators 6
  7. Redstone Content Solutions Oracle Platinum Partner - WebCenter focus ▪

    VAR & SI ▪ Public Sector Addendum ▪ Employee Staff that Started Stellent Partner Program Premier product training ▪ 3T™ Methodology ▪ Oracle Certified Training Delivery Partner Technical competency ▪ Java-Certified Developers ▪ PMP-Certified PMs 7
  8. Distributed Elastic Search (DES) Overview • Elasticsearch search engine •

    Redstone Content Solutions – Distributed Index Integration to Oracle WebCenter Content • DES is a Game Changing WebCenter Content Searching and Indexing Solution ▪ Oracle Validated Integration • WebCenter Content / Elasticsearch integration ▪ Indexing – Multiple indices definable o Rules-based and highly configurable o Indexing Engine is engineered for speed ▪ Searching o Intercept calls to GET_SEARCH_RESULTS service o Reroutes conditional queries to separate indices o Returns the result of the search to the WebCenter Results Interface 8
  9. The Enterprise Content Management Global Environment 
 Instance  Name Size

    #  of  Docs #  of  Contributors #  of  Users eDocs  Asia 0.25  TB  0.250M 100 250  I eDocsDB 7.00  TB  7.000M programmatic 1000  I eDocs  Europe 1.50  TB  1.250M 200 2000  I eDocs  Eur  Ext 1.00  TB  1.000M programmatic 5000  E eDocs/FishWeb 1.50  TB  1.650M 1500  &  prog 6000  I eDocs  Ext  NA 1.80  TB  3.100M programmatic 60000  E eDocsReports 0.25  TB  0.200M programmatic 200  I MMIDocs 0.25  TB  0.200M 100  &  prog 200  I OCM 6.00  TB 18.900M 1300  &  prog 8000 Total 19.55  TB 33.55M 3200  &  prog 65K  E    11.6K  I 9 E  =  external  users      I  =  internal  users
  10. OCM Instance Environnent WebCenter Content Environment ▪ 10gR3 ▪ Physical

    Server ▪ Dedicated SQL Server 10 Sales Order/Quote Repository ▪ 15,000,000 records ▪ 5 TB metadata index ▪ User and programmatic filing: o Check in Screens o WebDAV Drag & Drop from Desktop or from Outlook o eMail Listener o Oracle EBS o And Other Business Systems ▪ 150,000+ check-ins/month
  11. Our Problems Slow Searching – Metadata Index ▪ 45 seconds

    for a search across the whole database ▪ 12 to 15 second for a search across 1M records Slow Indexing ▪ Months to Re-index o Started Dec 23 Completed Mar 26
  12. Impact Dissatisfied users Long-running queries Increasing # of tickets for

    slow searching High database utilization - impacted document loading Unable to add database fields because of slow index rebuild speed 12
  13. We needed a solution that meet Our Objectives No UI

    modifications No security group modifications No database architecture modifications 13 Our Goals Improve searching by 75% Improve index rebuild speed by 90% Improve user satisfaction
  14. Redstone’s Distributed Elastic Search Integration Single Master Index No change

    for end users ▪ Same search and retrieval interface and results screens DES Integration with WebCenter ▪ Intercepts search prior to being passed SQL database ▪ Runs search against DES Index ▪ Runs multiple term searches simultaneously o SQL runs sequential searches for each term ▪ Passes the Search Results back to WebCenter results pages for display 14
  15. The  Results  -­‐  FAST Search  Speed     – Searching

       all  content    with  multiple     terms  in  <  2  seconds   – 1,000,000  records  <  1  second   Index  Rebuild  Speed       – From  3  months  to  9  hours  for   17,000,000  records   USERS  ARE  EXSTATIC!
  16. 16

  17. Value  -­‐  Distributed  Elasticsearch Objectives   No  UI  modifications  

    No  security  group  modifications   No  database  architecture  modifications Goals     Improve  searching  by  75%  à 99%   Improve  index  rebuild  speed  by  90%  à 99%       Improved  WebCenter  user  satisfaction   17
  18. 950%  Return  On  Investment   WebCenter  Usage     –

    6,000  employees   – 1,440,000  searches/month   – 17,280,000  searches/year   – $40/hr  (average  fully  loaded  FTE)   Time  Saved  in  Doing  Searches   – 11  Seconds    per  Search     – 2,247  Hours  /  Rebuild   What  that  adds  up  to:     – 17,280,000  searches     – x  11  seconds       – =  190,080,000  seconds     – =  52,800  hours  saved   – x  $40/hr     – =  $2,112,000  annual  savings   • Satisfied  users   • Mission  Critical  Application   that  the  users  can’t  do  with   out! 18
  19. Features/Benefits  -­‐  In-­‐use 99%  improved  search  and  index  rebuild  

    speeds   Supports  unlimited  number  of  content   items/search  terms   Rebuild  segments  of  an  index   Create  unlimited  number  of  indices   Create/rebuild  search  indices  without   restarting  content  server   Minimal  user  impact  during  index   rebuild   Easy  implementation  and  configuration   Horizontal/vertical  scalability  for   extreme  performance 19
  20. Features/Benefits  -­‐  In-­‐use Support  for  distributed  architecture   for  indices

     and  hardware   Real-­‐time  feedback—estimates   remaining  index  rebuild  time   Search  remains  operational  during   index  rebuild   Multi-­‐threaded  rebuild  process Maintains  current  out-­‐of-­‐the-­‐box   index   Add  new  documents/revisions  or   delete  content  during  rebuilds   Live  reload—real-­‐time  tuning  of  the   index  rebuild  process 20
  21. Features/Benefits  –  Not  In  Use  Yet Clustering     Create/maintain

     multiple  indices   based  on  content/metadata   Index  priority  for  documents   matching  multiple  indices Ability  to  simultaneously  rebuild   multiple  indices     Flexible  faceted  search  results   – End  users  can  filter  search  results   – End  users  can  control  facets  per   search  or  saved  search   – Administrators  can  create   different  search  facets  per  profile 21
  22. Future  of  DES  at  Emerson We  are  or  will  be

     use  DES  to  index  ALL  our  WebCenter  Content  instances   In-­‐production   – OCM  (18M  items)  metadata-­‐only   In-­‐process     – eDocs  /  Fishweb  (2M  items)  GSA  full-­‐text   – eDocsDB  (5M  items)  metadata-­‐only Future     – eDocs  Extranet  (3M  items)  full-­‐text   – MMI  Docs  (200,000  items)  full-­‐text   – eDocs  Europe  (2M  items)  full-­‐text   – eDocs  Asia  (300,000  items)  full-­‐text   End-­‐Game     – All  WebCenter  Content  Instances                        with  a  Master    Index  (32M  items)                        to  search  them  all  at  once 22
  23. Future  of  DES  at  Emerson We  are  in  discussions  on

     how  we  can  use  DES   solution  for  eDiscovery  purposes  on     – File  Servers   – SharePoint 23
  24. Wrap Up & Summary • We Had a Major Indexing

    and Searching Problems • Redstone Developed an Integration of Elastic Search with the Oracle WebCenter Content • The Integration required NO Changes for how the users were use to using the system • The solution was Game Changing for the users with Searching – it is FAST • The solution was Game Changing for the administrators with the Indexing - it is FAST • The Value of the Solution can be Quantified – in real time and money • The Value of the Solution is measured by the Users Response – Ecstatic, Satisfied, See the Application as Critical • Big Future for DES at Emerson Process Management 24
  25. Customer Integration Solution & Technology
 With Problem Implementation Partner Solution

    Mark  Heindselman   Mark.Heindselman@emerson.com   641  754-­‐3161 John  Klein   jklein@redstonecontentsolutioins.com   (563)  424-­‐7563 info@elastic.co