Fast and Flexible Donor Segmentation at Red Cross Using Elasticsearch

Christiaan Stoffer, CTO at Nexios.com, talks about how they implemented tools for volunteers, a platform built on top of Elasticsearch, for the administration of all 30000 volunteers of the Dutch Red Cross.

Christian Stoffer | Elastic{ON} Tour Amsterdam | October 29, 2015

Transcript

1. Christiaan Stoffer - October 29 1   Fast and flexible Donor Segmentation at Red Cross using Elasticsearch
2. 2 Christiaan Stoffer CTO / Architect / Developer
3. 3 National emergency aid First aid International Assistance
4. Web-application: “Tools for volunteers” 4 •  Platform for all volunteer related activities •  Broad adoption through focus on: §  Ease of use §  Fast results §  Self-service •  Thousands of users each week
5. What about fundraising? 5  
6. Some facts and figures •  ~ 2 million contacts §  Roles like donor, member, volunteer, etc. •  ~ 25 million financial transactions §  Small donations to major donor contributions •  ~ 300 million campaign activities and responses §  Direct marketing, telemarketing, etc. 6
7. Fundraising application landscape – planned architecture 7 CRM   Finance  
8. CRM   Finance   Fundraising application landscape – actual architecture 8 POOR PERFORMANCE!
9. 9   “It takes way too long to build our market segmentation”
10. 10   “We cannot import campaign results during work hours”
11. Solution: stick to original plan & solve performance issues! Business evolution not revolution •  Same business processes & principles •  Just need a very powerful engine! 11 CRM   CRM  
12. Solution: Enter Elasticsearch! With Elasticsearch we are able to… -  Provide all the power needed -  Reduce TCO by almost 30% -  Be more flexible; better respond to peaks in usage 12
13. Main goals for fast and flexible segmentation •  One system for all processes §  Easy to use §  Productivity •  Fast segmentation results §  # change as you adjust segments definitions §  Near real-time data for up-to-date results •  Work whenever you want! §  No noticeable performance degradation §  Even with bulk operations 13
14. Results of proof of concept Using Elasticsearch we are able to: •  Get insights really fast §  Milliseconds, not minutes or even hours •  Provide an overall better user-experience §  Focus on results; not waste time on queries •  Build to “keep scaling” 14
15. What was our key challenge? 15  
16. Lots of relations need to be traversed or “joined” •  Parent – Child §  Slows down really quick: 2+ seconds and slower §  More and bigger hardware does not provide enough improvement §  CPU intensive: noticeable performance degradation •  Nested §  Response-time stays within milliseconds Even with around 20 nested filters in 1 query (!!) §  Slightly more complicated index strategy needed 16
17. There are boundaries with this solution… Input from Elasticsearch Support: Take the number of nested documents into account: •  1M ES docs with ~100 nested docs results in ~100M Lucene docs •  1 shard can hold ~2B Lucene docs And be aware of other things like: •  Complicated mapping biggest performance killer •  Large documents may become a problem at index time 17
18. Conclusion 18  
19. Using Elasticsearch… •  We are able to consolidate all systems (reducing IT costs by ~30%) •  Provide excellent performance and scalability •  Really empower users with real-time business insight 19 =  
20. Christiaan Stoffer christiaan@nexios.com Thank you! 20  
21. www.elas6c.co   21