Findery: an Elasticsearch case study

A presentation for the January 2014 Elasticsearch MeetUp in San Francisco. It shows how Findery uses Elasticsearch to solve their unique geo challenges and how Elasticsearch helps small teams implement robust and successful search features.

Transcript

1. M A G G I E N E L S O N ( @ M A G G I E 1 0 0 0 ) Findery: an Elasticsearch case study
2. R E D I S C O V E R Y O U R W O R L D What is Findery?
6. F I N D E RY T I M E L I N E
7. L O N G , L O N G A G O …
8. • Young startup
9. • RoR application
10. • Easy to prototype
11. • Easy to build in technical debt
12. “Hey, Maggie, can you add search?”
13. S E A R C H : F I R S T I T E R AT I O N
14. • using Solr • welcoming and informative community
15. • Solr isn’t AWS’s target audience • lots of babysitting of machines
16. • very robust, lots of features • geo features not in by default pre 4.x release - manage via plugins
17. • query language has a steep learning curve • difficult to share code with new/inexperienced developers
18. S E A R C H : E L A S T I C S E A R C H
19. “Y’know, for search!”
20. • a welcoming and informative community
21. • easy to set up and monitor on AWS • chef driven setup, multiple data nodes behind a load balancer • new nodes discoverable via well-maintained plugins • additional internal monitoring using Sematext
22. • easy to share with new developers (JSON ftw!) • geo features in by default
23. A S T H E C O M PA N Y G R O W S …
24. • Moving portions of the app from RoR to Node.js • Easy transition
25. • JSON queries remain usually unchanged • some changes to indexing strategy
27. G E N E R A L S I T E S E A R C H • ES 101 • index objects, search objects using basic match / query_string queries
28. W O R D A U T O C O M P L E T E • I <3 ngrams • tags autocomplete • username autocomplete
29. H AV E R S I N E F O R M U L A D I S TA N C E B E T W E E N P O I N T S O N T H E S U R FA C E O F A S P H E R E
30. A U T O C O M P L E T E W I T H A G E O C O M P O N E N T • “find venues near me” • about 60M records • ngram autocomplete for names of venues • combined with sorting by latitude and longitude • good performance when queries are pre warmed (in 10’s of ms) • even better performance if you can set a max on distance
31. D ATA O N A M A P : I S S U E S
32. • potentially lots of notes in a small area
33. • potentially lots of notes in a small area • potentially very few notes in a big area
34. • potentially lots of notes in a small area • potentially very few notes in a big area • some notes are (very subjectively) better than others
35. • potentially lots of notes in a small area • potentially very few notes in a big area • some notes are (very subjectively) better than others • some notes are (very objectively) better than others
36. D ATA O N A M A P : S O L U T I O N S
37. “ M A N Y N O T E S I N A S M A L L A R E A ” • simple bounding box query • boost notes based on scores • prevent “rich get richer” with score half life • boost newer content where it makes sense
38. “ F E W N O T E S I N A B I G A R E A ” • expand bbox until you find something? • order by distance AND score? (kind of expensive) • determine some sort of a base score? • quality vs. performance
39. “ S O M E N O T E S A R E ( S U B J E C T I V E LY ) B E T T E R T H A N O T H E R S ” • people tend to like what their friends like (incorporate the friend graph into queries) • people who liked X also liked Y (generate a few Y for every X) • a little bit of editorial content (easiest tech-wise, needs lots of people)
40. “ S O M E N O T E S A R E ( O B J E C T I V E LY ) B E T T E R T H A N O T H E R S ” • spam detection • note scoring per-user, per-IP, per-anything
42. “ P O P U L A R TA G S I N T H I S A R E A ” • fun to implement • lots of lat/long pairs • essentially a simple geo search based on object’s attributes
43. C O L L E C T I O N S O F N O T E S • multiple lat/long points in a collection • what does “near me” mean? • shape queries
44. “The interesting problems are the ones that are difficult.”
45. Q & A
46. • Photo credits: • http://www.flickr.com/photos/m2w2/1400035437/ • https://www.artsjournal.com/aestheticgrounds/bunnies-and-rabbits-and-hares-passivity-in-publicart/roas- rabbit-in-hackney-2008a/ • http://www.florentijnhofman.nl/dev/project.php?id=181 • http://exhibitioninquisition.wordpress.com/2012/02/10/four-facts-this-will-have-been/ • http://silence-design.deviantart.com/art/Rabbit-Street-Art-159681282 • http://www.flickr.com/photos/boonovista/391812573/ ! • More rabbits to visit: https://findery.com/heather/sets/rabbits-to-visit
47. M A G G I E N E L S O N @ M A G G I E 1 0 0 0 T H A N K S ! ! ! ! ! ! F I N D E RY. C O M