Hippo GetTogether: The architecture behind Hippos relevance platform

These slides were from my Hippo GetTogether 2013 presentation. During this presentation I went into detail about the architecture behind our high performance relevance platform. The talk will also cover why we chose CouchBase for storage and how Elasticsearch can be used for search and analytics. I shared how we integrated and leverage both products full-circle from within our Hippo CMS product.

Transcript

1. Building a relevance platform with Couchbase and Elasticsearch Hippo GetTogether, 21 June 2013 Jeroen Reijn | @jreijn | #hgt2013 Hippo GetTogether 2013 follow the Hippo trail
2. follow the Hippo trail Hippo GetTogether 2013 About me • Architect @ Hippo • DevOps guy • Blogger @ http://blog.jeroenreijn.com
3. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Relevance?
4. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto “The capability of a search engine or function to retrieve data appropriate to a user's needs.” http://www.thefreedictionary.com/relevance
5. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto
6. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto How we deliver relevant content @Hippo
7. follow the Hippo trail Hippo GetTogether 2013 Registration Visitor - entity making HTTP requests Collector - records data about a visitor or his behavior Example: location collector (GeoIPCollector) Targeting Data - all data about a specific visitor Example: IP address is located in Amsterdam
8. follow the Hippo trail Hippo GetTogether 2013 Matching Characteristic - a type of fact about visitors Example: "comes from a city", "experiences a type of weather" Target Group - the specification of a Characteristic Example: "comes from a European city", "comes from Amsterdam" Persona - one or more target groups that describe a certain type of visitor Example: "Jim, the European urban consumer", "Alice, the Pet owner"
9. follow the Hippo trail Hippo GetTogether 2013 What do we store? Request log Targeting data Statistics Averages, e.g. how many visitors became which persona
10. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto BIG DATA !!
11. follow the Hippo trail Hippo GetTogether 2013 Real-time analysis
12. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Architecture
13. follow the Hippo trail Hippo GetTogether 2013 RDBMS Hippo Delivery Tier Hippo Repository App server XML JSON (X)HTML
14. follow the Hippo trail Hippo GetTogether 2013 Delivery Tier URL Matching Fetch content Compose output Request Response
15. follow the Hippo trail Hippo GetTogether 2013 Delivery Tier URL Matching Targeting Data Collection Compose output Request Response Fetch content Scoring
16. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Scaling
17. follow the Hippo trail Hippo GetTogether 2013 RDBMS Hippo Delivery Tier Hippo Repository App server Hippo Delivery Tier Hippo Repository App server Scaling out
18. follow the Hippo trail Hippo GetTogether 2013 RDBMS Delivery Tier Repository App server Delivery Tier Repository App server Scaling out Targeting Datastore
19. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto What kind of ‘storage’?
20. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Question?
21. follow the Hippo trail Hippo GetTogether 2013 Distributed Cache?
22. follow the Hippo trail Hippo GetTogether 2013 We have a winner!
23. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Requirements change!
24. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto NoSQL to the rescue
25. follow the Hippo trail Hippo GetTogether 2013 Suitable types • Key-value store • Document database
26. follow the Hippo trail Hippo GetTogether 2013 Assessment Criteria Maturity Data model Consistency model Performance Replication Caching model Query model Monitoring Scalability Reliability Support
27. follow the Hippo trail Hippo GetTogether 2013 Selection Criteria • Performance • Scalability • Schema flexibility • Simplicity • Monitoring • Support
28. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Performance !! Performance !!!!
29. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Scalability
30. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Schema flexibility
31. follow the Hippo trail Hippo GetTogether 2013 { "visitorId": "7a1c7e75-8539-40", "pageUrl": "http://localhost:8080/site/news", "pathInfo": "/news", "remoteAddr": "127.0.0.1", "referer": "http://localhost:8080/site/", "timestamp": 1371419505909, "collectorData": { "geo": { "country": "", "city": "", "latitude": 0, "longitude": 0 }, "returningvisitor": false, "channel": "English Website" }, "personaIdScores": [], "globalPersonaIdScores": [] } Request log document
32. follow the Hippo trail Hippo GetTogether 2013 { "geo": { "collectorId": "geo", "city": "", "country": "", "latitude": 0, "longitude": 0 }, "channel": { "collectorId": "channel", "channels": [ "English Website" ], "lastVisitedChannel": "English Website" } } Visitor document
33. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Simplicity
34. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Monitoring
35. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Support
36. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Couchbase
37. follow the Hippo trail Hippo GetTogether 2013 Why Couchbase? • Drop-in replacement for memcached • Read/Write-through cache • High throughput • Easy scalability • Schema flexibility • Low latency
38. follow the Hippo trail Hippo GetTogether 2013 Couchbase • Open Source • Document-oriented • Easy Scalable • Consistent High Performance • Apache license
39. follow the Hippo trail Hippo GetTogether 2013 Performance • Object managed cache • Write Queue to disk • Avoids Cold Cache
40. follow the Hippo trail Hippo GetTogether 2013 Source: http://www.slideshare.net/Couchbase/benchmarking-couchbase Copyright © Altoros Systems, Inc.
41. follow the Hippo trail Hippo GetTogether 2013 Easy scalable • Auto sharding • Cross cluster replication (XDCR) • Master - Master replication
42. follow the Hippo trail Hippo GetTogether 2013 Flexible data model • Native JSON support • Incremental Map Reduce • Gives power to the developer
43. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto How we run Couchbase @Hippo
44. follow the Hippo trail Hippo GetTogether 2013 Load Balancer Database cluster Hippo Delivery Tier Couchbase cluster •Request log data •Targeting data •Statistics data
45. follow the Hippo trail Hippo GetTogether 2013 Query capabilities • Querying via views • Secondary indexes via views • Views based on Map - Reduce • Lacks some advanced query capabilities
46. follow the Hippo trail Hippo GetTogether 2013 Elasticsearch • Apache Lucene • Designed to be distributed • Schema free • Apache license • RESTful API
47. follow the Hippo trail Hippo GetTogether 2013 Added value of ES • Full text search • Faceted search • Geo spatial search • All in (near) real-time
48. follow the Hippo trail Hippo GetTogether 2013 Couchbase Server Cluster Elasticsearch Server Cluster Hippo Delivery Tier Java API Write Read XDCR Couchbase ES Transport plugin Replicating to ES
49. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto What’s Next?
50. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto What’s Next?
51. follow the Hippo trail Hippo GetTogether 2013 Advanced analytics
52. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Demo time!
53. follow the Hippo trail Hippo GetTogether 2013 OneHippo @ Goto Thank you! Questions? j.reijn@onehippo.com | @jreijn ps. We’re hiring!