Architecting Polyglot-Persistent Solutions

DevConf 2016 (Johannesburg, South Africa)

Transcript

1. TUGBERK UGURLU Architecting Polyglot-Persistent Solutions Persistence & Data
2. @tourismgeek tugberkugurlu.com github.com/tugberkugurlu stackoverflow.com/users/463785/tugberk
3. @redgate @RedGateCareers red-gate.com/careers
4. What is/isn’t this talk about? > Not an introduction to Polyglot Persistence > More about how and why with practical examples > Looking at it more in terms of developer point of view
5. Polyglot Persistence “A fancy term meaning using multiple data storage technologies in a solution.
6. Traditional Look MySuperShoppingApp My GIANT RDBMS Shopping Cart
7. Traditional Look from https://speakerdeck.com/chrisann/cd-making-it-work-with-windows-and-net
8. Polyglot Persistence Look MySuperShoppingApp MongoDB Elasticsearch Neo4j SQL Server
9. Advantages > Tapping into the strengths of each data storage technology > One of the key drivers towards componentization component based architecture > Sometimes, it’s the only way
10. Drawbacks > Learning + understanding data storage technology-specific interfaces > Different needs: • Performance • Monitoring • Backups • Compliance
11. Living Examples: Foursquare
12. Living Examples: Foursquare > Mainly on MongoDB, Elasticsearch and Hadoop > Search, recommendation, social graph, listing, ratings and reviews, venue detail > More on engineering.foursquare.com
13. Living Examples: Foursquare
14. Living Examples: Zleek
15. Living Examples: Zleek > MongoDB as the primary source of truth > Elasticsearch for answering search questions and log data > Neo4j for recommendation and answering interesting questions > Future: Redis for caching
16. Living Examples: Zleek - MongoDB > Primary source of truth > Simple ID lookups and listing queries > Changes communicated to RabbitMQ
17. Living Examples: Zleek - Elasticsearch > Syncing data from MongoDB > Mainly all search requests • Full-text search queries • Range queries • Geo Location queries
18. Living Examples: Zleek - Neo4j > Syncing the data from MongoDB > Questions to answer: • Which grocery store has the ingredients for a recipe? • Which recipes can be made with the items in a store? • How my friends interacted with this recipe?
19. Living Examples: Zleek - Neo4j
20. Living Examples: Zleek – Neo4j “Which recipes can I cook with the items available in grocery store 1?
21. Living Examples: Zleek - Key Points > Immutability of the data > RabbitMQ as main event store > Be sure you need it > Measure, measure and measure
22. What about development?
23. What about development?
24. What about development?
25. What about development? > Docker > Docker Compose
26. What about development?
27. What about development?
28. Shaping the product “The way you deal with your data will shape the product.
29. Shaping the product: Foursquare Example
30. Shaping the product: Twitter Example
31. Shaping the product: Twitter Example
32. One last thing... “Well-tune the requirements, consider the trade offs and come up with a solid, functional plan to serve your data instead of blindly getting them out through the same door.
33. Here is more information > Polyglot Persistence from Martin Fowler http://martinfowler.com/bliki/PolyglotPersistence.html > An evolving example: https://github.com/tugberkugurlu/ModernShopping
34. /* THANK YOU*/ TUGBERK UGURLU Redgate Software tugberkugurlu.com @tourismgeek http://www.devconf.co.za/