Architecting Polyglot-Persistent Solutions

DevConf 2016 (Johannesburg, South Africa)

Transcript

1. TUGBERK UGURLU
   Architecting Polyglot-Persistent Solutions
   Persistence & Data
   

   View Slide

2. @tourismgeek
   tugberkugurlu.com
   github.com/tugberkugurlu
   stackoverflow.com/users/463785/tugberk
   

   View Slide

3. @redgate
   @RedGateCareers
   red-gate.com/careers
   

   View Slide

4. What is/isn’t this talk about?
   > Not an introduction to Polyglot Persistence
   > More about how and why with practical examples
   > Looking at it more in terms of developer point of view
   

   View Slide

5. Polyglot Persistence
   “A fancy term meaning using multiple data storage
   technologies in a solution.
   

   View Slide

6. Traditional Look
   MySuperShoppingApp
   My GIANT RDBMS
   Shopping Cart
   

   View Slide

7. Traditional Look
   from https://speakerdeck.com/chrisann/cd-making-it-work-with-windows-and-net
   

   View Slide

8. Polyglot Persistence Look
   MySuperShoppingApp
   MongoDB
   Elasticsearch
   Neo4j
   SQL Server
   

   View Slide

9. Advantages
   > Tapping into the strengths of each data storage technology
   > One of the key drivers towards componentization component based
   architecture
   > Sometimes, it’s the only way
   

   View Slide

10. Drawbacks
    > Learning + understanding data storage technology-specific interfaces
    > Different needs:
    • Performance
    • Monitoring
    • Backups
    • Compliance
    

    View Slide

11. Living Examples: Foursquare
    

    View Slide

12. Living Examples: Foursquare
    > Mainly on MongoDB, Elasticsearch and Hadoop
    > Search, recommendation, social graph, listing, ratings and reviews, venue detail
    > More on engineering.foursquare.com
    

    View Slide

13. Living Examples: Foursquare
    

    View Slide

14. Living Examples: Zleek
    

    View Slide

15. Living Examples: Zleek
    > MongoDB as the primary source of truth
    > Elasticsearch for answering search questions and log data
    > Neo4j for recommendation and answering interesting questions
    > Future: Redis for caching
    

    View Slide

16. Living Examples: Zleek - MongoDB
    > Primary source of truth
    > Simple ID lookups and listing queries
    > Changes communicated to RabbitMQ
    

    View Slide

17. Living Examples: Zleek - Elasticsearch
    > Syncing data from MongoDB
    > Mainly all search requests
    • Full-text search queries
    • Range queries
    • Geo Location queries
    

    View Slide

18. Living Examples: Zleek - Neo4j
    > Syncing the data from MongoDB
    > Questions to answer:
    • Which grocery store has the ingredients for a recipe?
    • Which recipes can be made with the items in a store?
    • How my friends interacted with this recipe?
    

    View Slide

19. Living Examples: Zleek - Neo4j
    

    View Slide

20. Living Examples: Zleek – Neo4j
    “Which recipes can I cook with the items available in grocery store 1?
    

    View Slide

21. Living Examples: Zleek - Key Points
    > Immutability of the data
    > RabbitMQ as main event store
    > Be sure you need it
    > Measure, measure and measure
    

    View Slide

22. What about development?
    

    View Slide

23. What about development?
    

    View Slide

24. What about development?
    

    View Slide

25. What about development?
    > Docker
    > Docker Compose
    

    View Slide

26. What about development?
    

    View Slide

27. What about development?
    

    View Slide

28. Shaping the product
    “The way you deal with your data will shape the product.
    

    View Slide

29. Shaping the product: Foursquare Example
    

    View Slide

30. Shaping the product: Twitter Example
    

    View Slide

31. Shaping the product: Twitter Example
    

    View Slide

32. One last thing...
    “Well-tune the requirements, consider the trade offs and
    come up with a solid, functional plan to serve your data
    instead of blindly getting them out through the same door.
    

    View Slide

33. Here is more information
    > Polyglot Persistence from Martin Fowler
    http://martinfowler.com/bliki/PolyglotPersistence.html
    > An evolving example:
    https://github.com/tugberkugurlu/ModernShopping
    

    View Slide

34. /* THANK YOU*/
    TUGBERK UGURLU
    Redgate Software
    tugberkugurlu.com
    @tourismgeek
    http://www.devconf.co.za/
    

    View Slide