Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Architecting Polyglot-Persistent Solutions

Architecting Polyglot-Persistent Solutions

DevConf 2016 (Johannesburg, South Africa)

Tugberk Ugurlu

March 08, 2016
Tweet

More Decks by Tugberk Ugurlu

Other Decks in Programming

Transcript

  1. TUGBERK UGURLU
    Architecting Polyglot-Persistent Solutions
    Persistence & Data

    View Slide

  2. @tourismgeek
    tugberkugurlu.com
    github.com/tugberkugurlu
    stackoverflow.com/users/463785/tugberk

    View Slide

  3. @redgate
    @RedGateCareers
    red-gate.com/careers

    View Slide

  4. What is/isn’t this talk about?
    > Not an introduction to Polyglot Persistence
    > More about how and why with practical examples
    > Looking at it more in terms of developer point of view

    View Slide

  5. Polyglot Persistence
    “A fancy term meaning using multiple data storage
    technologies in a solution.

    View Slide

  6. Traditional Look
    MySuperShoppingApp
    My GIANT RDBMS
    Shopping Cart

    View Slide

  7. Traditional Look
    from https://speakerdeck.com/chrisann/cd-making-it-work-with-windows-and-net

    View Slide

  8. Polyglot Persistence Look
    MySuperShoppingApp
    MongoDB
    Elasticsearch
    Neo4j
    SQL Server

    View Slide

  9. Advantages
    > Tapping into the strengths of each data storage technology
    > One of the key drivers towards componentization component based
    architecture
    > Sometimes, it’s the only way

    View Slide

  10. Drawbacks
    > Learning + understanding data storage technology-specific interfaces
    > Different needs:
    • Performance
    • Monitoring
    • Backups
    • Compliance

    View Slide

  11. Living Examples: Foursquare

    View Slide

  12. Living Examples: Foursquare
    > Mainly on MongoDB, Elasticsearch and Hadoop
    > Search, recommendation, social graph, listing, ratings and reviews, venue detail
    > More on engineering.foursquare.com

    View Slide

  13. Living Examples: Foursquare

    View Slide

  14. Living Examples: Zleek

    View Slide

  15. Living Examples: Zleek
    > MongoDB as the primary source of truth
    > Elasticsearch for answering search questions and log data
    > Neo4j for recommendation and answering interesting questions
    > Future: Redis for caching

    View Slide

  16. Living Examples: Zleek - MongoDB
    > Primary source of truth
    > Simple ID lookups and listing queries
    > Changes communicated to RabbitMQ

    View Slide

  17. Living Examples: Zleek - Elasticsearch
    > Syncing data from MongoDB
    > Mainly all search requests
    • Full-text search queries
    • Range queries
    • Geo Location queries

    View Slide

  18. Living Examples: Zleek - Neo4j
    > Syncing the data from MongoDB
    > Questions to answer:
    • Which grocery store has the ingredients for a recipe?
    • Which recipes can be made with the items in a store?
    • How my friends interacted with this recipe?

    View Slide

  19. Living Examples: Zleek - Neo4j

    View Slide

  20. Living Examples: Zleek – Neo4j
    “Which recipes can I cook with the items available in grocery store 1?

    View Slide

  21. Living Examples: Zleek - Key Points
    > Immutability of the data
    > RabbitMQ as main event store
    > Be sure you need it
    > Measure, measure and measure

    View Slide

  22. What about development?

    View Slide

  23. What about development?

    View Slide

  24. What about development?

    View Slide

  25. What about development?
    > Docker
    > Docker Compose

    View Slide

  26. What about development?

    View Slide

  27. What about development?

    View Slide

  28. Shaping the product
    “The way you deal with your data will shape the product.

    View Slide

  29. Shaping the product: Foursquare Example

    View Slide

  30. Shaping the product: Twitter Example

    View Slide

  31. Shaping the product: Twitter Example

    View Slide

  32. One last thing...
    “Well-tune the requirements, consider the trade offs and
    come up with a solid, functional plan to serve your data
    instead of blindly getting them out through the same door.

    View Slide

  33. Here is more information
    > Polyglot Persistence from Martin Fowler
    http://martinfowler.com/bliki/PolyglotPersistence.html
    > An evolving example:
    https://github.com/tugberkugurlu/ModernShopping

    View Slide

  34. /* THANK YOU*/
    TUGBERK UGURLU
    Redgate Software
    tugberkugurlu.com
    @tourismgeek
    http://www.devconf.co.za/

    View Slide