Beyond The Buzzwords - Microservices and Event-Driven Architecture

Talk given at Dreamforce 2018, September 26th 2018 in Moscone West, San Francisco.

Let's set aside the marketing spin for a moment and focus on the real-world problems that microservices and event streams can solve for us. We'll explore why you should consider these patterns, the exciting technologies that support them, such as Apache Kafka, and most importantly, how you can set aside your fears and be confident in what you're building now and the future.

Transcript

1. Beyond the Buzzwords:
   Microservices and Event-Driven Architecture
   
   with Apache Kafka on Heroku
   Ryan Townsend
   SHIFT Commerce CTO
   Chris Castle
   Heroku Developer Advocate
   

   View Slide

2. This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the
   assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we
   make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber
   growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief,
   any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services.
   The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new
   products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays
   in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the
   immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth,
   new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger
   enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form
   10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important
   disclosures are available on the SEC Filings section of the Investor Information section of our Web site.
   Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be
   delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available.
   Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.
   Statement under the Private Securities Litigation Reform Act of 1995
   Forward-Looking Statement
   

   View Slide

3. Why are we here?
   @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
   

   View Slide

4. Anxiety
   @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
   

   View Slide

5. Monoliths and Microservices
   @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
   

   View Slide

6. “There is no industry consensus yet regarding the properties
   of microservices, and an official definition is missing as well.”

   - Wikipedia
   @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
   

   View Slide

7. ryan.townsend@shiftcommerce.com
   Ryan Townsend, CTO
   chris.castle@salesforce.com
   Chris Castle, Developer Advocate
   @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
   

   View Slide

8. 20 minutes
   No time for questions!
   @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
   

   View Slide

9. Why the hype around microservices?
   @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
   

   View Slide

10. All the cool kids are doing it
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

11. But what are the real benefits?
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

12. @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    Strong Module
    Boundaries
    Independent
    Deployment
    Technology
    Diversity
    Source: Martin Fowler, ThoughtWorks
    

    View Slide

13. If this isn’t you, that’s ok!
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

14. Because there is a price to pay
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

15. Data / Functionality Distribution
    Eventual Consistency
    Operational Complexity
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    Source: Martin Fowler, ThoughtWorks
    

    View Slide

16. How do we mitigate these?
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

17. Demo Time!
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

18. Solution: Data / Functionality Distribution
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

19. Solution: Data / Functionality Distribution
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    n * (n-1) / 2
    Max possible connections
    

    View Slide

20. Solution: Data / Functionality Distribution
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    Source: AWS
    

    View Slide

21. Solution: Data / Functionality Distribution
    Digital Asset
    Management
    Product
    Information
    Management
    Upload product-123.jpg!
    Admin Panel
    Photographer
    Database or
    Search Index
    Merchandiser
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    Chronological Event Stream (Kafka)
    Uploaded Event
    What images can I
    attach to my products?
    Uploaded Event
    Add image to index
    

    View Slide

22. Solution: Eventual Consistency
    Digital Asset
    Management
    Product
    Information
    Management
    Delete product-123.jpg!
    Admin Panel
    Photographer
    Merchandiser
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    Event
    Create Product #123 with
    product-123.jpg
    Event
    Send to trash
    Event
    Untrash
    Admin Panel
    Chronological Event Stream (Kafka)
    

    View Slide

23. Start with monolith and
    extract pieces – build
    up your maturity
    Solution: Operational Complexity
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

24. Cost of engineers vs
    cost of Heroku
    Solution: Operational Complexity
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    Opportunity Cost!
    

    View Slide

25. Maturity of engineers
    vs maturity of Heroku
    Solution: Operational Complexity
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

26. Simplicity of using
    Apache Kafka in
    Heroku
    Solution: Operational Complexity
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

27. • Start with a monolith and break
    out services from there
    • Use Heroku ¯\_(ツ)_/¯
    • Isolate consistency concerns
    • Ensure each service is self-
    serving

    e.g. own data stores
    • Handle via UI, acknowledgement
    • Use event stream

    e.g. Apache Kafka
    Distribution
    Eventual
    Consistency
    Operational
    Complexity
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

28. How the approach grows
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

29. @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    Streaming
    Platform
    Store 1 Store 2 Online Store Distribution Centre Online Marketing
    POS
    Stream
    Inventory
    Stream
    Remarketing
    Stream
    Retail
    Monitoring Reporting
    Data
    Warehouse
    Inventory
    Service
    Logistics
    Service
    Purchasing
    Service
    Profile
    Update
    Service
    Remarketing
    Service
    Source: Confluent
    

    View Slide

30. @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    Source: AWS
    

    View Slide

31. Elephant in the room:
    Scaling
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    

    View Slide

32. @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    Monoliths can scale more than you might think…
    e.g.
    

    View Slide

33. take a photo of this slide, or download at https://bit.ly/2zx3TW2
    @ryantownsend / @crc Slides: https://bit.ly/2zx3TW2
    •
    Last year’s talk - more on Event Streams

    https://hrku.co/df17-event-streams
    •
    Microservices Resource Guide

    https://hrku.co/microservices-guide
    •
    Deconstructing Monolithic Applications into Services

    https://hrku.co/deconstructing-monoliths
    •
    How Small Should Your Microservices Be?

    https://hrku.co/microservice-sizing
    Next Steps
    Summary
    •
    Monoliths are okay - microservices aren’t a silver
    bullet
    •
    Start with a monolith - migrate as a journey not a
    single project
    •
    Having a centralised source of truth will set you up
    for success - Apache Kafka is a great solution
    •
    Heroku makes it quick and easy to get started and
    provides the DevOps maturity
    

    View Slide

34. View Slide