Revitalizing Aging Architectures (Gartner Catalyst)

We take a look into the Walmart Canada case study and how Scala, Play, and Akka was leveraged to modernize a heritage architecture. We then dive into the latest in reactive microservices development and walk through a modern architecture on how a heritage architecture can be re-architected as an event-driven real-time system.

Transcript

1. Revitalizing Aging Architectures
   Kevin Webber
   Enterprise Architect, Lightbend
   

   View Slide

2. View Slide

3. The world by 2020
   » 4 billion connected people
   » 25+ million apps
   » 25+ billion embedded systems
   » 40 zettabytes (40 trillion gigabytes)
   » 5,200 GB of data for every person on Earth
   

   View Slide

4. View Slide

5. View Slide

6. Goals
   » ~100% availability ("nine nines")
   

   View Slide

7. Goals
   » ~100% availability ("nine nines")
   » Consistent responsiveness under varying load
   conditions
   

   View Slide

8. Goals
   » ~100% availability ("nine nines")
   » Consistent responsiveness under varying load
   conditions
   » Predictable spikes of traffic, e.g, Black Friday
   » Less predictable spikes of traffic, e.g,
   marketing campaign driving traffic
   

   View Slide

9. Goals
   » ~100% availability ("nine nines")
   » Consistent responsiveness under varying load
   conditions
   » Predictable spikes of traffic, e.g, Black Friday
   » Less predictable spikes of traffic, e.g,
   marketing campaign driving traffic
   » Enable LOBs to rapidly adapt to market conditions
   

   View Slide

10. walmart.ca
    » Added Play in front of
    customer experience
    » API enabled heritage
    architecture for scale
    » Added Akka in front of
    back-office
    » Circuit breakers for
    resilience
    » Async message-passing
    for parallelism
    

    View Slide

11. Walmart's Business Uplift
    » Conversions up 20%
    » Mobile orders up 98%
    » No downtime Black Friday or Boxing Day
    

    View Slide

12. Walmart's Operational Savings
    » Moved off expensive hardware
    » On cheap virtual x86 servers
    » 20% - 50% cost savings
    » ~ 40% compute cycles
    

    View Slide

13. Heritage challenges
    

    View Slide

14. Challenges of scale
    Heritage solution:
    » Replicate stateless servers in front of middlware
    » Push out state to the database tier
    Reality:
    » The database becomes a single point of failure
    

    View Slide

15. Challenges of resilience
    Heritage solution:
    » DR, active/passive
    Reality:
    » No graceful way to handle cascading failure in a
    heritage system without downtime
    

    View Slide

16. Challenges in development
    Heritage solution:
    » Hire more developers!
    Reality:
    » ...
    

    View Slide

17. heritage
    applications
    » Single points of failure
    » Difficult to scale out (and
    back in) due to state
    » Different non-functional
    requirements between
    logical boundaries
    » Long, risky release cycles
    

    View Slide

18. “Architecture is
    hard enough
    without adding
    unnecessary
    complexity.”
    Ross Mason, CTO, MuleSoft
    

    View Slide

19. View Slide

20. SOA
    Stateless:
    messaging, message exchange patterns (sync req/resp,
    async p2p messaging, pub/sub, streams), routing,
    service orchestration (service aggregation), security
    (encryption and signing), monitoring, protocol
    conversion, legacy adapters, payload transformation,
    schema validation, governance (the ability to apply
    business rules uniformly), service catalog
    

    View Slide

21. SOA
    Stateful:
    process choreography (BPM), complex event processing
    (CEP), reliable delivery, transaction management,
    split and merge (conflating, windowing, ordering)
    

    View Slide

22. heritage
    systems
    » Productivity depends on
    working within the system
    context
    » Creates difficult
    productivity barriers
    » E.g, systems-level
    exploratory development
    on a laptop?
    

    View Slide

23. "Jack of all trades, master of none,
    often times better than the master of one."
    Give up control to gain flexibility.
    Embrace new patterns to achieve balance.
    

    View Slide

24. Solution
    

    View Slide

25. We need to build these types of systems.
    asynchronous, non-blocking, real-time, highly-
    available, loosely coupled, scalable, fault-tolerant,
    concurrent, reactive, event-driven, push instead of
    pull, distributed, low latency, high throughput
    

    View Slide

26. View Slide

27. View Slide

28. Reactive
    A maturity model for modern
    distributed systems.
    » Responsive
    » Resilient
    » Elastic
    » Message-driven
    

    View Slide

29. Principles
    Reactive:
    » Responsive
    » Resilient
    » Elastic
    » Message-driven
    

    View Slide

30. Strategy
    Microservices:
    » Domain Driven Design (DDD)
    » Bounded contexts
    » Process isolation
    » Distributed systems
    » Event sourcing
    » CQRS
    » Eventual consistency
    

    View Slide

31. Tools
    Lightbend Reactive Platform:
    » Lagom
    » Play
    » Akka
    » Spark
    

    View Slide

32. The Lightbend
    Platform
    

    View Slide

33. Overview
    » Lagom - Microservices
    framework
    » Play - RESTful API
    framework
    » Akka - Distributed
    computing toolkit
    » Spark - General purpose
    compute engine
    

    View Slide

34. View Slide

35. Lagom
    

    View Slide

36. Lagom
    » Core abstract are
    Persistent Entities
    » They move around the
    cluster automatically
    » Can replay all state
    changes
    » One command to launch all
    services in a developer
    environment
    

    View Slide

37. Play
    

    View Slide

38. Play
    » Familiar MVC paradigm
    » Embraces flows of data
    » WebSockets, SSE
    » Reactive Streams
    » File uploads
    » Integration with Akka
    » Distribute work via
    messaging
    

    View Slide

39. Akka
    

    View Slide

40. Actor model
    » Actors
    » Conceptually they are
    very lightweight threads
    (~300 bytes each) that
    communicate with each
    other by messaging
    » Inspired by Erlang
    

    View Slide

41. View Slide

42. Spark
    

    View Slide

43. View Slide

44. View Slide

45. Fast Data
    Architecture
    » Akka Streams
    » < 1 second latency per-
    event processing
    » Spark Streaming
    » 1-10+ second latency for
    mini-batch event
    processing
    » Kafka captures the complete
    event stream
    

    View Slide

46. A reactive
    microservices
    architecture
    

    View Slide

47. Microservices
    » Single responsibility
    » Independently deployable,
    scalable, resilient
    » Communicate via async
    messaging
    » Own their data
    » Don't expose a public API
    

    View Slide

48. API gateway
    » Authentication,
    authorization
    » Quality of service
    » Protocol conversions
    » Well-documented (self-
    documented) API
    » Completely stateless
    

    View Slide

49. Services &
    integration
    » Stateful services
    » Async messaging within
    service boundaries
    » Publish/subscribe rather
    than request/response
    between services
    » A system made up entirely
    of commands, queries, and
    events
    

    View Slide

50. Kafka
    » Publish-subscribe messaging
    rethought as a distributed
    commit log
    » The event stream becomes
    the database
    » Distributed and replicated
    for resilience
    » Ordering and at-least once
    delivery
    

    View Slide

51. Clustered
    microservices
    » Responsive to events, load,
    failure, users
    » Clusters are resilient
    » Across nodes, regions,
    data centers
    » Actors can fail on one
    node and come back on
    another node, with state
    intact
    

    View Slide

52. booth 317
    Reactive Microservices
    Architecture:
    » Go deeper into the theory
    behind microservices
    » Learn more about the new
    architectural patterns
    required for a successful
    microservices project
    » Come by the booth and pick
    up a copy!
    

    View Slide

53. View Slide

54. Thank you!
    Visit https://www.lightbend.com to get started
    Get in touch:
    Kevin Webber, Enterprise Architect, Lightbend
    Email: kevin.webber@lightbend.com
    Medium: medium.com/@kvnwbbr
    Twitter: @kvnwbbr
    Booth: 317
    

    View Slide