Designing and Visualizing Cloud Architectures for Humans

F4612f9492cccc0c61c9b5f80d0ca8a1?s=47 jng
April 27, 2019

Designing and Visualizing Cloud Architectures for Humans

Today's software architectures are organic, scaling with cloud infrastructure and changing with agile development cycles. As software matures and teams grow, so does complexity. Combining d3.js visualizations and monitoring, we'll explore different dimensions of cloud architectures, both in design and operation to focus less on technical details and more on what matters: people.

F4612f9492cccc0c61c9b5f80d0ca8a1?s=128

jng

April 27, 2019
Tweet

Transcript

  1. Designing and Visualizing Cloud Architectures for Humans • Julie Ng

    Cloud Solution Architect Microsoft • Frontconf Munich 27 April 2019 Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 1
  2. What is Good Architecture? In the opinion of a former

    Enterprise Architect 1
  3. Hello World! ONCE UPON A TIME…

  4. Decouple Frontends! DIFFERENT UI REQUIREMENTS

  5. Backends for Frontends (BFF) DIFFERENT UX REQUIREMENTS

  6. • Decoupling has continued: • microservices, containers, serverless… what’s next?

    • on-prem à cloud à where’s my data? • Failures more complex • Debugging harder • Does it have to be this way? • WHY is it this way? Did I order this Architecture Spaghe>? IT-TRANSFORMATION AT SCALE Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 6
  7. • One that works • Has no downtime • Enterprise

    Architecture = bridging technology and business • Allianz Germany 2018 • 36.4 € Billion Revenue • 2.4 € Billion Operating Profit • 129 years old • So how much does 1 hour of downtime cost? What is a good architecture? LARGE SCALE - ENTERPRISE ARCHITECTURE COMPANY STATISTICS FROM h/ps://www.allianzdeutschland.de/die-allianz-deutschland-im-portraet/ Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 7
  8. Cloud Context The Human Element What causes DownAme? PER GOOGLE

    NEXT (YELLOW), INDEPENDENT ORACLE USER GROUP (BLUE) 3rd Party or Cloud Server Failures Storage Failures Application Errors Usage Spikes/Surges Power Outages Network Outage Human Error Bad Deployments Monitoring Gaps Scaling Boundaries Uneven Sharding ' Humans cause downtime The cloud just works… when configured properly (by humans) Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 8
  9. • Measure quality of architecture by downtime • Assume the

    cloud just works • Assume we humans caused the failure • Goal is to understand human factor in architecture • How do we behave? • Is there a pattern? • Why do we behave the way we do? The Human Factor on Architecture WHY I BUILT NEWTON.JS We’re pushing the buttons Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 9
  10. How do you Document Architecture? What did we intend to

    build? 2
  11. Architecture Diagrams CLASSIC DOCUMENTATION Diagram Source – Scalable Web Application

    Reference Architecture https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/app-service-web-app/scalable-web-app • Classic diagrams • Show intent • But very static • Not real-time. Quickly outdated. • What do arrows indicate? • Dependencies? • Data Flow? Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 11
  12. Typical Real Time Monitoring Dashboards INTERACTIVE DOCUMENTATION? • Latency •

    CPU Usage • Uptime • Storage Consumption • Memory Consumption • Nodes Count • Temperature Does this make sense for a cloud application? No. The cloud just works, remember? Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 12
  13. • For 10.000 feet overview - too much information •

    Missing relationships – remember those arrows? • To understand how everything is connected, we have to go return to the humans… Monitor uptime for hundreds of components? JUST UPTIME Graphic Source https://community.grafana.com/t/status-panel-for-more-than-200-servers/2710 Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 13
  14. OrganizaAons in the Cloud The people behind the code 3

  15. What we imagine – 1 team ORGANIZATIONS – UNIFORM AND

    STANDARDIZED IN LARGE OPEN SPACE OFFICES Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 15
  16. Reality – many chaoAc teams, but diverse ORGANIZATIONS So do

    you really expect 1 architecture? Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 16
  17. Conway’s Law BEHAVIORAL PSYCHOLOGY “ Organizations which design systems... are

    constrained to produce designs which are copies of the communication structures of these organizations. - Melvin Conway. "How Do Committees Invent?” Datamation (April 1968). So despite independent teams, is there only a single possible overall result? Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 17
  18. • Installed on base images so every team uses them

    – easy win? • Large organizations and systems à too much noise • Arrows indicate request flow, but not intent - the relationships we wanted to build • How can we indicate intent? Automated graphs with trace agents AUTOMATE RELATIONSHIPS Example automated graph Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 18
  19. Junior designers… make it look good JULIE ZHOU, VP PRODUCT

    FACEBOOK ON GOALS https://medium.com/the-year-of-the-looking-glass/junior-designers-vs-senior-designers-fbe483d3b51e Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 19
  20. Senior designers… make new value JULIE ZHOU, VP PRODUCT FACEBOOK

    ON GOALS https://medium.com/the-year-of-the-looking-glass/junior-designers-vs-senior-designers-fbe483d3b51e Before we automate, let’s create. How can we reintroduce intent to real time monitoring? Remember, that’s how we choose to measure good architecture. Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 20
  21. Visualizations with d3.js Data Driven Transformations on the DOM 4

  22. • There are no loops over the DOM • Based

    on data() • selectAll() does it all: • Adds Circles • Updates Circles • Removes Circles Data Driven – describe what, not how D3.JS - NO FOR LOOPS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 22
  23. • Nodes array • IDs must be unique • Links

    Array • source & target reference IDs • References are key based, not Array index based #PROTIP Source and targets are not array index based – although d3.js documents and tutorials say they are. Example Data D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 23
  24. General Update Pattern D3.JS DATA DOM ELEMENTS Enter Exit Update

    Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 24
  25. • General Update Pattern • nodes.data(data) // join • nodes.exit()

    • nodes.enter() • Nodes.merge() // update Nodes D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 25
  26. • selection.data(data, (d) => d.id) // does not work for

    links * Links, Key Functions D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 26
  27. • Default Force, Particles Engine • Too uniform? D3.js Force

    Engine D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 27
  28. Webcola Force Layout D3.JS Frontconf - 27 April 2019 Designing

    and Visualizing Cloud Architectures for Humans 28
  29. • D3 Layout iterations: tick • Custom events via EventEmitter

    • this.emit('update') • Standard events • mouseover, mouseout • dragstart • etc. Event Driven D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 29
  30. • Runtime vs. State • Transactions Visualize Dependencies – intent!

    D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 30
  31. Live Demo! Newton.js Cloud Architecture Graphs for Humans https://github.com/julie-ng/newtonjs-graph Frontconf

    - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 31
  32. • Follow Me! • julie.io • github.com/julie-ng • twitter.com/jng5 •

    Credits - all stock photos are from amazing photographers on Unsplash https://unsplash.com/collections/4638940/frontconf-talk-cloud-architectures- visualizations/98850e9cfebb9cc0d350ce787e6986e4 • Note - all opinions are my own and do not reflect those of my employers past and present. Thank you! BYE BYE Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 32