ChaosSpark.pdf

Transcript

3. Testing High Availability In Apache Spark With Chaos Engineering February 27th 2020 Bogotá Colombia
4. 4 Yury Niño Roa DevOps Engineer | Chaos Engineering Advocate @yurynino Ingeniera de Sistemas. Especialista en Ingeniería de Software Coorganizadora de GDG Bogotá, GDG Cloud Bogotá y Women Techmakers Bogotá “A single point of failure can trigger a chain reaction across the value chain, including suppliers to the final customer, and cause a severe business interruption” Dimitar Pachov
5. How many of you have seen these messages interacting with your Bank?
6. Or these messages?
7. Agenda Why the world needs High Availability? How to achieve High Availability? Apache Spark is Highly Available The promise: Resilience Patterns Chaos Engineering Chaos Principles … Testing High Availability with Chaos!
8. Do you have clear the difference between: Fault Tolerance, High Availability & Disaster Recovery
10. High Availability Principles Component Redundancy. No Single Points of Failure. Failure Detection and Response.
11. Why Banks need High Availability?
13. Modern applications are distributed, HA and resilient by default! However … 38% of customers at traditional banks experienced disruption to their service every year, compared with 21% of challenger bank customers. The FCA says bank outages have risen 138% in the past year in the world!
16. Banking Software Systems must be distributed, reliable and resilient!
17. How to achieve that?
18. Banks are data-driven organizations that places emphasis on the quality! Take advantage of this!
20. Apache … • Spark is an analytics engine for large-scale data processing. • Spark is available for both batch and streaming data. • Spark allows to write applications in Java, Scala, Python, and SQL. • Spark makes easy to build parallel apps. • Spark combines SQL, streaming, and complex analytics. • Spark runs on Hadoop, Apache Mesos, Kubernetes, standalone, or in the cloud.
21. Apache Spark is composed by …
22. Apache Spark Cluster Mode
23. Apache Spark is Highly Available
24. Apache Spark provides HA Standby Masters with Single-Node Recovery
25. Single-Node Recovery with LFS If you just want to be able to restart the Master if it goes down, Filesystem mode can take care of it. When Applications and Workers register, they have enough state written to the provided directory so that they can be recovered upon a restart of the Master.
26. Standby Master with Zookeeper ZooKeeper provides leader election and state storage, you can launch multiple Masters in your cluster connected to the same ZooKeeper instance. One will be elected “leader” and the others will remain in standby mode. If the current leader dies, another Master will be elected, recover the old Master’s state, and then resume scheduling.
27. Standby Master with Zookeeper
28. The Promise: Resilience Patterns
29. Circuit Breaker Pattern Bulkhead Pattern Leader Election Compensating Transactions Health Endpoint Monitoring Resilience Patterns
30. Leader Election
31. How to test that Master is broken: Chaos Engineering
32. Chaos Engineering It is the discipline of experimenting in production on a distributed system in order to reveal their weakness and to build confidence in their resilience capability. https://principlesofchaos.org/
33. 2008 Chaos Engineering began at Netflix 2010 Chaos Monkey was launched 2018 A lot of resources for Chaos Engineering. 2014 Role of Chaos Engineer was created. History of Chaos Engineering Kolton Andrus
34. What my mom thinks I do What my friends thinks I do What software engineers think I do What I really do Who is a Chaos Engineer? Help service owners to increase their resilience through education, tools and encouragement.
35. Who are doing Chaos Engineering?
36. Hypothesize about Steady State Run Experiments Vary Real-World Events Automate Experiments Chaos Engineering Principles
37. Testing with Chaos
38. Netflix Chaos Experiment with Spark
39. Experiment: Hypothesis Validate that there is no interruption in computing metrics when the different Spark components fail. To simulate such failures, we employed a whack-a-mole approach and killed the various Spark components.
40. Experiment: Running
41. Netflix Chaos Results with Spark
42. How to begin ... https://chaosengineering.slack.com https://github.com/dastergon/ awesome-chaos-engineering https://www.infoq.com/chaos-engineering @yurynino
44. If we want to have Banking Systems distributed, highly available, reliable and resilient! We must be reliable and resilient! Take care of yourself!
45. Thanks for coming!!! @yurynino