Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Chaos Gamedays for training Engineering Teams

Chaos Gamedays for training Engineering Teams

Yury Nino

March 11, 2021
Tweet

More Decks by Yury Nino

Other Decks in Technology

Transcript

  1. If you know the enemy and know yourself, you need

    not fear the result of a hundred battles … The Art of War. Sun Tzu
  2. Black Swans 1. The event is a surprise. 2. The

    event has a major effect. 3. After the first recorded, it is rationalized by hindsight!
  3. “Don't worry about the future. Or worry, but know that

    worrying is as effective as trying to solve an algebra equation by chewing a bubble gum. The real troubles in your life are things that never crossed your worried mind, the kind that blindside you at 4 p.m. on some idle Tuesday" Mary Schmich
  4. Security Attacks Denial of Service Phishing Man in Middle Attacker

    overwhelms a system’s resources so that it cannot respond to service requests. DoS doesn’t provide direct benefits for attackers! Attacker hijacks a session between a trusted client and network server. Session hijacking, IP spoofing and replay! Attacker sends emails that appear to be from trusted sources to gain access. Social engineering and Technical trickery. https://blog.netwrix.com
  5. Attacks Attacker executes a SQL query via an input data

    from the client to server. “SELECT * FROM users WHERE account = ‘’ or ‘1’ = ‘1’;” Attacker uses third-party web resources to run scripts in browsers or applications. Steal cookies, keystrokes and collect information. Attacker installs malicious software in the system without consentment of the owner. File infectors, trojans, worms, ransomware. https://blog.netwrix.com Security SQL Injection Malware Cross-Site Scripting
  6. Cyberwar is everywhere! In the media, in the military, among

    politicians and in academia. https://www.yurynino.com/
  7. The World is Chaotic! and Insecure Black swans take our

    systems down and keep them down for a long time. Laura Nolan, SRE in Slack
  8. What is Chaos Engineering? It is the discipline of experimenting

    failures in production in order to reveal their weakness and to build confidence in their resilience capability. https://principlesofchaos.org/
  9. What is Security Chaos Engineering? It is the identification of

    security control failures through proactive experimentation to build confidence in the system’s ability to defend against malicious conditions in production. Chaos Engineering Book. 2020
  10. History 1986 Artificial Immune Systems 2008 Chaos Engineering was born

    2018 2020 Chapter dedicated to Security CE 2019 Aaron Rinehart first articles Artificial Intelligence for data security
  11. Principles Chaos Engineering Principles Injecting failure to achieve resilience! Hypothesize

    about Steady State Run Experiments Vary Real-World Events Automate Experiments
  12. 1. Pick a Hypothesis: Recipe! 2. Choose the tools: Ingredients!

    3. Launch an attack: Cook! 4. Notify the Org: Invite! 5. Run the Experiment: Enjoy! 6. Analyze the Results 7. Automate Chaos Principles
  13. More Chaos Security Engineering With Security Chaos Engineering we can

    introduce false positives into production, to check whether procedures are capable of identifying security failures under controlled conditions.
  14. Human factors in cybersecurity are perhaps the biggest challenge when

    building an effective threat prevention strategy. Vircom
  15. What my mom thinks I do What my friends thinks

    I do What software engineers think I do What I really do Who is a Security Chaos Engineer? Help service owners to increase their security and resilience through education, tools and encouragement.
  16. Red Team Exercises • They were originated with the US

    Armed Forces by Bryce Hoffman. • Adversarial approach that imitates the behaviors and techniques of attackers in the most realistic way possible. • Two common forms of Red Teaming seen in the enterprise are: • Ethical hacking • Penetration testing. • Blue Teams are the defensive counterparts to the Red teams in these exercises. • Recommendations: Think-Write-Share! https://whatis.techtarget.com Training
  17. Purple Team Exercises • They were intended as an evolution

    of Red Team exercises by delivering a more cohesive experience between the offensive and defensive teams. • The “Purple” in Purple Teaming reflects the cohesion of Red and Blue Teaming. • The goal of these exercises is the collaboration of offensive and defensive tactics to improve the effectiveness of both groups in the event of an attempted compromise. • The intention is to increase transparency as well as provide a conduit for the security apparatus to learn about how effective their preparation is when subjected to a live fire exercise. https://whatis.techtarget.com Training
  18. Chaos GameDays GameDays are an interactive, real-world and learning exercises.

    They are designed to give players a chance to put their skills in a technology to test. GameDays were created by Jesse Robbins inspired by his experience & training as a firefighter.
  19. Before After During • Pick a hypothesis. • Pick a

    style. • Decide who. • Decide where. • Decide when. • Document. • Get approval! • Detect the situation. • Take a deep breath. • Communicate. • Visit dashboards. • Analyze data. • Propose solutions. • Apply and solve! • Write a postmortem. • What Happened • Impact • Duration • Resolution Time • Resolution • Timeline • Action Items Gameday Methodology
  20. First on Call Monitors, triages, and tries to mitigate failures

    caused by the Master of Disaster. Master of Disaster Decides the failure and declares start of incident and attack!!! Team Find and solve the exhibited issues, and write up postmortem. Chaos Roles
  21. ChaoSlingr • Serverless app in AWS. • Written in Python.

    • 100% Native in AWS. • Configuration as a Code. • Configurable Operational Mode. • Open Framework. • With example codes. Tools
  22. • Introduce latency on security controls. • Drop a folder

    like a script would do in production. • Software secret clear text disclosure. • Permission collision in a shared IAM role policy. • Disable service event logging. • API gateway shutdown. • Unencrypted S3 Bucket. • Disable MFA. https://www.yurynino.dev/ Experiment
  23. Hypothesis: After the owner of Root account in AWS left

    the company, we could use our cloud in a normal way. Result: Hypothesis disproved. In this experiment the access to AWS was connected to the Active Directory. When an employee left the company his account is dropped and we lost the access to AWS. Side Effect: Thinking in this scenario allows to consider another applications connected to Active Directory. https://www.yurynino.dev/ Experiment
  24. As Henry Ford said, "Failure is only the opportunity to

    begin again, this time more intelligently." Security Chaos Engineering and Security Chaos Testing give us that opportunity. Taken from DevOpsSec by Jim Bird
  25. Who