Control the Chaos - testing akka-based applications against persistence failures

Akka doesn't guarantee message delivery, especially for remoting or persistent actors.

During this talk, you will learn how a very simple idea of how to test with minimal effort if your Akka application is resilient in case of persistence failures. Chaos testing is very broad and complex topic. If you would like to test your application against one specific point of failure, it may be too expensive to use standard chaos testing techniques. In case of persistence failures you will discover how to use "ChaosJournal" - journal that randomly fails.

You will explore a simple case study, practical advices, ideas and pitfalls. The talk will be about testing, not how to prevent loosing messages itself. Similar techniques may be used for testing against eg. remoting failures.

Transcript

1. Control the Chaos Piotr Śliwa 
 @psliwa
2. The story https://commons.wikimedia.org/wiki/File:Tell_Me_a_Story_130920-M-TH017-014.jpg
3. The story 1. Async API, lack of persistence 

4. The story 1. Async API, lack of persistence 
 https://pixabay.com/en/baby-boy-diaper-blue-smile-crawl-306799/
5. The story 2. Introduce persistence https://pixabay.com/en/baby-girl-neutral-child-cute-kid-528887/
6. The story 3. Database issues - loosing messages https://pixabay.com/en/explosion-fireball-fire-brand-123690/
7. The story 4. Introduce AtLeastOnceDelivery http://www.afcec.af.mil/News/Photos/igphoto/2000917976/
8. The story 4. Introduce AtLeastOnceDelivery http message AtLeastOnce Delivery The business logic https://pixabay.com/en/email-letter-send-envelope-mail-297632/
9. The story 4. Introduce AtLeastOnceDelivery http message AtLeastOnce Delivery The business logic
10. The story 4. Introduce AtLeastOnceDelivery http message AtLeastOnce Delivery The business logic HTTP 200 https://pixabay.com/en/tick-mark-ok-perfect-check-done-305245/
11. The story 4. Introduce AtLeastOnceDelivery http message AtLeastOnce Delivery The business logic
12. The story 4. Introduce AtLeastOnceDelivery http message AtLeastOnce Delivery The business logic HTTP 500
13. The story 5. Still loosing messages... https://www.flickr.com/photos/kaibabnationalforest/4479870532
14. The story 6. Introduce persistence fault tolerance tests https://pixabay.com/en/fire-bonfire-dancing-people-2755499/
15. Testing
16. Testing: Chaos Testing
17. Testing: Chaos Testing https://pixabay.com/en/animal-fly-insect-1295030/
18. Testing: Chaos Testing https://pixabay.com/en/cannon-artillery-weapon-military-309152/ https://pixabay.com/en/animal-fly-insect-1295030/
19. Testing: Chaos Journal Simulate random persistence failures during E2E tests

20. Testing: Chaos Journal Simulate random persistence failures during E2E tests
 https://pixabay.com/en/animal-fly-insect-1295030/
21. Testing: Chaos Journal Simulate random persistence failures during E2E tests
 https://pixabay.com/en/swatter-insecticide-fly-insect-147936/ https://pixabay.com/en/animal-fly-insect-1295030/
22. Testing: Chaos Journal Simulate random persistence failures during E2E tests
 https://pixabay.com/en/animal-fly-insect-1295030/
23. Chaos Journal Basic Chaos Journal implementation used in Akka Persistence tests
24. Chaos Journal Basic Chaos Journal implementation used in Akka Persistence tests 
 then Wrapper for akka-persistence-inmemory plugin
25. Chaos Journal Basic Chaos Journal implementation used in Akka Persistence tests
 then Wrapper for akka-persistence-inmemory plugin
 1. Random persistence failures (reads, writes) 2. Random persistence latency (writes) 3. Persistence Query support
26. Tests preparation
27. Tests preparation Problem 1: the application sometimes responded 500 error in tests
28. Tests preparation Problem 1: the application sometimes responded 500 error in tests Solution: retry policy in E2E test for http messages
29. Tests preparation Problem 2: some messages were allowed to be lost because eg. they were sent periodically
30. Tests preparation Problem 2: some messages were allowed to be lost because eg. they were sent periodically Solution: repeat such messages periodically while asserting the result
31. Tests preparation Problem 3: unstable tests because circuit breakers open, backoff supervisors work too slow...
32. Tests preparation Problem 3: unstable tests because circuit breakers open, backoff supervisors work too slow... Solution: tune backoff supervisors, retries interval, circuit breakers etc. in order to make test stable
33. Tests preparation Problem 4: Chaos Journal tests were few times slower than naked tests
34. Tests preparation Problem 4: Chaos Journal tests were few times slower than naked tests Solution: run tests in parallel in the pipeline
35. Tests preparation Problem 5: Tests were not deterministic, even if there was a bug - they could pass
36. Tests preparation Problem 5: Tests were not deterministic, even if there was a bug - they could pass Solution: run them hundred times during night build
37. The epilogue 7. A lot bugs discovered and fixed with low cost, more confidence to application resiliency https://pixabay.com/en/green-forest-beech-trees-wood-2830942/
38. Demo 
 
 https://github.com/psliwa/akka-persistence-chaos-journal
39. Thanks! Q&A? Links: • https://goo.gl/BnC39D - ChaosJournal from Akka Persistence test sources • https://github.com/psliwa/akka-persistence-chaos-journal Piotr Śliwa 
 @ psliwa