Simulant in Anger; An Experience Report

As a community, we talk a lot about generative testing, especially in the small. What happens when we apply these approaches at the systems level? (Hint, it's Simulant!) In this talk, we'll explore how we applied Simulation Testing to a modern financial system to prepare it for it's big, initial launch. At the end, you'll have an idea what Simulation Testing is capable of, and how to begin entertaining applying it to your own systems.

Transcript

1. Simulant in Anger An Experience Report
2. Ryan Neufeld @rkneufeld – rkn.io Ryan Neufeld @rkneufeld – rkn.io
5. A Talk, in Three Parts • What is MI-X? • What is Simulation Testing? • What did we learn from it?
7. MI-X Value-added Services Acquirers Issuers Merchants Consumers
8. MI-X Value-added Services Acquirers Acquirer API Tx Engine Auth API VAS API Issuers
9. Landing on Mars
10. mendhak • flic.kr/p/drgWYD
11. Big Launch 100s Stores 1000s of Tellers
12. Big Launch No Second Chances
13. What is success?
14. What is success? 1. Speed of Delivery
15. 3.5
16. What is success? 2. Consistent Speed
17. What is success? 3. Scalability
18. mendhak • flic.kr/p/drgWYD
19. Simulating Reality
20. But how? ab? JMeter? Selenium? siege? httperf? integration tests?
21. Enter Simulation Testing
22. Quintessential Black-box ?
23. So… Integration Tests? Not Really.
24. Integration
25. Simulation
26. Validate… Correctness
27. Explore Correctness
28. Well-functioning System + ? Features
29. Well-functioning System + Qualities Features
30. Validate… Qualities
31. Enter Simulant
32. Simulant A Framework for Simulation Testing
33. A Test in Four Parts* Model Test Sim Validate * Well, four big ones…
34. Model A C B D E 75% 25% 100% 75% 25% 100%
35. Test A B D E B A C B D E 75% 25% 100% 75% 25% 100%
36. A B D E B A B D E B Test A B D E B A C B D E 75% 25% 100% 75% 25% 100%
37. Sim A B D E B ✓ ✓ ? A C E D ✓ ✓ ✗ ✗ A B D E ✓ ✓ ✓ ? Setup Teardown Action Log
38. Validation Action Log Correct? No Errors? No Slow Queries? Valid Run? … … ✓ ✓ ✓ ✓ ✓
39. Simulant x MI-X
40. Step One Goals
41. Gather a Baseline
42. Incrementally Improve
43. 0 → [simple] 1 week
44. [simple] → [advanced] 1 Month
45. Why so slow? Bill Lopp • flic.kr/p/5hAi7Q
46. –Someone on Clojure “Hard to learn, but it pays you back.”
47. And what did we find…
48. Finding #0 “Did this ever work?”
49. Finding #1 “It’s bogged down”
50. Finding #2 “Auto-scaling don’t”
51. Finding #3 “Create’s got it bad”
52. Under Pressure Tx Acq Auth VAS
53. Encrypted Payloads As a Token JWTs
54. Mean Response Time 2x Improvement
55. Max Response Time 3x Improvement
56. Reduced Errors approx. 75% → 0%
57. Overall 4.5x Throughput 30 TPM Constant up to 170 TPM Proportional
58. What else?
59. Eroded False Confidences
60. Gained True Confidence
61. A Tool For Change
62. In Closing…
63. Fantastic for Qualities
64. Is Simulation Testing for you?
65. High Risk necessitates High Confidence
66. Reputation Money Lives…
67. Moderate Risk necessitates High Confidence
68. Unsession!
69. Thanks! • Slides: bit.ly/angry-sim • homegrown.io/ask-me-anything • Learn more via our mailing list @homegrownio homegrown.io