Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Simulant in Anger; An Experience Report

Simulant in Anger; An Experience Report

As a community, we talk a lot about generative testing, especially in the small. What happens when we apply these approaches at the systems level? (Hint, it's Simulant!) In this talk, we'll explore how we applied Simulation Testing to a modern financial system to prepare it for it's big, initial launch. At the end, you'll have an idea what Simulation Testing is capable of, and how to begin entertaining applying it to your own systems.

58dc7342c17dbc3a0848fe78e5b6d3a4?s=128

Ryan Neufeld

April 20, 2015
Tweet

Transcript

  1. Simulant in Anger An Experience Report

  2. Ryan Neufeld @rkneufeld – rkn.io Ryan Neufeld @rkneufeld – rkn.io

  3. None
  4. None
  5. A Talk, in Three Parts • What is MI-X? •

    What is Simulation Testing? • What did we learn from it?
  6. None
  7. MI-X Value-added Services Acquirers Issuers Merchants Consumers

  8. MI-X Value-added Services Acquirers Acquirer API Tx Engine Auth API

    VAS API Issuers
  9. Landing on Mars

  10. mendhak • flic.kr/p/drgWYD

  11. Big Launch 100s Stores 1000s of Tellers

  12. Big Launch No Second Chances

  13. What is success?

  14. What is success? 1. Speed of Delivery

  15. 3.5

  16. What is success? 2. Consistent Speed

  17. What is success? 3. Scalability

  18. mendhak • flic.kr/p/drgWYD

  19. Simulating Reality

  20. But how? ab? JMeter? Selenium? siege? httperf? integration tests?

  21. Enter Simulation Testing

  22. Quintessential Black-box ?

  23. So… Integration Tests? Not Really.

  24. Integration

  25. Simulation

  26. Validate… Correctness

  27. Explore Correctness

  28. Well-functioning System + ? Features

  29. Well-functioning System + Qualities Features

  30. Validate… Qualities

  31. Enter Simulant

  32. Simulant A Framework for Simulation Testing

  33. A Test in Four Parts* Model Test Sim Validate *

    Well, four big ones…
  34. Model A C B D E 75% 25% 100% 75%

    25% 100%
  35. Test A B D E B A C B D

    E 75% 25% 100% 75% 25% 100%
  36. A B D E B A B D E B

    Test A B D E B A C B D E 75% 25% 100% 75% 25% 100%
  37. Sim A B D E B ✓ ✓ ? A

    C E D ✓ ✓ ✗ ✗ A B D E ✓ ✓ ✓ ? Setup Teardown Action Log
  38. Validation Action Log Correct? No Errors? No Slow Queries? Valid

    Run? … … ✓ ✓ ✓ ✓ ✓
  39. Simulant x MI-X

  40. Step One Goals

  41. Gather a Baseline

  42. Incrementally Improve

  43. 0 → [simple] 1 week

  44. [simple] → [advanced] 1 Month

  45. Why so slow? Bill Lopp • flic.kr/p/5hAi7Q

  46. –Someone on Clojure “Hard to learn, but it pays you

    back.”
  47. And what did we find…

  48. Finding #0 “Did this ever work?”

  49. Finding #1 “It’s bogged down”

  50. Finding #2 “Auto-scaling don’t”

  51. Finding #3 “Create’s got it bad”

  52. Under Pressure Tx Acq Auth VAS

  53. Encrypted Payloads As a Token JWTs

  54. Mean Response Time 2x Improvement

  55. Max Response Time 3x Improvement

  56. Reduced Errors approx. 75% → 0%

  57. Overall 4.5x Throughput 30 TPM Constant up to 170 TPM

    Proportional
  58. What else?

  59. Eroded False Confidences

  60. Gained True Confidence

  61. A Tool For Change

  62. In Closing…

  63. Fantastic for Qualities

  64. Is Simulation Testing for you?

  65. High Risk necessitates High Confidence

  66. Reputation Money Lives…

  67. Moderate Risk necessitates High Confidence

  68. Unsession!

  69. Thanks! • Slides: bit.ly/angry-sim • homegrown.io/ask-me-anything • Learn more via

    our mailing list @homegrownio homegrown.io