Deploying large-scale, serverless and asynchronous systems - without integrated tests

Modern distributed architectures are increasingly composed of large numbers of decoupled, asynchronous components. In AWS, these components are plumbed together via services like SQS, SNS, Kinesis and S3 often integrated via small and frequent numbers of microservices or lambdas. But how do you test these architectures if they are cloud native?

It’s 2018, and we can do better than deploying the entire stack and running a battery of E2E tests against them.

In his talk, Matt will demonstrate how you can scale development of large-scale systems across teams, technology and process, and unlock the agility of your cloud-native architecture.

WARNING: there will be code (https://github.com/mefellows/serverless-testing-example)

Transcript

1. Deploying large-scale, serverless and asynchronous systems Matt Fellows @matthewfellows ...without integrated tests!
2. Architecture
3. 3 Evolution Architecture
4. 4 Release Architecture > Evolution More Evolved Time Monolith SOA Serverless Microservices Testing Culture Unit tests, TDD, BDD Contract tests Waterfall Outsourcing API Testing Scenario Tests Fn Not all primitives are invisible! Co-evolved practices: XP Agile\Lean Multiple E2E Test Env. Chaos Engineering DevOps CD CI
5. 5 If you can’t build a good monolith you shouldn’t be doing microservices Architecture > Microservices
6. Entry criteria to make µServices • Ability to write a modular monolith • Strong grasp on your domain model • Ability to design loosely coupled services ◦ = state + transactions • Ability to scale and decouple teams • Ability to deploy continuously • A strong “DevOps” culture • Be comfortable with decentralised, complex systems + uncertainty Architecture > Microservices
7. ...the µServices ride • Ability to write a modular monolith • Strong grasp on your domain model • Ability to design loosely coupled services ◦ = state + transactions • Ability to scale and decouple teams • Ability to deploy continuously • A strong “DevOps” culture • Be comfortable with decentralised, complex systems + uncertainty Architecture > Microservices
8. ...the µServices ride • Ability to write a modular monolith • Strong grasp on your domain model • Ability to design loosely coupled services ◦ = state + transactions • Ability to scale and decouple teams • Ability to deploy continuously • A strong “DevOps” culture • Be comfortable with decentralised, complex systems + uncertainty Architecture > Microservices
9. ...the µServices ride • Ability to write a modular monolith • Strong grasp on your domain model • Ability to design loosely coupled services ◦ = state + transactions • Ability to scale and decouple teams • Ability to deploy continuously • A strong “DevOps” culture • Be comfortable with decentralised, complex systems + uncertainty Architecture > Microservices
10. WHAT WE ARE KNOWN FOR 10 Microservices Evolution Drivers for levelling up • Increase time-to-value • Increase utilisation / cost reduction • Raise abstraction; focus on composition • Scale product teams • Massively distributed, real-time apps • API contracts (e.g. OAS) • Ubiquity of cloud computing • Rise of containers • Unreliable networking / components • Rise of the Service Mesh • Improved observability Architecture > Microservices
11. Functions + Serverless
12. 12 Architecture > Functions + Serverless (Cloud Native) https://aws.amazon.com/retail/data-analytics/
13. 13 https://www.slideshare.net/AmazonWebServices/new-launch-data-driven-apps-with-graphql-aws-appsync-deep-dive-mbl402-reinvent-2017 Architecture > Functions + Serverless (Cloud Native)
14. 14 https://aws.amazon.com/blogs/big-data/analyze-a-time-series-in-real-time-with-aws-lambda-amazon-kinesis-and-amazon-dynamodb-streams/ Architecture > Functions + Serverless (Cloud Native)
15. 15 https://aws.amazon.com/blogs/big-data/automating-analytic-workflows-on-aws/ Architecture > Functions + Serverless (Cloud Native)
16. 16 https://www.slideshare.net/nateware/aws-gdc2015-gameanalyticsfinal Architecture > Functions + Serverless (Cloud Native)
17. 17 Architecture > Functions + Serverless (Cloud Native) https://aws.amazon.com/retail/data-analytics/
18. 18 Much Compose! So speed! Architecture > Functions + Serverless (Cloud Native) Architecture > Functions + Serverless (Cloud Native)
19. 19 BUT. Architecture > Functions + Serverless (Cloud Native)
20. 20 If you can’t build good microservices you shouldn’t be doing serverless Architecture > Functions + Serverless (Cloud Native)
21. 21 If you can’t build a good Fn you shouldn’t be doing FaaS Architecture > Functions + Serverless (Cloud Native)
22. 22 So, what practices do we need? Architecture > Functions + Serverless (Cloud Native)
23. AWS Meetup Lounge
24. 24 Demo: AWS Meetup Lounge CloudFront API Gateway Authentication You! awslounge.onegeek.com.au Twitter Fetch Twitter feed Twitter Push IoT Message Broker Sentiment Analysis Checkpoint Amazon ML #awsmelb
25. Challenges w\ Serverless Blasphemy!
26. 26 Vendor lock-in Challenges > Vendor lock-in
27. 27 https://twitter.com/alambert/status/958050456101642241 Challenges > Vendor lock-in
28. WHAT WE ARE KNOWN FOR 28 Options • Multi-cloud = lowest common denominator • There is always a vendor (even if it’s you!) • You’ll never be completely vendor agnostic • Multi-cloud abstraction • Avoid altogether • Structure your code better Challenges Challenges > Vendor lock-in
29. WHAT WE ARE KNOWN FOR 29 Options • Multi-cloud = lowest common denominator • There is always a vendor (even if it’s you!) • You’ll never be completely vendor agnostic • Multi-cloud abstraction • Avoid altogether • Structure your code better Challenges Challenges > Vendor lock-in
30. Lambda Handler Challenges > Vendor lock-in > Lambda Anti-pattern { "Records": [ { "EventVersion": "1.0", "EventSubscriptionArn": "somearn", "EventSource": "aws:sns", "Sns": { ... } } ] } { "Success": true }
31. MQ Handler WebSocket handler MQ Adapter WS Adapter Azure Function Adapter Google Cloud Fn Adapter Azure Function Handler Google Cloud Fn Handler Lambda Handler Lambda Adapter Service Repository Adapter (Anti-corruption layer) Ports Business Logic Collaborators ... Challenges > Vendor lock-in > Ports and Adaptors
32. 32 Challenges > Vendor lock-in > Ports and Adaptors > Sentiment Analysis Example Twitter feed Sentiment Analysis IoT Message Broker
33. Challenges > Vendor lock-in > Ports and Adaptors > Lambda Port
34. Challenges > Vendor lock-in > Ports and Adaptors > SNS Adapter
35. Challenges > Vendor lock-in > Ports and Adaptors > Lambda Sentiment Service
36. Summary Architecture > Microservices • Separate protocol handling from business logic • Business logic shouldn’t change with introduction of a new protocol • Port, Adapter and Business Logic are independently testable
37. 37 How do you test locally? Challenges > Testing Locally
38. WHAT WE ARE KNOWN FOR 38 Options Challenges > Testing Locally • Stub services (e.g. Localstack) • Stub API (e.g. Moto) • Multiple handlers • Unit tests
39. 39 Challenges > Testing Locally
40. WHAT WE ARE KNOWN FOR 40 Options • Feature arms race • Trustworthy? • Integration style = slower • All cloud providers? • Stub services (e.g. Localstack) • Stub API (e.g. Moto) • Multiple handlers • Unit tests Challenges > Testing Locally Challenges
41. WHAT WE ARE KNOWN FOR 41 Writing good unit tests • Follow test pyramid • Stub out AWS APIs • Test your business logic separately from any serverless features (e.g. lambda handler) • Callback pyramid of death • Tests are hard to write • Your business logic knows about AWS • You have to stub multiple services per test • Large methods • Confusing control flow Challenges > Testing Locally > Unit Test Code Smells
42. 42 Challenges > Testing Locally > Unit Test > Twitter Fetch Checkpoint Twitter Fetch
43. Challenges > Testing Locally > Unit Test > Lambda Service Look ma, no AWS Code
44. Challenges > Testing Locally > Unit Test > Lambda Service Look ma, no AWS Code
45. 45 Challenges > Testing Locally > Unit Test > Sentiment Analysis Example Twitter feed Sentiment Analysis
46. Integration-style test of handler Challenges > Testing Locally > Unit Test > Lambda Handler
47. Summary Architecture > Microservices • Business logic should be tested separately from port + adapter • Majority of tests, should be fast unit tests • Stub AWS Services where required • (Lambda) Handlers can be locally integration tested with pre-canned JSON events
48. 48 Async - who is my consumer? Challenges > Async
49. 49 Async - delay in feedback Challenges > Async
50. WHAT WE ARE KNOWN FOR 50 Options • Local integration testing • Cloud env. integration testing • Contract testing Challenges > Async
51. WHAT WE ARE KNOWN FOR 51 Options • Can’t run complete local environment • Many moving parts • Problems with integration testing... • Local integration testing • Cloud env. integration testing • Contract testing Challenges Challenges > Async
52. WHAT WE ARE KNOWN FOR 52 Options • Can’t run complete local environment • Many moving parts • Problems with integration testing... • Local integration testing • Cloud env. integration testing • Contract testing Challenges Challenges > Async
53. 53 Contract tests Challenges > Async
54. 54 IoT Message Broker Sentiment Analysis Contract { 'Positive': 0.234, 'Negative': 10.123, 'Neutral': 7.334, 'Mixed': 0 } Consumer Provider Challenges > Async > Contract Tests
55. 55 CloudFront API Gateway Authentication You! awslounge.onegeek.com.au Twitter Fetch Twitter feed Twitter Push IoT Message Broker Sentiment Analysis Checkpoint Amazon ML #awsmelb Challenges > Async > Contract Tests
56. 56 Authentication Twitter Fetch Twitter feed Twitter Push IoT Message Broker Sentiment Analysis Checkpoint #awsmelb Challenges > Async > Contract Tests Contract Contract Contract Contract Contract Contract Contract Contract
57. WHAT WE ARE KNOWN FOR 57 Problems with automated e2e tests • Slow • Easy to break • Hard to fix • Scales badly across teams • Lots of set up \ maintenance • $$ potentially costly Challenges > Async > Contract Tests
58. 58 C P mock Challenges > Async > Contract Tests Understanding Test Symmetry
59. WHAT WE ARE KNOWN FOR 59 Test symmetry Solved problems New problems • Hard to keep both sides in sync • Fast feedback • Few dependencies • No dedicated environment • Reliable • Easy to debug Challenges > Async > Contract Tests
60. 60 C P mock Challenges > Async > Contract Tests
61. 61 C P mock Challenges > Async > Contract Tests
62. P P C C mock contract Challenges > Async > Contract Tests
63. Bug turnaround - minutes Contracts tests Challenges > Async > Contract Tests
64. Know before you commit Contracts tests Challenges > Async > Contract Tests
65. Make changes with speed and confidence Contracts tests Challenges > Async > Contract Tests
66. Deploy independently Contracts tests Challenges > Async > Contract Tests
67. Better API design (Consumer) Contracts tests Challenges > Async > Contract Tests
68. 68 • Open source • Multiple languages ◦ Javascript ◦ Go ◦ Python ◦ JVM ◦ .NET ◦ + more pact.io • HTTP contracts • Message contracts
69. How to be taller ...and ride that Fn
70. Summary 70 Summary Problems Solutions • Microservices+ • Create cloud and protocol anti-corruption layer • Mock out AWS SDK in unit tests • Use contract tests • Reduce reliance on end-to-end environments • Ad-hoc exploratory testing • Serverless complexity > uServices • Testing cloud native services • Avoiding vendor lock-in • Handling asynchronous Fn’s
71. Deploying large-scale, serverless and asynchronous systems Matt Fellows @matthewfellows ...without integrated tests!