A holistic approach for testing serverless architectures

One of the peculiar aspects of microservices architecture is the possibility of designing a system as a set of independent but collaborative components. At the root of this collaboration there are well-defined contracts among all the microservices: breaking even a single contract might compromise the health of the entire system. With AWS Lambda, we have the ability to simply create an ever-increasing number of microservices and, therefore, it becomes crucial to find a way to catch errors in advance whenever one of these contracts has been broken.

Transcript

1. Beyond the unit: a holistic approach for testing serverless architectures Mariano Calandra – XPeppers (a Claranet group company) www.wpc2019.it 2
2. www.wpc2019.it 3 Who am I? Mariano Calandra Cloud Engineer AWS Authorized Instructor @MarianoCalandra
3. www.wpc2019.it 4 Who am I? Mariano Calandra Full time developer (.NET & AngularJS) Digital payments @MarianoCalandra 3 yrs ago 3 yrs ago
4. www.wpc2019.it 5 SERVERLESS
5. Serverless is about removing complexity by allowing the services that others provide to provide the complexity for you. – Paul Johnston www.wpc2019.it 6
6. www.wpc2019.it 7 CreateOrder Functions are sleeping and waiting to be invoked
7. www.wpc2019.it 8 ConfirmOrder ...until an event wake them up. Lambda event source Amazon API Gateway Amazon S3 Amazon DynamoDB Amazon SQS/SNS Amazon CloudWatch Scheduled events ….
8. www.wpc2019.it 9 ConfirmOrder ConfirmOrder ConfirmOrder In case of concurrent requests: do the same
9. www.wpc2019.it 10 ConfirmOrder ConfirmOrder ConfirmOrder Once the work is done the functions are disposed.
10. www.wpc2019.it 11 Fully managed REST API service Integrated caching Canary release API Keys or Lambda Authorizers Models validation Seamless integration with AWS services Amazon API Gateway
11. www.wpc2019.it 12 Fully managed NoSQL database Multi-master replication Built-in backup/restore Near real time autoscaling Up to 20 mln req/sec Amazon DynamoDB
12. www.wpc2019.it 13 ... SO WHAT ARE WE TRYING TO ADDRESS?
13. www.wpc2019.it 14 ... SO WHAT ARE WE TRYING TO ADDRESS? E N H A N C E D E V E LO P M E N T F E E D B AC K
14. www.wpc2019.it 15 ... SO WHAT ARE WE TRYING TO ADDRESS? E N H A N C E D E V E LO P M E N T F E E D B AC K I M P R O V E YO U R C O D E T E S TA B I L I T Y
15. www.wpc2019.it 16 ... SO WHAT ARE WE TRYING TO ADDRESS? E N H A N C E D E V E LO P M E N T F E E D B AC K I M P R O V E YO U R C O D E T E S TA B I L I T Y B E YO N D U N I T T E S T I N G
16. Order topic Product table ConfirmOrder function API endpoint www.wpc2019.it 17 the Order service
17. Order topic Product table ConfirmOrder function API endpoint www.wpc2019.it 18 POST /orders { "customer_id": "mariano", "product_ids": ["1", "2"], "address": "Via Roma, 12, Roma" } the Order service
18. Order topic Product table ConfirmOrder function API endpoint www.wpc2019.it 19 { "httpMethod": "POST", "body": "{\"customer_id\": \"mariano\",\"product_ids\": [\"1\", \"2\"],\"address\": \"Via Roma, 12, Roma\"}", "resource": "/orders", ... } the Order service
19. Order topic Product table ConfirmOrder function API endpoint www.wpc2019.it 20 Check whether products 1 and 2 are stored into the database. In that case, get the price; otherwise raise an error. the Order service
20. Order topic Product table ConfirmOrder function API endpoint www.wpc2019.it 21 { "type": "order_confirmed", "order_id": "4121", "customer_id": "mariano", "products": [ { "price": 128, "id": "1" }, { "price": 321, "id": "2" } ], "address": "Via Roma, 12, Roma" } the Order service
21. Order topic Product table ConfirmOrder function API endpoint www.wpc2019.it 22 { "statusCode":200, "body":"{\"order_id\":\"4121\"}" } the Order service
22. Order topic Product table ConfirmOrder function API endpoint www.wpc2019.it 23 HTTP/1.0 200 OK Content-Type: application/json Content-Length: 19 Server: Werkzeug/0.15.2 Python/3.7.3 Date: Fri, 29 Nov 2019 18:54:53 GMT {"order_id":"4121"} the Order service
23. www.wpc2019.it 24 ENHANCE DEVELOPMENT FEEDBACK
24. // filename: order/src/create-order.js (first version) let docClient = new AWS.DynamoDB.DocumentClient(); let sns = new AWS.SNS(); exports.handler = function(event) { try { let order_request = validateOrderRequest(event.body); let results = docClient.batchGet(order_request); let order_confirmed_evt = { "type": "order_confirmed", // ... } sns.publish(order_confirmed_evt); let retBody = {"order_id": order_confirmed_evt.order_id}; return { "status_code": 200, "body": JSON.stringify(retBody) } } catch (error) { /* ... */ } }; www.wpc2019.it 25
25. // filename: order/src/create-order.js (first version) let docClient = new AWS.DynamoDB.DocumentClient(); let sns = new AWS.SNS(); exports.handler = function(event) { try { let order_request = validateOrderRequest(event.body); let results = docClient.batchGet(order_request); let order_confirmed_evt = { "type": "order_confirmed", // ... } sns.publish(order_confirmed_evt); let retBody = {"order_id": order_confirmed_evt.order_id}; return { "status_code": 200, "body": JSON.stringify(retBody) } } catch (error) { /* ... */ } }; www.wpc2019.it 26
26. // filename: order/src/create-order.js (first version) let docClient = new AWS.DynamoDB.DocumentClient(); let sns = new AWS.SNS(); exports.handler = function(event) { try { let order_request = validateOrderRequest(event.body); let results = docClient.batchGet(order_request); let order_confirmed_evt = { "type": "order_confirmed", // ... } sns.publish(order_confirmed_evt); let retBody = {"order_id": order_confirmed_evt.order_id}; return { "status_code": 200, "body": JSON.stringify(retBody) } } catch (error) { /* ... */ } }; www.wpc2019.it 27
27. www.wpc2019.it 28 $ pip install localstack $ localstack start
28. // filename: order/src/create-order.js (second version) let docClient = new AWS.DynamoDB.DocumentClient({ endpoint: process.env.DDB_ENDPOINT, sslEnabled: process.env.SSL_ENABLED }); let sns = new AWS.SNS({ endpoint: process.env.SNS_ENDPOINT, sslEnabled: process.env.SSL_ENABLED }); exports.handler = function(event) { try { // ... let results = docClient.batchGet(order_request); // ... sns.publish(order_confirmed_evt); // ... } catch (error) { /* ... */ } }; www.wpc2019.it 29
29. order$ npm run test:local > sam local invoke -e scripts/create-order.input.json -n scripts/create-order.local.env.json CreateOrderFunction > Invoking src/create-order-lambda.handler (nodejs8.10) Fetching lambci/lambda:nodejs8.10 Docker container image... START RequestId: XXX Version: $LATEST END RequestId: XXX REPORT RequestId: XXX Init Duration: 1290.03 ms Duration: 93.97 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 52 MB {"statusCode":200,"body":"{\"order_id\":\"7363\"}"} www.wpc2019.it 30
30. www.wpc2019.it 31 https://commandeerit.com/
31. Order topic Product table ConfirmOrder function API endpoint www.wpc2019.it 32
32. Order topic Product table ConfirmOrder function API endpoint www.wpc2019.it 33 $ sam local start-api -n scripts/create-order.local.json Mounting CreateOrderFunction: http://127.0.0.1:3000/orders [POST] Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
33. Order topic Product table ConfirmOrder function API endpoint www.wpc2019.it 34
34. www.wpc2019.it 35 //filename: order/tests/create-order.integration.js test(‘raise error if product_ids is invalid', done => { let json = { "customer_id": "mariano", "product_ids": [3, "8"], // numbers are not allowed "address": "Via Roma, 12, Roma" }; return axios.post("http://localhost:3000/orders", json) .catch(function (error) { expect(error.response.status).toBe(500); expect(error.response.data) .toBe("product_ids must be a list of strings!"); done(); }); });
35. www.wpc2019.it 36 //filename: order/tests/create-order.integration.js test(‘raise error if product_ids is invalid', done => { let json = { "customer_id": "mariano", "product_ids": [3, "8"], // numbers are not allowed "address": "Via Roma, 12, Roma" }; return axios.post("http://localhost:3000/order", json) .catch(function (error) { expect(error.response.status).toBe(500); expect(error.response.data) .toBe("product_ids must be a list of strings!"); done(); }); });
36. www.wpc2019.it 37 //filename: order/tests/create-order.integration.js test(‘raise error if product_ids is invalid', done => { let json = { "customer_id": "mariano", "product_ids": [3, "8"], // numbers are not allowed "address": "Via Roma, 12, Roma" }; return axios.post("http://localhost:3000/orders", json) .catch(function (error) { expect(error.response.status).toBe(500); expect(error.response.data) .toBe("product_ids must be a list of strings!"); done(); }); });
37. www.wpc2019.it 38 //filename: order/tests/create-order.integration.js test(‘raise error if product_ids is invalid', done => { let json = { "customer_id": "mariano", "product_ids": [3, "8"], // numbers are not allowed "address": "Via Roma, 12, Roma" }; return axios.post("http://localhost:3000/order", json) .catch(function (error) { expect(error.response.status).toBe(500); expect(error.response.data) .toBe("product_ids must be a list of strings!"); done(); }); });
38. www.wpc2019.it 39 IMPROVE YOUR CODE TESTABILITY
39. Order topic Product table ConfirmOrder function API endpoint www.wpc2019.it 40
40. www.wpc2019.it 41 Business logic SNS API call DynamoDB API call An example of bad-design for our services (everething is wired into a single component)
41. www.wpc2019.it 42 Business logic Topic port Database port A better approach using hexagonal architecture.
42. www.wpc2019.it SNS API DynamoDB API Business logic Topic port Database port Lambda handler order/src/create-order-lambda.js
43. www.wpc2019.it 44 //filename: order/src/create-order-lambda.js let deps = { topic: new AWS.SNS(), db: new AWS.DynamoDB.DocumentClient() } let order = new Order(deps); let parser = new LambdaParser(); module.exports.handler = async function(event) { try { let body = parser.extractInput(event); let created_order = await order.create(body); return parser.buildSuccessfulResponse(created_order) } catch (error) { return parser.buildFailureResponse(error); } }
44. www.wpc2019.it 45 //filename: order/src/create-order-lambda.js let deps = { topic: new AWS.SNS(), db: new AWS.DynamoDB.DocumentClient() } let order = new Order(deps); let parser = new LambdaParser(); module.exports.handler = async function(event) { try { let body = parser.extractInput(event); let created_order = await order.create(body); return parser.buildSuccessfulResponse(created_order) } catch (error) { return parser.buildFailureResponse(error); } }
45. www.wpc2019.it 46 //filename: order/src/create-order-lambda.js let deps = { topic: new AWS.SNS(), db: new AWS.DynamoDB.DocumentClient() } let order = new Order(deps); let parser = new LambdaParser(); module.exports.handler = async function(event) { try { let body = parser.extractInput(event); let created_order = await order.create(body); return parser.buildSuccessfulResponse(created_order) } catch (error) { return parser.buildFailureResponse(error); } }
46. www.wpc2019.it 47 SNS API DynamoDB API Business logic Topic port Database port Lambda handler order/src/create-order-lambda.js SNS API (mock) DDB API (mock) Business logic Topic port Database port Test runner order/tests/create-order.unit.js
47. www.wpc2019.it 48 // filename: order/test/__mocks__/ddb.wrapper.js async getItemsById(ids) { let response = { // AWS DynamoDB Query Model "Responses": { "Product":[] }, "UnprocessedKeys":{} } if (ids.includes("1")) response.Responses.Product.push({"price":128,"id":"1"}); if (ids.includes("2")) response.Responses.Product.push({"price":321,"id":"2"}); return response; }
48. www.wpc2019.it 49 // filename: order/test/__mocks__/ddb.wrapper.js async getItemsById(ids) { let response = { // AWS DynamoDB Query Model "Responses": { "Product":[] }, "UnprocessedKeys":{} } if (ids.includes("1")) response.Responses.Product.push({"price":128,"id":"1"}); if (ids.includes("2")) response.Responses.Product.push({"price":321,"id":"2"}); return response; }
49. www.wpc2019.it 50 //filename: order/tests/create-order.unit.js test(the order request is correct', async () => { // ARRANGE let spy = jest.spyOn(deps.topic, 'publish’); let order = new Order(deps); let json = { "customer_id": "mariano", "product_ids": ["1", "2"], "address": "Via Roma, 12, Roma" } let body = JSON.stringify(json) // ACT (exec business logic) await order.create(body); // ASSERT expect(spy).toHaveBeenCalled(); });
50. www.wpc2019.it 51 //filename: order/tests/create-order.unit.js test(the order request is correct', async () => { // ARRANGE let spy = jest.spyOn(deps.topic, 'publish’); let order = new Order(deps); let json = { "customer_id": "mariano", "product_ids": ["1", "2"], "address": "Via Roma, 12, Roma" } let body = JSON.stringify(json) // ACT (exec business logic) await order.create(body); // ASSERT expect(spy).toHaveBeenCalled(); });
51. www.wpc2019.it 52 //filename: order/tests/create-order.unit.js test(the order request is correct', async () => { // ARRANGE let spy = jest.spyOn(deps.topic, 'publish’); let order = new Order(deps); let json = { "customer_id": "mariano", "product_ids": ["1", "2"], "address": "Via Roma, 12, Roma" } let body = JSON.stringify(json) // ACT (exec business logic) await order.create(body); // ASSERT expect(spy).toHaveBeenCalled(); });
52. www.wpc2019.it 53 //filename: order/tests/create-order.unit.js test(the order request is correct', async () => { // ARRANGE let spy = jest.spyOn(deps.topic, 'publish’); let order = new Order(deps); let json = { "customer_id": "mariano", "product_ids": ["1", "2"], "address": "Via Roma, 12, Roma" } let body = JSON.stringify(json) // ACT (exec business logic) await order.create(body); // ASSERT expect(spy).toHaveBeenCalled(); });
53. www.wpc2019.it 54 //filename: order/tests/create-order.unit.js test(the order request is correct', async () => { // ARRANGE let spy = jest.spyOn(deps.topic, 'publish’); let order = new Order(deps); let json = { "customer_id": "mariano", "product_ids": ["1", "2"], "address": "Via Roma, 12, Roma" } let body = JSON.stringify(json) // ACT (exec business logic) await order.create(body); // ASSERT expect(spy).toHaveBeenCalled(); });
54. www.wpc2019.it 55 BEYOND UNIT TESTING
55. www.wpc2019.it 56 Order (provider) This service accept an order request (products and customer address) from an HTTP endpoint and if correct it publish an order-confirmed event. Unit and integration tests in place.
56. www.wpc2019.it 57 Order (provider) { "type": "order_confirmed", "order_id": "4121", "customer_id": "mariano", "products": [ { "price": 128, "id": "1" }, { "price": 321, "id": "2" } ], "address": "Via Roma, 12, Roma" }
57. www.wpc2019.it 58 Order (provider) ListOfOrders (consumer) This service capture the order-confirmed event and populate the order’s history for a given customer. Unit and integration tests are in place.
58. www.wpc2019.it 59 Order (provider) ListOfOrders (consumer) { "type": "order_confirmed", "order_id": "4121", "customer_id": "mariano", "products": [ { "price": 128, "id": "1" }, { "price": 321, "id": "2" } ], "address": "Via Roma, 12, Roma" }
59. www.wpc2019.it 60 Order (provider) ListOfOrders (consumer) function validateMessage(msg) { if(!msg.customer_id) throw "Something went wrong: customer_id is missing"; // other validation rules... }
60. www.wpc2019.it 61 Order (provider) ListOfOrders (consumer) function makeOrderConfirmedEvt(username, prods, addr) { return { "type": "order_confirmed", "customer_id": username, // ...
61. www.wpc2019.it 62 Order (provider) ListOfOrders (consumer) function makeOrderConfirmedEvt(username, prods, addr) { return { "customer_id": username, . "customer_name": username, . // ...
62. www.wpc2019.it 63 Order (provider) ListOfOrders (consumer) { "type": "order_confirmed", "order_id": "4121", "customer_name": "mariano", "products": [ { "price": 128, "id": "1" }, { "price": 321, "id": "2" } ], "address": "Via Roma, 12, Roma" }
63. www.wpc2019.it 64 Order (provider) ListOfOrders (consumer) { "type": "order_confirmed", "order_id": "4121", "customer_name": "mariano", "products": [ { "price": 128, "id": "1" }, { "price": 321, "id": "2" } ], "address": "Via Roma, 12, Roma" }
64. 65 //filename: list-of-orders/tests/add-order.consumer.js ... let loo = new ListOfOrders(); let messagePact = new Pact.MessageConsumerPact({ consumer: "ListOfOrders", dir: path.resolve(process.cwd(), "..", "pacts"), provider: "Order", }); ... return messagePact .expectsToReceive("an order_confirmed event") .withContent({ type: "order_confirmed", order_id: { matcher: "^[0-9]{1,4}$", generate: "4121" }) customer_id: "mariano", products: [{ "price": 128, "id": "1" }], address: "Via Roma, 12, Roma" }) // verify consumer's ability to handle messages .verify(synchronousBodyHandler(loo.validateMessage)); ... www.wpc2019.it
65. 66 //filename: list-of-orders/tests/add-order.consumer.js ... let loo = new ListOfOrders(); let messagePact = new Pact.MessageConsumerPact({ consumer: "ListOfOrders", dir: path.resolve(process.cwd(), "..", "pacts"), provider: "Order", }); ... return messagePact .expectsToReceive("an order_confirmed event") .withContent({ type: "order_confirmed", order_id: { matcher: "^[0-9]{1,4}$", generate: "4121" }) customer_id: "mariano", products: [{ "price": 128, "id": "1" }], address: "Via Roma, 12, Roma" }) // verify consumer's ability to handle messages .verify(synchronousBodyHandler(loo.validateMessage)); ... www.wpc2019.it
66. 67 //filename: list-of-orders/tests/add-order.consumer.js ... let loo = new ListOfOrders(); let messagePact = new Pact.MessageConsumerPact({ consumer: "ListOfOrders", dir: path.resolve(process.cwd(), "..", "pacts"), provider: "Order", }); ... return messagePact .expectsToReceive("an order_confirmed event") .withContent({ type: "order_confirmed", order_id: { matcher: "^[0-9]{1,4}$", generate: "4121" }) customer_id: "mariano", products: [{ "price": 128, "id": "1" }], address: "Via Roma, 12, Roma" }) // verify consumer's ability to handle messages .verify(synchronousBodyHandler(loo.validateMessage)); ... www.wpc2019.it
67. 68 //filename: list-of-orders/tests/add-order.consumer.js ... let loo = new ListOfOrders(); let messagePact = new Pact.MessageConsumerPact({ consumer: "ListOfOrders", dir: path.resolve(process.cwd(), "..", "pacts"), provider: "Order", }); ... return messagePact .expectsToReceive("an order_confirmed event") .withContent({ type: "order_confirmed", order_id: { matcher: "^[0-9]{1,4}$", generate: "4121" }) customer_id: "mariano", products: [{ "price": 128, "id": "1" }], address: "Via Roma, 12, Roma" }) // verify consumer's ability to handle messages .verify(synchronousBodyHandler(loo.validateMessage)); ... www.wpc2019.it
68. 69 //filename: list-of-orders/tests/add-order.consumer.js ... let loo = new ListOfOrders(); let messagePact = new Pact.MessageConsumerPact({ consumer: "ListOfOrders", dir: path.resolve(process.cwd(), "..", "pacts"), provider: "Order", }); ... return messagePact .expectsToReceive("an order_confirmed event") .withContent({ type: "order_confirmed", order_id: { matcher: "^[0-9]{1,4}$", generate: "4121" }) customer_id: "mariano", products: [{ "price": 128, "id": "1" }], address: "Via Roma, 12, Roma" }) // verify consumer's ability to handle messages .verify(synchronousBodyHandler(loo.validateMessage)); ... www.wpc2019.it
69. www.wpc2019.it 70 list-of-orders $ npm run can-i-deploy PASS tests/add-order.consumer.test.js Given the ListOfOrders service When an order_confirmed event is received ✓ Then it should respect the pact (429ms) ...behind the scenes a JSON pact has been created.
70. www.wpc2019.it 71 //filename: orders/tests/order.provider.js ... // arrange the order_confirmed event let order = new Order(); let cid = "mariano"; let products = [{ "price": 128, "id": "1" }]; let addr = "Via Roma, 12, Roma"; ... // Pact setup let pact = new Pact.MessageProviderPact({ messageProviders: { "an order_confirmed event": () => order.makeOrderConfirmedEvent(cid, products, addr), }, provider: "Order", pactUrls: /* path to JSON pact file */ }) ... // verify the pact return pact.verify(); ...
71. www.wpc2019.it 72 //filename: orders/tests/order.provider.js ... // arrange the order_confirmed event let order = new Order(); let cid = "mariano"; let products = [{ "price": 128, "id": "1" }]; let addr = "Via Roma, 12, Roma"; ... // Pact setup let pact = new Pact.MessageProviderPact({ messageProviders: { "an order_confirmed event": () => order.makeOrderConfirmedEvent(cid, products, addr), }, provider: "Order", pactUrls: /* path to JSON pact file */ }) ... // verify the pact return pact.verify(); ...
72. www.wpc2019.it 73 //filename: orders/tests/order.provider.js ... // arrange the order_confirmed event let order = new Order(); let cid = "mariano"; let products = [{ "price": 128, "id": "1" }]; let addr = "Via Roma, 12, Roma"; ... // Pact setup let pact = new Pact.MessageProviderPact({ messageProviders: { "an order_confirmed event": () => order.makeOrderConfirmedEvent(cid, products, addr), }, provider: "Order", pactUrls: /* path to JSON pact file */ }) ... // verify the pact return pact.verify(); ...
73. www.wpc2019.it 74 //filename: orders/tests/order.provider.js ... // arrange the order_confirmed event let order = new Order(); let cid = "mariano"; let products = [{ "price": 128, "id": "1" }]; let addr = "Via Roma, 12, Roma"; ... // Pact setup let pact = new Pact.MessageProviderPact({ messageProviders: { "an order_confirmed event": () => order.makeOrderConfirmedEvent(cid, products, addr), }, provider: "Order", pactUrls: /* path to JSON pact file */ }) ... // verify the pact return pact.verify(); ...
74. www.wpc2019.it 75 order $ npm run can-i-deploy FAIL test/order.provider.test.js Given the Order service When an order_confirmed event is published ✕ Then it should respect the pact (362ms) Expected: { "customer_id": "mariano" } Description: Could not find key "customer_id" (keys present are: type, order_id, customer_name, products, address).
75. Takeaways 76 Decouple your code Don’t forget patterns Always unit test your code... ...but sometimes unit test is not enough!
76. @MarianoCalandra @mariano.calandra
77. Contatti OverNet Education – Info@OverNetEducation.it – www.OverNetEducation.it – Rozzano (MI) +39 02 365738 – Bologna +39 051 269911 – www.wpc-overneteducation.it – #wpc19it www.wpc2019.it 78