Your Lambda function might execute twice. Be prepared! (ServerlessDays Zurich)

Are you confused when scheduled Lambdas execute twice, SNS messages trigger an invocation three times, your handmade S3 inventory is out of sync because events occurred twice? Bad news: Sooner or later, your Lambda function executes multiple times. You have to prepare yourself! The reasons are system failures, retries on errors, and event sources that guarantee at-least-once delivery (e.g., CloudWatch Events, SNS, …). In this talk, you learn why Lambda functions might execute twice and what you can do about it focusing on Lambda, DynamoDB, and Kinesis Data Streams.

Transcript

1. 1 Your Lambda function might execute twice. Be prepared! Michael Wittig cloudonaut.io
2. Michael Wittig 2 • AWS in Action • Rapid Docker on AWS • cloudonaut.io • Freelancer michael@widdix.de @hellomichibye
3. 3 marbot.io - Incident Management via Slack 100% serverless
4. Demo “Something is wrong here” 4 https://github.com/widdix/serverlessdays-zurich-2019
5. 5 Cron rate(5 mins)?!
6. 6 Cron rate(5 mins)?!
8. Retry In case of an (internal) error, Lambda executes your code again. 8
9. Surprised? 9
10. No exactly once Lambda functions / event sources do not guarantee exactly-once delivery! 10
11. Lambda internals on-demand invocation 1
12. 12 1 Execution RequestResponse Invoke Success Data Error Error Not 200 Error
13. 13 “Event” Retries
14. 14 1 Execution 2 1. Retry after random delay Execution 3 2. Retry after random delay Execution Error Error Error Dead Letter Queue Event Invoke Success Success Success Not 202 Error
15. 15 Random delays between “Event” retries 1 2 3 1 2 3 Time Invoke A Invoke B
16. “ Invocations occur at least once in response to an event and functions must be idempotent to handle this. - AWS in 2017 16 https://web.archive.org/web/20171208134129/http://docs.aws.amazon.com/de_de/lambda/latest/dg/API_Invoke.html
18. “ When an error occurs, your function may be invoked multiple times. Retry behavior varies by error type, client, event source, and invocation type. - AWS in 2019 18 https://docs.aws.amazon.com/de_de/lambda/latest/dg/API_Invoke.html
19. 2 Idempotence A powerful concept
20. “ Executing your Lambda function [1...N] times with the same event must eventually lead to the same result (after retries) 20
21. 21 Error Success Output = Input Invoke ... { "a": 7 } exports.h = async (e) => { return {b: e.a}; }; { "b": 7 }
22. 22 Error Success Output = f(Input) Invoke ... { "a": 7 } exports.h = async (e) => { return {b: e.a*e.a}; }; { "b": 49 }
23. 23 Error Success Store data in data store Invoke ... { "mail": "y" } exports.h = async (e) => { u = await db.create(e.mail); return {id: u.id}; }; { "id": ? }
24. 24 Error Success Make HTTP POST Invoke ... { "mail": "y" } exports.h = async (e) => { await http.post({ url: 'mailch.imp/add, data: e }); return {}; }; {}
25. 25 Error Success Get data from data store Invoke ... { "mail": "x" } exports.h = async (e) => { u = await db.get(e.mail); if (u === null) { return {ok: false}; } else { return {ok: true}; } }; { "ok": ? }
26. “ Executing your Lambda function [1...N] times with the same event must eventually lead to the same result (after retries) when only our functions mutaes the external state 26
27. 27 Error Success Get data from data store Invoke ... { "mail": "x" } exports.h = async (e) => { u = await db.get(e.mail); if (u === null) { return {ok: false}; } else { return {ok: true}; } }; { "ok": true } x: u123 y: u456
28. Tricks How to create idempotent functions 3
29. Tricks ￮ Client generated token for uniqueness ￮ Avoid system time ￮ Let the client pass the state into the system 29 https://github.com/widdix/serverlessdays-zurich-2019
30. ConditionExpression DynamoDB supports conditional writes: ￮ AND, OR, NOT ￮ =, <>, <, <=, >, >= ￮ attribute_not_exists(attr) ￮ attribute_exists(attr) ￮ begins_with(attr, str) 30 https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.OperatorsAndFunctions.html
31. 31 Serialize work with Kinesis Data Streams A Time Iterator B Iterator C Iterator
32. Credits Special thanks to all the people who made and released these awesome resources for free: ￮ Presentation template by SlidesCarnival ￮ Photographs by Unsplash and Pexels 32
33. 33 Michael Wittig • AWS in Action • Rapid Docker on AWS • cloudonaut.io • Freelancer • marbot.io michael@widdix.de @hellomichibye