Your Lambda function might execute twice. Be prepared!

Are you confused when scheduled Lambdas execute twice, SNS messages trigger an invocation three times, your handmade S3 inventory is out of sync because events occurred twice? Bad news: Sooner or later, your Lambda function will be invoked multiple times. You have to be prepared! The reasons are retries on errors and event sources that guarantee at-least-once delivery (e.g., CloudWatch Events, SNS, …).In this talk, you will learn why Lambda functions might execute twice and what you can do about it.

Transcript

1. Frankfurt | 26 & 27 September 2018
2. Your Lambda function might execute twice. Be prepared! Michael Wittig – cloudonaut.io 2
3. Michael Wittig ￮ Author of AWS in Action ￮ https://cloudonaut.io ￮ freelance AWS consultant and programmer ￮ AWS Community Hero 3
4. Demo “Something is wrong here” 4 https://github.com/widdix/aws-community-days-2018-ffm
5. 5 Cron rate(5 mins)?!
6. 6 Cron rate(5 mins)?!
8. Retry In case of an error, Lambda executes your code again. 8
9. Surprised? 9
10. 10 A On-Demand Invocation By calling the API directly B Event Source Invocation 1 Invocation Type RequestResponse 2 Invocation Type Event 1 Poll based Executed by Lambda a Stream based Kinesis, DynamoDB b Others SQS 2 Push based Executed by Service a Synchronous API Gateway, Cognito b Asynchronous S3, SNS, SES, CW Exactly once guaranteed?
11. No exactly once Lambda functions / event sourced do not guarantee exactly-once delivery! 11
12. Lambda internals on-demand invocation 1
13. On Handled or Unhandled error: Lambda executes your code once. But when StatusCode <> 200 there is no invocation at all. https://docs.aws.amazon.com/lambda/lates t/dg/API_Invoke.html 13 A On-Demand Invocation 1 Invocation Type RequestResponse
14. 14 1 Execution RequestResponse Invoke Success Data Error Error Not 200 Error
15. On Handled or Unhandled error: Lambda retries executing your code twice and therefore executes your code at-least-once [1...3]. But when StatusCode <> 202 there is no invocation at all. https://docs.aws.amazon.com/lambda/lates t/dg/retries-on-errors.html 15 A On-Demand Invocation 1 Invocation Type Event
16. 16 “Event” Retries
17. 17 1 Execution 2 1. Retry after random delay Execution 3 2. Retry after random delay Execution Error Error Error Dead Letter Queue Event Invoke Success Success Success Not 202 Error
18. 18 Random delays between “Event” retries 1 2 3 1 2 3 Time Invoke A Invoke B
19. “ If you use [...] Event [...], the function will be invoked at least once [...] and the function must be idempotent to handle this. 19 https://docs.aws.amazon.com/de_de/lambda/latest/dg/API_Invoke.html
20. 2 Idempotence An invaluable concept
21. “ With the same event execute your Lambda function code [1...N] times the result is eventually the same (after retries) 21
22. 22 Error Success Output = Input Invoke ... { "a": 7 } exports.h = async (e) => { return {b: e.a}; }; { "b": 7 }
23. 23 Error Success Output = f(Input) Invoke ... { "a": 7 } exports.h = async (e) => { return {b: e.a*e.a}; }; { "b": 49 }
24. 24 Error Success Store data in data store Invoke ... { "mail": "y" } exports.h = async (e) => { u = await db.create(e.mail); return {id: u.id}; }; { "id": ? }
25. 25 Error Success Make HTTP POST Invoke ... { "mail": "y" } exports.h = async (e) => { await http.post({ url: 'mailch.imp/add, data: e }); return {}; }; {}
26. 26 Error Success Get data from data store Invoke ... { "mail": "x" } exports.h = async (e) => { u = await db.get(e.mail); if (u === null) { return {ok: false}; } else { return {ok: true}; } }; { "ok": ? }
27. “ with the same event execute your Lambda function code [1...N] times the result is eventually the same (after retries) when only our code mutaes the external state 27
28. 28 Error Success Get data from data store Invoke ... { "mail": "x" } exports.h = async (e) => { u = await db.get(e.mail); if (u === null) { return {ok: false}; } else { return {ok: true}; } }; { "ok": true } x: u123 y: u456
29. Tricks How to create idempotent functions 3
30. Primary Key Partition key plus optional sort key identifies item. DynamoDB No enforced schema You can add attributes to an item at any time. 30 Read GetItem Query Scan Mutate PutItem UpdateItem DeleteItem
31. Demo “Twitter like status updates” 31
32. 32 Cause for retries
33. 33 Status updates are mixed up status might be late or lost
34. Idea 1 Status update creation time as an input (part of the event) Keep order Idea 2 Previous status update as an input (part of the event) 34 Idea 3 Use Kinesis Data Stream in between Idea 4 Use Kinesis Data Stream in between with sequence numbers
35. “ PutItem If an item that has the same primary key as an existing item, the new item completely replaces the existing item. 35 https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html
36. Code “Status update creation time as an input” 36 https://github.com/widdix/aws-community-days-2018-ffm
37. 37 Order is maintained status might be late or lost Idea 1
38. Solution? Order relies on “absolute client time” 38
39. ConditionExpression DynamoDB supports conditional writes: ￮ AND, OR, NOT ￮ =, <>, <, <=, >, >= ￮ attribute_not_exists(attr) ￮ attribute_exists(attr) ￮ begins_with(attr, str) 39 https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.Op eratorsAndFunctions.html
40. Code “Previous status update as an input” 40 https://github.com/widdix/aws-community-days-2018-ffm
41. 41 Order is maintained status might be lost Idea 2
42. 42 Solution? Order relies on “logical time”
43. 43 How can we fix the retry issue?
44. 44 Kinesis Data Stream A Time Iterator B Iterator C Iterator
45. “ UpdateItem If an item that has the same primary key as an existing item, only the attributes in UpdateExpression are touched. 45 https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.ht ml
46. Code “Use Kinesis Data Stream in between” 46 https://github.com/widdix/aws-community-days-2018-ffm
47. 47 Order is maintained all status updates are stored Idea 3
48. “ SequenceNumberForOrdering Set the SequenceNumber ForOrdering of record n to the sequence number of record n-1 (as returned in the result when putting record n-1). 48 https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html
49. Code “Use Kinesis + sequence numbers” 49 https://github.com/widdix/aws-community-days-2018-ffm
50. 50 Order is maintained all status updates are stored Idea 4
51. Credits Special thanks to all the people who made and released these awesome resources for free: ￮ Presentation template by SlidesCarnival ￮ Photographs by Unsplash and Pexels 51
52. Michael Wittig ￮ Author of AWS in Action ￮ https://cloudonaut.io ￮ freelance AWS consultant and programmer ￮ AWS Community Hero 52
53. Place your screenshot here 53 AWS in Action 2nd edition now available Use code ctwawscd18 on manning.com and save 40%