Your Lambda function might execute twice. Be prepared!

Slide 1

Slide 1 text

Frankfurt | 26 & 27 September 2018

Slide 2

Slide 2 text

Your Lambda function might execute twice. Be prepared! Michael Wittig – cloudonaut.io 2

Slide 3

Slide 3 text

Michael Wittig ￮ Author of AWS in Action ￮ https://cloudonaut.io ￮ freelance AWS consultant and programmer ￮ AWS Community Hero 3

Slide 4

Slide 4 text

Demo “Something is wrong here” 4 https://github.com/widdix/aws-community-days-2018-ffm

Slide 5

Slide 5 text

5 Cron rate(5 mins)?!

Slide 6

Slide 6 text

6 Cron rate(5 mins)?!

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Retry In case of an error, Lambda executes your code again. 8

Slide 9

Slide 9 text

Surprised? 9

Slide 10

Slide 10 text

10 A On-Demand Invocation By calling the API directly B Event Source Invocation 1 Invocation Type RequestResponse 2 Invocation Type Event 1 Poll based Executed by Lambda a Stream based Kinesis, DynamoDB b Others SQS 2 Push based Executed by Service a Synchronous API Gateway, Cognito b Asynchronous S3, SNS, SES, CW Exactly once guaranteed?

Slide 11

Slide 11 text

No exactly once Lambda functions / event sourced do not guarantee exactly-once delivery! 11

Slide 12

Slide 12 text

Lambda internals on-demand invocation 1

Slide 13

Slide 13 text

On Handled or Unhandled error: Lambda executes your code once. But when StatusCode <> 200 there is no invocation at all. https://docs.aws.amazon.com/lambda/lates t/dg/API_Invoke.html 13 A On-Demand Invocation 1 Invocation Type RequestResponse

Slide 14

Slide 14 text

14 1 Execution RequestResponse Invoke Success Data Error Error Not 200 Error

Slide 15

Slide 15 text

On Handled or Unhandled error: Lambda retries executing your code twice and therefore executes your code at-least-once [1...3]. But when StatusCode <> 202 there is no invocation at all. https://docs.aws.amazon.com/lambda/lates t/dg/retries-on-errors.html 15 A On-Demand Invocation 1 Invocation Type Event

Slide 16

Slide 16 text

16 “Event” Retries

Slide 17

Slide 17 text

17 1 Execution 2 1. Retry after random delay Execution 3 2. Retry after random delay Execution Error Error Error Dead Letter Queue Event Invoke Success Success Success Not 202 Error

Slide 18

Slide 18 text

18 Random delays between “Event” retries 1 2 3 1 2 3 Time Invoke A Invoke B

Slide 19

Slide 19 text

“ If you use [...] Event [...], the function will be invoked at least once [...] and the function must be idempotent to handle this. 19 https://docs.aws.amazon.com/de_de/lambda/latest/dg/API_Invoke.html

Slide 20

Slide 20 text

2 Idempotence An invaluable concept

Slide 21

Slide 21 text

“ With the same event execute your Lambda function code [1...N] times the result is eventually the same (after retries) 21

Slide 22

Slide 22 text

22 Error Success Output = Input Invoke ... { "a": 7 } exports.h = async (e) => { return {b: e.a}; }; { "b": 7 }

Slide 23

Slide 23 text

23 Error Success Output = f(Input) Invoke ... { "a": 7 } exports.h = async (e) => { return {b: e.a*e.a}; }; { "b": 49 }

Slide 24

Slide 24 text

24 Error Success Store data in data store Invoke ... { "mail": "y" } exports.h = async (e) => { u = await db.create(e.mail); return {id: u.id}; }; { "id": ? }

Slide 25

Slide 25 text

25 Error Success Make HTTP POST Invoke ... { "mail": "y" } exports.h = async (e) => { await http.post({ url: 'mailch.imp/add, data: e }); return {}; }; {}

Slide 26

Slide 26 text

26 Error Success Get data from data store Invoke ... { "mail": "x" } exports.h = async (e) => { u = await db.get(e.mail); if (u === null) { return {ok: false}; } else { return {ok: true}; } }; { "ok": ? }

Slide 27

Slide 27 text

“ with the same event execute your Lambda function code [1...N] times the result is eventually the same (after retries) when only our code mutaes the external state 27

Slide 28

Slide 28 text

28 Error Success Get data from data store Invoke ... { "mail": "x" } exports.h = async (e) => { u = await db.get(e.mail); if (u === null) { return {ok: false}; } else { return {ok: true}; } }; { "ok": true } x: u123 y: u456

Slide 29

Slide 29 text

Tricks How to create idempotent functions 3

Slide 30

Slide 30 text

Primary Key Partition key plus optional sort key identifies item. DynamoDB No enforced schema You can add attributes to an item at any time. 30 Read GetItem Query Scan Mutate PutItem UpdateItem DeleteItem

Slide 31

Slide 31 text

Demo “Twitter like status updates” 31

Slide 32

Slide 32 text

32 Cause for retries

Slide 33

Slide 33 text

33 Status updates are mixed up status might be late or lost

Slide 34

Slide 34 text

Idea 1 Status update creation time as an input (part of the event) Keep order Idea 2 Previous status update as an input (part of the event) 34 Idea 3 Use Kinesis Data Stream in between Idea 4 Use Kinesis Data Stream in between with sequence numbers

Slide 35

Slide 35 text

“ PutItem If an item that has the same primary key as an existing item, the new item completely replaces the existing item. 35 https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html

Slide 36

Slide 36 text

Code “Status update creation time as an input” 36 https://github.com/widdix/aws-community-days-2018-ffm

Slide 37

Slide 37 text

37 Order is maintained status might be late or lost Idea 1

Slide 38

Slide 38 text

Solution? Order relies on “absolute client time” 38

Slide 39

Slide 39 text

ConditionExpression DynamoDB supports conditional writes: ￮ AND, OR, NOT ￮ =, <>, <, <=, >, >= ￮ attribute_not_exists(attr) ￮ attribute_exists(attr) ￮ begins_with(attr, str) 39 https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.Op eratorsAndFunctions.html

Slide 40

Slide 40 text

Code “Previous status update as an input” 40 https://github.com/widdix/aws-community-days-2018-ffm

Slide 41

Slide 41 text

41 Order is maintained status might be lost Idea 2

Slide 42

Slide 42 text

42 Solution? Order relies on “logical time”

Slide 43

Slide 43 text

43 How can we fix the retry issue?

Slide 44

Slide 44 text

44 Kinesis Data Stream A Time Iterator B Iterator C Iterator

Slide 45

Slide 45 text

“ UpdateItem If an item that has the same primary key as an existing item, only the attributes in UpdateExpression are touched. 45 https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.ht ml

Slide 46

Slide 46 text

Code “Use Kinesis Data Stream in between” 46 https://github.com/widdix/aws-community-days-2018-ffm

Slide 47

Slide 47 text

47 Order is maintained all status updates are stored Idea 3

Slide 48

Slide 48 text

“ SequenceNumberForOrdering Set the SequenceNumber ForOrdering of record n to the sequence number of record n-1 (as returned in the result when putting record n-1). 48 https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html

Slide 49

Slide 49 text

Code “Use Kinesis + sequence numbers” 49 https://github.com/widdix/aws-community-days-2018-ffm

Slide 50

Slide 50 text

50 Order is maintained all status updates are stored Idea 4

Slide 51

Slide 51 text

Credits Special thanks to all the people who made and released these awesome resources for free: ￮ Presentation template by SlidesCarnival ￮ Photographs by Unsplash and Pexels 51

Slide 52

Slide 52 text

Michael Wittig ￮ Author of AWS in Action ￮ https://cloudonaut.io ￮ freelance AWS consultant and programmer ￮ AWS Community Hero 52

Slide 53

Slide 53 text

Place your screenshot here 53 AWS in Action 2nd edition now available Use code ctwawscd18 on manning.com and save 40%