Your Lambda function might execute twice. Be prepared!

Your Lambda function might execute twice. Be prepared!

Are you confused when scheduled Lambdas execute twice, SNS messages trigger an invocation three times, your handmade S3 inventory is out of sync because events occurred twice? Bad news: Sooner or later, your Lambda function will be invoked multiple times. You have to be prepared! The reasons are retries on errors and event sources that guarantee at-least-once delivery (e.g., CloudWatch Events, SNS, …).In this talk, you will learn why Lambda functions might execute twice and what you can do about it.

E8f66870d1204779ecc45f2695faa73e?s=128

Michael Wittig

September 26, 2018
Tweet

Transcript

  1. Frankfurt | 26 & 27 September 2018

  2. Your Lambda function might execute twice. Be prepared! Michael Wittig

    – cloudonaut.io 2
  3. Michael Wittig ○ Author of AWS in Action ○ https://cloudonaut.io

    ○ freelance AWS consultant and programmer ○ AWS Community Hero 3
  4. Demo “Something is wrong here” 4 https://github.com/widdix/aws-community-days-2018-ffm

  5. 5 Cron rate(5 mins)?!

  6. 6 Cron rate(5 mins)?!

  7. 7

  8. Retry In case of an error, Lambda executes your code

    again. 8
  9. Surprised? 9

  10. 10 A On-Demand Invocation By calling the API directly B

    Event Source Invocation 1 Invocation Type RequestResponse 2 Invocation Type Event 1 Poll based Executed by Lambda a Stream based Kinesis, DynamoDB b Others SQS 2 Push based Executed by Service a Synchronous API Gateway, Cognito b Asynchronous S3, SNS, SES, CW Exactly once guaranteed?
  11. No exactly once Lambda functions / event sourced do not

    guarantee exactly-once delivery! 11
  12. Lambda internals on-demand invocation 1

  13. On Handled or Unhandled error: Lambda executes your code once.

    But when StatusCode <> 200 there is no invocation at all. https://docs.aws.amazon.com/lambda/lates t/dg/API_Invoke.html 13 A On-Demand Invocation 1 Invocation Type RequestResponse
  14. 14 1 Execution RequestResponse Invoke Success Data Error Error Not

    200 Error
  15. On Handled or Unhandled error: Lambda retries executing your code

    twice and therefore executes your code at-least-once [1...3]. But when StatusCode <> 202 there is no invocation at all. https://docs.aws.amazon.com/lambda/lates t/dg/retries-on-errors.html 15 A On-Demand Invocation 1 Invocation Type Event
  16. 16 “Event” Retries

  17. 17 1 Execution 2 1. Retry after random delay Execution

    3 2. Retry after random delay Execution Error Error Error Dead Letter Queue Event Invoke Success Success Success Not 202 Error
  18. 18 Random delays between “Event” retries 1 2 3 1

    2 3 Time Invoke A Invoke B
  19. “ If you use [...] Event [...], the function will

    be invoked at least once [...] and the function must be idempotent to handle this. 19 https://docs.aws.amazon.com/de_de/lambda/latest/dg/API_Invoke.html
  20. 2 Idempotence An invaluable concept

  21. “ With the same event execute your Lambda function code

    [1...N] times the result is eventually the same (after retries) 21
  22. 22 Error Success Output = Input Invoke ... { "a":

    7 } exports.h = async (e) => { return {b: e.a}; }; { "b": 7 }
  23. 23 Error Success Output = f(Input) Invoke ... { "a":

    7 } exports.h = async (e) => { return {b: e.a*e.a}; }; { "b": 49 }
  24. 24 Error Success Store data in data store Invoke ...

    { "mail": "y" } exports.h = async (e) => { u = await db.create(e.mail); return {id: u.id}; }; { "id": ? }
  25. 25 Error Success Make HTTP POST Invoke ... { "mail":

    "y" } exports.h = async (e) => { await http.post({ url: 'mailch.imp/add, data: e }); return {}; }; {}
  26. 26 Error Success Get data from data store Invoke ...

    { "mail": "x" } exports.h = async (e) => { u = await db.get(e.mail); if (u === null) { return {ok: false}; } else { return {ok: true}; } }; { "ok": ? }
  27. “ with the same event execute your Lambda function code

    [1...N] times the result is eventually the same (after retries) when only our code mutaes the external state 27
  28. 28 Error Success Get data from data store Invoke ...

    { "mail": "x" } exports.h = async (e) => { u = await db.get(e.mail); if (u === null) { return {ok: false}; } else { return {ok: true}; } }; { "ok": true } x: u123 y: u456
  29. Tricks How to create idempotent functions 3

  30. Primary Key Partition key plus optional sort key identifies item.

    DynamoDB No enforced schema You can add attributes to an item at any time. 30 Read GetItem Query Scan Mutate PutItem UpdateItem DeleteItem
  31. Demo “Twitter like status updates” 31

  32. 32 Cause for retries

  33. 33 Status updates are mixed up status might be late

    or lost
  34. Idea 1 Status update creation time as an input (part

    of the event) Keep order Idea 2 Previous status update as an input (part of the event) 34 Idea 3 Use Kinesis Data Stream in between Idea 4 Use Kinesis Data Stream in between with sequence numbers
  35. “ PutItem If an item that has the same primary

    key as an existing item, the new item completely replaces the existing item. 35 https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html
  36. Code “Status update creation time as an input” 36 https://github.com/widdix/aws-community-days-2018-ffm

  37. 37 Order is maintained status might be late or lost

    Idea 1
  38. Solution? Order relies on “absolute client time” 38

  39. ConditionExpression DynamoDB supports conditional writes: ○ AND, OR, NOT ○

    =, <>, <, <=, >, >= ○ attribute_not_exists(attr) ○ attribute_exists(attr) ○ begins_with(attr, str) 39 https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.Op eratorsAndFunctions.html
  40. Code “Previous status update as an input” 40 https://github.com/widdix/aws-community-days-2018-ffm

  41. 41 Order is maintained status might be lost Idea 2

  42. 42 Solution? Order relies on “logical time”

  43. 43 How can we fix the retry issue?

  44. 44 Kinesis Data Stream A Time Iterator B Iterator C

    Iterator
  45. “ UpdateItem If an item that has the same primary

    key as an existing item, only the attributes in UpdateExpression are touched. 45 https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.ht ml
  46. Code “Use Kinesis Data Stream in between” 46 https://github.com/widdix/aws-community-days-2018-ffm

  47. 47 Order is maintained all status updates are stored Idea

    3
  48. “ SequenceNumberForOrdering Set the SequenceNumber ForOrdering of record n to

    the sequence number of record n-1 (as returned in the result when putting record n-1). 48 https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html
  49. Code “Use Kinesis + sequence numbers” 49 https://github.com/widdix/aws-community-days-2018-ffm

  50. 50 Order is maintained all status updates are stored Idea

    4
  51. Credits Special thanks to all the people who made and

    released these awesome resources for free: ○ Presentation template by SlidesCarnival ○ Photographs by Unsplash and Pexels 51
  52. Michael Wittig ○ Author of AWS in Action ○ https://cloudonaut.io

    ○ freelance AWS consultant and programmer ○ AWS Community Hero 52
  53. Place your screenshot here 53 AWS in Action 2nd edition

    now available Use code ctwawscd18 on manning.com and save 40%