Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Your Lambda function might execute twice. Be prepared! (ServerlessDays Zurich)

Your Lambda function might execute twice. Be prepared! (ServerlessDays Zurich)

Are you confused when scheduled Lambdas execute twice, SNS messages trigger an invocation three times, your handmade S3 inventory is out of sync because events occurred twice? Bad news: Sooner or later, your Lambda function executes multiple times. You have to prepare yourself! The reasons are system failures, retries on errors, and event sources that guarantee at-least-once delivery (e.g., CloudWatch Events, SNS, …). In this talk, you learn why Lambda functions might execute twice and what you can do about it focusing on Lambda, DynamoDB, and Kinesis Data Streams.

E8f66870d1204779ecc45f2695faa73e?s=128

Michael Wittig

April 11, 2019
Tweet

Transcript

  1. 1 Your Lambda function might execute twice. Be prepared! Michael

    Wittig cloudonaut.io
  2. Michael Wittig 2 • AWS in Action • Rapid Docker

    on AWS • cloudonaut.io • Freelancer michael@widdix.de @hellomichibye
  3. 3 marbot.io - Incident Management via Slack 100% serverless

  4. Demo “Something is wrong here” 4 https://github.com/widdix/serverlessdays-zurich-2019

  5. 5 Cron rate(5 mins)?!

  6. 6 Cron rate(5 mins)?!

  7. 7

  8. Retry In case of an (internal) error, Lambda executes your

    code again. 8
  9. Surprised? 9

  10. No exactly once Lambda functions / event sources do not

    guarantee exactly-once delivery! 10
  11. Lambda internals on-demand invocation 1

  12. 12 1 Execution RequestResponse Invoke Success Data Error Error Not

    200 Error
  13. 13 “Event” Retries

  14. 14 1 Execution 2 1. Retry after random delay Execution

    3 2. Retry after random delay Execution Error Error Error Dead Letter Queue Event Invoke Success Success Success Not 202 Error
  15. 15 Random delays between “Event” retries 1 2 3 1

    2 3 Time Invoke A Invoke B
  16. “ Invocations occur at least once in response to an

    event and functions must be idempotent to handle this. - AWS in 2017 16 https://web.archive.org/web/20171208134129/http://docs.aws.amazon.com/de_de/lambda/latest/dg/API_Invoke.html
  17. 17

  18. “ When an error occurs, your function may be invoked

    multiple times. Retry behavior varies by error type, client, event source, and invocation type. - AWS in 2019 18 https://docs.aws.amazon.com/de_de/lambda/latest/dg/API_Invoke.html
  19. 2 Idempotence A powerful concept

  20. “ Executing your Lambda function [1...N] times with the same

    event must eventually lead to the same result (after retries) 20
  21. 21 Error Success Output = Input Invoke ... { "a":

    7 } exports.h = async (e) => { return {b: e.a}; }; { "b": 7 }
  22. 22 Error Success Output = f(Input) Invoke ... { "a":

    7 } exports.h = async (e) => { return {b: e.a*e.a}; }; { "b": 49 }
  23. 23 Error Success Store data in data store Invoke ...

    { "mail": "y" } exports.h = async (e) => { u = await db.create(e.mail); return {id: u.id}; }; { "id": ? }
  24. 24 Error Success Make HTTP POST Invoke ... { "mail":

    "y" } exports.h = async (e) => { await http.post({ url: 'mailch.imp/add, data: e }); return {}; }; {}
  25. 25 Error Success Get data from data store Invoke ...

    { "mail": "x" } exports.h = async (e) => { u = await db.get(e.mail); if (u === null) { return {ok: false}; } else { return {ok: true}; } }; { "ok": ? }
  26. “ Executing your Lambda function [1...N] times with the same

    event must eventually lead to the same result (after retries) when only our functions mutaes the external state 26
  27. 27 Error Success Get data from data store Invoke ...

    { "mail": "x" } exports.h = async (e) => { u = await db.get(e.mail); if (u === null) { return {ok: false}; } else { return {ok: true}; } }; { "ok": true } x: u123 y: u456
  28. Tricks How to create idempotent functions 3

  29. Tricks ○ Client generated token for uniqueness ○ Avoid system

    time ○ Let the client pass the state into the system 29 https://github.com/widdix/serverlessdays-zurich-2019
  30. ConditionExpression DynamoDB supports conditional writes: ○ AND, OR, NOT ○

    =, <>, <, <=, >, >= ○ attribute_not_exists(attr) ○ attribute_exists(attr) ○ begins_with(attr, str) 30 https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.OperatorsAndFunctions.html
  31. 31 Serialize work with Kinesis Data Streams A Time Iterator

    B Iterator C Iterator
  32. Credits Special thanks to all the people who made and

    released these awesome resources for free: ○ Presentation template by SlidesCarnival ○ Photographs by Unsplash and Pexels 32
  33. 33 Michael Wittig • AWS in Action • Rapid Docker

    on AWS • cloudonaut.io • Freelancer • marbot.io michael@widdix.de @hellomichibye