Python × AWS × Serverless 初学者が次の一歩を踏み出すためのテクニック/Python & AWS & Serverless - Step to the next stage from a beginner

B84123138372542a55401c92b170b528?s=47 hassaku63
August 28, 2020

Python × AWS × Serverless 初学者が次の一歩を踏み出すためのテクニック/Python & AWS & Serverless - Step to the next stage from a beginner

PyCon JP 2020 ( https://pycon.jp/2020/en/ ) の登壇資料

https://pycon.jp/2020/en/timetable/?id=203588

(2020/08/30 追記) Qiita 補足ブログ書きました https://qiita.com/hassaku_63/items/875ba4694538c6484526

B84123138372542a55401c92b170b528?s=128

hassaku63

August 28, 2020
Tweet

Transcript

  1. Python × AWS × Serverless ॳֶऀ͕࣍ͷҰาΛ౿Έग़ͨ͢ΊͷςΫχοΫ Python & AWS &

    Serverless - Step to the next stage from a beginner PyCon JP 2020 #pyconjp_2 @hassaku
  2. ͸͡Ίʹ

  3. Speaker hassaku (Twitter: @hassaku_63) Serverworks Co., Ltd.

  4. Speaker ࣮ફ AWS CDK - TypeScript Ͱ
 Πϯϑϥ΋ΞϓϦ΋ʂ ڞஶ͠·ͨ͠ʢٕज़ॻయ8ʣ BOOTH:

    https://booth.pm/ja/items/1881928
  5. Serverworks • AWS ಛԽͷ SIer • ʮΫϥ΢υͰɺੈքΛɺ΋ͬͱɺ͸ͨΒ͖΍͘͢ʯ • AWSύʔτφʔωοτϫʔΫ
 ʮϓϨϛΞίϯαϧςΟϯάύʔτφʔʯೝఆ

  6. ൃදͷલʹ - ͓͜ͱΘΓ - • ࠂ஌ࡁΈͷΞδΣϯμ͔Βएׯͷมߋ͕͋Γ·͢ʢݟղͷมԽʹ൐͏΋ͷʣ • ૝ఆಡऀ • AWS

    ʹ͍ͭͯɺଟগͷར༻ܦݧ͕͋Δ • ʮαʔόʔϨεʯͷ֓೦΍ɺྑ͞͸ͳΜͱͳ͘஌͍ͬͯΔ • Lambda Function ΛσϓϩΠͨ͜͠ͱ͕͋Δʗ֓೦Λཧղ͍ͯ͠Δ • ʮαʔόʔϨεʯೖ໳هࣄͷ࣍ʹԿΛͨ͠Βྑ͍ͷ͔Θ͔Βͳ͍ • Զͨͪ͸งғؾͰαʔόʔϨεΛ΍͍ͬͯΔ
  7. Agenda 1. AWSʹ͓͚ΔαʔόʔϨεΞϓϦέʔγϣϯߏஙͷجຊ 2. Serverless Framework جຊͷ͓͞Β͍ 3. ϓϩδΣΫτߏ੒ͷྫ 4.

    ʮεςʔδʯͷ؅ཧ 5. ϩΪϯά 6. ·ͱΊ
  8. AWSʹ͓͚ΔαʔόʔϨε
 ΞϓϦέʔγϣϯߏஙͷجຊ

  9. αʔόʔϨεͷԿ͕೉͍͠ʁ • ֶशίετͷߴ͞ • ֓೦ʹೃછΈ͕ͳ͍ • ʮೖ໳ॻͷ࣍ʯ໰୊

  10. αʔόʔϨεͳʮࢥߟʯ • ඇಉظ • ΠϕϯτυϦϒϯ • εςʔτϨε

  11. αʔόʔϨεͳʮࢥߟʯ • ඇಉظ • ΠϕϯτυϦϒϯ • εςʔτϨε

  12. αʔόʔϨεͳొ৔ਓ෺ • ϒϩʔΧʔ • Πϕϯτϋϯυϥ • σʔλετΞ CSPLFS FWFOU IBOEMFS

    %BUBTUPSF CSPLFS e.g SQS, SNS, Kinesis e.g DynamoDB Lambda
  13. See also • [PyCon JP 2017] The theory of Serverless

    development by Python (ཧ࿦ ͔ΒֶͿPythonʹΑΔαʔόϨε։ൃ) @Masashi Terui
  14. Serverless Framework جຊͷ͓͞Β͍

  15. Serverless Framework • OSSͷσϓϩΠπʔϧ • Infrastructure as Code (IaC) •

    ϚϧνΫϥ΢υରԠ • Serverless Frameworkͷ࢖͍ํ·ͱΊ https://www.serverless.com/
  16. Serverless Framework 1. ϓϩδΣΫτ࡞੒ 2. ίʔυΛॻ͘ (yaml, python) 3. σϓϩΠύοέʔδΛ࡞੒

    4. σϓϩΠ … sls create … sls deploy (or sls package) … sls deploy
  17. σϓϩΠͷಈ͖ > sls deploy —stage dev

  18. σϓϩΠͷಈ͖ > sls deploy —stage dev $MPVE'PSNBUJPOςϯϓϨ -BNCEBσϓϩΠύοέʔδ Λ࡞੒

  19. σϓϩΠͷಈ͖ S3 όέοτΛੜ੒͠ɺ
 ҎԼΛΞοϓϩʔυ ɾLambda ͷσϓϩΠύοέʔδ
 ɾCloudFormation tempalte "84 4

    ΁Ξοϓϩʔυ
  20. σϓϩΠͷಈ͖ $MPVE'PSNBUJPO σϓϩΠ CloudFormation deploy Λ࣮ߦ

  21. Serverless Framework ͷಠࣗ֓೦
 ˞σϓϩΠઌ؀ڥΛ۠ผ͢ΔͨΊͷ༻్ > sls deploy —stage dev >

    sls deploy —stage prod → “dev” ༻ͷ CloudFormation ελοΫ͕࡞੒͞ΕΔ → “prod” ༻ͷ CloudFormation ελοΫ͕࡞੒͞ΕΔ Note: εςʔδ
  22. See also • [Qiita] Serverless Framework ͷ࢖͍ํ·ͱΊ @horike37

  23. ϓϩδΣΫτߏ੒ͷྫ

  24. ྫʣఆظδϣϒͷ࣮ߦ

  25. ྫʣఆظδϣϒͷ࣮ߦ δϣϒύϥϝʔλੜ੒

  26. ྫʣఆظδϣϒͷ࣮ߦ δϣϒ࣮ߦج൫ δϣϒύϥϝʔλΛ%ZOBNP%#͔Βऔಘ

  27. ྫʣఆظδϣϒͷ࣮ߦ service dir lambda functions scheduled task step functions task

    utility libs jinja2 tempalte files tests ࣮૷ྫ: Qiita هࣄࣗಈߋ৽ https://github.com/hassaku63/qiita-autogenaration
  28. σΟϨΫτϦߏ੒ͷҙਤ ʮԿʹؔ͢Δίʔυͳͷ͔ʯͷݟ௨͠Λྑ͔ͨͬͨ͘͠ • αʔόʔϨε͸ߏ੒ਤ͕ϐλΰϥεΠονʹͳΓ͕ͪ • ී௨ʹϞδϡʔϧ෼ׂ͍ͯ͘͠ײ֮ͱ΄ͱΜͲಉ͡ͰOK • ϚΠΫϩαʔϏε͝ͱʹτοϓϨϕϧͷ֊૚Λ෼͚Δ • ϚΠΫϩαʔϏε಺͸

    Lambda ͱͦΕҎ֎ͷϞδϡʔϧͰ෼཭ • lambda functions ʢτϦΨʔ͝ͱʹαϒϞδϡʔϧʹ෼͚Δʣ • ൚༻Ϟδϡʔϧʗઃఆ༻ͷม਺؅ཧ
  29. ʮεςʔδʯͷ؅ཧ

  30. ΍Γ͍ͨ͜ͱ • ʮεςʔδʯʹґଘ͠ͳ͍ίʔυϕʔεʹ͍ͨ͠ • ϩʔΧϧͰ΋ؾָʹಈ͔͍ͨ͠

  31. ΍Γ͍ͨ͜ͱ • ʮεςʔδʯʹґଘ͠ͳ͍ίʔυϕʔεʹ͍ͨ͠ • ϩʔΧϧͰ΋ؾָʹಈ͔͍ͨ͠ → ʮεςʔδʯݻ༗ͳ஋Λ͍͍ײ͡ʹ؅ཧ͢Δ → ϩʔΧϧͰ΋ lambda

    ͷϩδοΫίʔυΛ࣮ߦ͍ͨ͠
 ˞ ίϯςφϕʔεͷϩʔΧϧϥϯλΠϜͷ࿩͸ࠓճׂѪ
  32. 1. εςʔδʹґଘ͠ͳ͍ίʔυϕʔεʹ͍ͨ͠

  33. εςʔδ͝ͱʹ஋͕มΘΔ΋ͷ • Ϧιʔε໊ͷࢀরɹ • AWS Ϧιʔεͷ Resource Name, ARN •

    Endpoint (API Gateway, SQS Queue URL) • ΞϓϦέʔγϣϯݻ༗ͷઃఆ৘ใɹ • ϩάϨϕϧ • ར༻͢ΔAWSϦιʔεͷAPIࢀর༻ͷ໊લʢARN ΍ Ϧιʔε໊ɺURL ͳͲʣ • ઀ଓ͢Δ֎෦αʔϏεͷઃఆ (e.g Slack ͷ௨஌ઌνϟϯωϧʣ
  34. εςʔδ͝ͱʹ஋͕มΘΔ΋ͷ • Ϧιʔε໊ͷࢀর → ελοΫͷఆٛͰରॲ • AWS Ϧιʔεͷ Resource Name,

    ARN • Endpoint (API Gateway, SQS Queue URL) • ΞϓϦέʔγϣϯݻ༗ͷઃఆ৘ใ → ʢLambda ͷʣίʔυϕʔεͰରॲ • ϩάϨϕϧ • ར༻͢ΔAWSϦιʔεͷAPIࢀর༻ͷ໊લʢARN ΍ Ϧιʔε໊ɺURL ͳͲʣ • ઀ଓ͢Δ֎෦αʔϏεͷઃఆ (e.g Slack ͷ௨஌ઌνϟϯωϧʣ
  35. ؀ڥม਺ʹΑΔεςʔδݻ༗ม਺ͷ؅ཧ େࡶ೺ͳ؅ཧํ਑ɿ • ੩తͳ஋͔ͭػີ৘ใͰͳ͍ͳΒ؀ڥม਺ • ੩తͳ஋͔ͭػີ৘ใͳΒ Secret Manager/ Parameter Store

    • ಈతͳ஋ͳΒσʔλετΞ (e.g. DynamoDB)
  36. ؀ڥม਺ʹΑΔεςʔδݻ༗ม਺ͷ؅ཧ • python-dotenv • .env ϑΝΠϧΛѻ͏ύοέʔδ • serverless-dotenv-plugin • Serverless

    Framework ϓϥάΠϯ • .env.<stage> ϑΝΠϧͷએݴΛελοΫఆٛ಺ͰࢀরͰ͖Δ • ϦϦʔε໊ʢͷҰ෦͋Δ͍͸શ෦ʣͷએݴ • Lambda ؀ڥม਺ʹ౉͢ → ${provider.environment.XXX}
  37. 2. ϩʔΧϧͰ΋ؾָʹಈ͔͍ͨ͠

  38. جຊ • લड़ͷ͜ͱ͕Ͱ͖͍ͯΕ͹ɺ͋ͱ͸΄΅ʮςετ༰қੑʯͷ࿩ • Lambda handler ͷؔ਺ʹϕλॻ͖͗͢͠ͳ͍ • ϏδωεϩδοΫ΍֎෦αʔϏεݺͼग़͠͸ผϞδϡʔϧʹ෼཭ •

    ϋϯυϥ಺ͷ࢓ࣄ͸ event ͷύʔεͳͲɺ࠷௿ݶʹཹΊΔ • ϩʔΧϧίϯςφϥϯλΠϜͷ࢖༻΋ࣗવͱ΍Γ΍͘͢ͳΔ • e.g) SAM Local, DynamoDB Local, localstack
  39. αʔόʔϨεͬΆ͍෦෼ • εςʔδґଘͷ஋͸؀ڥม਺ͱͯ͠ѻ͏ • ؅ཧ͢ΔϞδϡʔϧΛ࡞ͬͯू໿ • e.g) settings.py, constants.py •

    example: hassaku63/sls-env-example settings.py
  40. Example • https://github.com/hassaku63/sls-env-example ૝ఆߏ੒ਤ

  41. Example • https://github.com/hassaku63/sls-env-example ૝ఆߏ੒ਤ 4242VFVF63- Λ஌͍ͬͯΔඞཁ͕͋Δ

  42. templates.yml

  43. templates.yml MBNCEBͷ؀ڥม਺ʹ Ϧιʔεಛఆʹ ඞཁͳ৘ใΛ౉͢

  44. settings.py

  45. settings.py εςʔδ༻ͷ EPUFOWΛϩʔυ 2VFVF63-ͷಛఆʹ ඞཁͳม਺Λϩʔυ

  46. ϩΪϯά

  47. ͠Ό΂Δ͜ͱʢ༧ఆʣ • JSON ϩΨʔΛ࢖ͬͯߏ଄Λ࣋ͨͤΔ • τϨʔε༻ͷIDΛ෇͚Δ • σίϨʔλͰڞ௨Խɹ

  48. ͠Ό΂Δ͜ͱʢ༧ఆมߋʣ • JSON ϩΨʔΛ࢖ͬͯߏ଄Λ࣋ͨͤΔ • τϨʔε༻ͷIDΛ෇͚Δɹ→ɹߏ੒ਤ͕೐Θ͖ͬͯͨΒ΍Δ • σίϨʔλͰڞ௨Խɹ→ɹ࣮૷ऀͷ޷Έ࣍ୈ

  49. JSONͰߏ଄Խ CloudWatch Logs Insights ͷݕࡧ͕ḿΔ
 (υοτ۠੾ΓͰωετͨ͠ϑΟʔϧυʹΞΫηεՄೳ) Amazon CloudWatch Logs -

    JSON ϩάͷϑΟʔϧυ https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html#CWL_AnalyzeLogData-discoverable-JSON-logs
  50. JSONͰߏ଄Խ ࣮૷ͷબ୒ࢶ 1. ϩΪϯά࣌ͷ msg Ҿ਺ʹ͸ඞͣ json.dump() ͔ͯ͠Β஋Λ౉͢ 2. JSON

    ϩΨʔΛαϙʔτ͢ΔύοέʔδΛ࢖͏ 3. ࣗલͰॻ͘ 1. Logger, Formatter Λܧঝ 2. Logging Adapter ͷར༻ʢݕ౼தʣ
  51. JSONͰߏ଄Խ Logger ʹٻΊ͍ͨཁ݅ʢ໛ࡧதʣ • JSONܗࣜͰग़ྗͰ͖Δ͜ͱ • Ұఆͷߏ଄ԽΛڧ੍Ͱ͖Δ͜ͱ • ʢগͳ͘ͱ΋ʣτοϓϨϕϧͷJSONϓϩύςΟ͕Ұ؏͍ͯ͠Δ •

    Lambda ͷ࣮ߦ؀ڥ৘ใΛউखʹ஫ೖͯ͘͠ΕΔ
  52. JSONͰߏ଄Խ ࢀߟʣLambda Context ͷ৘ใ͸༗༻ͳͷͰϩά౳Ͱ׆༻Ͱ͖Δ (doc) • Lambda ؔ਺໊ʗARNʗόʔδϣϯ • LogGroup

    / LogStream • Invocation timeout ·Ͱͷ࢒Γ࣌ؒ • etc ..
  53. τϨʔε༻IDͷ෇༩ • αʔόʔϨεߏ੒Ͱ͸ɺ1ͭͷϢʔεέʔε͕ෳ਺ͷϚΠΫϩ αʔϏεΛލ͍Ͱॲཧ͞ΕΔ • ӡ༻্ͷτϥϒϧγϡʔςΟϯάͰ͸Ϣʔεέʔε୯ҐͰϩά ΛߜΓࠐΈ͍ͨ • αʔϏεԣஅͰ࣮ߦ͝ͱʹҰҙͱͳΔID͕औΕΕ͹Good

  54. τϨʔε༻IDͷ෇༩ • جຊ͸ AWS X-Ray Λ࢖͓͚ͬͯ͹OK • τϨʔε༻IDʹ x-amzn-trace-id Λར༻Մೳ

    • Serverless Framework ͳΒ༗ޮԽ͸؆୯ • Step Functions Λ࢖͏ߏ੒ͩͱҰ೧Γඞཁ • Step Functions ͸ X-Ray ʹඇରԠʢ2020೥8݄ݱࡏʣ
  55. τϨʔε༻IDͷ෇༩ - X-Ray ͷ༗ޮԽ - template.yml ʹ2-3ߦ௥Ճ
 ʢ͓ΘΓʣ

  56. σίϨʔλͰڞ௨Խ Lambda ϋϯυϥ༻ͷσίϨʔλΛ࡞ͬͯΞλον͢Δ • e.g) ϋϯυϥͷ࣮ߦલޙͰඞͣϩΪϯάΛڬΉΑ͏ʹ͢Δ • e.g) ෳ਺ϨίʔυΛόϥͯ͠σίʔυ͔ͯ͠Β౉͢ʢSQS/Kinesis τϦΨʔʣ

  57. https://github.com/hassaku63/qiita-autogenaration/blob/master/qp/logs.py ࢖༻ྫ (Decorator)

  58. ࢖༻ྫ (lambda handler) https://github.com/hassaku63/qiita-autogenaration/blob/master/qp/functions/sfn/publish.py#L17-L28

  59. σίϨʔλͰڞ௨Խ Pros • ϋϯυϥ಺ͷهड़ྔ͕ݮΓɺຊ࣭తͳϩδοΫ෦෼͕ڧௐ͞ΕΔ • ൚༻తͳڞ௨ॲཧͷ࠶ར༻ • ։ൃϙϦγʔͷڧ੍ • ࣮ࡍʹ͜ͷख๏Λ࢖࣮ͬͨ૷͕͋Δ

    (e.g: awslabs/aws-lambda-powertools-python)
  60. σίϨʔλͰڞ௨Խ Cons • ΍Γ͗͢͸٫ͬͯίʔυ͕ಡΈͮΒ͘ͳΔʢओ؍ʣ • ଟগ௕͘ͳͬͯ΋ʮԿͯ͠Δ͔ʯ͕௚ײతʹಡΈԼͤΔͳΒ໰୊ͳ͍ɺͱ͍͏ߟ͑΋͋Δ • logging ࣌ͷελοΫϑϨʔϜ͕σίϨʔλʹͳͬͯ͠·͏ •

    ※ͨͩ͠ɺ΍ΓΑ͏͸͋Δʢݺͼग़͠ݩͷελοΫϑϨʔϜΛḷΔʗσίϨʔλͷҾ਺ͰରԠʣ • ϋϯυϥͷ్தͰ΋ϩΪϯά͍ͨ͠έʔε͸ग़ͯ͘Δ
  61. ·ͱΊ

  62. ·ͱΊ • αʔόʔϨεʹͳͬͯ΋ɺΞϓϦέʔγϣϯͷ෦෼͸͘͝ී௨ͷ Python • ߏ੒೺Ѳ͕΍Γ΍͘͢ͳΔΑ͏ʹιʔεΛߏ੒͢Δ • εςʔδݻ༗ͷύϥϝʔλ͸؀ڥม਺ͳͲ࢖ͬͯ؅ཧ • ϩΪϯά

    • JSONϩΨʔΛ࢖͍ͭͭɺσʔλߏ଄ΛҰ؏ͤ͞Δ • ՄೳͳΒτϨʔεIDΛৼΔ • σίϨʔλʹΑΔڞ௨Խ͸͓޷ΈͰ
  63. ʢࢀߟʣ৘ใݯ • AWS • AWS Ϋϥ΢υαʔϏε׆༻ࢿྉू: https://aws.amazon.com/jp/aws-jp-introduction/ • Webinar: https://aws.amazon.com/jp/about-aws/events/webinars/

    • Connpass: https://aws-serverless.connpass.com/ • Serverless community • Connpass: https://serverless.connpass.com/ • Serverless Operations • Blog archives: https://serverless.co.jp/blog-archives • Qiita: https://qiita.com/organizations/serverless-operations • Twitter • Yan Cui (@theburningmonk / AWS Serverless Hero): https://twitter.com/theburningmonk • Kensuke Shimokawa (@kensh /AWS Japan Serverless Specialist SA): https://twitter.com/_kensh • Keisuke Nishitani (@Keisuke69 / AWS Japan): https://twitter.com/Keisuke69 • Takahiro Horike (@horike37 / Serverless Operations): https://twitter.com/horike37
  64. We here hiring!