$30 off During Our Annual Pro Sale. View Details »

Python × AWS × Serverless 初学者が次の一歩を踏み出すためのテクニック/Python & AWS & Serverless - Step to the next stage from a beginner

hassaku63
August 28, 2020

Python × AWS × Serverless 初学者が次の一歩を踏み出すためのテクニック/Python & AWS & Serverless - Step to the next stage from a beginner

PyCon JP 2020 ( https://pycon.jp/2020/en/ ) の登壇資料

https://pycon.jp/2020/en/timetable/?id=203588

(2020/08/30 追記) Qiita 補足ブログ書きました https://qiita.com/hassaku_63/items/875ba4694538c6484526

hassaku63

August 28, 2020
Tweet

More Decks by hassaku63

Other Decks in Programming

Transcript

  1. Python × AWS × Serverless
    ॳֶऀ͕࣍ͷҰาΛ౿Έग़ͨ͢ΊͷςΫχοΫ
    Python & AWS & Serverless - Step to the next stage from a beginner
    PyCon JP 2020
    #pyconjp_2
    @hassaku

    View Slide

  2. ͸͡Ίʹ

    View Slide

  3. Speaker
    hassaku (Twitter: @hassaku_63)
    Serverworks Co., Ltd.

    View Slide

  4. Speaker
    ࣮ફ AWS CDK - TypeScript Ͱ

    Πϯϑϥ΋ΞϓϦ΋ʂ
    ڞஶ͠·ͨ͠ʢٕज़ॻయ8ʣ
    BOOTH: https://booth.pm/ja/items/1881928

    View Slide

  5. Serverworks
    • AWS ಛԽͷ SIer
    • ʮΫϥ΢υͰɺੈքΛɺ΋ͬͱɺ͸ͨΒ͖΍͘͢ʯ
    • AWSύʔτφʔωοτϫʔΫ

    ʮϓϨϛΞίϯαϧςΟϯάύʔτφʔʯೝఆ

    View Slide

  6. ൃදͷલʹ - ͓͜ͱΘΓ -
    • ࠂ஌ࡁΈͷΞδΣϯμ͔Βएׯͷมߋ͕͋Γ·͢ʢݟղͷมԽʹ൐͏΋ͷʣ
    • ૝ఆಡऀ
    • AWS ʹ͍ͭͯɺଟগͷར༻ܦݧ͕͋Δ
    • ʮαʔόʔϨεʯͷ֓೦΍ɺྑ͞͸ͳΜͱͳ͘஌͍ͬͯΔ
    • Lambda Function ΛσϓϩΠͨ͜͠ͱ͕͋Δʗ֓೦Λཧղ͍ͯ͠Δ
    • ʮαʔόʔϨεʯೖ໳هࣄͷ࣍ʹԿΛͨ͠Βྑ͍ͷ͔Θ͔Βͳ͍
    • Զͨͪ͸งғؾͰαʔόʔϨεΛ΍͍ͬͯΔ

    View Slide

  7. Agenda
    1. AWSʹ͓͚ΔαʔόʔϨεΞϓϦέʔγϣϯߏஙͷجຊ
    2. Serverless Framework جຊͷ͓͞Β͍
    3. ϓϩδΣΫτߏ੒ͷྫ
    4. ʮεςʔδʯͷ؅ཧ
    5. ϩΪϯά
    6. ·ͱΊ

    View Slide

  8. AWSʹ͓͚ΔαʔόʔϨε

    ΞϓϦέʔγϣϯߏஙͷجຊ

    View Slide

  9. αʔόʔϨεͷԿ͕೉͍͠ʁ
    • ֶशίετͷߴ͞
    • ֓೦ʹೃછΈ͕ͳ͍
    • ʮೖ໳ॻͷ࣍ʯ໰୊

    View Slide

  10. αʔόʔϨεͳʮࢥߟʯ
    • ඇಉظ
    • ΠϕϯτυϦϒϯ
    • εςʔτϨε

    View Slide

  11. αʔόʔϨεͳʮࢥߟʯ
    • ඇಉظ
    • ΠϕϯτυϦϒϯ
    • εςʔτϨε

    View Slide

  12. αʔόʔϨεͳొ৔ਓ෺
    • ϒϩʔΧʔ
    • Πϕϯτϋϯυϥ
    • σʔλετΞ
    CSPLFS
    FWFOU
    IBOEMFS
    %BUBTUPSF
    CSPLFS
    e.g SQS, SNS, Kinesis
    e.g DynamoDB
    Lambda

    View Slide

  13. See also
    • [PyCon JP 2017] The theory of Serverless development by Python (ཧ࿦
    ͔ΒֶͿPythonʹΑΔαʔόϨε։ൃ) @Masashi Terui

    View Slide

  14. Serverless Framework جຊͷ͓͞Β͍

    View Slide

  15. Serverless Framework
    • OSSͷσϓϩΠπʔϧ
    • Infrastructure as Code (IaC)
    • ϚϧνΫϥ΢υରԠ
    • Serverless Frameworkͷ࢖͍ํ·ͱΊ
    https://www.serverless.com/

    View Slide

  16. Serverless Framework
    1. ϓϩδΣΫτ࡞੒
    2. ίʔυΛॻ͘ (yaml, python)
    3. σϓϩΠύοέʔδΛ࡞੒
    4. σϓϩΠ
    … sls create
    … sls deploy (or sls package)
    … sls deploy

    View Slide

  17. σϓϩΠͷಈ͖
    > sls deploy —stage dev

    View Slide

  18. σϓϩΠͷಈ͖
    > sls deploy —stage dev
    $MPVE'PSNBUJPOςϯϓϨ
    -BNCEBσϓϩΠύοέʔδ
    Λ࡞੒

    View Slide

  19. σϓϩΠͷಈ͖
    S3 όέοτΛੜ੒͠ɺ

    ҎԼΛΞοϓϩʔυ
    ɾLambda ͷσϓϩΠύοέʔδ

    ɾCloudFormation tempalte
    "84 4
    ΁Ξοϓϩʔυ

    View Slide

  20. σϓϩΠͷಈ͖ $MPVE'PSNBUJPO
    σϓϩΠ
    CloudFormation deploy Λ࣮ߦ

    View Slide

  21. Serverless Framework ͷಠࣗ֓೦

    ˞σϓϩΠઌ؀ڥΛ۠ผ͢ΔͨΊͷ༻్
    > sls deploy —stage dev
    > sls deploy —stage prod
    → “dev” ༻ͷ CloudFormation ελοΫ͕࡞੒͞ΕΔ
    → “prod” ༻ͷ CloudFormation ελοΫ͕࡞੒͞ΕΔ
    Note: εςʔδ

    View Slide

  22. See also
    • [Qiita] Serverless Framework ͷ࢖͍ํ·ͱΊ @horike37

    View Slide

  23. ϓϩδΣΫτߏ੒ͷྫ

    View Slide

  24. ྫʣఆظδϣϒͷ࣮ߦ

    View Slide

  25. ྫʣఆظδϣϒͷ࣮ߦ
    δϣϒύϥϝʔλੜ੒

    View Slide

  26. ྫʣఆظδϣϒͷ࣮ߦ
    δϣϒ࣮ߦج൫
    δϣϒύϥϝʔλΛ%ZOBNP%#͔Βऔಘ

    View Slide

  27. ྫʣఆظδϣϒͷ࣮ߦ
    service dir
    lambda functions
    scheduled task
    step functions task
    utility libs
    jinja2 tempalte files
    tests
    ࣮૷ྫ: Qiita هࣄࣗಈߋ৽ https://github.com/hassaku63/qiita-autogenaration

    View Slide

  28. σΟϨΫτϦߏ੒ͷҙਤ
    ʮԿʹؔ͢Δίʔυͳͷ͔ʯͷݟ௨͠Λྑ͔ͨͬͨ͘͠
    • αʔόʔϨε͸ߏ੒ਤ͕ϐλΰϥεΠονʹͳΓ͕ͪ
    • ී௨ʹϞδϡʔϧ෼ׂ͍ͯ͘͠ײ֮ͱ΄ͱΜͲಉ͡ͰOK
    • ϚΠΫϩαʔϏε͝ͱʹτοϓϨϕϧͷ֊૚Λ෼͚Δ
    • ϚΠΫϩαʔϏε಺͸ Lambda ͱͦΕҎ֎ͷϞδϡʔϧͰ෼཭
    • lambda functions ʢτϦΨʔ͝ͱʹαϒϞδϡʔϧʹ෼͚Δʣ
    • ൚༻Ϟδϡʔϧʗઃఆ༻ͷม਺؅ཧ

    View Slide

  29. ʮεςʔδʯͷ؅ཧ

    View Slide

  30. ΍Γ͍ͨ͜ͱ
    • ʮεςʔδʯʹґଘ͠ͳ͍ίʔυϕʔεʹ͍ͨ͠
    • ϩʔΧϧͰ΋ؾָʹಈ͔͍ͨ͠

    View Slide

  31. ΍Γ͍ͨ͜ͱ
    • ʮεςʔδʯʹґଘ͠ͳ͍ίʔυϕʔεʹ͍ͨ͠
    • ϩʔΧϧͰ΋ؾָʹಈ͔͍ͨ͠
    → ʮεςʔδʯݻ༗ͳ஋Λ͍͍ײ͡ʹ؅ཧ͢Δ
    → ϩʔΧϧͰ΋ lambda ͷϩδοΫίʔυΛ࣮ߦ͍ͨ͠

    ˞ ίϯςφϕʔεͷϩʔΧϧϥϯλΠϜͷ࿩͸ࠓճׂѪ

    View Slide

  32. 1. εςʔδʹґଘ͠ͳ͍ίʔυϕʔεʹ͍ͨ͠

    View Slide

  33. εςʔδ͝ͱʹ஋͕มΘΔ΋ͷ
    • Ϧιʔε໊ͷࢀরɹ
    • AWS Ϧιʔεͷ Resource Name, ARN
    • Endpoint (API Gateway, SQS Queue URL)
    • ΞϓϦέʔγϣϯݻ༗ͷઃఆ৘ใɹ
    • ϩάϨϕϧ
    • ར༻͢ΔAWSϦιʔεͷAPIࢀর༻ͷ໊લʢARN ΍ Ϧιʔε໊ɺURL ͳͲʣ
    • ઀ଓ͢Δ֎෦αʔϏεͷઃఆ (e.g Slack ͷ௨஌ઌνϟϯωϧʣ

    View Slide

  34. εςʔδ͝ͱʹ஋͕มΘΔ΋ͷ
    • Ϧιʔε໊ͷࢀর → ελοΫͷఆٛͰରॲ
    • AWS Ϧιʔεͷ Resource Name, ARN
    • Endpoint (API Gateway, SQS Queue URL)
    • ΞϓϦέʔγϣϯݻ༗ͷઃఆ৘ใ → ʢLambda ͷʣίʔυϕʔεͰରॲ
    • ϩάϨϕϧ
    • ར༻͢ΔAWSϦιʔεͷAPIࢀর༻ͷ໊લʢARN ΍ Ϧιʔε໊ɺURL ͳͲʣ
    • ઀ଓ͢Δ֎෦αʔϏεͷઃఆ (e.g Slack ͷ௨஌ઌνϟϯωϧʣ

    View Slide

  35. ؀ڥม਺ʹΑΔεςʔδݻ༗ม਺ͷ؅ཧ
    େࡶ೺ͳ؅ཧํ਑ɿ
    • ੩తͳ஋͔ͭػີ৘ใͰͳ͍ͳΒ؀ڥม਺
    • ੩తͳ஋͔ͭػີ৘ใͳΒ Secret Manager/ Parameter Store
    • ಈతͳ஋ͳΒσʔλετΞ (e.g. DynamoDB)

    View Slide

  36. ؀ڥม਺ʹΑΔεςʔδݻ༗ม਺ͷ؅ཧ
    • python-dotenv
    • .env ϑΝΠϧΛѻ͏ύοέʔδ
    • serverless-dotenv-plugin
    • Serverless Framework ϓϥάΠϯ
    • .env. ϑΝΠϧͷએݴΛελοΫఆٛ಺ͰࢀরͰ͖Δ
    • ϦϦʔε໊ʢͷҰ෦͋Δ͍͸શ෦ʣͷએݴ
    • Lambda ؀ڥม਺ʹ౉͢ → ${provider.environment.XXX}

    View Slide

  37. 2. ϩʔΧϧͰ΋ؾָʹಈ͔͍ͨ͠

    View Slide

  38. جຊ
    • લड़ͷ͜ͱ͕Ͱ͖͍ͯΕ͹ɺ͋ͱ͸΄΅ʮςετ༰қੑʯͷ࿩
    • Lambda handler ͷؔ਺ʹϕλॻ͖͗͢͠ͳ͍
    • ϏδωεϩδοΫ΍֎෦αʔϏεݺͼग़͠͸ผϞδϡʔϧʹ෼཭
    • ϋϯυϥ಺ͷ࢓ࣄ͸ event ͷύʔεͳͲɺ࠷௿ݶʹཹΊΔ
    • ϩʔΧϧίϯςφϥϯλΠϜͷ࢖༻΋ࣗવͱ΍Γ΍͘͢ͳΔ
    • e.g) SAM Local, DynamoDB Local, localstack

    View Slide

  39. αʔόʔϨεͬΆ͍෦෼
    • εςʔδґଘͷ஋͸؀ڥม਺ͱͯ͠ѻ͏
    • ؅ཧ͢ΔϞδϡʔϧΛ࡞ͬͯू໿
    • e.g) settings.py, constants.py
    • example: hassaku63/sls-env-example settings.py

    View Slide

  40. Example
    • https://github.com/hassaku63/sls-env-example
    ૝ఆߏ੒ਤ

    View Slide

  41. Example
    • https://github.com/hassaku63/sls-env-example
    ૝ఆߏ੒ਤ
    4242VFVF63-
    Λ஌͍ͬͯΔඞཁ͕͋Δ

    View Slide

  42. templates.yml

    View Slide

  43. templates.yml
    MBNCEBͷ؀ڥม਺ʹ
    Ϧιʔεಛఆʹ
    ඞཁͳ৘ใΛ౉͢

    View Slide

  44. settings.py

    View Slide

  45. settings.py
    εςʔδ༻ͷ
    EPUFOWΛϩʔυ
    2VFVF63-ͷಛఆʹ
    ඞཁͳม਺Λϩʔυ

    View Slide

  46. ϩΪϯά

    View Slide

  47. ͠Ό΂Δ͜ͱʢ༧ఆʣ
    • JSON ϩΨʔΛ࢖ͬͯߏ଄Λ࣋ͨͤΔ
    • τϨʔε༻ͷIDΛ෇͚Δ
    • σίϨʔλͰڞ௨Խɹ

    View Slide

  48. ͠Ό΂Δ͜ͱʢ༧ఆมߋʣ
    • JSON ϩΨʔΛ࢖ͬͯߏ଄Λ࣋ͨͤΔ
    • τϨʔε༻ͷIDΛ෇͚Δɹ→ɹߏ੒ਤ͕೐Θ͖ͬͯͨΒ΍Δ
    • σίϨʔλͰڞ௨Խɹ→ɹ࣮૷ऀͷ޷Έ࣍ୈ

    View Slide

  49. JSONͰߏ଄Խ
    CloudWatch Logs Insights ͷݕࡧ͕ḿΔ

    (υοτ۠੾ΓͰωετͨ͠ϑΟʔϧυʹΞΫηεՄೳ)
    Amazon CloudWatch Logs - JSON ϩάͷϑΟʔϧυ https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html#CWL_AnalyzeLogData-discoverable-JSON-logs

    View Slide

  50. JSONͰߏ଄Խ
    ࣮૷ͷબ୒ࢶ
    1. ϩΪϯά࣌ͷ msg Ҿ਺ʹ͸ඞͣ json.dump() ͔ͯ͠Β஋Λ౉͢
    2. JSON ϩΨʔΛαϙʔτ͢ΔύοέʔδΛ࢖͏
    3. ࣗલͰॻ͘
    1. Logger, Formatter Λܧঝ
    2. Logging Adapter ͷར༻ʢݕ౼தʣ

    View Slide

  51. JSONͰߏ଄Խ
    Logger ʹٻΊ͍ͨཁ݅ʢ໛ࡧதʣ
    • JSONܗࣜͰग़ྗͰ͖Δ͜ͱ
    • Ұఆͷߏ଄ԽΛڧ੍Ͱ͖Δ͜ͱ
    • ʢগͳ͘ͱ΋ʣτοϓϨϕϧͷJSONϓϩύςΟ͕Ұ؏͍ͯ͠Δ
    • Lambda ͷ࣮ߦ؀ڥ৘ใΛউखʹ஫ೖͯ͘͠ΕΔ

    View Slide

  52. JSONͰߏ଄Խ
    ࢀߟʣLambda Context ͷ৘ใ͸༗༻ͳͷͰϩά౳Ͱ׆༻Ͱ͖Δ (doc)
    • Lambda ؔ਺໊ʗARNʗόʔδϣϯ
    • LogGroup / LogStream
    • Invocation timeout ·Ͱͷ࢒Γ࣌ؒ
    • etc ..

    View Slide

  53. τϨʔε༻IDͷ෇༩
    • αʔόʔϨεߏ੒Ͱ͸ɺ1ͭͷϢʔεέʔε͕ෳ਺ͷϚΠΫϩ
    αʔϏεΛލ͍Ͱॲཧ͞ΕΔ
    • ӡ༻্ͷτϥϒϧγϡʔςΟϯάͰ͸Ϣʔεέʔε୯ҐͰϩά
    ΛߜΓࠐΈ͍ͨ
    • αʔϏεԣஅͰ࣮ߦ͝ͱʹҰҙͱͳΔID͕औΕΕ͹Good

    View Slide

  54. τϨʔε༻IDͷ෇༩
    • جຊ͸ AWS X-Ray Λ࢖͓͚ͬͯ͹OK
    • τϨʔε༻IDʹ x-amzn-trace-id Λར༻Մೳ
    • Serverless Framework ͳΒ༗ޮԽ͸؆୯
    • Step Functions Λ࢖͏ߏ੒ͩͱҰ೧Γඞཁ
    • Step Functions ͸ X-Ray ʹඇରԠʢ2020೥8݄ݱࡏʣ

    View Slide

  55. τϨʔε༻IDͷ෇༩ - X-Ray ͷ༗ޮԽ -
    template.yml ʹ2-3ߦ௥Ճ

    ʢ͓ΘΓʣ

    View Slide

  56. σίϨʔλͰڞ௨Խ
    Lambda ϋϯυϥ༻ͷσίϨʔλΛ࡞ͬͯΞλον͢Δ
    • e.g) ϋϯυϥͷ࣮ߦલޙͰඞͣϩΪϯάΛڬΉΑ͏ʹ͢Δ
    • e.g) ෳ਺ϨίʔυΛόϥͯ͠σίʔυ͔ͯ͠Β౉͢ʢSQS/Kinesis τϦΨʔʣ

    View Slide

  57. https://github.com/hassaku63/qiita-autogenaration/blob/master/qp/logs.py
    ࢖༻ྫ (Decorator)

    View Slide

  58. ࢖༻ྫ (lambda handler)
    https://github.com/hassaku63/qiita-autogenaration/blob/master/qp/functions/sfn/publish.py#L17-L28

    View Slide

  59. σίϨʔλͰڞ௨Խ
    Pros
    • ϋϯυϥ಺ͷهड़ྔ͕ݮΓɺຊ࣭తͳϩδοΫ෦෼͕ڧௐ͞ΕΔ
    • ൚༻తͳڞ௨ॲཧͷ࠶ར༻
    • ։ൃϙϦγʔͷڧ੍
    • ࣮ࡍʹ͜ͷख๏Λ࢖࣮ͬͨ૷͕͋Δ (e.g: awslabs/aws-lambda-powertools-python)

    View Slide

  60. σίϨʔλͰڞ௨Խ
    Cons
    • ΍Γ͗͢͸٫ͬͯίʔυ͕ಡΈͮΒ͘ͳΔʢओ؍ʣ
    • ଟগ௕͘ͳͬͯ΋ʮԿͯ͠Δ͔ʯ͕௚ײతʹಡΈԼͤΔͳΒ໰୊ͳ͍ɺͱ͍͏ߟ͑΋͋Δ
    • logging ࣌ͷελοΫϑϨʔϜ͕σίϨʔλʹͳͬͯ͠·͏
    • ※ͨͩ͠ɺ΍ΓΑ͏͸͋Δʢݺͼग़͠ݩͷελοΫϑϨʔϜΛḷΔʗσίϨʔλͷҾ਺ͰରԠʣ
    • ϋϯυϥͷ్தͰ΋ϩΪϯά͍ͨ͠έʔε͸ग़ͯ͘Δ

    View Slide

  61. ·ͱΊ

    View Slide

  62. ·ͱΊ
    • αʔόʔϨεʹͳͬͯ΋ɺΞϓϦέʔγϣϯͷ෦෼͸͘͝ී௨ͷ Python
    • ߏ੒೺Ѳ͕΍Γ΍͘͢ͳΔΑ͏ʹιʔεΛߏ੒͢Δ
    • εςʔδݻ༗ͷύϥϝʔλ͸؀ڥม਺ͳͲ࢖ͬͯ؅ཧ
    • ϩΪϯά
    • JSONϩΨʔΛ࢖͍ͭͭɺσʔλߏ଄ΛҰ؏ͤ͞Δ
    • ՄೳͳΒτϨʔεIDΛৼΔ
    • σίϨʔλʹΑΔڞ௨Խ͸͓޷ΈͰ

    View Slide

  63. ʢࢀߟʣ৘ใݯ
    • AWS
    • AWS Ϋϥ΢υαʔϏε׆༻ࢿྉू: https://aws.amazon.com/jp/aws-jp-introduction/
    • Webinar: https://aws.amazon.com/jp/about-aws/events/webinars/
    • Connpass: https://aws-serverless.connpass.com/
    • Serverless community
    • Connpass: https://serverless.connpass.com/
    • Serverless Operations
    • Blog archives: https://serverless.co.jp/blog-archives
    • Qiita: https://qiita.com/organizations/serverless-operations
    • Twitter
    • Yan Cui (@theburningmonk / AWS Serverless Hero): https://twitter.com/theburningmonk
    • Kensuke Shimokawa (@kensh /AWS Japan Serverless Specialist SA): https://twitter.com/_kensh
    • Keisuke Nishitani (@Keisuke69 / AWS Japan): https://twitter.com/Keisuke69
    • Takahiro Horike (@horike37 / Serverless Operations): https://twitter.com/horike37

    View Slide

  64. We here hiring!

    View Slide