Pro Yearly is on sale from $80 to $50! »

ECSとSQSでスケーラブルなバッチを作った

 ECSとSQSでスケーラブルなバッチを作った

Ab963de01b53e635d6e699d6d7d162b1?s=128

Takafumi Yoshida

August 17, 2019
Tweet

Transcript

  1. &$4ͱ424Ͱ εέʔϥϒϧͳόονΛ࡞ͬͨ ٢ాوจ ![FQIJSBOTBT  Ϋϥεϝιουגࣜձࣾ

  2. εϥΠυ͸ޙͰೖख͢Δ͜ͱ͕ग़དྷ·͢ͷͰ ൃදதͷ಺༰ΛϝϞ͢Δඞཁ͸͋Γ·ͤΜɻ ࣸਅࡱӨΛ͢Δ৔߹͸ ϑϥογϡɾγϟολʔԻ͕ग़ͳ͍Α͏ʹ͝഑ྀ͍ͩ͘͞ Attention

  3.   #jawsug #jawsoka #soracomug

  4. ࣗݾ঺հ   ٢ాوจ ![FQIJSBOTBT  wΫϥεϝιουגࣜձࣾ w$9ࣄۀຊ෦αʔόαΠυΤϯδχΞ wԬࢁ+BWBϢʔβձ୅දΦʔϓϯη ϛφʔԬࢁ࣮ߦҕһ

    w޷͖ͳ"84ͷαʔϏε w&$4 %ZOBNP%#
  5. ΞδΣϯμ   wΞʔΩςΫνϟ֓ཁ w4XJUDI3PMFʹ͍ͭͯ w424Ͱ΍ͬͨ͜ͱ w&$4Ͱ΍ͬͨ͜ͱ w΍ͬͯΈͨ

  6.   ΞʔΩςΫνϟ֓ཁ

  7. ΍Γ͍ͨ͜ͱ   wσʔλͷҰׅߋ৽ wݩσʔλ͸$47 wσʔλྔ͸े਺ສ݅ఔ౓ w*%ͱɺߋ৽಺༰͕ೖ͍ͬͯΔ wߋ৽ʹ͸֎෦ͷ"1*Λୟ͘

  8.  

  9.   Switch RoleͰ ΍ͬͨ͜ͱ

  10.   w424΁ͷσʔλૹ৴ॲཧͰ4XJUDI3PMF͍ͨ͠ w4XJUDI3PMF͢Δʹ͸.'"ඞਢ wBXTDMJͰ͋Ε͹్தͰτʔΫϯΛೖྗͰ͖Δ w4%,ͩͱࣗલͰΫϨσϯγϟϧΛऔಘͯ͠΍ Δඞཁ͕͋Δ

  11. BXTDPOpH   [default] region = ap-northeast-1 output = json

    [profile hoge] region = ap-northeast-1 source_profile = default role_arn = arn:aws:iam::ACCOUNT_ID:role/john-doe mfa_serial = arn:aws:iam::ACCOUNT_ID:mfa/john-doe
  12. BXTDMJͰ4XJUDI3PMF͢Δ৔߹   $ AWS_PROFILE=hoge aws s3 ls Enter MFA

    code for arn:aws:iam::ACCOUNT_ID:mfa/john-doe [MFAτʔΫϯΛೖྗ͢Δ]
  13. "3/ɺ.'"τʔΫϯɺTUTΫϥΠΞϯτ   sts_client = Aws::STS::Client.new(region: 'ap-northeast-1') role_arn = `aws

    configure get role_arn --profile hoge`.chomp serial_number = `aws configure get mfa_serial --profile hoge`.chomp puts "Input MFA token code..." token_code = gets.chomp
  14. ΫϨσϯγϟϧੜ੒͠424ΫϥΠΞϯτΛ࡞੒   role_credentials = Aws::AssumeRoleCredentials.new( client: sts_client, role_arn: role_arn,

    role_session_name: "hoge_session", serial_number: serial_number, token_code: token_code) Aws::SQS::Client.new(credentials: role_credentials)
  15.   w؀ڥม਺"84@130'*-&͸ར༻͠ͳ͍ wBXTDPOpHVSFHFUͰඞཁͳ"3/Λऔಘ͢Δ w.'"τʔΫϯ͸ผ్ɺೖྗͤ͞Δ wTUTΫϥΠΞϯτΛ࡞੒͠ɺ "TTVNF3PMF$SFEFOUJBMTͰɺΫϨσϯγϟϧ Λऔಘ͢Δ

  16.   SQSͰ΍ͬͨ͜ͱ

  17.   wෳ਺ͷλεΫ͔ΒΞΫηε͞ΕΔͷͰɺ͜ΕΛ ͍͍ײ͡ʹॲཧͰ͖Δ wॲཧͰ͖ͳ͔ͬͨσʔλΛɺผΩϡʔʹҠͯ͠ ϦτϥΠ͠΍͘͢͢Δ w424ͷ%FBE-FUUFS2VFVFͷ࢓૊ΈΛ࢖͏

  18.  

  19.   #PEZ 3FDFJWF$PVOU 7JTJCMF   536&

  20.   #PEZ 3FDFJWF$PVOU 7JTJCMF   '"-4&

  21.   #PEZ 3FDFJWF$PVOU 7JTJCMF   536& VisibilityTimeoutΛա͗ͯ΋Delete͞Εͳ͔ͬͨ৔߹

  22.   #PEZ 3FDFJWF$PVOU 7JTJCMF Receive Count͕࠷େReceive CountΛ௒͑ͨ৔߹ #PEZ 3FDFJWF$PVOU

    7JTJCMF   536& DLQ΁Ҡಈ
  23.   ECSͰ΍ͬͨ͜ͱ

  24.   w'BSHBUFͰϦιʔε؅ཧͷखؒΛݮΒ͍ͨ͠ wฒྻͰ࣮ߦͰ͖ΔΑ͏ʹ͍ͨ͠ wঢ়گʹԠͯ͡ɺλεΫͷ਺Λௐ੔͍ͨ͠ w$MJFOU4FDSFUͳͲΛ҆શʹѻ͍͍ͨ

  25.  

  26. ύϥϝʔλετΞʹઃఆ஋Λ֨ೲ   aws ssm put-parameter \ --name /ClientId \

    --value CLIENT_ID_XXXX \ --type String
  27. λεΫఆ͔ٛΒࢀর   ContainerDefinitions: - Name: app ... Secrets: -

    Name: CLIENT_ID ValueFrom: !Sub "arn:aws:ssm:ap-northeast-1:${AWS::AccountId}:parameter/ClientId" - Name: CLIENT_SECRET ValueFrom: !Sub "arn:aws:ssm:ap-northeast-1:${AWS::AccountId}:parameter/ClientSecret" λεΫ಺ͷ؀ڥม਺Ͱ஋ΛऔಘͰ͖Δ
  28.   ΍ͬͯΈͨ

  29.   wର৅σʔλສ݅ w424΁ͷσʔλૹ৴ʹ࣌ؒ wʢͳΜ͔վળ͍ͨ͠ؾ͕͢Δ w&$4ͷόονॲཧ͕࣌ؒະຬͰऴྃ

  30.   w4%,Ͱ4XJUDI3PMF͢Δʹ͸ͻͱखؒඞཁ w424͸࢓૊ΈΛཧղ͔ͯͭ͑͠͹ɺ͘͢͝ศ ར w&$4ͷฒྻλεΫΛ࢖ͬͯɺεέʔϥϒϧʹ͠ Α͏

  31. None