CEDEC 2018 の講演資料です https://2018.cedec.cesa.or.jp/session/detail/s5abc66488c745
Amazon ECS ͱϚωʔδυαʔϏεΛ׆༻ͨ͠ϑϧίϯςφߏʹΑΔήʔϜαʔόͷߏஙͱӡ༻2018.08.22 CEDEC 2018໘ന๏ਓΧϠοΫ ౻ݪढ़Ұ
View Slide
ࣗݾհ@fujiwaragithub.com/fujiwarasfujiwara.hatenablog.com
Game & Community
Agenda• Amazon ECS Ͱͷαʔόߏஙͱӡ༻• σϓϩΠख๏ͱൿಗใͷཧ• ϩάͷूͱετϦʔϛϯάॲཧ• GoݴޠʹΑΔӡ༻πʔϧ/ϛυϧΣΞ։ൃ
࠷ॳʹ͓ͼ·ͨɺຊ൪ϦϦʔεલʹඞਢͰ͋ΔෛՙࢼݧίϯςφΛ׆༻͢Δ͜ͱͰॊೈʹߦ͑ΔͨΊɺͦͷख๏ʹ͍ͭͯઆ໌͠·͢ɻ0ෛՙࢼݧʹ͍ͭͯऩ·Γ͖Βͳ͔ͬͨͨΊผͷࢿྉΛ͝ࢀর͍ͩ͘͞!GoͱίϯςφͰ࡞ΔWebΞϓϦέʔγϣϯϕϯνϚʔΧʔspeakerdeck.com/fujiwara3/benchmarker-built-with-go-and-container0 https://2018.cedec.cesa.or.jp/session/detail/s5abc66488c745
Amazon ECS Ͱͷαʔόߏஙͱӡ༻
Amazon ECSAWSͷίϯςφཧϚωʔδυαʔϏε͍ΘΏΔʮΦʔέετϨʔγϣϯπʔϧʯෳͷίϯςφͰىಈͨ͠ϓϩηε܈Λཧ͢Δͷଞʹ༗໊ͳͷ Kubernetes(k8s)
ECS ͷ֓೦
λεΫ - Taskؔ࿈͢ΔίϯςφϓϩηεΛҰͭʹ·ͱΊͨͷλεΫͷίϯςφ૬ޓʹ௨৴Ͱ͖Δ(localhost or hostname)k8s ͷ Pod ʹ૬
λεΫఆٛ - TaskDefinitionλεΫͷߏΛఆٛ͢Δͷؚ·ΕΔ(ෳͷ)ίϯςφɺλεΫࣗମͷωοτϫʔΫઃఆͳͲΛఆٛ• Πϝʔδ• CPU• ϝϞϦ• ϙʔτׂΓͯ ͳͲ
αʔϏε - ServiceλεΫΛෳݸଋͶͨͷ• ϩʔυόϥϯαʔ (ALB, NLB) ʹऩ༰͢Δ୯ҐʹͳΔ• ࢦఆͨ͠λεΫΛҡ࣋͢Δ• λεΫΛΦʔτεέʔϧͰ͖Δk8s ͷ Deployment ʹ૬
Ϋϥελʔ - ClusterλεΫ͕ஔ͞ΕΔ EC2 Πϯελϯεͷू߹EC2 Πϯελϯεࣗମ͕Ϛωʔδυʹͳͬͨ Fargate
ࠓ·Ͱͷ EC2 Ͱͷαʔόӡ༻ͱҧ͏EC2 ্ʹΞϓϦέʔγϣϯ/ϛυϧΣΞΛىಈ→ λεΫʹϓϩηεΛ·ͱΊΔϩʔυόϥϯαʔʹEC2ΠϯελϯεΛొ→ αʔϏεΛొΦʔτεέʔϧEC2ΠϯελϯεΛ૿ݮ͢Δ→ λεΫΛ૿ݮ͢Δ…ͨͩ͠ EC2 ίϯςφΠϯελϯε૿ݮ͠ͳ͍ͱ࣮ࡍʹ͑ΔϦιʔεมΘΒͳ͍ / Fargate Ͱղܾ
ECS Ͱͷαʔόߏͷํঢ়ଶΛ࣋ͨͳ͍ΞϓϦέʔγϣϯͷΈΛಈ͔࣌ؒ͢ͷঢ়ଶΛ࣋ͭϛυϧΣΞಈ͔͞ͳ͍(RDBMSͳͲͷετϨʔδ)ঢ়ଶͯ͢ϚωʔδυαʔϏεอଘ͢Δ(RDS, S3, ElastiCache...)
ECS Ͱͷαʔόߏͷํঢ়ଶΛ࣋ͨͳ͍ΞϓϦέʔγϣϯͷΈΛಈ͔͢σϓϩΠͷͨͼʹ৽͍͠λεΫ͕ىಈ͠ɺݹ͍λεΫམͪΔλεΫ͕ऴྃ/མͪΔͱϑΝΠϧͷॻ͖ࠐΈࣦΘΕΔEC2ϗετͷϑΝΠϧγεςϜΛϚϯτͰ͖Δ͕…ΫϥελͷͲͷEC2ͰλεΫΛىಈ͢Δ͔ECS͕ܾΊΔ→ Ӭଓ͢Δঢ়ଶͯ͢ϚωʔδυαʔϏεʂϩάϩʔΧϧϑΝΠϧʹ࣋ͨͣϚωʔδυαʔϏεసૹ
ECS ʹͯ͠Α͔ͬͨ͜ͱEC2্ͰͷϛυϧΣΞͷߏཧ͕ෆཁ͜Ε·Ͱ Chef ͰཧੈؒͰ͍͏΄Ͳਏ͘ͳ͍͕…ႈʹ͢Δͷ͕໘ɺΞϓϦέʔγϣϯΤϯδχΞʹෛ୲→ Dockerfile ͰΫϦʔϯͳঢ়ଶ͔ΒΠϯετʔϧ͢Δ͚ͩ
ECS ʹͯ͠Α͔ͬͨ͜ͱαʔόͷՃআָ͕ΫϥελͷEC2Πϯελϯεͯ͢ಉ͡ͷ͜Ε·Ͱαʔόͷछྨ͝ͱʹΠϯελϯεΛ࡞͍ͬͯͨApp, WebSocket, Batch, ϩάू etc...OS ʹ࠶ىಈඞਢͳύονΛ͍ͯͨ߹1.৽͍͠ AMI ͔ΒΠϯελϯεىಈ2.λεΫΛ৽͍͠ΠϯελϯεʹҠಈ3.ݹ͍ΠϯελϯεΛࣺͯΔ
ECS ʹͯ͠େมͩͬͨ͜ͱ͜Ε·Ͱͱͷҧ͍Λҙࣝͯ͠Γӽ͑Δ֓೦ΛΞϓϦέʔγϣϯΤϯδχΞʹཧղͯ͠Β͏ͷ͕େมϩʔΧϧϑΝΠϧʹґଘ͠ͳ͍ΈΛ࡞Δඞཁ͕͋Δ→ ঢ়ଶΛ࣋ͨͳ͍͜ͱΛపఈ͢ΔʲྫʳϩάΛ tail -f ͍͚ͨ͠ͲϑΝΠϧʹॻ͚ͳ͍…→ ϚωʔδυαʔϏεʴࣗ࡞πʔϧͰղܾ (ޙड़)
ECS ʹͯ͠େมͩͬͨ͜ͱϓϩηεͷঢ়ଶΛΈΔௐ͕ࠪ໘ʮstrace ͍ͨ͠ʂʯɾૂͬͨλεΫ͕ಈ࡞͍ͯ͠ΔEC2ͷϗετΛݟ͚ͭΔɾEC2 ʹ ssh ͯ͠ docker exec sh & straceίϯςφʹೖͬͯඞཁͳπʔϧ͕ͳ͔ͬͨΓ͢Δ
ECS / ίϯςφԽͷ෭࡞༻ঢ়ଶΛ࣋ͨͳ͍ɺϗετ͍ͭফ͑ͯͳ͍Α͏ʹͳͬͨ→ Spot Πϯελϯε͕׆༻Ͱ͖Δ௨ৗͷΦϯσϚϯυΠϯελϯεͷ༨ϦιʔεΛ҆͘(30%͙Β͍Ͱ) ఏڙ͢Δͷͨͩ͠༨Ϧιʔε͕ͳ͘ͳΔͱམͱ͞ΕΔ
Spot FleetෳͷΠϯελϯελΠϓAZ ΛΈ߹ΘͤͯϦιʔεΛ֬อམͪͨࣗಈతʹସ͕ىಈ
ECS Ͱ Spot ΠϯελϯεΛ҆શʹΓ͢ΦϯσϚϯυͷधཁ͕૿͑ͯ Spot ׂΓͯΔϦιʔε͕ͳ͘ͳΔͱ120ඵલʹ termination ௨͕དྷΔhttp://169.254.169.254/latest/meta-data/spot/instance-actionΛ polling→ ঢ়ଶมԽΛݕͨ͠ΒࣗࣗΛΫϥελ͔Β֎͢͜ΕΛ͠ͳ͍ͱλεΫ͕ಥવࢮ͢Δ
֤ΠϯελϯεͰୀॲཧλεΫΛಈ͔͢#!/bin/bashwhile sleep 5; doCONTENT=$(curl -sf http://169.254.169.254/latest/meta-data/spot/instance-action)if [ -z "$CONTENT" ]; thencontinuefiCLUSTER=$(curl -s http://localhost:51678/v1/metadata | jq -r .Cluster)CONTAINER_INSTANCE=$(curl -s http://localhost:51678/v1/metadata | jq -r .ContainerInstanceArn)aws ecs update-container-instances-state \--cluster "$CLUSTER" \--container-instances "$CONTAINER_INSTANCE" \--status DRAINING \&& exit 0donelocalhost:51678 = ECS agent!ਖ਼͜Ε͙Β͍ ECS agent ͕ͬͯ΄͍͠
Amazon ECS Ͱͷαʔόߏஙͱӡ༻ ·ͱΊ• ΞϓϦέʔγϣϯʹঢ়ଶΛ࣋ͨͳ͍͜ͱΛపఈ͢Δ• ঢ়ଶΛ࣋ͨͳ͍͜ͱͰಘΒΕΔϝϦοτΛڗड͠Α͏
σϓϩΠख๏ͱൿಗใͷཧ
͜Ε·Ͱͷ EC2 ্ͷσϓϩΠ• ֤ EC2 ʹΞʔΧΠϒϑΝΠϧΛ• S3 ʹ tar.gz ΛΞοϓϩʔυ• ֤ϗετ͕ S3 ͔Βμϯϩʔυͯ͠ల։• ϓϩηεΛ graceful ʹ࠶ىಈConsul + Stretcher (ࣗ࡞OSS) Ͱ࣮ݱ
ECSʹ͓͚ΔσϓϩΠϢʔβ͕Δ͜ͱ৽͍͠ΞϓϦέʔγϣϯؚ͕·ΕͨΠϝʔδΛ ECR ʹొ৽͍͠ΠϝʔδΛ͏λεΫఆٛΛొαʔϏεઃఆͰ৽͍͠λεΫఆٛΛ͏Α͏ʹมߋ
ECSʹ͓͚ΔσϓϩΠECS ͕Δ͜ͱ৽͍͠λεΫఆٛͰλεΫΛىಈ(LB ͕͋Δ߹ LB ʹΈࠐΉ)ਖ਼ৗʹىಈɺΈࠐΊͨΒݹ͍λεΫΛΛམͱ͢ਖ਼ৗʹىಈ͠ͳ͔ͬͨΒݹ͍ͷམͪͳ͍αʔϏεͷͯ͢ͷλεΫ͕৽͍͠ͷʹͳͬͨΒྃ
ϩʔϧόοΫݹ͍λεΫఆٛΛ͏Α͏ʹαʔϏεΛมߋ͢Δಛʹ ECS ʹػೳ͕͋ΔΘ͚Ͱͳ͍Ϣʔβ͕ࣗͰʮҰͭલʹσϓϩΠ͍ͯͨ͠λεΫఆٛʯΛσϓϩΠ͢͠!ىಈޙʹಈతͳ͜ͱΛͳΔ͘͠ͳ͍ͷ͕ॏཁ!ίϯςφىಈޙʹ࠷৽ίʔυΛϨϙδτϦ͔Βऔಘ"ίʔυΠϝʔδʹম͖ࠐΉ
σϓϩΠπʔϧ͍Ζ͍Ζ͋Δecs-cli github.com/aws/amazon-ecs-cliɹGo / docker-compose Ͱఆٛͨ͠ઃఆΛ ECS ʹσϓϩΠhako github.com/eagletmt/hakoɹRuby / ఆٛ YAMLecs-deploy github.com/silinternational/ecs-deployɹbash + aws-cli
σϓϩΠπʔϧࣗ࡞ecspresso github.com/kayac/ecspresso1. ݩʑίϯιʔϧૢ࡞ͰσϓϩΠ͍ͯͨ͠ͷ͕͋ͬͨ2. aws-cli ͰऔಘͰ͖ΔλεΫఆٛJSONΛͦͷ··͏ͨΊʹshell script Λॻ͍ͨ3. ίʔυ͕ෳࡶʹͳͬͨͷͰ Go + aws-sdk-go Ͱॻ͖ͦ͏͍ͭͷؒʹ͔ҭͬͯ͠·ͬͨ
ecspresso ͷಛ৭λεΫఆٛ aws-cli Ͱѻ͏JSONΛͦͷ··͑Δ→ݱঢ়ಈ͍͍ͯΔͷΛͦͷ··ཧରʹͰ͖ΔJSON ͷதͷΛ࣮ߦ࣌ͷڥมͰஔͯ͠ొͰ͖Δ{"taskDefinition": {"cpu": "4 vCPU","containerDefinitions": [{"name": "app","image": "prod/app:f6fb4dcc20",
{"taskDefinition": {"cpu": "{{ env `VCPU_NUM` `4` }} vCPU","containerDefinitions": [{"name": "app","image": "{{ must_env `SERVICE` }}/app:{{ must_env `TAG` }}",VCPU_NUM : prod | staging ͰҟͳΔCPUΛՄมʹSERVICE : prod | staging σϓϩΠઌͷڥΛΓସ͑ΔTAG : σϓϩΠ͝ͱʹมΘΔΠϝʔδͷλάͻͱͭͷ JSON Λෳڥʹ͍ճͤΔͷͰҡ࣋ཧָ͕
ίϯςφͷൿಗใͷઃఆΠϝʔδʹൿಗใΛম͖ࠐΉͷආ͚͍ͨম͖ࠐΉͱ docker pull ͨ͠Βͦ͜ʹΔഁغ͍ͨ͠߹ɹΠϝʔδΛͯ͢ݸผʹഁغ͢Δ͔͠ͳ͍→ ڥมͰ͢λεΫఆٛͰ environment ઃఆͰ͖Δ͕→ λεΫఆٛࣗମฏจͰอଘ͞ΕΔɹݹ͍ใͷഁغΠϝʔδಉ༷ʹʹͳΔ
ͲͷΑ͏ʹڥมΛઃఆ͢Δ͔λεΫىಈ࣌ʹ҆શͳͱ͜Ζ͔Βऔಘڥมʹઃఆ͔ͯ͠ΒϓϩηεΛىಈ͢Δ҆શͳઃఆอଘॴAWS Systems Manager (SSM) ύϥϝʔλετΞ
SSMύϥϝʔλετΞ
SSMύϥϝʔλετΞ• ֊Խͨ͠ path ໊ͰΛอଘ• ઃఆͷόʔδϣχϯάɺมߋऀͷཤྺཧ• KMSͰ҉߸Խͨ͠ΛอଘͰ͖Δ• KMSͷݖݶ͕ͳ͚ΕಡΊͳ͍• ίϯςφىಈ࣌ʹ͔͜͜ΒΛऔΓग़ͯ͠ڥมʹઃఆ͢ΕΑ͍
ύϥϝʔλετΞ͔ΒΛऔΓग़͢aws-ssm-env github.com/jamietsao/aws-ssm-env$ aws-ssm-env --paths=/prod/API_KEY=xxxxxxDB_PASS=productionpassentry point ͷ sh Ͱ࣮ߦɺexport ͔ͯ͠Β exec ͢Δ#!/bin/shexport AWS_REGION=ap-northeast-1export $(aws-ssm-env --paths=/prod/)exec /path/to/myapp
ύϥϝʔλετΞͷΛ export & execssmwrap github.com/handlename/ssmwrapฐࣾಉ྅࡞$ ssmwrap -paths=/prod/ -- /path/to/myappssm ͔Βऔಘͨ͠Λ env ʹઃఆͯ͠ exec ·Ͱͯ͘͠ΕΔɾexec ·Ͱ͢ΔͷͰ Docker ͷ entrypoint ʹࢦఆͰ͖Δɾshell scirpt Λܦ༝͠ͳ͍ͷͰվߦΛؚΉͷ͕ͳ͍ɾ-retries ΦϓγϣϯͰϦτϥΠճΛࢦఆͰ͖Δ
SSMύϥϝʔλετΞͷAPI Rate Limit ͕ݫ͠ΊେྔͷλεΫΛҰؾʹىಈ͢Δͱ Rate Limit Error(੍ݶαϙʔτʹ͍߹Θ͕ͤͨඇެ։)ɾaws-ssm-env ΤϥʔʹͳΔͱ panic Ͱࢮ͵ɹσϓϩΠ࣌ʹࢮΜͩλεΫ࠶ىಈ͞ΕΔ͕…ɾssmwrap -retries ઃఆͰϦτϥΠͰ͖Δ(͓નΊ)ɾىಈ࣌ʹ random sleep ͰλΠϛϯάΛͣΒͯ͠؇!ਖ਼ECSଆͰڥมʹઃఆͯ͠ىಈͯ͠΄͍͠
σϓϩΠख๏ͱൿಗใͷཧ ·ͱΊ• Blue-Green ͷརΛੜ͔ͤΔΑ͏ʹ͏• SSMύϥϝʔλετΞศར• Ͱͬͱڧ͘ͳͬͯ΄͍͠• ֤͍͖ࣗͯͬͯ͠·͠ΐ͏
ϩάͷूͱετϦʔϛϯάॲཧ
ϩάूͷੲͱࠓੲ (ʙ2011 Before Fluentd)ɾϑΝΠϧΛఆظతʹճऩɾλΠϜϥάେ
ϩάूͷੲͱࠓۙ (2012ʙ After Fluentd)ϑΝΠϧΛ fluentd ͕ஞ࣍ಡΈऔΓ ૹ৴ΞϓϦέʔγϣϯ͕ fluentd ૹ৴λΠϜϥάখ
ECS ͔Βͷϩάूൃੜ͢Δϩάେ·͔ʹ3छྨ1.ΞΫηεϩά2.ΞϓϦέʔγϣϯ/ϛυϧΣΞ͕ɹඪ४ग़ྗɺඪ४Τϥʔग़ྗʹు͖ग़͢ϩά3.ΞϓϦέʔγϣϯతʹҙຯ͕͋ΔߦಈϩάΞΫηεϩά nginx ͔Β /dev/stdout ʹग़ͤ2छྨͲͷΑ͏ʹूΊΔ͔
֤ίϯςφ͕ STDOUT, STDERR ʹు͖ग़ͨ͠ͷDocker logging driver ʹΑͬͯѻΘΕΔλεΫఆٛͰίϯςφ͝ͱʹࢦఆͰ͖Δawslogs : CloudWatch Logs ૹ৴fluend : Fluentd ૹ৴json-file : ϑΝΠϧอଘsyslog : syslogd ૹ৴(ଞʹ͋Δ)
awslogs or fluentd?CloudWatch Logsɹpros: ϚωʔδυͳͷͰԿߟ͑ͳͯ͘Α͍ɹcons: ͔ͦ͜Βߋʹผʹྲྀ͢ͷ͕ଟগ໘ (Lambda ͕ඞཁ)Fluentd:ɹpros: ߦಈϩάଞͱಉ༷ͷѻ͍͕Ͱ͖Δɹcons: fluentdΛࣗͰཱͯΔඞཁ͕͋Δ
Fluentd ʹ౷Ұ͢ΔߦಈϩάͲͪΒʹ͠Ζ Fluentd Ͱѻ͍͍ͨͷͰඞཁߦಈϩάɾλεΫʹΞϓϦέʔγϣϯͱҰॹʹ fluentd Λஔ (sidecar)ɾΞϓϦ͔Β localhost:24224 ʹૹ৴͢Δ(ߏԽϩά)STDOUT, STDERRɾECSαʔϏεͱͯ͠ NLB Λհͯ͠ fluentd Λஔɾ֤λεΫ Docker logging driver Ͱ NLB:24224 ૹ৴͢Δ
ECS Ͱ fluentd ΛͲ͏͏͔AWS ͷ߹ fluentd + plugin S3 Ͱ S3 อ͢Δͷ͕ఆ൪S3 ͷॻ͖ग़͠ 1ʙ5 ఔʹ͢Δɹ͋·Γ͍ͱ S3 ͷΦϒδΣΫτ͕ࡉΕʹͳΔ)ো࣌ͷ͜ͱΛߟ͑ΔͱఔͰίϯςφͷόοϑΝ(ϝϞϦ or ϩʔΧϧϑΝΠϧ)ʹอଘ͢ΔͷফࣦϦεΫ→ ৴པͷஔ͚ΔόοϑΝ͕΄͍͠
Kinesis Data Streams͛ࠐΜͩϝοηʔδ(ϩάʹݶΒͳ͍)Λ24࣌ؒɺॱংΛอͬͯอͯ͘͠ΕΔετϦʔϛϯάॲཧج൫ͷϚωʔδυαʔϏεOSS Ͱ Apache Kafka ͕ྨࣅϩάΛ S3 ʹॻ͖ग़͢લͷʮ৴པͰ͖ΔʯόοϑΝͱͯ͠࠷ద
Kinesis Data FirehoseετϦʔϛϯάσʔλΛఆظతʹɾS3ɾRedshiftɾElasticsearchʹॻ͖ग़͢ϚωʔδυαʔϏειʔεͱͯ͠ Kinesis Streams Λ͑Δ
࠷ऴతʹ͜͏
ʮtail -f ͍ͨ͠ΜͰ͕͢ʯϩά͕ϑΝΠϧʹॻ͔Εͳ͘ͳͬͨ→ tail -f Ͳ͏͢Δ?kinesis-tailfgithub.com/fujiwara/kinesis-tailfGo + aws-sdk-go Ͱࣗ࡞$ kinesis-tailf -stream docker-logsKinesis Streams Λඌ͠ඪ४ग़ྗʹు͖ग़͢ CLI-start -end ΦϓγϣϯͰಛఆ࣌ؒଳΛநग़Մೳ
Τϥʔϩά͚ͩநग़͍ͨ͠શϩάΛ kinesis-tailf | grep ?ྲྀྔ͕ଟ͍ͱͭΒ͍Streams ͷσʔλΛ Lambda ͰॲཧύλʔϯʹϚονͨ͠ϩά͚ͩผͷStreams ʹྲྀ͢͠ɾαʔόΤϥʔ status:5ɾॲཧʹ͕࣌ؒ3ඵҎ্ֻ͔ͬͨͷɹapptime:([3-9]|[1-9][0-9]+)\.ɾ(ERROR|WARN)
ϩά͔ΒͷΤϥʔ௨S3 Event notificationS3ʹΦϒδΣΫτ͕࡞/আ͞ΕͨΒLambdaΛݺͿΤϥʔநग़ετϦʔϜ͔Β S3 ͷΦϒδΣΫτ͕ੜ͞ΕͨΒ Slack ʹ௨ΠϕϯτυϦϒϯͳόονॲཧʹFirehose + S3 + Lambda ͕͑Δ
ϩά͔ΒͷΤϥʔ௨நग़͞ΕͨλάɺߦͱͱʹsnippetߘΤϥʔൃੜ͔Β1Ͱ௨େྔʹΤϥʔ͕ൃੜͯ͠1͝ͱʹ͔͠௨དྷͳ͍(ॏཁ)
ϩάͷूͱετϦʔϛϯάॲཧ ·ͱΊ• ϩά STDOUT/STDERR ʹग़ྗDocker Logging Driver Ͱૹ৴• Fluentd + ৴པͰ͖ΔόοϑΝͱͯ͠ͷ Kinesis Streams• Firehose + S3 + Lambda ͰΠϕϯτυϦϒϯͳσʔλॲཧ
GoݴޠʹΑΔӡ༻πʔϧ/ϛυϧΣΞ։ൃ
AWS / ECS 伱͕ؒଟ͍ࠓճࣗલͰ։ൃͨ͠ͷecspresso github.com/kayac/ecspressoσϓϩΠπʔϧssmwrap github.com/handlename/ssmwrapSSM ύϥϝʔλετΞͷΛڥมʹઃఆͯ͠ execkinesis-tailf github.com/fujiwara/kinesis-tailfKinesis StreamsΛඌ͢Δ
伱ؒՈ۩Λࣗ࡞͢Δ伱ؒΛຒΊΔπʔϧɺϛυϧΣΞΛ։ൃͯ͠ӡ༻վળখ͘͞ɺదʹ൚༻తͳͷΛ࡞ΔຊՈ͕伱ؒΛຒΊͨΒࣺͯΒΕΔΑ͏ʹ
伱ؒՈ۩ͷྫRin github.com/fujiwara/RinRedshift data Importer by SQSmessaging.S3 Πϕϯτ௨Ͱ SQS ʹૹ৴SQS ͷϝοηʔδ͔Β Redshiftʹ copyͯ͠औΓࠐΈΛߦ͏πʔϧ2015.05 ʹ։ൃ2015.10 Firehose ൃද2017.07 ౦ژϦʔδϣϯʹ Firehose
Before / AfterFluentd ͔ΒϩάΛૹ৴S3 / Redshift ʹॱ࣍औΓ͜·ΕΔͱ͍͏ߏͦͷ··Ϛωʔδυʹ
πʔϧ։ൃݴޠͱͯ͠ͷ GoγϯάϧόΠφϦʹͳΔϥϯλΠϜෆཁίϯςφͷϕʔεΠϝʔδΛબͳ͍ॻ͖͢͞ΑΓಡΈ͢͞Λॏࢹ→ ϝϯςφϯεੑ͕ߴ͍aws-sdk-goػೳɺΞοϓσʔτਃ͠ͳ͍͍উखਖ਼ඍົͳͱ͜Ζ͕…
ίϯςφڥ͚ͷ։ൃ TipsίϚϯυϥΠϯΦϓγϣϯͷΛڥม͔ΒಡΉ1func main() {var port intvar host stringflag.IntVar(&port, "port", 8080, "port number")flag.StringVar(&host, "host", "localhost", "hostname")flag.VisitAll(func(f *flag.Flag) {if s := os.Getenv(strings.ToUpper(f.Name)); s != "" {f.Value.Set(s)}})flag.Parse()fmt.Printf("%s:%d\n", host, port)}1 https://mattn.kaoriya.net/software/lang/go/20170609110526.htm
ίϚϯυϥΠϯΦϓγϣϯͷΛڥม͔ΒಡΉ$ myapp -host example.com -port 9999example.com:9999$ HOST=example.com PORT=9999 myappexample.com:9999ίϯςφͰίϚϯυϥΠϯΑΓڥมͷ΄͏͕ઃఆ͍͢͠
ઃఆϑΝΠϧʹڥม͔ΒΛຒΊࠐΉgithub.com/kayac/go-configGo ͷઃఆϑΝΠϧಡΈࠐΈ package# yamlaccount_id: "{{ must_env `MY_ACCOUNT_ID` }}"profile: "{{ env `PROFILE` `default` }}"must_env: ઃఆ͞Εͳ͍ঢ়ଶͰ࣮ߦ͞ΕΔͱ panicenv: ઃఆ͞Ε͍ͯͳ͍߹σϑΥϧτΛࢦఆYAML / JSON / TOML ʹରԠ
ઃఆϑΝΠϧΛϦϞʔτ͔Βऔಘ͢Δઃఆ͕ϑΝΠϧʹͳ͍ͬͯΔͱมߋͷࡍʹίϯςφͷ࠶Ϗϧυ & Ξοϓϩʔυ͕ඞཁ(ࢼߦࡨޡ͕໘)File / S3 / HTTP ͰऔಘͰ͖ΔΑ͏ʹ࡞͓ͬͯ͘ͱศར2$ CONFIG=s3://my-config-bucket/config.yaml mydaemon2 GoͰHTTPͱS3Λಁաతʹѻ͏ https://shogo82148.github.io/blog/2018/06/09/go-s3-protocol/
OSS ͱͯ͠࡞ΔOSS ʹͯ͠͠·͏ (ԾʹࣗΒ͔͠Θͳͯ͘)README ͙Β͍ؤுͬͯॻ͘(ϞνϕʔγϣϯʹͳΔ)աͳࣾࣄͷࠞೖΛ͙ʮͦͷϓϩδΣΫτҎ֎ʹҙຯ͕͋Δػೳͳͷ͔?ʯίϐϖͰຐվ൛͕૿৩͢ΔͷΛ͙ʮGitHubͰόΠφϦఏڙͯ͠ΔͷͰͦΕΛ͍ͬͯͩ͘͞ʯ
GoݴޠʹΑΔӡ༻πʔϧ/ϛυϧΣΞ։ൃ ·ͱΊ• AWS / ECS ͷ伱ؒΛখ͘͞ॻ͍ͨπʔϧͰຒΊΑ͏• ͋͑ͯOSSʹ͢Δ͜ͱͰ͖Ε͍ʹอͭ• Go ίϯςφڥͰ͍উख͕Α͍
Questions?• Amazon ECS Ͱͷαʔόߏஙͱӡ༻• σϓϩΠख๏ͱൿಗใͷཧ• ϩάͷूͱετϦʔϛϯάॲཧ• GoݴޠʹΑΔӡ༻πʔϧ/ϛυϧΣΞ։ൃ
fujiwara3
satoshirobatofujimoto
nnstt1
banjun
hankehly
kawaguti
takashi1029
shimap_sampo
microcms
keithyokoma
tenforward
m3hiro3
gotok365
stephaniewalter
sachag
keavy
danielanewman
akosma
zakiyama
roundedbygravity
notwaldorf
reverentgeek
yeseniaperezcruz
frogandcode
paulrobertlloyd