#hbstudy の 79 回目で SRE の話をしました。
SRE at CookpadΫοΫύουגࣜձࣾ ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ٢ ཽଠ ( @rrreeeyyy )hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 1
View Slide
Agenda• SRE ʹ͍ͭͯ(վΊͯ)• ΫοΫύουʹ͓͚ΔΠϯϑϥߏ• EC2 with spot instances• hako + ECS ڥ• ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ• ࣄͷྲྀΕɾऔΓΈʹ͍ͭͯɹhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 2
!SSSFFFZZZ!SSSFFFZZZIUUQTSSSFFFZZZDPN:PTIJLBXB3ZPUBMe• Yoshikawa Ryota ( @rrreeeyyy [reɪ] )• גࣜձࣾϋʔτϏʔπ (2010/11 ʙ 2016/12)• ΫοΫύουגࣜձࣾ (2017/01 ʙ)• ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ• ڵຯྖҬ• ϞχλϦϯάɾ࣌ܥྻσʔλϕʔε• ࢄγεςϜɾϩʔυόϥϯα• झຯ• League of Legends, ΑΑΫϩχΫϧhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 3
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 4
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 5
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 6
(վΊͯ) Site Reliability Engineering• αΠτͷ৴པੑͷશͯʹൣғΛ࣋ͭ• ಛʹɺιϑτΣΞɾγεςϜΤϯδχΞϦϯάΛ༻͍ͯ՝Λղܾ͢Δ• ྫ͑ɺύϑΥʔϚϯε• ύϑΥʔϚϯεʹ͕͋Δίʔυ SRE ͕ࣗίʔυΛॻ͍ͯमਖ਼͢Δ• ྫ͑ɺεέʔϥϏϦςΟɾΞϕΠϥϏϦςΟ• ਓؒʹΑΔखಈͷ࡞ۀΛίʔυʹΑΓࣗಈԽ͢Δ• ଐਓੑΛഉআͯ҆͠ఆͤ͞ΔɾਓؒΛ૿͞ͳͯ͘αʔϏεΛεέʔϧͤ͞Δ• ن͕૿େͯ҆͠ఆͯ͠αʔϏεΛఏڙͰ͖Δঢ়ଶʹ͢Δhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 7
SRE ͷ׆ಈSRE ͷ׆ಈେ͖͚ͯ࣍͘ͷ 4 ͭʹྨͰ͖Δͱ͞Ε͍ͯΔ• So#ware Engineering• Systems Engineering• Toil• Overheadhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 8
SRE ͷ׆ಈ• So#ware Engineering• ίʔυͷ࡞मਖ਼, ͦΕʹؔ࿈͢ΔઃܭυΩϡϝϯςʔγϣϯͷ࡞ۀͳͲ• ࣗಈԽεΫϦϓτͷ࡞, πʔϧϑϨʔϜϫʔΫͷ࡞ ...• αʔϏεͷ৴པੑΛߴΊΔػೳͷՃ ...• Systems Engineering• 1 ճͷ࡞ۀͰӬଓతͳվળΛੜΉ࣍ͷΑ͏ͳ࡞ۀ• ຊ൪γεςϜͷઃఆมߋ ...• ։ൃνʔϜͷαʔϏεΞʔΩςΫνϟઃܭ ...hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 9
SRE ͷ׆ಈ• Toil• αʔϏεΛՔಇͤ͞ΔͨΊ͚ͩʹ܁Γฦ͠ߦΘΕΔखಈͰͷ࡞ۀ• ظతͳՁΛ࣋ͨͳ͍, αʔϏεͷʹରͯ͠ O(n) Ͱ͋Δ ...• ͭ·Βͳ͍ࣄͰظతͳՁ͕͋Ε Toil Ͱͳ͍• Overhead• αʔϏεΛՔಇͤ͞Δ͜ͱʹ݁͠ͳ͍ཧతͳ࡞ۀ• ࠾༻, ϛʔςΟϯά, ධՁ, τϨʔχϯά ...hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 10
ΫοΫύουʹ͓͚Δ SREhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 11
ΫοΫύουʹ͓͚Δ SRE• ΠϯϑϥετϥΫνϟʔ෦• ͷதʹ SRE άϧʔϓ͕͋Δ• DWH, Security, ࣾIT ͕ಉ͡෦ॺʹ͋Δ• ϝϯόʔ SRE ͚ͩͰ 10 ਓఔ + ւ֎ʹ 2 ໊ఔ• ࠃ 7 ׂɾάϩʔόϧ 1 3 ׂ͙Β͍ͷൺ1 h$ps://speakerdeck.com/sorah/building-infrastructure-for-our-global-servicehbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 12
ΫοΫύουʹ͓͚ΔΠϯϑϥߏhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 13
ΫοΫύουʹ͓͚ΔΠϯϑϥߏ• ༷ʑ͋Δ͕ϝΠϯͰಈ͍͍ͯΔͷେ͖͚ͯ͘ 2 ͭ• EC2 + spotscaler ڥ• EC2 ʹΞϓϦέʔγϣϯΛσϓϩΠ͍ͯ͠Δߏ• EC2 Πϯελϯεεϙοτʹͳ͍ͬͯΔ• ECS + hako ڥ• ECS Ϋϥελʹରͯ͠ίϯςφΛσϓϩΠ͍ͯ͠Δߏ• σϓϩΠϝϯτʹ eagletmt/hako Λར༻͍ͯ͠Δhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 14
EC2 + spotscaler ڥ• ΫοΫύουͷ͍ΘΏΔը໘ͷඳը෦ͳͲ͕͜Εʹ͋ͨΔ• PC ൛αΠτ, εϚϗ༻ͷ API ... ͳͲ• Route53 + ALB + EC2 ͷ AWS ͷҰൠతͳߏ• EC2 spot instance Λ spotscaler 2 Λͬͯཧ͍ͯ͠Δ• deployment serf + sorah/mamiya 3 Λͬͯߦ͍ͬͯΔ3 h$ps://github.com/eagletmt/hako2 h$ps://speakerdeck.com/ryotarai/spot-instances-in-cookpad-number-cookpadtechconf-2017hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 15
ECS + hako ڥ• ΫοΫύουͷ༷ʑͳϚΠΫϩαʔϏε͕͜Εʹ͋ͨΔ• λΠϜϥΠϯ, ࠂ, ೝূ... ͳͲ• ػցֶशͷج൫͜͜ʹ͍ͬͯΔʢg2 Πϯελϯεͷ ECS Ϋϥελʣ• ECS ͷΫϥελʹରͯ͠ hako 3 Λར༻ͯ͠σϓϩΠ͍ͯ͠Δ• GHE ্ͷϦϙδτϦʹΞϓϦέʔγϣϯఆٛͷ PR Λग़ͯ͠Β͏• σϓϩΠ Slack ্͔Β։ൃऀ͕جຊతʹࣗ༝ʹߦ͑Δ• PR ຖͷεςʔδϯάɾ౷߹ίϯιʔϧɾτϨʔγϯάͳͲ͕͋Δ 44 h$p://techlife.cookpad.com/entry/2016/03/16/1000433 h$ps://github.com/eagletmt/hakohbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 16
ECS + hako ڥ ʹ͓͚Δ YAML ઃఆscheduler:<<: !include schedulers/default.ymlelb_v2:vpc_id: vpc-xxxxxxhealth_check_path: /healthchecklisteners:- port: 80protocol: HTTP# :autoscaling:min_capacity: 2max_capacity: 24policies:# : policies for autoscalinghbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 17
ECS + hako ڥ ʹ͓͚Δ YAML ઃఆapp:image: xxxxxxxxxx/service # ECR Λࢦఆ͢Δ͜ͱ͕ଟ͍cpu: 128memory: 512links:- fluentdenv:$providers:- <<: !include env_providers/vault.ymldirectory: hako/group/service # vault hako/group/service ͷ- !include env_providers/vault_shared.yml # ֘ͷάϧʔϓʹॴଐ͢Δਓ͕ࣗ༝ʹઃఆͰ͖ΔRAILS_ENV: productionWORKER_NUM: '2'additional_containers:front:!include containers/front.ymlfluentd:!include containers/fluentd.ymlvolumes:td-agent:source_path: /var/lib/hako/td-agent/servicehbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 18
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 19
ΫοΫύουʹ͓͚ΔϚΠΫϩαʔϏεͷ࣮ફ• ৫ਓɾνʔϜɾαʔϏεͷʹͭΕͯݖݶͷҕৡ͕ඞཁʹͳͬͨ• SRE ج൫νʔϜ͕શͯΛཧ͢ΔͷෆՄೳͩ͠ෆ݈શ• ΞϓϦέʔγϣϯࣗମ͕େ͖͘ͳΓ༷͗ͯ͢ʑͳ͕ग़ΔΑ͏ʹͳͬͨ• ίʔυͷݟ௨͕͠ѱ͘ͳΔɾςετʹֻ͕͔࣌ؒΓ͗͢Δ ...• લ͔ΒঃʑʹඋΛਐΊ͍ͯΔ 44 h$p://techlife.cookpad.com/entry/2016/03/16/100043hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 20
ΫοΫύουʹ͓͚ΔϚΠΫϩαʔϏεͷ࣮ફ• جຊతʹϝΠϯͷΞϓϦέʔγϣϯΛ hako ڥʹΓग़͢• ৽نʹ࡞͞ΕΔΞϓϦέʔγϣϯ hako ڥΛඪ४ʹ͢Δ• ಉ࣌ʹɺmicroservice ԽʹΑΔ՝ʹ͍͔ͭ͘औΓΜͰ͍Δ• αʔϏεؒͷ Tracing ʹ AWS X-Ray Λར༻͍ͯ͠Δ• αʔϏεϝογϡͷ࣮ݱʹ envoyproxy/envoy Λར༻͍ͯ͠Δ• ͍͔ͭ͘՝͕͋Γ࣮ͭͭࡍʹಈ͔ͯ͠ΛਐΊ͍ͯΔhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 21
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 22
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 23
ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 24
ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ• ΫοΫύουͷ SRE ͷ׆ಈΛ͍͔ͭ͘հ• αʔόߏͷཧɾมߋ• ৽نαʔϏεʹର͢Δ SRE ͷಈ͖• ϞχλϦϯάͷվળ• ϩʔυόϥϯγϯάͷվળ• ৽ଔݚम• ...hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 25
αʔόߏͷมߋɾཧ• αʔόߏجຊతʹશͯίʔυԽ͞Ε͍ͯΔ• Codenize.tools 5 Itamae 6, Terraform ͳͲ• IAM User Route53 GHE ্Ͱී௨ʹཧ͞Ε͍ͯΔ• ։ൃऀ͕ࣗ༝ʹ PR Λग़͢͜ͱ͕ग़དྷΔ• αʔόߏͷมߋίʔυͷ Pull request ͱͯ͠ग़͢• ϨϏϡʔΛड͚ɺϚʔδ͔ͯ͠Βө• Ұ෦Ϛʔδͯࣗ͠ಈͰө͞ΕΔ6 h$ps://github.com/itamae-kitchen/itamae5 h$ps://codenize.tools/hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 26
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 27
৽نαʔϏεʹର͢Δ SRE ͷಈ͖hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 28
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 29
৽نαʔϏεʹର͢Δ SRE ͷಈ͖• େ͖ΊͳػೳαʔϏε୯ҐͰׂͯ͠։ൃ͢Δ• ߹ʹΑΔ͕ SRE ͕ 1 ਓΞαΠϯ͞ΕΔ• ֘ͷαʔϏεͷ৴པੑʹؔ͢Δ͜ͱجຊతʹશ෦Δ• ઃܭͷ૬ஊɾϨϏϡʔ• ϦϦʔεલͷෛՙࢼݧɾΩϟύγςΟϓϥϯχϯά• ύϑΥʔϚϯεʹର͢ΔΞϓϩʔν• ...hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 30
ઃܭͷ૬ஊɾϨϏϡʔ• ։ൃͷલʹҎԼͷΑ͏ͳ͜ͱΛڞ༗ͯ͠Β͍ٞ͢Δ• ͲͷΑ͏ͳΞϓϦέʔγϣϯΛ࡞Δ͔• ͲͷΑ͏ͳίϯϙʔωϯτ͕ඞཁ͔• ͲͷΑ͏ͳߏɾεέʔϧઓུʹ͢Δ͔• Design Document Λॻ͍ͯΒ͍ͦΕΛϕʔεʹΛͨ͠Γ͢Δ• େ͖ΊͷػೳͳͲΛ։ൃ͢Δ࣌ Design Document Λॻ͘͜ͱ͕ଟ͍ (SRE )hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 31
ෛՙࢼݧ• ͋Δఔग़དྷ͖ͯͨลΓͰෛՙࢼݧͷΛ͢Δ• Ͳͷ͘Β͍ͷϦΫΤετ͕ఆ͞ΕΔ͔• ͲͷΑ͏ͳϢʔβϦΫΤετ͕ఆ͞ΕΔ͔• SRE ଆͰෛՙࢼݧͷγφϦΦڥΛ࡞Δ• ෛՙࢼݧΛߦ͍ͳ͕Βຊ൪ڥͰͷΩϟύγςΟΛݟੵΔ• ΞϓϦέʔγϣϯ্ͷϘτϧωοΫ͕Ͳ͜ʹ͋Δ͔ΛݟΔhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 32
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 33
ύϑΥʔϚϯεʹର͢ΔΞϓϩʔν• ৴པੑʹؔ͢Δ͕͋Δ߹ࣗͰίʔυΛॻ͘• ྫ͑ύϑΥʔϚϯε্ͷͳͲ• ΞϓϦέʔγϣϯͷ༷Λཧղͯ͠దͳΞϓϩʔνΛऔΔ• ྫ͑Ͳ͏͍͏ը໘ͷͲ͏͍͏ΫΤϦͳͷ͔• Ͳ͏͍͏ใ͕ཉͯ͘͠Ͳ͏͍͏੍͕͋Δͷ͔• ͍ͬͯΔϛυϧΣΞͷ༷ཧղͯ͠దͳΞϓϩʔνΛऔΔ• ྫ͑ SQL ͷվળͳΒ MySQL ͷΠϯσοΫεͷߏ෦ॲཧͷ͕ࣝ͋Δͱྑ͍hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 34
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 35
৽نαʔϏεʹର͢Δ SRE ͷಈ͖• ͜ͷลΓͨΓલͷࣄΛී௨ʹ͍ͬͯΔ͚ͩ• ͬͱվળग़དྷΔՕॴ͕ͨ͘͞Μ͋Δͱߟ͍͑ͯΔ• ྫ͑ෛՙࢼݧΛࣗಈͰߦ͏Α͏ʹ͢ΔɺͳͲ• ΞϓϦέʔγϣϯΛཧղ͠ɺඞཁͳΒίʔυΛॻ͘• SRE αΠτͷ৴པੑʹؔΘΔશͯʹΛෛ͏• ΞϓϦέʔγϣϯͷಛੑઃܭΛཧղ͠ϛυϧΣΞΛΈ߹ΘͤΔ Systems Engineering• ༷ϛυϧΣΞΛཧղ࣮͠ࡍʹΞϓϦέʔγϣϯͷϩδοΫΛॻ͘ So.ware Engineeringhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 36
ϞχλϦϯάͷվળhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 37
ϞχλϦϯάͷվળ• ݱࡏϝτϦΫεऔಘʹ Zabbix Λ༻͍ͯ͠Δ• ΑΓεέʔϧ͠ɺߴڃݴޠͰϝτϦΫεΞϥʔτΛѻ͑ΔΑ͏ʹ͍ͨ͠• SRE ຊ ͷ 10 ষʹॻ͔Ε͍ͯΔΑ͏ͳঢ়ଶʹͳΓ͍ͨ• Prometheus ͷಋೖΛঃʑʹ࣮ࢪ͍ͯ͠Δ• ϝτϦΫεͷղ૾ 15 ඵ୯ҐͳͲͰऔಘ͢ΔΑ͏ʹ͍ͯ͠Δ• ୯७ͳ HTTP ϨεϙϯεͰϝτϦΫεΛऩूग़དྷΔ• ΞϓϦέʔγϣϯࣗମͷϝτϦΫεͳͲ͕ൺֱత؆୯ʹऩूग़དྷΔhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 38
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 39
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 40
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 41
ϞχλϦϯάͷվળ• Ͳ͏͍ͬͨϛυϧΣΞΛબఆ͢Δ͔• ϞχλϦϯάͰ͋Ε࣌ܥྻσʔλϕʔεͳͲʹؔ͢Δਂ͍ཧղΛ͔ͯ͠ΒબͿ• Ͳ͏͍ͬͨಛੑ͕͋Δ͔ʁࣗͨͪͷنʹ߹͍ͬͯΔ͔ʁεέʔϧ͢Δ͔ʁ• Γͳ͍ίϯϙʔωϯτͳ͍͔ʁແ͍ͳΒࣗ࡞Ͱ͖ͦ͏ͳن͔ʁ• Prometheus ͳΒྫ͑ Long-Term Storage पΓͷίϯϙʔωϯτͳͲ• ͔ͬ͠Γͱͨ͠ Systems Engineering ͷݟΛར༻ͯ͠બఆ͢Δhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 42
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 43
ϩʔυόϥϯγϯάͷվળhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 44
ϩʔυόϥϯγϯάͷվળ• ैདྷΞϓϦέʔγϣϯαʔόͷϩʔυόϥϯγϯάʹ haproxy Λར༻͍ͯͨ͠• Weighted Round-robin (wrr) Λར༻ͯ͠όϥϯγϯάΛߦ͍ͬͯͨ• ෛՙͷ͍ΤϯυϙΠϯτͱߴ͍ΤϯυϙΠϯτ͕͋Δ߹ෛՙ͕Βͭ͘• wrr ͩͱಛఆͷαʔόʹෛՙͷߴ͍ΤϯυϙΠϯτ͕ूத͢ΔՄೳੑ͕͋Δ• leastconn Λ͑ྑͦ͞͏͕ͩ HTTP keepalive ͍ͯ͠Δ߹ʹ͕͋Δ• ίωΫγϣϯ͕ΔͷͰ࣮ࡍʹॲཧ͍ͯ͠ΔϦΫΤετͱണ͢Δɹhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 45
ryotarai/simproxy• h#ps://github.com/ryotarai/simproxy• Go Ͱॻ͔ΕͨγϯϓϧͳϦόʔεϓϩΩγ• ϔϧενΣοΫͳͲͷجຊతͳػೳ͋Δ• Balancing Method ʹ leastreq Λ࣮͍ͯ͠Δ• ࢦඪͱͯ͠୯७ͳίωΫγϣϯͰͳ࣍͘ͷΑ͏ͳΛར༻͢Δ• όοΫΤϯυ͕ड͚औ͕ͬͨԠ͕ฦ͖͍ͬͯͯͳ͍ϦΫΤετͷ• ͜ΕʹΑΓ HTTP keepalive ͕͋ͬͯ leastconn ͷΑ͏ͳόϥϯγϯά͕Մೳ• ϩʔυόϥϯγϯάͷΞϧΰϦζϜʹؔ͢Δ Systems Engineering ͱ࣮͢Δ So.ware Engineeringhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 46
before simproxyhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 47
a"er simproxyhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 48
৽ଔݚम• ৽ଔʹରͯ͠ΠϯϑϥͷݚमΛ 3 ؒߦ͍ͬͯΔ• MySQL ͷΠϯσοΫεͷߏΛཧղ͠ΫΤϦνϡʔχϯάΛߦ͏ ...• AWS ͷ ALB RDS ͳͲΛ͍ࣗͷΞϓϦέʔγϣϯΛεέʔϧͤ͞Δ ...• ࣮ࡍͷΞϓϦέʔγϣϯΛॻ͘ϝϯόʔ͕Πϯϑϥͷཧղ͕͋Δํ͕ྑ͍• શͯͷΞϓϦέʔγϣϯίʔυΛ SRE ͕νΣοΫ͍ͯ͠Δͱεέʔϧ͠ͳ͍• ྨ্ Overhead ͔͠Εͳ͍͕αʔϏεΛεέʔϧͤ͞ΔͨΊʹඞཁ• ࣮ࡍʹ Systems Engineering ͷਂ͍ݟ͕ແ͍ͱڭ͑Δ͜ͱ͕ग़དྷͳ͍hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 49
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 50
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 51
PRhbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 52
We are hiring!• ΫοΫύουͰຊհͨ͠Α͏ͳऔΓΈΛߦ͍ͬͯ·͢• ·ͩ·ͩղܾ͍ͨ͠ɾऔΓΈ͍ͨ՝͕ͨ͘͞Μ͋Δ• ָ͠Έͳ͕ΒҰॹʹΕΔϝϯόʔΛืू͍ͯ͠·͢→ h$ps://cookpad.jobshbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 53
rrreeeyyy
yumayabe
shimap_sampo
oracle4engineer
miyayou
vividtone
shohata
optim
okazuki
mthomps
danielanewman
holman
chriscoyier
cromwellryan
brettharned
iamctodd
lauravandoore
sferik
jlugia
eileencodes