Web サービスの信頼性を守るための取り組み / jtf-2017-site-reliability-engineering

#jtf2017 ( http://2017.techfesta.jp/ ) にて『Web サービスの信頼性を守るための取り組み』というタイトルで発表しました。

1. Web αʔϏεͷ৴པੑΛ कΔͨΊͷऔΓ૊Έ ΫοΫύουגࣜձࣾ ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ ٢઒ ཽଠ ( @rrreeeyyy ) July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 1
2. Agenda • Site Reliability Engineering (SRE) ͱ͸ • Ұൠతͳ SRE ͷ׆ಈʹ͍ͭͯ • ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈʹ͍ͭͯ • JTF2017 ͷςʔϚʮIT ΤϯδχΞϦϯάͷຊ࣭ʯ • SRE ʹ͓͚ΔʮΤϯδχΞϦϯάʯͱ͸Կ͔ʁ • SRE ͷจ຺͔Β IT ΤϯδχΞϦϯάͷຊ࣭Λߟ͑Δ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 2
3. !SSSFFFZZZ !SSSFFFZZZ IUUQTSSSFFFZZZDPN :PTIJLBXB
   3ZPUB Me • Yoshikawa Ryota ( @rrreeeyyy [reɪ] ) • ΫοΫύουגࣜձࣾ (2017/01 ʙ) • ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ • ڵຯྖҬ • ϞχλϦϯάɾ࣌ܥྻσʔλϕʔε • ෼ࢄγεςϜɾϩʔυόϥϯα • झຯ • League of Legends, ΀Α΀ΑΫϩχΫϧ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 3
4. !SSSFFFZZZ !SSSFFFZZZ IUUQTSSSFFFZZZDPN :PTIJLBXB
   3ZPUB Me • Yoshikawa Ryota ( @rrreeeyyy [reɪ] ) • ΫοΫύουגࣜձࣾ (2017/01 ʙ) • ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ • ڵຯྖҬ • ϞχλϦϯάɾ࣌ܥྻσʔλϕʔε • ෼ࢄγεςϜɾϩʔυόϥϯα • झຯ • League of Legends, ΀Α΀ΑΫϩχΫϧ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 4
5. July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 5
6. Site Reliability Engineering ͱ͸ • αΠτͷ৴པੑͷશͯʹ੹೚ൣғΛ࣋ͭ • ಛʹɺιϑτ΢ΣΞɾγεςϜΤϯδχΞϦϯάΛ༻͍ͯ՝୊Λղܾ͢Δ • ྫ͑͹ɺύϑΥʔϚϯε • ύϑΥʔϚϯεʹ໰୊͕͋Δίʔυ͸ SRE ࣗ਎͕ίʔυΛॻ͍ͯमਖ਼͢Δ • ྫ͑͹ɺεέʔϥϏϦςΟɾΞϕΠϥϏϦςΟ • ਓؒʹΑΔखಈͷ࡞ۀΛίʔυʹΑΓࣗಈԽ͢Δ • ଐਓੑΛഉআͯ҆͠ఆͤ͞ΔɾਓؒΛ૿΍͞ͳͯ͘΋αʔϏεΛεέʔϧͤ͞Δ • ن໛͕૿େͯ͠΋҆ఆͯ͠αʔϏεΛఏڙͰ͖Δঢ়ଶʹ͢Δ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 6
7. SRE ͷ׆ಈ SRE ͷ׆ಈ͸େ͖͘෼͚ͯ࣍ͷ 4 ͭʹ෼ྨͰ͖Δͱ͞Ε͍ͯΔ • So#ware Engineering • Systems Engineering • Toil • Overhead July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 7
8. SRE ͷ׆ಈ • So#ware Engineering • ίʔυͷ࡞੒΍मਖ਼, ͦΕʹؔ࿈͢Δઃܭ΍υΩϡϝϯςʔγϣϯͷ࡞ۀͳͲ • ࣗಈԽεΫϦϓτͷ࡞੒, πʔϧ΍ϑϨʔϜϫʔΫͷ࡞੒ ... • αʔϏε΁ͷ৴པੑΛߴΊΔػೳͷ௥Ճ ... • Systems Engineering • 1 ճͷ࡞ۀͰӬଓతͳվળΛੜΉ࣍ͷΑ͏ͳ࡞ۀ • ຊ൪γεςϜͷઃఆมߋ ... • ։ൃνʔϜͷαʔϏεΞʔΩςΫνϟઃܭ ... July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 8
9. SRE ͷ׆ಈ • Toil • αʔϏεΛՔಇͤ͞ΔͨΊ͚ͩʹ܁Γฦ͠ߦΘΕΔखಈͰͷ࡞ۀ • ௕ظతͳՁ஋Λ࣋ͨͳ͍, αʔϏεͷ੒௕ʹରͯ͠ O(n) Ͱ͋Δ ... • ͭ·Βͳ͍࢓ࣄͰ΋௕ظతͳՁ஋͕͋Ε͹ Toil Ͱ͸ͳ͍ • Overhead • αʔϏεΛՔಇͤ͞Δ͜ͱʹ௚݁͠ͳ͍؅ཧతͳ࡞ۀ • ࠾༻, ϛʔςΟϯά, ධՁ, τϨʔχϯά ... July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 9
10. ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 10
11. ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ • SRE ຊʹ͋ΔΑ͏ͳ׆ಈΛΫοΫύου಺Ͱ΋༷ʑߦ͍ͬͯΔ • ΫοΫύουͷ SRE ͷ׆ಈΛ͍͔ͭ͘঺հ • ৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • ϞχλϦϯάͷվળ • ϩʔυόϥϯγϯάͷվળ • ৽ଔݚम • ... July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 11
12. ৽نαʔϏεʹର͢Δ SRE ͷಈ͖ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 12
13. ৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • େ͖Ίͳػೳ͸αʔϏε୯ҐͰ෼ׂͯ͠։ൃ͢Δ • ৔߹ʹ΋ΑΔ͕ SRE ͕ 1 ਓΞαΠϯ͞ΕΔ • ౰֘ͷαʔϏεͷ৴པੑʹؔ͢Δ͜ͱ͸جຊతʹશ෦΍Δ • ઃܭͷ૬ஊɾϨϏϡʔ • ϦϦʔεલͷෛՙࢼݧɾΩϟύγςΟϓϥϯχϯά • ύϑΥʔϚϯεʹର͢ΔΞϓϩʔν • ... July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 13
14. ઃܭͷ૬ஊɾϨϏϡʔ • ։ൃͷલʹҎԼͷΑ͏ͳ͜ͱΛڞ༗ͯ͠΋Β͍ٞ࿦͢Δ • ͲͷΑ͏ͳΞϓϦέʔγϣϯΛ࡞Δ͔ • ͲͷΑ͏ͳίϯϙʔωϯτ͕ඞཁ͔ • ͲͷΑ͏ͳߏ੒ɾεέʔϧઓུʹ͢Δ͔ • Design Document Λॻ͍ͯ΋Β͍ͦΕΛϕʔεʹ࿩Λͨ͠Γ͢Δ • େ͖ΊͷػೳͳͲΛ։ൃ͢Δ࣌͸ Design Document Λॻ͘͜ͱ͕ଟ͍ (SRE ΋) July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 14
15. ෛՙࢼݧ • ͋Δఔ౓ग़དྷ͖ͯͨลΓͰෛՙࢼݧͷ࿩Λ͢Δ • Ͳͷ͘Β͍ͷϦΫΤετ͕૝ఆ͞ΕΔ͔ • ͲͷΑ͏ͳϢʔβϦΫΤετ͕૝ఆ͞ΕΔ͔ • SRE ଆͰෛՙࢼݧͷγφϦΦ΍؀ڥΛ࡞Δ • ෛՙࢼݧΛߦ͍ͳ͕Βຊ൪؀ڥͰͷΩϟύγςΟΛݟੵ΋Δ • ΞϓϦέʔγϣϯ্ͷϘτϧωοΫ͕Ͳ͜ʹ͋Δ͔ΛݟΔ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 15
16. July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 16
17. ύϑΥʔϚϯεʹର͢ΔΞϓϩʔν • ৴པੑʹؔ͢Δ໰୊͕͋Δ৔߹͸ࣗ෼ͰίʔυΛॻ͘ • ྫ͑͹ύϑΥʔϚϯε্ͷ໰୊ͳͲ • ΞϓϦέʔγϣϯͷ࢓༷Λཧղͯ͠ద੾ͳΞϓϩʔνΛऔΔ • ྫ͑͹Ͳ͏͍͏ը໘ͷͲ͏͍͏ΫΤϦͳͷ͔ • Ͳ͏͍͏৘ใ͕ཉͯ͘͠Ͳ͏͍͏੍໿͕͋Δͷ͔ • ࢖͍ͬͯΔϛυϧ΢ΣΞͷ࢓༷΋ཧղͯ͠ద੾ͳΞϓϩʔνΛऔΔ • ྫ͑͹ SQL ͷվળͳΒ MySQL ͷΠϯσοΫεͷߏ଄΍಺෦ॲཧͷ஌͕ࣝ͋Δͱྑ͍ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 17
18. July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 18
19. ৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • ͜ͷลΓ͸౰ͨΓલͷࣄΛී௨ʹ΍͍ͬͯΔ͚ͩ • ΋ͬͱվળग़དྷΔՕॴ͕ͨ͘͞Μ͋Δͱߟ͍͑ͯΔ • ྫ͑͹ෛՙࢼݧΛࣗಈͰߦ͏Α͏ʹ͢ΔɺͳͲ • ΞϓϦέʔγϣϯΛཧղ͠ɺඞཁͳΒ͹ίʔυΛॻ͘ • SRE ͸αΠτͷ৴པੑʹؔΘΔશͯʹ੹຿Λෛ͏ • ΞϓϦέʔγϣϯͷಛੑ΍ઃܭΛཧղ͠ϛυϧ΢ΣΞΛ૊Έ߹ΘͤΔ Systems Engineering • ࢓༷΍ϛυϧ΢ΣΞΛཧղ࣮͠ࡍʹΞϓϦέʔγϣϯͷϩδοΫΛॻ͘ So.ware Engineering July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 19
20. ϞχλϦϯάͷվળ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 20
21. ϞχλϦϯάͷվળ • ݱࡏ͸ϝτϦΫεऔಘʹ Zabbix Λ࢖༻͍ͯ͠Δ • ΑΓεέʔϧ͠ɺߴڃݴޠͰϝτϦΫε΍ΞϥʔτΛѻ͑ΔΑ͏ʹ͍ͨ͠ • SRE ຊ ͷ 10 ষʹॻ͔Ε͍ͯΔΑ͏ͳঢ়ଶʹͳΓ͍ͨ • Prometheus ͷಋೖΛঃʑʹ࣮ࢪ͍ͯ͠Δ • ϝτϦΫεͷղ૾౓΋ 15 ඵ୯ҐͳͲͰऔಘ͢ΔΑ͏ʹ͍ͯ͠Δ • ୯७ͳ HTTP ϨεϙϯεͰϝτϦΫεΛऩूग़དྷΔ • ΞϓϦέʔγϣϯࣗମͷϝτϦΫεͳͲ͕ൺֱత؆୯ʹऩूग़དྷΔ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 21
22. July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 22
23. July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 23
24. July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 24
25. ϞχλϦϯάͷվળ • Ͳ͏͍ͬͨϛυϧ΢ΣΞΛબఆ͢Δ͔ • ϞχλϦϯάͰ͋Ε͹࣌ܥྻσʔλϕʔεͳͲʹؔ͢Δਂ͍ཧղΛ͔ͯ͠ΒબͿ • Ͳ͏͍ͬͨಛੑ͕͋Δ͔ʁࣗ෼ͨͪͷن໛ʹ߹͍ͬͯΔ͔ʁεέʔϧ͢Δ͔ʁ • ଍Γͳ͍ίϯϙʔωϯτ͸ͳ͍͔ʁແ͍ͳΒࣗ࡞Ͱ͖ͦ͏ͳن໛͔ʁ • Prometheus ͳΒྫ͑͹ Long-Term Storage पΓͷίϯϙʔωϯτͳͲ • ͔ͬ͠Γͱͨ͠ Systems Engineering ͷ஌ݟΛར༻ͯ͠બఆ͢Δ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 25
26. July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 26
27. ϩʔυόϥϯγϯάͷվળ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 27
28. ϩʔυόϥϯγϯάͷվળ • ैདྷ͸ΞϓϦέʔγϣϯαʔόͷϩʔυόϥϯγϯάʹ haproxy Λར༻͍ͯͨ͠ • Weighted Round-robin (wrr) Λར༻ͯ͠όϥϯγϯάΛߦ͍ͬͯͨ • ෛՙͷ௿͍ΤϯυϙΠϯτͱߴ͍ΤϯυϙΠϯτ͕͋Δ৔߹ෛՙ͕͹Βͭ͘ • wrr ͩͱಛఆͷαʔόʹෛՙͷߴ͍ΤϯυϙΠϯτ͕ूத͢ΔՄೳੑ͕͋Δ • leastconn Λ࢖͑͹ྑͦ͞͏͕ͩ HTTP keepalive ͍ͯ͠Δ৔߹ʹ໰୊͕͋Δ • ίωΫγϣϯ͕࢒ΔͷͰ࣮ࡍʹॲཧ͍ͯ͠ΔϦΫΤετ਺ͱണ཭͢Δɹ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 28
29. ryotarai/simproxy • h#ps:/ /github.com/ryotarai/simproxy • Go Ͱॻ͔ΕͨγϯϓϧͳϦόʔεϓϩΩγ • ϔϧενΣοΫͳͲͷجຊతͳػೳ͸͋Δ • Balancing Method ʹ leastreq Λ࣮૷͍ͯ͠Δ • ࢦඪͱͯ͠୯७ͳίωΫγϣϯ਺Ͱ͸ͳ࣍͘ͷΑ͏ͳ஋Λར༻͢Δ • όοΫΤϯυ͕ड͚औ͕ͬͨԠ౴͕ฦ͖͍ͬͯͯͳ͍ϦΫΤετͷ਺ • ͜ΕʹΑΓ HTTP keepalive ͕͋ͬͯ΋ leastconn ͷΑ͏ͳόϥϯγϯά͕Մೳ • ϩʔυόϥϯγϯάͷΞϧΰϦζϜ౳ʹؔ͢Δ Systems Engineering ͱ࣮૷͢Δ So.ware Engineering July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 29
30. before simproxy July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 30
31. a"er simproxy July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 31
32. ৽ଔݚम • ৽ଔʹରͯ͠ΠϯϑϥͷݚमΛ 3 ೔ؒߦ͍ͬͯΔ • MySQL ͷΠϯσοΫεͷߏ଄Λཧղ͠ΫΤϦνϡʔχϯάΛߦ͏ ... • AWS ͷ ALB ΍ RDS ͳͲΛ࢖͍ࣗ෼ͷΞϓϦέʔγϣϯΛεέʔϧͤ͞Δ ... • ࣮ࡍͷΞϓϦέʔγϣϯΛॻ͘ϝϯόʔ͕Πϯϑϥͷཧղ͕͋Δํ͕ྑ͍ • શͯͷΞϓϦέʔγϣϯίʔυΛ SRE ͕νΣοΫ͍ͯ͠Δͱεέʔϧ͠ͳ͍ • ෼ྨ্͸ Overhead ͔΋͠Εͳ͍͕αʔϏεΛεέʔϧͤ͞ΔͨΊʹ͸ඞཁ • ࣮ࡍʹ͸ Systems Engineering ͷਂ͍஌ݟ͕ແ͍ͱڭ͑Δ͜ͱ͕ग़དྷͳ͍ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 32
33. July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 33
34. July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 34
35. ࠓճͷ JTF ͷςʔϚ ʮITΤϯδχΞϦϯάͷຊ࣭ʯ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 35
36. ࠓճͷ JTF ͷςʔϚ ʮITΤϯδχΞϦϯάͷຊ࣭ʯ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 36
37. SRE ʹ͓͚Δ "ΤϯδχΞϦϯά" • The work of reducing toil and scaling up services is the "Engineering" in Site Reliability Engineering. • Engineering work is novel and intrinsically requires human judgment. It produces a permanent improvement in your service, and is guided by a strategy. It is frequently creaBve and innovaBve, taking a design-driven approach to solving a problem—the more generalized, the beFer. — Site Reliability Engineering, Chapter 5 - EliminaBng Toil July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 37
38. SRE ʹ͓͚Δ "ΤϯδχΞϦϯά" • Toil ΛݮΒ͠αʔϏεΛεέʔϧͤ͞Δ͜ͱ͕ͦ͜ SRE ͷ "ΤϯδχΞϦϯά" • ΤϯδχΞϦϯάͷ࡞ۀ͸... • ৽͍͜͠ͱΛ͢Δ΋ͷͰ͋Δ • ຊ࣭తʹਓؒͷ൑அΛඞཁͱ͢Δ • αʔϏεʹ߃ٱతͳվળΛ༩͑Δ • ઓུʹΑͬͯಋ͔ΕΔ • ΫϦΤΠςΟϒ͔ͭΠϊϕʔςΟϒͰ͋Δ • ໰୊Λղܾ͢ΔͨΊʹઃܭओಋͷΞϓϩʔνΛऔΔ • ൚༻ੑ͕͋Δ΄Ͳ༏Ε͍ͯΔ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 38
39. ࣗ෼͕ߟ͑Δ IT ΤϯδχΞϦϯάͷຊ࣭ • IT ΤϯδχΞϦϯά͸޻ֶͷதͰ΋ಛʹʮ৘ใʯΛऔΓѻ͏ • ৘ใͱ͸(weblioࣙॻΑΓ): • ࣄ෺ɾग़དྷࣄͳͲͷ಺༰ɾ༷ࢠɻ·ͨɼͦͷ஌Βͤɻ • ػցܥ΍ੜମܥʹ༩͑ΒΕΔࢦྩ΍৴߸ɻ • ͋Δಛఆͷ໨తʹ͍ͭͯɼద੾ͳ൑அΛԼͨ͠Γɼߦಈͷҙ ࢥܾఆΛ͢ΔͨΊʹ໾ཱͭࢿྉ΍஌ࣝɻ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 39
40. ࣗ෼͕ߟ͑Δ IT ΤϯδχΞϦϯάͷຊ࣭ • Web αʔϏεͳͲͷ৘ใ఻ୡͷखஈΛ࢖ͬͯϢʔβʹՁ஋Λఏڙ͢Δ • ྫ͑͹ɺશੈքʹ͓͚ΔϨγϐͷڞ༗ͳͲ • ػցʹ༷ʑͳࢦྩ΍৴߸Λ༩͑ɺਓؒͷ࡞ۀΛޮ཰Խͤ͞Δ • ྫ͑͹ɺϓϩάϥϛϯάʹΑΔࣗಈԽ΍༏ΕͨઃܭʹΑΔޮ཰Խ • → ৘ใ఻ୡ΍ਓؒͷࢥߟ΍൑அΛػցʹߦΘͤΔ͜ͱʹΑͬͯՁ஋Λग़͢ • SRE ͱ͍͏จԽ͸ैདྷΑΓࣗ෼ͷߟ͑Δ͜ͷຊ࣭ʹ͍ۙͱࢥ͍ͬͯΔ • ߴ౓ͳࣗಈԽɺੜମͷΑ͏ʹࣗಈͰಈ࡞͢ΔγεςϜ͸ࣗ෼ͷڵຯྖҬ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 40
41. ·ͱΊ • Ұൠతͳ SRE ͷ׆ಈͱΫοΫύουʹ͓͚Δ SRE ͷऔΓ૊ΈΛ঺հ • Ұൠతͳ SRE ͷ׆ಈ͸ So'ware, Systems Engineering Λओ࣠ʹ Toil, Overhead ͕͋Δ • SRE ຊʹ΋͋ΔΑ͏ͳΤοηϯεΛऔΓೖΕͭͭࣗ෼ͨͪͷ໨తɾن໛ʹ͋ͬͨΤϯ δχΞϦϯάΛΫοΫύουͰ͸΍͍ͬͯΔ • SRE ʹ͓͚ΔΤϯδχΞϦϯάͱɺࣗ෼͕ߟ͑Δ IT ΤϯδχΞϦϯάͷຊ࣭ʹ͍ͭͯ • Toil ΛݮΒ͠αʔϏεΛεέʔϧͤ͞Δ͜ͱ • ৘ใ఻ୡ΍ਓؒͷࢥߟ΍൑அΛػցʹޮ཰తʹߦΘͤΔ͜ͱͰՁ஋Λग़͢͜ͱ July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 41
42. PR July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 42
43. We are hiring! • ΫοΫύουͰ͸ຊ೔঺հͨ͠Α͏ͳऔΓ૊ΈΛߦ͍ͬͯ·͢ • ·ͩ·ͩղܾ͍ͨ͠໰୊ɾऔΓ૊Έ͍ͨ՝୊͕ͨ͘͞Μ͋Δ • ָ͠Έͳ͕ΒҰॹʹ΍ΕΔϝϯόʔΛืू͍ͯ͠·͢ → h$ps:/ /cookpad.jobs July Tech Festa 2017 (2017/08/27) | Yoshikawa Ryota ( @rrreeeyyy ) 43