第40回インターネットと運用技術研究発表会の招待講演で『Web サービスの信頼性と運用の自動化について』というタイトルで発表しました。
Web αʔϏεͷ৴པੑͱӡ༻ͷࣗಈԽʹ͍ͭͯΫοΫύουגࣜձࣾ ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ٢ ཽଠ ( @rrreeeyyy )ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 1
View Slide
Agenda• Site Reliability Engineering (SRE) ʹ͍ͭͯ• SRE ͱ• ӡ༻ͷݱͱӡ༻ٕज़ݚڀ• ӡ༻ͷࣗಈԽʹ͍ͭͯ• ࣌ܥྻσʔλͷॲཧͱҟৗݕʹ͍ͭͯୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 2
Me!SSSFFFZZZ!SSSFFFZZZIUUQTSSSFFFZZZDPN:PTIJLBXB3ZPUBୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 3
Me• Yoshikawa Ryota ( @rrreeeyyy [reɪ] )• ిؾ௨৴େֶ (2010/04 ʙ 2014/03)• ใཧֶ෦ ใɾ௨৴ֶՊ• גࣜձࣾϋʔτϏʔπ (2010/11ʙ2016/12)• MSP ࣄۀ෦ ӡ༻ελοϑɾӡ༻ΤϯδχΞ• ΫοΫύουגࣜձࣾ (2017/01 ʙ)• ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 4
Me• ిؾ௨৴େֶ (2010/04 ʙ 2014/03)• mruby Λ༻͍ͨLinux ϩʔυόϥϯαΠϯλϑΣʔεͷ࣮Ͱଔۀ• גࣜձࣾϋʔτϏʔπ (2010/11ʙ2016/12)• ʙઍنͷӡ༻ɾߏஙɾཧΛܦݧ• ϝσΟΞɾήʔϜɾEC ͷ༷ʑͳͷӡ༻Λܦݧ• ΫοΫύουגࣜձࣾ (2017/01 ʙ)• ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ• ઍنͷγεςϜͷ৴པੑʹؔΘΔ෦શͯΛ୲ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 5
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 6
Site Reliability Engineering ͱ• αΠτͷ৴པੑͷશͯʹൣғΛ࣋ͭɺͱ͢ΔΤϯδχΞϦϯάख๏• ಛʹɺιϑτΣΞɾγεςϜΤϯδχΞϦϯάΛ༻͍ͯ՝Λղܾ͢Δ• ྫ͑ɺύϑΥʔϚϯε• ύϑΥʔϚϯεʹ͕͋Δίʔυ SRE ͕ࣗίʔυΛॻ͍ͯमਖ਼͢Δ• ྫ͑ɺεέʔϥϏϦςΟɾΞϕΠϥϏϦςΟ• ਓؒʹΑΔखಈͷ࡞ۀΛίʔυʹΑΓࣗಈԽ͢Δ• نΞΫηε͕૿େͯ҆͠ఆͯ͠αʔϏεΛఏڙͰ͖Δঢ়ଶʹ͢Δ• ΠϯϑϥΤϯδχΞͷൣғ͕૿େͨ͠தͰ͋Δछɺ৬Λ࠶ఆٛͨ͠ͱଊ͍͑ͯΔୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 7
SRE ͷ׆ಈSRE ͷ׆ಈେ͖͚ͯ࣍͘ͷ 4 ͭʹྨͰ͖Δͱ͞Ε͍ͯΔ• So#ware Engineering• Systems Engineering• Toil• Overheadୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 8
SRE ͷ׆ಈ• So#ware Engineering• ίʔυͷ࡞मਖ਼, ͦΕʹؔ࿈͢ΔઃܭυΩϡϝϯςʔγϣϯͷ࡞ۀͳͲ• ࣗಈԽεΫϦϓτͷ࡞, πʔϧϑϨʔϜϫʔΫͷ࡞ ...• αʔϏεͷ৴པੑΛߴΊΔػೳͷՃ ...• Systems Engineering• 1 ճͷ࡞ۀͰӬଓతͳվળΛੜΉ࣍ͷΑ͏ͳ࡞ۀ• ຊ൪γεςϜͷઃఆมߋ ...• ։ൃνʔϜͷαʔϏεΞʔΩςΫνϟઃܭ ...ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 9
SRE ͷ׆ಈ• Toil• αʔϏεΛՔಇͤ͞ΔͨΊ͚ͩʹ܁Γฦ͠ߦΘΕΔखಈͰͷ࡞ۀ• ظతͳՁΛ࣋ͨͳ͍, αʔϏεͷʹରͯ͠ O(n) Ͱ͋Δ ...• ͭ·Βͳ͍ࣄͰظతͳՁ͕͋Ε Toil Ͱͳ͍• Overhead• αʔϏεΛՔಇͤ͞Δ͜ͱʹ݁͠ͳ͍ཧతͳ࡞ۀ• ࠾༻, ϛʔςΟϯά, ධՁ, τϨʔχϯά ...ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 10
ӡ༻ͷݱͱӡ༻ٕज़ݚڀ• ΤϯδχΞϦϯάΛͪΌΜͱΔࣄ͕ݱϨϕϧͰධՁ͞ΕΔΑ͏ʹ• ͜ͷจ຺ͷҰͭͱͯ͠ SRE ͷΑ͏ͳख๏ɾ৬छ͕ग़͖ͯͨͱ͍͏ཧղ• จΛॻ͍ͨΓಡΜͰཧղɾ࣮͢Δࣄ͕ධՁ͞ΕΔΑ͏ʹͳ͖ͬͯͨ• ࣮ࡍͷӡ༻ͷݱͰಡΜͰཧղ͢Δࣄඞཁʹͳ͖ͬͯͨ• ٕज़બఆͷࢦඪͷҰͭͱͯ͠ϛυϧΣΞͷڍಈͷཧղͳͲ• ͍ΘΏΔݚڀ։ൃͷΑ͏ͳ෦Ͱͳͯ͘ৗతʹߦ͏Α͏ʹ• ଟ͘ͷٕज़اۀͰӡ༻ɾϛυϧΣΞͷจ͕ଟ͘ग़ͯ͘ΔΑ͏ʹୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 11
େखΫϥυϕϯμͷจ• Google, Amazon, Facebook, Ne3lix...• ଞʹେখؔΘΒ༷ͣʑͳاۀ͔Βจ͕ग़͍ͯΔ• USENIX ͷΧϯϑΝϨϯεͰٕज़اۀ͕ੵۃతʹൃද• FAST, OSDI, Security ...• ΫϥυαʔϏεͷ෦࣮ϛυϧΣΞؔΘΒͣ• ΫϥυαʔϏεͷڍಈΛਖ਼͘͠ཧղ͢ΔͨΊʹจΛಡΉ• ৽͍͠ΞʔΩςΫνϟɾϛυϧΣΞͷબఆͷͨΊʹจΛಡΉୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 12
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 13
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 14
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 15
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 16
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 17
จͷ༰Λ࣮ӡ༻Ͱ͑ΔϛυϧΣΞ• HashiCorp• Vagrant, Terraform Λ࡞͍ͬͯΔϕϯνϟʔ• จͷ༰Λ࣮ӡ༻Ͱ͍͍͢ܗʹམͱ͠ࠐΜͰ͍Δ• Serf, Consul, Vault • ͜ΕΒͷϛυϧΣΞΛਖ਼͘͠ཧղͯ͠ӡ༻͢ΔͨΊʹจΛಡΉ• ͳͥ Consul ͷϊʔυ࠷ 3 ඞཁͳͷ͔ʁୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 18
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 19
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 20
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 21
ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 22
ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ• SRE ͱͯ͠ͷ׆ಈΛΫοΫύουͰ༷ʑߦ͍ͬͯΔ• ΫοΫύουͷ SRE ͷ׆ಈΛ͍͔ͭ͘հ• ৽نαʔϏεʹର͢Δ SRE ͷಈ͖• ϞχλϦϯάͷվળ• ϩʔυόϥϯγϯάͷվળ• ৽ଔݚम• ...ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 23
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 24
ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ• SRE ͱͯ͠ͷ׆ಈΛΫοΫύουͰ༷ʑߦ͍ͬͯΔ• ΫοΫύουͷ SRE ͷ׆ಈΛ͍͔ͭ͘հ• ৽نαʔϏεʹର͢Δ SRE ͷಈ͖• ϞχλϦϯάͷվળ• ϩʔυόϥϯγϯάͷվળ• ৽ଔݚम• ...ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 25
ϞχλϦϯάͷվળ• େྔͷσʔλΛೖɾॲཧ͢ΔΑ͏ͳ࣌ܥྻσʔλϕʔεͷج൫͖ͬͯͨ• ΫοΫύουͰ Prometheus Λ TSDB ͱͯ͠࠾༻• Long-term storage ͱͯ͠ S3 ΛόοΫΤϯυͱͯ͠ݕ౼த• औಘͰ͖ΔϝτϦΫεͷղ૾ͷ্ɾछྨͷ૿ڧ• 15 ඵ୯ҐͷϝτϦΫεऔಘ͕ग़དྷΔΑ͏ʹͳ͍ͬͯΔ• ϚΠΫϩαʔϏεͷαʔϏεؒͷ࿈ܞͷ༷ࢠՄࢹԽΛݕ౼தୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 26
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 27
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 28
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 29
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 30
࣌ܥྻσʔλͷॲཧʹ͍ͭͯ• ͜Ε·Ͱ΄ͱΜͲ੩తͳᮢͰԿͱ͔͍ͯͨ͠ঢ়ଶ• ͜Ε͔Β࣌ܥྻσʔλʹ؆୯ͳ౷ܭతॲཧ͕ࢪͤΔΑ͏ʹ• ઢܗճؼ, ࢦฏԽ, ҠಈฏۉͳͲ...• ࣄલʹى͖͏Δҟৗ͕͔͍ͬͯΔ࣌ʹΑ͘ಈ͘• Πϯγσϯτোͷ "ੵΈॏͶ" ͰݕͰ͖ΔΑ͏ʹ• ࣮ࡍʹऔΕΔϝτϦΫε͕૿͑ҟৗ࣌ʹͲ͏ͳΔ͔༧ଌࠔͳࣄୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 31
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 32
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 33
࣌ܥྻσʔλͱҟৗݕ• ࣌ܥྻσʔλͷҟৗ(֎Ε)ݕ༷ʑͳख๏͕طʹݚڀ͞Ε͖͍ͯͯΔ• Gaussian Mixture Model, Isola5on forest ͳͲͳͲ...• ࣮ࡍͷӡ༻Ͱग़ݱ͢ΔϝτϦΫεͰඇઢܗͳཁૉ͕ଟ͘ग़ݱ͢Δ• ैདྷͷख๏ෳࡶͳඇઢܗมͳͲͷදݱʹऑ͍͜ͱ͕͋ͬͨ• ࠷ۙͷख๏Ͱ CNN Λ༻͍ͯҟৗݕߦ͏Α͏ͳख๏͕ొͨ͠• ෳͷೖྗ͕͋ΔΑ͏ͳ૬ؔҟৗͰൺֱత؆୯ʹ༧ଌ͢Δࣄ͕Մೳୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 34
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 35
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 36
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 37
LSTM ʹΑΔ࣌ܥྻσʔλͷҟৗݕ• LSTM Λ༻͍ͯਖ਼ৗͳ࣌ܥྻσʔλͷֶशΛߦ͏• ߦֶͬͨशʹج͍ͯɺ࣍ͷσʔλͷ༧ଌΛߦ͏• ࣮ࡍͷσʔλͱ༧ଌΛൺֱ͢Δ͜ͱͰҟৗΛଌఆ͢Δ• ֶश͢Δσʔλͷೖྗͱͯ͠ϝτϦΫεҎ֎ʹαʔόͷίϯςΩετΛೖΕΔ• ྫ͑σϓϩΠதɾόον࣮ߦதɾΦʔτεέʔϧத...• ͦͷ࣌ͷαʔόͷίϯςΩετΛؚΊͯ༧ଌΛ࡞ΕΔ• ΑΓαʔόͷঢ়گΛՃຯͨ͠ҟৗΛଌΔ͜ͱ͕ग़དྷΔୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 38
༧ଌରʹͳΔ࣌ܥྻσʔλͷྫtimestamp,cpu,deploying,autoscaling,batching2018-03-05 12:00:00,80,1,0,02018-03-05 12:00:15,81,1,0,02018-03-05 12:00:30,82,1,0,02018-03-05 12:00:45,82,1,1,02018-03-05 12:01:00,75,1,1,02018-03-05 12:01:15,60,1,1,02018-03-05 12:01:30,55,1,1,02018-03-05 12:01:45,40,0,1,02018-03-05 12:02:00,35,0,0,0• ࣮ࡍʹ Prometheus ͷϥϕϧػೳΛ༻্͍ͯهͷίϯςΩετΛೖྗୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 39
Keras Λ༻͍࣮ͨɾ࣮ݧ• ϊΠζΛؚ·ͤͨਖ਼ݭΛ LSTM Ͱֶशͤ͞Δ• ݩͱͳͬͨਖ਼ݭʹҟৗΛؚ·ͤΔ• ҟৗͷ͋ΔܗͱֶशࡁΈͷϞσϧ͔Βੜ͞ΕͨܗͷೋޡࠩΛऔΔ• ߋʹ࣮ࡍʹલϖʔδͷ࣌ܥྻσʔλΛֶशͯ͠༧ଌΛग़ͯ͠ΈΔ• ࣮ࡍͷσʔλʹݶΓͳ͍ۙ͘ܗͷ toy-data ͱͯ͠ར༻• CPU ༻ʹؔͯ͠ 0-1 ͷൣғʹਖ਼نԽֶͯ͠शΛऩଋͤ͞Δ• ز͔ͭͷ CPU ༻ɾσϓϩΠपظͷϞσϧΛֶशͤͯ͞༧ଌͤͯ͞ΈΔୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 40
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 41
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 42
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 43
ୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 44
·ͱΊ• SRE ͱ͍͏ΤϯδχΞϦϯάख๏ʹ͍ͭͯհ• ΠϯϑϥΤϯδχΞͷ৬͕͕͖ͬͯͨ• αΠτͷ৴པੑࣗମʹ৬Λ࣋ͭ͜ͱΛ໌֬ʹ͢Δ SRE ͕ొ• αΠτͷ৴པੑΛकΔͨΊʹ "ਖ਼͘͠ΤϯδχΞϦϯάΛ͢Δ" ࣄ͕ॏཁ• ΫοΫύουͰͷ SRE ͷऔΓΈͱͯ͠ϞχλϦϯάͷվળΛհ• ࣌ܥྻσʔλΛରʹอଘɾղੳ͕ߦ͑Δج൫͕͖ͬͯͨ• ࣍ͳΔεςʔδͱͯ࣌͠ܥྻσʔλͷҟৗݕʹ͍༷ͭͯʑͳख๏Λࡧதୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 45
ࢀߟࢿྉ• h#ps://landing.google.com/sre/book.html• h#ps://www.allthingsdistributed.com/files/p1041-verbitski.pdf• h#p://lamport.azurewebsites.net/tla/formal-methods-amazon.pdf• h#ps://staAc.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf• h#p://techlife.cookpad.com/entry/Ameseries-database-001• h#p://www.vldb.org/pvldb/vol8/p1816-teller.pdf• h#ps://qiita.com/rrreeeyyy/items/6bbd1f09c96e2af4a8ce• h#p://ramcloud.stanford.edu/raH.pdf• h#p://www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf• h#ps://speakerdeck.com/rrreeeyyy/jK-2017-site-reliability-engineering• h#ps://speakerdeck.com/yuukit/the-study-of-Ame-series-database-for-server-monitoring• h#ps://speakerdeck.com/rrreeeyyy/introducAon-to-prometheus-pracAce• h#ps://github.com/NeKlix/vizceral• h#ps://github.com/nghialv/promvizୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 46
PRୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 47
We are hiring!• ΫοΫύουͰຊհͨ͠Α͏ͳऔΓΈΛߦ͍ͬͯ·͢• ·ͩ·ͩղܾ͍ͨ͠ɾऔΓΈ͍ͨ՝͕ͨ͘͞Μ͋Δ• ָ͠Έͳ͕ΒҰॹʹΕΔϝϯόʔΛืू͍ͯ͠·͢→ h$ps://cookpad.jobsୈ 40 ճΠϯλʔωοτͱӡ༻ٕज़ݚڀൃදձ (2018/03/05) | Yoshikawa Ryota ( @rrreeeyyy ) 48
rrreeeyyy
knsg16
smt7174
h1kita
oracle4engineer
toshi0383
tomoaki25
colopl
ufried
etaroid
takesection
naohachi89
paigeccino
holman
destraynor
trishagee
erikaheidi
brettharned
chriscoyier
brad_frost
keavy
jeffersonlam
geeforr
bkeepers