ڧ͍Πϯϑϥ͕૊৫Λ૑Δ ŠSite ReliabilityɾDeveloper Productivityɾٕज़ઓུͷऔΓ૊ΈŠ July Tech Festa 2018 29 July 2018 Wantedly, Inc. | Kodai Sakabe @koudaiii 1

΋͠ɺࣗ෼͕௅ઓͰ͖ͳ͍ͱࢥ͏ͳΒͦΕ ͸ج൫ͱ૊৫͕ऑ͍ɻͦͯͦ͠Ε͸ɺΠϯ ϑϥνʔϜ $50΍Ϧʔμʔ ͷ੹೚ɻࣗ෼ ͕Ͳͷ͘Β͍௅ઓͰ͖Δج൫ͱ૊৫Ͱಇ͍ ͍ͯΔ͔طʹΤϯδχΞ͸ؾ͍͍ͮͯΔ @koudaiii 2

X X X 3 Τ ϯ δ χ Ξ ͔ Β ݟ ͨ ັ ྗ ௅ઓͰ͖Δ͜ͱ ࣾձతΠϯύΫτ͕͋Δ ͍͢͝ਓͨͪͱҰॹʹಇ͚Δ

CREATE A WORLD WHERE WORK DRIVES PASSION. Wantedly, Inc. 4

OUR GOALS Wantedly, Inc. D i s c o v e r C o n n e c t E n g a g e S N S C r e a t e a w o r l d w h e r e w o r k d r i v e s p a s s i o n 5

࿩͢͜ͱ Site ReliabilityɾDeveloper Productivityɾٕज़ઓུͷऔΓ૊Έ ೥͔ΒมԽʹڧ͍ΠϯϑϥΛ໨ࢦ͖ͯͨ͜͠ͱʹΑΔޮՌͱ՝୊ ม Խ ʹ ڧ ͍ Π ϯϑ ϥ ࡞ Γ ͷ ݁ Ռ ຊདྷΠϯϑϥνʔϜͷՌͨ͢΂͖՝୊ղܾ͸ͳʹ͔ʁ ໨ ࢦ ͢ ΰʔϧ ͷ ม Խ มԽʹڧ͍Πϯϑϥ͸ڧྗͳ෢ث͕ͩͬͨಉ࣌ʹ೉͍͠՝୊΋ੜΜͩ ϚΠΫ ϩ α ʔ Ϗε ͷ ՝ ୊ 6 Ͳ͏ ղ ܾ ͯ͠ ͍ ͘ ͔ ͍·ݱ࣌఺ͰऔΓ૊ΜͰ͍Δ΋ͷ΍ղܾ͍ͯ͠Δ΋ͷͳʹ͔

ΞδΣϯμ w มԽʹڧ͍ΠϯϑϥΛ໨ࢦͨ݁͠Ռ w Microservicesʹ͓͚Δ՝୊ w ΠϯϑϥνʔϜ͕໨ࢦ͢΋ͷͷมԽ w Ұਓ͋ͨΓͷΤϯδχΞͷ࢓ࣄྔΛ૿΍͢ = OKR ʹूதͰ͖ΔνʔϜ w Ͳ͏ղܾ͍͔ͯ͘͠ w Site ReliabilityʮαΠτͷ৴པੑʹͭͳ͕ΔऔΓ૊Έʯ w Developer ProductivityʮΤϯδχΞ͕ࠔ͍ͬͯΔ͜ͱͷղܾࡦʯ w ٕज़ઓུʮະདྷʹඞཁʹͳΔίΞٕज़ͱͳΔج൫ͮ͘Γʯ 7

Wantedly Vist ม Խ ʹ ڧ ͍ Π ϯ ϑ ϥ ࡞ Γ ͷ ݁ Ռ ೥ ʹ ɺ H e r o k u ͔ Β A W S ΁ ࡌ ͤ ସ ͑ ɺ Α Γ Ϟ ό Πϧ Λ ॏ ࢹ ͨ͠Ξʔ Ω ς Ϋ ν ϟ ʹ ม ߋ ӈ ͸ ౰ ࣌ ͷ ߏ ੒ ਤ ೥ ʹ U b u n t u ͔ Β C o r e O S ʹ ม ߋ ɻ ͜ ͷ ೥ ʹ C h e f Λ ΍ Ί ͯɺ D o c k e r f i l e Ұ ຊ ʹ ͨ͠ ɻ ͦ ͷ ޙ ɺ ೥ ͔ Β K u b e r n e t e s ΁ Ҡ ߦ Λ ͸ ͡ Ί ͯ ͍ ͘ ɻ 8

Wantedly People ม Խ ʹ ڧ ͍ Π ϯ ϑ ϥ ࡞ Γ ͷ ݁ Ռ 2 0 1 6 ೥ Wa n t e d l y P e o p l e M i c r o s e r v i c e s o n K u b e r n e t e s ͜ Ε · Ͱ ͷ R u b y o n R a i l s ( U n i c o r n ) ͷ Έ ͠ ͔ ࢖ ͑ ͳ ͍ σ ϓϩ Π ϑ ϩ ʔ Λ ΍ Ί ɺ ʮ ม Խ ʹ ڧ ͍ Π ϯϑ ϥʯ Λ ࠶ ݱ Ͱ ͖ Δ Α ͏ ͳ ج ൫ ʹ ͨ͠ ɻ ݱ ࡏ ͸ ɺ ౰ ࣌ ͷ α ʔ Ϗε ਺ ͷ ഒ Ҏ ্ ͷ α ʔ Ϗε ͕ Ք ಇ ͯ͠ ͍ Δ ʢ ӈ ͸ ౰ ࣌ ͷ ߏ ੒ ਤ ʣ 9

Why Microservices? Ϛ Π Ϋ ϩ α ʔ Ϗ ε Λ ਐ Ί Δ ཧ ༝ ίΞόϦϡʔΛ໌֬ʹ͢Δ ଞͷӨڹΛड͚ͳ͍Α͏ʹ͢Δ ί Ξό Ϧϡ ʔ Λ क Δ ʮ༑ୡͷ༑ୡͷ ID Λߴ଎ʹฦ͢ʯʮ໊ࢗը૾Λ៉ྷʹ͢Δʯ౳ʹಛԽ͢Δ ٕ ज़ ʹ ͜ ͩ Θ Δ νʔϜن໛֦େ͢Δ෼ɺҰਓ͋ͨΓͷҙࣝ͢Δྔͷ૿ՃΛ཈͑Δ ଞνʔϜͷґଘؔ܎͕ڧ͍΄Ͳ೉͘͠ɺௐ੔ͱશମͷ೺ѲͰେมʹͳΔ εέ ʔϧͰ ͖ Δ ૊ ৫ ʹ ͳΔ 10

Why Kubernetes? طଘʹ௥Ճͨ͠΄͏ָ͕ WT৽نʹ࡞Δ উखʹࢼͯ͠ɺউखʹऴΘΒ͍ͤͨ EC2, ELB, RDS, Cloudfront, S3, ElastiCache, Route53 Terraformʹͯ͠΋खؒʜ Π ϯ ϑ ϥ ࡞ ۀ ͷ ख ؒ ͕ গ ͳ ͍ ޷͖উखʹॻ͍ͨ΋ͷ͕ಈ͘ɻ ٯʹΊΜͲ͍͘͞෦෼΋͋Δ Rustͷ௅ઓ I/F͕໌֬ɻBϓϥϯ͕༗ΔɻMonitoringɻ࠷ѱॻ͖௚ͤΔɻσʔλҰ؏ੑ D o c k e r f i l e ʹ ޷ ͖ উ ख ʹ ॻ ͚Δ Ϋϥελʹ৐ͤΕ͹؂ࢹର৅ʹͳΔ ࠷ѱिؒͰॻ͖௚͢͜ͱ΋Ͱ͖ΔαʔϏε΋ଞͷαʔϏεͱಉ౳ͷ඼࣭ʹѻΘΕΔ ͲΜ ͳ α ʔ Ϗε΋ ಉ ౳ ͷ Π ϯ ϑ ϥ ඼ ࣭ 11

Why Kubernetes? Kubernetes ͷϨʔϧʹ৐Δ w ޮ ཰ త ʹ M a c h i n e Λ ಈ ͔ ͢ • H o r i z o n t a l P o d A u t o s c a l e r • C u s t o m m e t r i c s • C l u s t e r A u t o s c a l e r w ӡ ༻ ͷ ౷ Ұ • k u b e c t l • t h i r d p a r t y t o o l s ( i s t i o c t l … ) • x x x - o p e r a t o r ( v a u l t - o p e r a t o r, e t c d - o p e r a t o r … ) • m a n i f e s t f i l e Ͳͷ Ξ ϓ Ϧ έ ʔ γ ϣ ϯ ʹ Ҡ ͬͯ ΋ ɺ ಉ ͡ ί Ϛ ϯ υ Ͱ Ұ ௨ Γ ͷ ۀ ຿ ͕ Ͱ ͖ Δɻ ϛ υϧ ΢ΣΞ ΋ ͞ ͬ͘ ͱ ೖ Ε Δ ͷ Ͱ͋ Ε ͹ ଞ ͷ M a n i f e s t f i l e ͔ Β ഈ आ ͯ͠ ࢼ ͤΔ 12

Why Kubernetes? Kubernetes ͷϨʔϧʹ৐Δ w K u b e r n e t e s Λ औ Γ ר ͘ Τ ί γεςϜ ͷ Ը ܙ • S e r v i c e m e s h ( I s t i o , E n v o y … ) • N e t w o r k i n g ( A m b a s s a d o r, Te l e p l e s e n c e , Ko n g … ) • M o n i t o r i n g ( P r o m e t h e u s , D a t a d o g , N e w R e l i c … ) • m i n i k u b e • H e l m c h a r t • C l o u d S e r v i c e s • C o m m u n i t y Ϋϥελʹ͍Δ͚ͩͰɺΤίγεςϜͷԸܙ͕ಘΒΕΔϝϦοτ͸Ͱ͔͘ɺΫϥελʹ͍ͳ͍ ͜ͱͰى͜ΔίετͱϦεΫ͕ߴ͍ɻ·ͨڧྗͳ Community ͸ɺੈͷதͷΞʔΩςΫνϟ͕ Ͳ͏͍͏՝୊͕͋ΓɺͲ͜ʹ޲͔͍ͬͯΔͷ͔͕Θ͔Δɻࣗ෼ͨͪͷٕज़ͷίϯύεʹͳΔɻ 13

X ❤ 14 Wantedly on Kubernetes

Microservices on Kubernetes o u r p r o d u c t s i s Wa n t e d l y V i s i t , Wa n t e d l y Pe o p l e 60+ M i c r o s e r v i c e s 70+ K u b e r n e t e s N o d e s 3K+ C o n t a i n e r s

1.83 2.14 3.27 2.14 2.37 2.36 1.50 1.28 1.50 2.02 2.05 2.47 3.20 3.33 3.50 4.10 4.50 5.00 5.40 6.00 6.70 X X X M i c r o s e r v i c e s S e r v e r M a c h i n e Machine ͱ Server ม Խ ʹ ڧ ͍ Π ϯ ϑ ϥ ࡞ Γ ۭ ͍ͯ ͍ Δ Ϧ ιʔεΛ ଞ ͷ α ʔ ϏεͰ ར ༻ ͢Δ ͜ͱͰɺ ʮ ม Խ ʹ ڧ ͍ Π ϯϑ ϥʯ ͱ ޮ ཰ ੑ Λ ߴ Ί Δ 16

X X X ͜ͷ੒௕աఔͰɺ ͦΕ·Ͱ֖ͨ͠΋ ͷ͕஝ੵ͍͖ͯ͠ɺ ؾ͕͍ͭͨΒᷓΕ ͯἝΛ޲͍͖ͯͨɻ M i c r o s e r v i c e s ʹ ͓ ͚ Δ ՝ ୊ 17

Τ ϥ ʔ ɾ Π ϯ γ σ ϯ τ ؅ ཧ ɾ औ Γ ܾ Ί ґ ଘ ؔ ܎ ͦ ΋ͦ ΋ Τϥ ʔ ͸ Ͳ͜ Ͱ ى ͖ ͯ ͍ Δ ͷ ͔ ʁ ࠶ ݱ ํ ๏ ͸ ɺ ֬ ཱ ͞Εͯ ͍ Δ ͷ ͔ ʁ ͍ ͟ ݴ ͏ ࣌ ʹ ࣗ ෼ ͨ ͪͰϋ ϯ υϦ ϯ ά Ͱ͖Δ ͷ ͔ ʁ * ' ͸ Ͳ͏ ͍ ͏ ϧ ʔϧ ͳ ͷ ͔ ʁ Ξ ϥ ʔ τ͕ ໐ ͬͨ ࣌ ʹ ୭ ͕ ౓ ͷ α ʔ Ϗ ε ʹ ੹ ೚ Λ ࣋ ͬͯ ͍ Δ ͷ ͔ େ ྔ ͷ ϫ ʔ Ϋ ϩ ʔ υ Λ ଞ α ʔ Ϗε ʹ ରͯ͠౤͛ͯ͠·ͬ ͯ ͍ ͍ ͷ ͔ ʁ α ʔ Ϗε ໊ ͱ ػ ೳ ʹ ဃ ཭ ͕ ͳ ͍ ͔ ʁ Microservicesʹ͓͚Δ՝୊ ͜ ͷ ظ ؒ ʹ ۩ ମ త ʹ ى ͜ ͬ ͨ ͜ ͱ ਺ ݸ ͷ M i c r o s e r v i c e s ͷ ͱ͖ ͸ ɺ ਓ ؒ ʹ Α Δ ؅ ཧ Ͱ े ෼ ӡ ༻ Λ ճ ͢͜ͱ ͕ Ͱ͖ ͯ ͍ͨɻ ࣗ ಈ Խ ΍ t e m p l a t e ͕ ਐ Έɺ M i c r o s e r v i c e ͕ ࡞ Γ ΍ ͢ ͍ ੈ ք Ͱ ى ͜Δ ͜ͱɻ A α ʔ Ϗε ʹ ม ߋ ͕ ඞ ཁ ͱ Θ ͔ ͬͨ ࣌ ʹ ɺ ࣮ ͸ શ α ʔ ϏεͰ ม ߋ ͕ ඞ ཁ ͩͱ ؾ ͮ ͍ͨ ࣌ ͷ ઈ ๬ ײ ɻ 18

ڞ ௨ σ ʔ λ ஌ ݟ ڞ ༗ ։ ൃ ؀ ڥ R a n k i n g , P r o f i l e , Fr i e n d … ڞ ௨ σʔ λ ͸ͨ ͘ ͞Μ ͋ Δ Ͳ͏ ΍ ͬ ͯ ଞ α ʔ Ϗε ʹ ఻ ൖ ͨ͠ Γɺ Ωϟ ο γϡΫ Ϧ Ξ ͢Δ ϕε τϓϥ Ϋ ςΟ ε΍ ৽ ͠ ͍ ϛ υϧ ΢ΣΞΛ ೖ Εͯ ΋ ν ʔϜΛ ӽ ͑Δ ͱ ౰ ͨ Γ લ ͡Ό ͳ͘ ͳΔɻ ͢΂ͯͷ α ʔ Ϗε Λ ϩ ʔΧϧ 1 $ Ͱ ཱ ͯΔ ڞ ༗ ͷ ؀ ڥ ͸ ਖ਼ ͘͠ ἧ ͬͯ ͍ Δ ಈ ͍ͯ ͍ Δ ʁ σʔ λͷ ߋ ৽ ͸ ਖ਼ ͍͠ʁ Microservicesʹ͓͚Δ՝୊ ͜ ͷ ظ ؒ ʹ ۩ ମ త ʹ ى ͜ ͬ ͨ ͜ ͱ ਺ ݸ ͷ M i c r o s e r v i c e s ͷ ͱ͖ ͸ ɺ ਓ ؒ ʹ Α Δ ؅ ཧ Ͱ े ෼ ӡ ༻ Λ ճ ͢͜ͱ ͕ Ͱ͖ ͯ ͍ͨɻ ࣗ ಈ Խ ΍ t e m p l a t e ͕ ਐ Έɺ M i c r o s e r v i c e ͕ ࡞ Γ ΍ ͢ ͍ ੈ ք Ͱ ى ͜Δ ͜ͱɻ ϧ ʔ ςΟ ϯ ά α ʔ Ϗεͷ ෼ ׂ Λ͠ ͯ ͍͖ͨ ͍ ͚Ͳɺ ύ εϨ ϕ ϧ ͷ ม ߋ ΛͲ ͜ · Ͱ ख ಈ Λ ڐ ༰ ʁ " # ςε τ ͸ Ͳ͏ ͢Δ ʁ 19

α ʔ Ϗ ε ͕ ૿ ͑ Δ ͨ ͼ ʹ ௥ Ճ ͞ Ε Δ ࡞ ۀ w To i l τ Πϧ w Ku b e r n e t e s ͷ m a n i fe s t ͷ ม ߋ w C l u s t e r * m i c ro s e r v i c e * d e p l o y m e n t Ͱ ม ߋ ࿙ Ε ͕ ى ͖ ΍ ͢ ͍ w n * n ͷ ࡞ ۀ p u b l i c _ u r l ͔ Β i n t e r n a l _ u r l ͷ ม ߋ ౳ w Ξ ϓϦ έ ʔ γ ϣ ϯ ଆ ΋ ಉ ͡ Α͏ ͳ ί ʔ υ Λ ྔ ࢈ M o n i t o r i n g ܥ ΍ϝ τ ϦΫε ऩ ू w S S L ূ ໌ ॻ ߋ ৽ ͕ 2 0 0 ୆ ෼ ྫ A C M ͱ E L B ͷ S S L ΞΫη ϥ Ϩ ʔ λʔ Λ ࢖ ͍ ɺ S S L ূ ໌ ॻ Λ i m p o r t ͢Δͩ ͚ ʹ ͢Δ ͳͲɺ ࠜ ຊ త ʹ O ( 1 ) ʹ Ͱ͖ ͳ ͍ ͔ ߟ ͑Δ 20

Τϥʔ ͦ ͷ Τ ϥ ʔ ͸ ࢝ · Γ ͷ ࢝ · Γ w Τϥ ʔ͕ ෳ ࡶ ͢ ͗Δ w ࠶ ݱ ͞ ͤΑ͏ ͱ͢Δ ΋ ύ ϑΥ ʔϚ ϯε ͕ ҧ ͍ ͢ ͗ ͯɼ ผ ͷ ໰ ୊ ͕ ى ͖Δ w ʮ ຊ དྷ ղ ܾ ͢΂ ͖ ՝ ୊ ʯ Λ ݟ ۃ Ί Δ ʮ ඞ ཁ ͳ ৘ ใ Λ ୳ ͢ʯ ʮ ඞ ཁ ͳ ৘ ใ Λ ୳ ͢ʯ ʮ ࠶ ݱ ͢Δ ؀ ڥ Λ ੔ ͑Δʯ ຊ ࣭ త ͳ ࡞ ۀ · Ͱ ʹ ࣌ ؒ ͕ ͔ ͔ Δ w ҟ ৗ ঢ় ଶ ͷ ؅ ཧ w Y Y ܥ ɺ Y Y ܥ ͷ ࣦ ഊ Λ ౎ ౓ ઃ ఆ ͢Δ ඞ ཁ ͕͋ Δͨ Ί ࿙ Ε΍ ͢ ͍ɻ w Τϥ ʔ Ҏ ֎ ʹ ʮ ٧ ·ͬͯ ͍ Δʯ ঢ় ଶ ΋ ݕ ஌ ͢Δ ͷ ͕ ೉ ͠ ͍ w ӡ ༻ ճ ආ ΋ ݶ ք ͕͋ Δ w ϋ ϯ υϦ ϯά w ෳ ਺ ͷ ࣦ ഊ έ ʔε ʹ ࿙ Ε ͳ͘ ϋ ϯ υϦ ϯά͞ ͤΔ ͷ ͕ ೉ ͠ ͍ 21

σόοά α ʔ Ϗ ε Λ ލ ͙ w αʔϏεΛލ͍ͩσόοά͕େม w ελοΫτϨʔε͸ͦͷϦϙδτϦ͚ͩͷ৘ใͷΈ w ௨৴݁ՌͷσʔλΛॲཧ͢ΔҰ࿈ͷϑϩʔͰΤϥʔ͕ى͖ͨ৔߹ w ୭͕Ͳ͏͍͏ϦΫΤετΛͯ݁͠Ռ͕Ͳ͏ͩͬͨͷ͔ʁ w λΠϛϯάɾΠγϡʔͰ্ख͍͔͘ͳ͍έʔεͰͦͷલޙͷϦΫΤετΛΈ͍ͨ w ϚΠΫϩαʔϏεؒͰ n + 1 ͷϦΫΤετ͍ͯ͠Δ͜ͱʹؾ͚ͮͳ͍ 22

Πϯγσϯτ ࡂ ֐ ͸ ಥ વ ʹ w ͋Δ೔ಥવͷεύΠΫͰɺΠϯϑϥνʔϜʹΞϥʔϜ͕໐Δ w Өڹൣғͷಛఆ͢Δ΋Կ͕ى͖͍ͯΔ w Өڹൣғಛఆ͢Δ΋ɺͲ͜ΛݟΔ΂͖͔ʁ w ݟΔ΂͖৔ॴ͸ɺͦ΋ͦ΋ݟΔ͜ͱ͕Ͱ͖Δ͔ʁ w εύΠΫͷݩ͸Ͳ͔͜Βདྷ͍ͯΔʁ w Կ͕۩ମతʹى͍ͬͯ͜Δ͔ʁ w ݱঢ়ঢ়ଶ͔ΒͲ͏΍ͬͯҰ࣍෮چͤ͞Δ͔ʁ w ࠜຊରࡦ͸͍ͭͲ͜Ͱ୭͕ߦ͏͔ʁ 23

؅ཧɾऔΓܾΊ Ϩ ʔ ϧ Λ Ҿ ͘ w αʔϏεURLͷ؅ཧ͕ϑϨʔϜϫʔΫͰมΘΓ೉͍͠ w Rails͸ config/settings/{production,qa,development}.yml URLΛॻ͘ඞཁ w URLʹinternal_urlͱpublic_url w ൚༻తʹࢀর͞ΕΔΑ͏ͳαʔϏε͸ɺMicroservice෼ඞཁ w log ͷϨʔϧ w ΠϕϯτϩάΛ௥Ճ͢ΔϨʔϧ͸͋Δ͔ʁ w ΞΫηεϩά͸উखʹΑ͠ͳʹऔͬͯ͘Εͳ͍ͷ͔ʁ w ϑΥʔϚοτͷܾ·Γ͸ JSON , LSTV w ܽଛ཰͸ͦ΋ͦ΋Ͳͷ͘Β͍͋Δͷ͔ʁ 24

؅ཧɾऔΓܾΊ Ϩ ʔ ϧ Λ Ҿ ͘ w ΫϥΠΞϯτͱαʔόʔͷৼΔ෣͍ͷϨʔϧ w HTTP ΫϥΠΞϯτͷڍಈ͕౷Ұ͞Ε͍ͯͳ͍ w ίʔυʹڞ௨Խ͞Ε͍ͯͳ͍ w λΠϜΞ΢τͱϦτϥΠͷॳظઃఆ͸͋Δͷ͔ʁ w ϔομʔʹ͸Ͳ͏͍͏৘ใΛೖΕΔͷ͔ʁ w αʔϏεؒͷೝূΛͲͷΑ͏ʹ͢Δͷ͔ʁ w current userͷ఻ൖ͸Ͳ͏͢Δͷ͔ʁetc w ͋ΔΤϯυϙΠϯτ͕ԿΛظ଴͍ͯ͠Δͷ͔͕Θ͔Βͳ͍ w ύϥϝʔλͷεΩʔϚͷཧ༝͕Ԟ·Ͱಡ·ͳ͍ͱΘ͔Βͳ͍ 25

ґଘؔ܎ ಺ ෦ D D o s w େྔͷϫʔΫϩʔυΛଞαʔϏεʹґཔ w Ϣʔβʔ૿ՃͰɺେྔʹϦΫΤετΛૹΔ͜ͱʹͳΔ w αʔόʔ͔ΒɺRDS΍ElastiCacheʹৗʹؾΛ࢖͏ w ద౰ͳස౓ʹsleep͢Δͱ͍͏ঊͷٕ w ͦ΋ͦ΋ඇಉظͰྑ͍λεΫ͸ྑ͍ײ͡ʹ΍Εͳ͍ͷ͔ w ຊདྷͷ໾ׂΛେ͖͘௒͑ͨػೳΛ࣋ͭαʔϏε͕͋ΓɺԿނ͔ඞͣ໰͍߹Θͤͷதʹଘࡏ ͢Δ 26

ڞ௨σʔλ Ұ ൪ ෛ ՙ ͕ ू த ͠ ΍ ͢ ͍ w σʔλͷ఻ൖ w ͦΕͧΕͷMicroserviceͰCache͍ͯͨ͠৔߹ʹPurgeΛͲ͏͢Δ͔ w มߋΛͲ͏఻͑Δ͔ w ڞ௨ͷσʔλͷϋϯυϦϯά w ෳ਺ͷϨΠϠʔͰऔΓ૊Ήඞཁ͕͋Δ w ྫϓϩϑΟʔϧʹΧϥϜΛͭ௥Ճͯ͠ಡΈॻ͖͢Δ w A αʔϏεʹ schema ௥Ճ -> B αʔϏεͷ Garage Λมߋ -> C αʔϏεͷ garage proxy Ϟσϧमਖ਼ -> D αʔϏεʹॻ͖ࠐΈ endpoint • Microservice ʹͯ͠ɺϥΠϒϥϦͱͯ͠੾Γग़ͯ͠ɺશ෦࠷৽ʹ্͛Δͱ௥Ճ͞Εͳ͍͔ʁ 27

஌ݟڞ༗ d o c s ͩ ͚ Ͱ ͸ ଍ Γ ͳ ͍ ҉ ໧ ஌ w Microservicesͷجຊతͳ஌ࣝ w ྫDBͷڞ༗ͷ͸Ͱ͖Ε͹ආ͚Δ w ΤϯυϙΠϯτɺϦΫΤετͷத਎΍΍ΓͱΓํࣜ w ؔ܎ऀͷ஌ࣝʹ͹Β͖ͭ w ૊৫શମͰͷAPIͷ࡞ΓํʹϨʔϧ w API࣮૷ํ๏ w I/F ϧʔϧ w όϥόϥʹΑΔݺͿଆͷੜ࢈ੑ͕Լ͛ΒΕΔ 28

஌ݟڞ༗ E P D T ͩ ͚ Ͱ ͸ ଍ Γ ͳ ͍ w ར༻͢ΔOSSͷѻ͍ʹؔ͢ΔϨʔϧ w ԿΛͲͷϨϕϧ·Ͱѻ͏ͷ͔ w ج४͸ͳʹ͔ w DockerfileͲ͏͍͏෩ʹॻ͍ͨΒ͍͍͔ w ݴޠʹΑ͔ͬͯΘΔ w Alpine Linux Ͱফ໣͢Δͷ͕ͭΒ͍ w ٕज़Λप஌ͤ͞Δ৔ॴ w ͦͷ৔Ͱཧղ͠ͳͯ͘΋ɺ΋͏Ұ౓୳ͤΔ৔ॴ͕ͳ͍ 29

։ൃ؀ڥ(QA,Staging) ੜ ࢈ ੑ ʹ ؔ Θ Δ w ֤؀ڥͷσʔλಉظ w σʔλͷϦετΞ͢Δॱ൪ͰMicroservices͕յΕΔ w ͣͬͱݹ͍··ͷσʔλ w ͦͷ··ϦετΞͰ͖ͳ͍σʔλͷѻ͍ w ڞ༗؀ڥͰҙਤ͠ͳ͍topic branchͰಈ͖ଓ͚Δ 30

ϧʔςΟϯά Α ͠ ͳ ʹ ܨ ͙ w ϞόΠϧνʔϜͱαʔόʔαΠυνʔϜͰڞಉ։ൃ͢Δࡍ w Topic branch Ͱ֬ೝ͠ͳ͕Β։ൃ͍ͨ͠ w ಛఆͷϧʔςΟϯά͢Δज़͕ͳ͍ͨΊɺ master Ͱৗʹ֬ೝ͍ͯ͠Δςελʔ͞Μͱ࿈ܞ ͯ͠ڞ༗؀ڥʹͩͯ֬͠ೝ͢Δ͔͠ͳ͍ w A service -> B sevice -> C service ͱ͍͏ϚΠΫϩαʔϏε͕ؒ͋ͬͨ࣌ w B ͷ։ൃ͕೉͗͢͠Δ w ϦΫΤετΛ࠶ݱ͢Δ؆୯ͳํ๏͸ͳ͍͔ʁ 31

ϧʔςΟϯά ω ο τ ϫ ʔ Ϋ ى Ҽ ͷ ϋ ϯ υ Ϧ ϯ ά ͸ Ͳ ͜ Ͱ ΍ Δ ʁ w ΞʔΩςΫνϟͷมߋ΍αʔϏε෼ׂ͢Δࡍ w Ͳͷ͘Β͍ΠϯύΫτ͕͋Δʁ w Θ͔Βͣාͯ͘ग़͠ʹ͍͘ɺΧφϦʔσϓϩΠͯ͠ཉ͍͠ͱ͖͕͋Δ w A/B ؀ڥͰνΣοΫ͍ͨ͠ w inͱout͕ಉ͡Ͱ͋Δ͜ͱΛอূΛνΣοΫ͢Δ࢓૊Έ͕΄͍͠ w timeout ͱ retry Λશ෦ͷΞϓϦέʔγϣϯʹॻ͍͍ͯ͘ͷ͸͍͔ͭഁ୼͢Δ w αʔϏεΛ෼ׂ͢Δࡍʹ path ϕʔεͰ෼͚ΔͷΛ nginx.conf Ͱ΍Δͷ͸͍͔ͭݶք w API Gatewayʹ೚͍͖͍ͤͯͨ 32

՝୊͔Β໨తͱखஈΛߟ͑Δ ม Խ ʹ ڧ ͍ Π ϯ ϑ ϥ ͷ ࣍ ͷ ε ς ο ϓ ΁ SREνʔϜʹվ໊ʁ ໾ׂΛڱΊΔ͜ͱʹͳΒͳ ͍Α͏ݱࡏͷ໊લ͸ҡ࣋ɻ SREͷྖҬʹରͯ͠ɺίϛο τͰ͖ΔΑ͏αʔόʔαΠυ ͷΤϯδχΞΛܴ͑ͯۀ຿ͷ தʹSREۀΛ૊ΈࠐΉɻࣄ ۀΛ௒͑ͯΠϯϑϥج൫ɺ࢓ ૊ΈɺΞϓϦͷ͍ΖΜͳํ޲ ͔Β՝୊ղܾͰ͖ΔΑ͏ʹɻ ՝ ୊ ղ ܾ ͸ ɺ πʔϧ ࡞ ੒ ࠔ ͬͯ ͍ Δ ͜ͱ ͸ ৗ ʹ ҙ ݟ ͱͯ͠ د ͤΒ ΕΔɻ ͦ ͷ த Ͱɺ ༏ ઌ ॱ Ґ ͕ Ұ ൪ ߴ ͍ ͷ Λ ൑ அ ͢Δ ͷ ͸ ೉ ͠ ͍ɻ ੠ ͷ Ͱ ͔ ͍ ਓ ɺ ࢹ ࠲ ͷ ҧ ͍ ౳ Ͱ ॱ Ґ ͕ ٯ స ͢Δ ͜ͱ ΋ ͋ Δɻ ࢥ ͍ ੾ ͬͯ ·ͱ Ίͯ ੔ ཧ ͯ͠ ΋ Β ͑ ͨ ͜ͱͰ ՝ ୊ ղ ܾ ͷ ࢹ ք ͕ ޿ ͕ ͬͨɻ มԽʹڧ͍ΠϯϑϥΛ࡞Δ͜ ͱͱɺٻΊΒΕΔ࣍ͷίΞٕ ज़ͷ૑ग़͸Ͳͷ͘Β͍ؔ࿈͋ Δʁ৔͚ͩ༻ҙͯ͠ʮ͋ͱ͸ ΑΖ͘͠ʯͰίΞٕज़͸೉͠ ͍ɻऔΓ૊΋͏ͱ͍ͯ͠Δ͜ ͱɺτϨϯυɺ࿦จɺւ֎ͷ ϒϩάͳͲͷهࣄΛҰॹʹݟ ͯɺऔΓ૊ΉྖҬʹϑΥʔΧ εͰ͖Δج൫Λ੔͑Δɻ 33

ΠϯϑϥνʔϜ͕໨ࢦ͢ΰʔϧͷมԽ มԽʹڧ͍Πϯϑϥ࡞Γ͔Β ม Խ ʹ ڧ ͍ Π ϯϑ ϥ ͔ Β ɺ γ ΰ τͰί ί ϩΦ υϧ Λ ૿ ΍ ͢ ج ൫ Τϯ δχΞͷ ࢓ ࣄ ྔ Λ ૿ ΍ ͢͜ͱ ʹ γ ϑ τ αΠτͷ৴པੑ S i t e R e l i a b i l i t y ΤϯδχΞશମͷੜ࢈ੑ D e v e l o p e r P r o d u c t i v i t y ίΞٕज़ͱͳΔج൫ ٕ ज़ ઓ ུ 34

OKRʹूதͰ͖ΔνʔϜ Ұਓ͋ͨΓͷΤϯδχΞͷ࢓ࣄྔΛ૿΍͢ • OKR(Objectives and Key Results) w Oʢ໨ඪʣ w KRʢ੒Ռࢦඪʣ w Ұਓ͋ͨΓ͕͜ͳͤΔ࢓ࣄྔͷ֦େ w Ұਓ͋ͨΓͷػցֶशandࢪࡦͷPDCAͷαΠΫϧ͕ଟ͘ճͤΔ w தԝूݖͱ෼ׂ౷࣏ͷׂ߹͕ద੾ͰҰਓͻͱΓ͕νϟϨϯδͰ͖Δ w Ұਓ͋ͨΓͷ࢓ࣄྔ͕ݮΔ΋ͷΛ࡟ݮ w ໰୊ͷՕॴΛ͙͢ʹಛఆͰ͖Δ w ஌Γ͍ͨ৘ใ͕͙͢ʹΘ͔Δ w ౷Ұ͢Δ΂͖ͱ͜Ζ͕౷Ұ͞Ε͍ͯΔ 35

X X X 36 Τ ϯ δ χ Ξ ͔ Β ݟ ͨ ັ ྗ ௅ઓͰ͖Δ͜ͱ ࣾձతΠϯύΫτ͕͋Δ ͍͢͝ਓͨͪͱҰॹʹಇ͚Δ

Ͳ͏ղܾ͍͔ͯ͘͠ ໨ ඪ ͱ ՝ ୊ ͔ Β ώ Ξ Ϧ ϯά΍ ·ͱ Ίͯ ΋ Βͬͨ΋ ͷ Λ ࢀ ߟ ʹ ϓϥ ΠΦϦ ςΟύ εΛ ܾ Ί Δɻ ͦ ͷ த Ͱɺ ಉ ࣌ ʹ ಈ ͔ ͤΔ ͷ ͸ ɺ ݱ ν ʔϜͰ ͸ 3 ͭ ͕ ݶ ք ɺ ͦΕͧΕͰ Ұ ൪ ߴ ͍ ΋ ͷ ʹ ू த ͢Δ αΠτͷ৴པੑʹͭͳ͕ ΔऔΓ૊Έ S i t e R e l i a b i l i t y ΤϯδχΞ͕ࠔ͍ͬͯΔ ͜ͱͷղܾ D e v e l o p e r P r o d u c t i v i t y ະདྷʹඞཁͱͳΔίΞٕ ज़ͱͳΔج൫ͮ͘Γ ٕ ज़ ઓ ུ 37

Site Reliability α Π τ ͷ ৴ པ ੑ ʹ ͭ ͳ ͕ Δ औ Γ ૊ Έ U p t i m e ί Ξό Ϧϡʔ ࢮ क S c a l a b i l i t y Ku b e r n e t e s Ҡ ߦ p o s t m o r t e m s ਖ਼͘͠໰ ୊Λཧ ղɺ ֶ ͼΛ ஝ ੵ ͠ɺ ࠶ ൃ ๷ ࢭ ͱ ࠜ ຊ ର Ԡ D o c s a n d Ta l k ؔ ৺ ࣄ ͱ ҉ ໧ ஌ ͷ ڞ ༗ S c a l a b i l i t y ͱ p o s t m o r t e m s 38

ϙετϞʔςϜ ௕ ظ త ੒ ௕ Λ ಷ Խ ͞ ͤ ͳ ͍ bϙετϞʔςϜ͸ɺΠϯγσϯτͱͦͷΠϯύΫτɺͦͷ؇࿨΍ղফͷͨΊʹߦΘΕͨΞΫ γϣϯɺࠜຊݪҼʢ܈ʣɺΠϯγσϯτͷ࠶ൃΛආ͚ΔͨΊͷϑΥϩʔΞοϓͷΞΫγϣϯΛ ه࿥͢ΔͨΊʹॻ͔ΕΔ΋ͷͰ͢ɻ` ൈਮBetsy BeyerlSREαΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯάz w Πϯγσϯτ͔ΒֶͼΛಘΔͨΊͷఆࣜԽ͞Εͨϓϩηε͕ͳ͚Ε͹ɺಉ͡Α͏ͳΠϯγσ ϯτ͕ແݶʹ܁Γฦ͠ى͜Δ w ໺์͠ͷ··ʹͳͬͯ͠·͑͹ɺΠϯγσϯτͷෳࡶ͞͸Ճ଎౓తʹ૿͠ɺ͋Δ͍͸ੵΈ ॏͳͬͯγεςϜͷରॲ͕Ͱ͖ͳ͍ w ࣦഊ͔ΒֶͿͨΊʹৼΓฦΓͷϙετϞʔςϜΛॻ͘ 39

Developer Productivity ΤϯδχΞ͕ࠔ͍ͬͯΔ͜ͱͷղܾࡦ To o l શ Τϯ δχΞ͕ ۀ ຿ Λ ޮ ཰ త ʹ ߦ ͏ F l u e n t d ܾ ·ͬͨ ϩ ά ܗ ࣜ Λ ૹ Δ D i s t r i b u t e d Tr a c i n g ӈ ͷ ਤ ɺ ϚΠΫ ϩ αʔϏε ؒ ͷτϨʔ εΛ ࣮ ݱ ͢Δ Service mesh ϚΠΫ ϩ α ʔ Ϗε ͷ ωο τϫ ʔ Ϋ ؒ Λ ੔ ͑Δ O p e n C e n s u s , S e r v i c e M e s h , F l u e n t d e t c 40

OpenTracing? OpenCensus? Distributed tracing ͷΫϥΠΞϯτϥΠϒϥϦ OpenTracing ෼ࢄτϨʔγϯάΛ࣮ݱ͢ΔͨΊͷڞ௨ ͷϓϩτίϧͱAPI͕ఆٛ ֤ࣾͰ࣮૷ɻZipkin͸Twitterࣾɻ طʹ࢖ΘΕ͍ͯΔͱ͜Ζ͕ଟ͘ɺ OpenCensus͸ޙൃͰɺOpenTracingͷ ΄͏͕ྺ࢙͕͋Δɻͨͩɺ࢓্༷ࣗ෼ͨ ͪͰ࣮૷͍ͯ͘͠ඞཁ͕͋Δɻ OpenCensus metrics΍tracesΛऩूɾ഑৴͢ΔͨΊ ͷ֦ுੑͷ͋Δ౷߹తͳΠϯλʔϑΣʔ εΛఏڙ͢ΔϑϨʔϜϫʔΫ Googleͷ census library ͷ OSS ൛ ݴޠͱϕϯμʔͷࣗ༝͞Λҡ͍࣋ͯ͠Δ Go, Ruby, Python, Java, C++, Erlang, PHP StackDriver, Datadog APM, AWS X-RAY 41

ٕज़ઓུ ະདྷʹඞཁʹͳΔίΞٕज़ͱͳΔج൫ͮ͘Γ ܧ ଓ త ͳ ධ Ձ Φϑ ϥ Π ϯ ධ Ձ D a t a S t o r e σʔ λ ͷ ࠷ ద Խ Development A m a z o n S a g e M a ke r C I / C D ϓϩ μΫ τ· Ͱ ͷ σϦό Ϧ Ͱ͖Δ ٕ ज़ ྖ Ҭ Λ ૿ ΍ ͢ 42

ٕज़ઓུ keyword w ࠓޙΛ୲͏ίΞٕज़ΛϓϩμΫτνʔϜ΍$50ɺϦʔυΤϯδχΞͱҰॹʹઓུΛཱͯͯɺ ج൫ͱ࢓૊Έͱ࣮૷ w ίϛϡχέʔγϣϯ w ྠߨձ w ւ֎ࣄྫ • Tech blog w ώΞϦϯάௐࠪ w ૊৫શମ͔ΒݟͯӨڹྗ͕େ͖͍෼໺Ͱ΋ଞνʔϜͰऔΓ૊Έʹ͍͘΋ͷ w σʔλετΞͷMicroservices w ։ൃͰ͖Δ΋ͷͷϨϕϧΛ্͛Δ w ͍͘Βߴ଎ʹͰ͖ͯ΋ɺ։ൃͰ͖Δ΋ͷʹݶΓ͕͋Ε͹ͦΕҎ্ʹͳΔ͜ͱ͸ͳ͍ 43

Summary Site ReliabilityɾDeveloper Productivityɾٕज़ઓུͷऔΓ૊Έ 44 ೥͔ΒมԽʹڧ͍ΠϯϑϥΛ໨ࢦ͖ͯͨ͜͠ͱʹΑΔޮՌͱ՝୊ ม Խ ʹ ڧ ͍ Π ϯϑ ϥ ࡞ Γ ͷ ݁ Ռ ຊདྷΠϯϑϥνʔϜͷՌͨ͢΂͖՝୊ղܾ͸ͳʹ͔ʁ ໨ ࢦ ͢ ΰʔϧ ͷ ม Խ มԽʹڧ͍Πϯϑϥ͸ڧྗͳ෢ث͕ͩͬͨಉ࣌ʹ೉͍͠՝୊΋ੜΜͩ ϚΠΫ ϩ α ʔ Ϗε ͷ ՝ ୊ Ͳ͏ ղ ܾ ͯ͠ ͍ ͘ ͔ ͍·ݱ࣌఺ͰऔΓ૊ΜͰ͍Δ΋ͷ΍ղܾ͍ͯ͠Δ΋ͷͳʹ͔

X X X 45 Τ ϯ δ χ Ξ ͔ Β ݟ ͨ ັ ྗ ௅ઓͰ͖Δ͜ͱ ࣾձతΠϯύΫτ͕͋Δ ͍͢͝ਓͨͪͱҰॹʹಇ͚Δ

X X X 46 W e a r e H i r i n g ! https://www.wantedly.com/companies/wantedly