$30 off During Our Annual Pro Sale. View Details »

強いインフラが組織を創る / #jtf2018

強いインフラが組織を創る / #jtf2018

これまで目指していた「変化に強いインフラ」の話から、Microservices における課題、インフラチームが目指すものの変化、それをどう解決していくか?という話をしました

July Tech Festa 2018
at 産業技術大学院大学(公立大学法人 首都大学東京)

Kodai Sakabe

July 29, 2018
Tweet

More Decks by Kodai Sakabe

Other Decks in Technology

Transcript

  1. ڧ͍Πϯϑϥ͕૊৫Λ૑Δ
    ŠSite ReliabilityɾDeveloper Productivityɾٕज़ઓུͷऔΓ૊ΈŠ
    July Tech Festa 2018
    29 July 2018
    Wantedly, Inc. | Kodai Sakabe @koudaiii
    1

    View Slide

  2. ΋͠ɺࣗ෼͕௅ઓͰ͖ͳ͍ͱࢥ͏ͳΒͦΕ
    ͸ج൫ͱ૊৫͕ऑ͍ɻͦͯͦ͠Ε͸ɺΠϯ
    ϑϥνʔϜ $50΍Ϧʔμʔ
    ͷ੹೚ɻࣗ෼
    ͕Ͳͷ͘Β͍௅ઓͰ͖Δج൫ͱ૊৫Ͱಇ͍
    ͍ͯΔ͔طʹΤϯδχΞ͸ؾ͍͍ͮͯΔ
    @koudaiii
    2

    View Slide

  3. X
    X
    X
    3
    Τ ϯ δ χ Ξ ͔ Β ݟ ͨ ັ ྗ
    ௅ઓͰ͖Δ͜ͱ
    ࣾձతΠϯύΫτ͕͋Δ
    ͍͢͝ਓͨͪͱҰॹʹಇ͚Δ

    View Slide

  4. CREATE
    A WORLD WHERE WORK
    DRIVES PASSION.
    Wantedly, Inc.
    4

    View Slide

  5. OUR GOALS
    Wantedly, Inc.
    D i s c o v e r C o n n e c t
    E n g a g e S N S
    C r e a t e a w o r l d w h e r e w o r k d r i v e s p a s s i o n
    5

    View Slide

  6. ࿩͢͜ͱ
    Site ReliabilityɾDeveloper Productivityɾٕज़ઓུͷऔΓ૊Έ
    ೥͔ΒมԽʹڧ͍ΠϯϑϥΛ໨ࢦ͖ͯͨ͜͠ͱʹΑΔޮՌͱ՝୊
    ม Խ ʹ ڧ ͍ Π ϯϑ ϥ ࡞ Γ ͷ ݁ Ռ
    ຊདྷΠϯϑϥνʔϜͷՌͨ͢΂͖՝୊ղܾ͸ͳʹ͔ʁ
    ໨ ࢦ ͢ ΰʔϧ ͷ ม Խ
    มԽʹڧ͍Πϯϑϥ͸ڧྗͳ෢ث͕ͩͬͨಉ࣌ʹ೉͍͠՝୊΋ੜΜͩ
    ϚΠΫ ϩ α ʔ Ϗε ͷ ՝ ୊
    6
    Ͳ͏ ղ ܾ ͯ͠ ͍ ͘ ͔
    ͍·ݱ࣌఺ͰऔΓ૊ΜͰ͍Δ΋ͷ΍ղܾ͍ͯ͠Δ΋ͷͳʹ͔

    View Slide

  7. ΞδΣϯμ
    w มԽʹڧ͍ΠϯϑϥΛ໨ࢦͨ݁͠Ռ
    w Microservicesʹ͓͚Δ՝୊
    w ΠϯϑϥνʔϜ͕໨ࢦ͢΋ͷͷมԽ
    w Ұਓ͋ͨΓͷΤϯδχΞͷ࢓ࣄྔΛ૿΍͢ = OKR ʹूதͰ͖ΔνʔϜ
    w Ͳ͏ղܾ͍͔ͯ͘͠
    w Site ReliabilityʮαΠτͷ৴པੑʹͭͳ͕ΔऔΓ૊Έʯ
    w Developer ProductivityʮΤϯδχΞ͕ࠔ͍ͬͯΔ͜ͱͷղܾࡦʯ
    w ٕज़ઓུʮະདྷʹඞཁʹͳΔίΞٕज़ͱͳΔج൫ͮ͘Γʯ
    7

    View Slide

  8. Wantedly Vist
    ม Խ ʹ ڧ ͍ Π ϯ ϑ ϥ ࡞ Γ ͷ ݁ Ռ
    ೥ ʹ ɺ H e r o k u ͔ Β A W S ΁ ࡌ
    ͤ ସ ͑ ɺ Α Γ Ϟ ό Πϧ Λ ॏ ࢹ ͨ͠Ξʔ Ω
    ς Ϋ ν ϟ ʹ ม ߋ ӈ ͸ ౰ ࣌ ͷ ߏ ੒ ਤ

    ೥ ʹ U b u n t u ͔ Β C o r e O S ʹ
    ม ߋ ɻ ͜ ͷ ೥ ʹ C h e f Λ ΍ Ί ͯɺ
    D o c k e r f i l e Ұ ຊ ʹ ͨ͠ ɻ ͦ ͷ ޙ ɺ
    ೥ ͔ Β K u b e r n e t e s ΁ Ҡ ߦ Λ ͸
    ͡ Ί ͯ ͍ ͘ ɻ
    8

    View Slide

  9. Wantedly People
    ม Խ ʹ ڧ ͍ Π ϯ ϑ ϥ ࡞ Γ ͷ ݁ Ռ
    2 0 1 6 ೥ Wa n t e d l y P e o p l e
    M i c r o s e r v i c e s o n K u b e r n e t e s
    ͜ Ε · Ͱ ͷ R u b y o n R a i l s ( U n i c o r n )
    ͷ Έ ͠ ͔ ࢖ ͑ ͳ ͍ σ ϓϩ Π ϑ ϩ ʔ Λ ΍
    Ί ɺ ʮ ม Խ ʹ ڧ ͍ Π ϯϑ ϥʯ Λ ࠶ ݱ Ͱ
    ͖ Δ Α ͏ ͳ ج ൫ ʹ ͨ͠ ɻ ݱ ࡏ ͸ ɺ ౰ ࣌
    ͷ α ʔ Ϗε ਺ ͷ ഒ Ҏ ্ ͷ α ʔ Ϗε ͕ Ք
    ಇ ͯ͠ ͍ Δ ʢ ӈ ͸ ౰ ࣌ ͷ ߏ ੒ ਤ ʣ
    9

    View Slide

  10. Why Microservices?
    Ϛ Π Ϋ ϩ α ʔ Ϗ ε Λ ਐ Ί Δ ཧ ༝
    ίΞόϦϡʔΛ໌֬ʹ͢Δ
    ଞͷӨڹΛड͚ͳ͍Α͏ʹ͢Δ
    ί Ξό Ϧϡ ʔ Λ क Δ
    ʮ༑ୡͷ༑ୡͷ ID Λߴ଎ʹฦ͢ʯʮ໊ࢗը૾Λ៉ྷʹ͢Δʯ౳ʹಛԽ͢Δ
    ٕ ज़ ʹ ͜ ͩ Θ Δ
    νʔϜن໛֦େ͢Δ෼ɺҰਓ͋ͨΓͷҙࣝ͢Δྔͷ૿ՃΛ཈͑Δ
    ଞνʔϜͷґଘؔ܎͕ڧ͍΄Ͳ೉͘͠ɺௐ੔ͱશମͷ೺ѲͰେมʹͳΔ
    εέ ʔϧͰ ͖ Δ ૊ ৫ ʹ ͳΔ
    10

    View Slide

  11. Why Kubernetes?
    طଘʹ௥Ճͨ͠΄͏ָ͕ WT৽نʹ࡞Δ
    উखʹࢼͯ͠ɺউखʹऴΘΒ͍ͤͨ
    EC2, ELB, RDS, Cloudfront, S3, ElastiCache, Route53 Terraformʹͯ͠΋खؒʜ

    Π ϯ ϑ ϥ ࡞ ۀ ͷ ख ؒ ͕ গ ͳ ͍
    ޷͖উखʹॻ͍ͨ΋ͷ͕ಈ͘ɻ ٯʹΊΜͲ͍͘͞෦෼΋͋Δ

    Rustͷ௅ઓ I/F͕໌֬ɻBϓϥϯ͕༗ΔɻMonitoringɻ࠷ѱॻ͖௚ͤΔɻσʔλҰ؏ੑ

    D o c k e r f i l e ʹ ޷ ͖ উ ख ʹ ॻ ͚Δ
    Ϋϥελʹ৐ͤΕ͹؂ࢹର৅ʹͳΔ
    ࠷ѱिؒͰॻ͖௚͢͜ͱ΋Ͱ͖ΔαʔϏε΋ଞͷαʔϏεͱಉ౳ͷ඼࣭ʹѻΘΕΔ
    ͲΜ ͳ α ʔ Ϗε΋ ಉ ౳ ͷ Π ϯ ϑ ϥ ඼ ࣭
    11

    View Slide

  12. Why Kubernetes?
    Kubernetes ͷϨʔϧʹ৐Δ
    w ޮ ཰ త ʹ M a c h i n e Λ ಈ ͔ ͢
    • H o r i z o n t a l P o d A u t o s c a l e r
    • C u s t o m m e t r i c s
    • C l u s t e r A u t o s c a l e r
    w ӡ ༻ ͷ ౷ Ұ
    • k u b e c t l
    • t h i r d p a r t y t o o l s ( i s t i o c t l … )
    • x x x - o p e r a t o r ( v a u l t - o p e r a t o r, e t c d - o p e r a t o r … )
    • m a n i f e s t f i l e
    Ͳͷ Ξ ϓ Ϧ έ ʔ γ ϣ ϯ ʹ Ҡ ͬͯ ΋ ɺ ಉ ͡ ί Ϛ ϯ υ Ͱ Ұ ௨ Γ ͷ ۀ ຿ ͕ Ͱ ͖ Δɻ ϛ υϧ
    ΢ΣΞ ΋ ͞ ͬ͘ ͱ ೖ Ε Δ ͷ Ͱ͋ Ε ͹ ଞ ͷ M a n i f e s t f i l e ͔ Β ഈ आ ͯ͠ ࢼ ͤΔ
    12

    View Slide

  13. Why Kubernetes?
    Kubernetes ͷϨʔϧʹ৐Δ
    w K u b e r n e t e s Λ औ Γ ר ͘ Τ ί γεςϜ ͷ Ը ܙ
    • S e r v i c e m e s h ( I s t i o , E n v o y … )
    • N e t w o r k i n g ( A m b a s s a d o r, Te l e p l e s e n c e , Ko n g … )
    • M o n i t o r i n g ( P r o m e t h e u s , D a t a d o g , N e w R e l i c … )
    • m i n i k u b e
    • H e l m c h a r t
    • C l o u d S e r v i c e s
    • C o m m u n i t y
    Ϋϥελʹ͍Δ͚ͩͰɺΤίγεςϜͷԸܙ͕ಘΒΕΔϝϦοτ͸Ͱ͔͘ɺΫϥελʹ͍ͳ͍
    ͜ͱͰى͜ΔίετͱϦεΫ͕ߴ͍ɻ·ͨڧྗͳ Community ͸ɺੈͷதͷΞʔΩςΫνϟ͕
    Ͳ͏͍͏՝୊͕͋ΓɺͲ͜ʹ޲͔͍ͬͯΔͷ͔͕Θ͔Δɻࣗ෼ͨͪͷٕज़ͷίϯύεʹͳΔɻ
    13

    View Slide

  14. X

    14
    Wantedly on Kubernetes

    View Slide

  15. Microservices on Kubernetes
    o u r p r o d u c t s i s
    Wa n t e d l y V i s i t , Wa n t e d l y Pe o p l e
    60+
    M i c r o s e r v i c e s
    70+
    K u b e r n e t e s N o d e s
    3K+
    C o n t a i n e r s


    View Slide

  16. 1.83
    2.14
    3.27
    2.14
    2.37 2.36
    1.50
    1.28
    1.50
    2.02 2.05
    2.47
    3.20 3.33
    3.50
    4.10
    4.50
    5.00
    5.40
    6.00
    6.70
    X
    X
    X
    M i c r o s e r v i c e s S e r v e r M a c h i n e
    Machine ͱ Server
    ม Խ ʹ ڧ ͍ Π ϯ ϑ ϥ ࡞ Γ
    ۭ ͍ͯ ͍ Δ Ϧ ιʔεΛ ଞ ͷ α ʔ ϏεͰ ར ༻ ͢Δ ͜ͱͰɺ
    ʮ ม Խ ʹ ڧ ͍ Π ϯϑ ϥʯ ͱ ޮ ཰ ੑ Λ ߴ Ί Δ
    16

    View Slide

  17. X
    X
    X
    ͜ͷ੒௕աఔͰɺ
    ͦΕ·Ͱ֖ͨ͠΋
    ͷ͕஝ੵ͍͖ͯ͠ɺ
    ؾ͕͍ͭͨΒᷓΕ
    ͯἝΛ޲͍͖ͯͨɻ
    M i c r o s e r v i c e s ʹ ͓ ͚ Δ ՝ ୊
    17

    View Slide

  18. Τ ϥ ʔ ɾ Π ϯ γ σ ϯ τ ؅ ཧ ɾ औ Γ ܾ Ί ґ ଘ ؔ ܎
    ͦ ΋ͦ ΋ Τϥ ʔ ͸ Ͳ͜
    Ͱ ى ͖ ͯ ͍ Δ ͷ ͔ ʁ ࠶
    ݱ ํ ๏ ͸ ɺ ֬ ཱ ͞Εͯ
    ͍ Δ ͷ ͔ ʁ ͍ ͟ ݴ ͏ ࣌
    ʹ ࣗ ෼ ͨ ͪͰϋ ϯ υϦ ϯ
    ά Ͱ͖Δ ͷ ͔ ʁ
    * ' ͸ Ͳ͏ ͍ ͏
    ϧ ʔϧ ͳ ͷ ͔ ʁ Ξ
    ϥ ʔ τ͕ ໐ ͬͨ ࣌
    ʹ ୭ ͕ ౓ ͷ α ʔ Ϗ
    ε ʹ ੹ ೚ Λ ࣋ ͬͯ
    ͍ Δ ͷ ͔
    େ ྔ ͷ ϫ ʔ Ϋ ϩ ʔ
    υ Λ ଞ α ʔ Ϗε ʹ
    ରͯ͠౤͛ͯ͠·ͬ
    ͯ ͍ ͍ ͷ ͔ ʁ α ʔ
    Ϗε ໊ ͱ ػ ೳ ʹ ဃ
    ཭ ͕ ͳ ͍ ͔ ʁ
    Microservicesʹ͓͚Δ՝୊
    ͜ ͷ ظ ؒ ʹ ۩ ମ త ʹ ى ͜ ͬ ͨ ͜ ͱ
    ਺ ݸ ͷ M i c r o s e r v i c e s ͷ ͱ͖ ͸ ɺ ਓ ؒ ʹ Α Δ ؅ ཧ Ͱ े ෼ ӡ ༻ Λ ճ ͢͜ͱ ͕ Ͱ͖ ͯ ͍ͨɻ
    ࣗ ಈ Խ ΍ t e m p l a t e ͕ ਐ Έɺ M i c r o s e r v i c e ͕ ࡞ Γ ΍ ͢ ͍ ੈ ք Ͱ ى ͜Δ ͜ͱɻ
    A α ʔ Ϗε ʹ ม ߋ
    ͕ ඞ ཁ ͱ Θ ͔ ͬͨ
    ࣌ ʹ ɺ ࣮ ͸ શ α ʔ
    ϏεͰ ม ߋ ͕ ඞ ཁ
    ͩͱ ؾ ͮ ͍ͨ ࣌ ͷ
    ઈ ๬ ײ ɻ
    18

    View Slide

  19. ڞ ௨ σ ʔ λ ஌ ݟ ڞ ༗ ։ ൃ ؀ ڥ
    R a n k i n g , P r o f i l e ,
    Fr i e n d …
    ڞ ௨ σʔ λ ͸ͨ ͘
    ͞Μ ͋ Δ Ͳ͏ ΍ ͬ
    ͯ ଞ α ʔ Ϗε ʹ ఻
    ൖ ͨ͠ Γɺ Ωϟ ο
    γϡΫ Ϧ Ξ ͢Δ
    ϕε τϓϥ Ϋ ςΟ
    ε΍ ৽ ͠ ͍ ϛ υϧ
    ΢ΣΞΛ ೖ Εͯ ΋
    ν ʔϜΛ ӽ ͑Δ ͱ
    ౰ ͨ Γ લ ͡Ό ͳ͘
    ͳΔɻ
    ͢΂ͯͷ α ʔ Ϗε
    Λ ϩ ʔΧϧ 1 $ Ͱ ཱ
    ͯΔ ڞ ༗ ͷ ؀ ڥ ͸
    ਖ਼ ͘͠ ἧ ͬͯ ͍ Δ
    ಈ ͍ͯ ͍ Δ ʁ σʔ
    λͷ ߋ ৽ ͸ ਖ਼ ͍͠ʁ
    Microservicesʹ͓͚Δ՝୊
    ͜ ͷ ظ ؒ ʹ ۩ ମ త ʹ ى ͜ ͬ ͨ ͜ ͱ
    ਺ ݸ ͷ M i c r o s e r v i c e s ͷ ͱ͖ ͸ ɺ ਓ ؒ ʹ Α Δ ؅ ཧ Ͱ े ෼ ӡ ༻ Λ ճ ͢͜ͱ ͕ Ͱ͖ ͯ ͍ͨɻ
    ࣗ ಈ Խ ΍ t e m p l a t e ͕ ਐ Έɺ M i c r o s e r v i c e ͕ ࡞ Γ ΍ ͢ ͍ ੈ ք Ͱ ى ͜Δ ͜ͱɻ
    ϧ ʔ ςΟ ϯ ά
    α ʔ Ϗεͷ ෼ ׂ Λ͠
    ͯ ͍͖ͨ ͍ ͚Ͳɺ ύ
    εϨ ϕ ϧ ͷ ม ߋ ΛͲ
    ͜ · Ͱ ख ಈ Λ ڐ ༰ ʁ
    " # ςε τ ͸ Ͳ͏
    ͢Δ ʁ
    19

    View Slide

  20. α ʔ Ϗ ε ͕ ૿ ͑ Δ ͨ ͼ ʹ ௥ Ճ ͞ Ε Δ ࡞ ۀ
    w To i l τ Πϧ

    w Ku b e r n e t e s ͷ m a n i fe s t ͷ ม ߋ
    w C l u s t e r * m i c ro s e r v i c e * d e p l o y m e n t Ͱ ม ߋ ࿙ Ε ͕ ى ͖ ΍ ͢ ͍
    w n * n ͷ ࡞ ۀ p u b l i c _ u r l ͔ Β i n t e r n a l _ u r l ͷ ม ߋ ౳

    w Ξ ϓϦ έ ʔ γ ϣ ϯ ଆ ΋ ಉ ͡ Α͏ ͳ ί ʔ υ Λ ྔ ࢈ M o n i t o r i n g ܥ ΍ϝ τ ϦΫε ऩ ू
    w S S L ূ ໌ ॻ ߋ ৽ ͕ 2 0 0 ୆ ෼
    ྫ A C M ͱ E L B ͷ S S L ΞΫη ϥ Ϩ ʔ λʔ Λ ࢖ ͍ ɺ S S L ূ ໌ ॻ Λ i m p o r t ͢Δͩ
    ͚ ʹ ͢Δ ͳͲɺ ࠜ ຊ త ʹ O ( 1 ) ʹ Ͱ͖ ͳ ͍ ͔ ߟ ͑Δ
    20

    View Slide

  21. Τϥʔ
    ͦ ͷ Τ ϥ ʔ ͸ ࢝ · Γ ͷ ࢝ · Γ
    w Τϥ ʔ͕ ෳ ࡶ ͢ ͗Δ
    w ࠶ ݱ ͞ ͤΑ͏ ͱ͢Δ ΋ ύ ϑΥ ʔϚ ϯε ͕ ҧ ͍ ͢ ͗ ͯɼ ผ ͷ ໰ ୊ ͕ ى ͖Δ
    w ʮ ຊ དྷ ղ ܾ ͢΂ ͖ ՝ ୊ ʯ Λ ݟ ۃ Ί Δ ʮ ඞ ཁ ͳ ৘ ใ Λ ୳ ͢ʯ ʮ ඞ ཁ ͳ ৘ ใ Λ
    ୳ ͢ʯ ʮ ࠶ ݱ ͢Δ ؀ ڥ Λ ੔ ͑Δʯ ຊ ࣭ త ͳ ࡞ ۀ · Ͱ ʹ ࣌ ؒ ͕ ͔ ͔ Δ
    w ҟ ৗ ঢ় ଶ ͷ ؅ ཧ
    w Y Y ܥ ɺ Y Y ܥ ͷ ࣦ ഊ Λ ౎ ౓ ઃ ఆ ͢Δ ඞ ཁ ͕͋ Δͨ Ί ࿙ Ε΍ ͢ ͍ɻ
    w Τϥ ʔ Ҏ ֎ ʹ ʮ ٧ ·ͬͯ ͍ Δʯ ঢ় ଶ ΋ ݕ ஌ ͢Δ ͷ ͕ ೉ ͠ ͍
    w ӡ ༻ ճ ආ ΋ ݶ ք ͕͋ Δ
    w ϋ ϯ υϦ ϯά
    w ෳ ਺ ͷ ࣦ ഊ έ ʔε ʹ ࿙ Ε ͳ͘ ϋ ϯ υϦ ϯά͞ ͤΔ ͷ ͕ ೉ ͠ ͍
    21

    View Slide

  22. σόοά
    α ʔ Ϗ ε Λ ލ ͙
    w αʔϏεΛލ͍ͩσόοά͕େม
    w ελοΫτϨʔε͸ͦͷϦϙδτϦ͚ͩͷ৘ใͷΈ
    w ௨৴݁ՌͷσʔλΛॲཧ͢ΔҰ࿈ͷϑϩʔͰΤϥʔ͕ى͖ͨ৔߹
    w ୭͕Ͳ͏͍͏ϦΫΤετΛͯ݁͠Ռ͕Ͳ͏ͩͬͨͷ͔ʁ
    w λΠϛϯάɾΠγϡʔͰ্ख͍͔͘ͳ͍έʔεͰͦͷલޙͷϦΫΤετΛΈ͍ͨ
    w ϚΠΫϩαʔϏεؒͰ n + 1 ͷϦΫΤετ͍ͯ͠Δ͜ͱʹؾ͚ͮͳ͍
    22

    View Slide

  23. Πϯγσϯτ
    ࡂ ֐ ͸ ಥ વ ʹ
    w ͋Δ೔ಥવͷεύΠΫͰɺΠϯϑϥνʔϜʹΞϥʔϜ͕໐Δ
    w Өڹൣғͷಛఆ͢Δ΋Կ͕ى͖͍ͯΔ
    w Өڹൣғಛఆ͢Δ΋ɺͲ͜ΛݟΔ΂͖͔ʁ
    w ݟΔ΂͖৔ॴ͸ɺͦ΋ͦ΋ݟΔ͜ͱ͕Ͱ͖Δ͔ʁ
    w εύΠΫͷݩ͸Ͳ͔͜Βདྷ͍ͯΔʁ
    w Կ͕۩ମతʹى͍ͬͯ͜Δ͔ʁ
    w ݱঢ়ঢ়ଶ͔ΒͲ͏΍ͬͯҰ࣍෮چͤ͞Δ͔ʁ
    w ࠜຊରࡦ͸͍ͭͲ͜Ͱ୭͕ߦ͏͔ʁ
    23

    View Slide

  24. ؅ཧɾऔΓܾΊ
    Ϩ ʔ ϧ Λ Ҿ ͘
    w αʔϏεURLͷ؅ཧ͕ϑϨʔϜϫʔΫͰมΘΓ೉͍͠
    w Rails͸ config/settings/{production,qa,development}.yml URLΛॻ͘ඞཁ
    w URLʹinternal_urlͱpublic_url
    w ൚༻తʹࢀর͞ΕΔΑ͏ͳαʔϏε͸ɺMicroservice෼ඞཁ
    w log ͷϨʔϧ
    w ΠϕϯτϩάΛ௥Ճ͢ΔϨʔϧ͸͋Δ͔ʁ
    w ΞΫηεϩά͸উखʹΑ͠ͳʹऔͬͯ͘Εͳ͍ͷ͔ʁ
    w ϑΥʔϚοτͷܾ·Γ͸ JSON , LSTV
    w ܽଛ཰͸ͦ΋ͦ΋Ͳͷ͘Β͍͋Δͷ͔ʁ
    24

    View Slide

  25. ؅ཧɾऔΓܾΊ
    Ϩ ʔ ϧ Λ Ҿ ͘
    w ΫϥΠΞϯτͱαʔόʔͷৼΔ෣͍ͷϨʔϧ
    w HTTP ΫϥΠΞϯτͷڍಈ͕౷Ұ͞Ε͍ͯͳ͍
    w ίʔυʹڞ௨Խ͞Ε͍ͯͳ͍
    w λΠϜΞ΢τͱϦτϥΠͷॳظઃఆ͸͋Δͷ͔ʁ
    w ϔομʔʹ͸Ͳ͏͍͏৘ใΛೖΕΔͷ͔ʁ
    w αʔϏεؒͷೝূΛͲͷΑ͏ʹ͢Δͷ͔ʁ
    w current userͷ఻ൖ͸Ͳ͏͢Δͷ͔ʁetc
    w ͋ΔΤϯυϙΠϯτ͕ԿΛظ଴͍ͯ͠Δͷ͔͕Θ͔Βͳ͍
    w ύϥϝʔλͷεΩʔϚͷཧ༝͕Ԟ·Ͱಡ·ͳ͍ͱΘ͔Βͳ͍
    25

    View Slide

  26. ґଘؔ܎
    ಺ ෦ D D o s
    w େྔͷϫʔΫϩʔυΛଞαʔϏεʹґཔ
    w Ϣʔβʔ૿ՃͰɺେྔʹϦΫΤετΛૹΔ͜ͱʹͳΔ
    w αʔόʔ͔ΒɺRDS΍ElastiCacheʹৗʹؾΛ࢖͏
    w ద౰ͳස౓ʹsleep͢Δͱ͍͏ঊͷٕ
    w ͦ΋ͦ΋ඇಉظͰྑ͍λεΫ͸ྑ͍ײ͡ʹ΍Εͳ͍ͷ͔
    w ຊདྷͷ໾ׂΛେ͖͘௒͑ͨػೳΛ࣋ͭαʔϏε͕͋ΓɺԿނ͔ඞͣ໰͍߹Θͤͷதʹଘࡏ
    ͢Δ
    26

    View Slide

  27. ڞ௨σʔλ
    Ұ ൪ ෛ ՙ ͕ ू த ͠ ΍ ͢ ͍
    w σʔλͷ఻ൖ
    w ͦΕͧΕͷMicroserviceͰCache͍ͯͨ͠৔߹ʹPurgeΛͲ͏͢Δ͔
    w มߋΛͲ͏఻͑Δ͔
    w ڞ௨ͷσʔλͷϋϯυϦϯά
    w ෳ਺ͷϨΠϠʔͰऔΓ૊Ήඞཁ͕͋Δ
    w ྫϓϩϑΟʔϧʹΧϥϜΛͭ௥Ճͯ͠ಡΈॻ͖͢Δ
    w A αʔϏεʹ schema ௥Ճ -> B αʔϏεͷ Garage Λมߋ -> C αʔϏεͷ garage
    proxy Ϟσϧमਖ਼ -> D αʔϏεʹॻ͖ࠐΈ endpoint
    • Microservice ʹͯ͠ɺϥΠϒϥϦͱͯ͠੾Γग़ͯ͠ɺશ෦࠷৽ʹ্͛Δͱ௥Ճ͞Εͳ͍͔ʁ
    27

    View Slide

  28. ஌ݟڞ༗
    d o c s ͩ ͚ Ͱ ͸ ଍ Γ ͳ ͍ ҉ ໧ ஌
    w Microservicesͷجຊతͳ஌ࣝ
    w ྫDBͷڞ༗ͷ͸Ͱ͖Ε͹ආ͚Δ
    w ΤϯυϙΠϯτɺϦΫΤετͷத਎΍΍ΓͱΓํࣜ
    w ؔ܎ऀͷ஌ࣝʹ͹Β͖ͭ
    w ૊৫શମͰͷAPIͷ࡞ΓํʹϨʔϧ
    w API࣮૷ํ๏
    w I/F ϧʔϧ
    w όϥόϥʹΑΔݺͿଆͷੜ࢈ੑ͕Լ͛ΒΕΔ
    28

    View Slide

  29. ஌ݟڞ༗
    E P D T ͩ ͚ Ͱ ͸ ଍ Γ ͳ ͍
    w ར༻͢ΔOSSͷѻ͍ʹؔ͢ΔϨʔϧ
    w ԿΛͲͷϨϕϧ·Ͱѻ͏ͷ͔
    w ج४͸ͳʹ͔
    w DockerfileͲ͏͍͏෩ʹॻ͍ͨΒ͍͍͔
    w ݴޠʹΑ͔ͬͯΘΔ
    w Alpine Linux Ͱফ໣͢Δͷ͕ͭΒ͍
    w ٕज़Λप஌ͤ͞Δ৔ॴ
    w ͦͷ৔Ͱཧղ͠ͳͯ͘΋ɺ΋͏Ұ౓୳ͤΔ৔ॴ͕ͳ͍
    29

    View Slide

  30. ։ൃ؀ڥ(QA,Staging)
    ੜ ࢈ ੑ ʹ ؔ Θ Δ
    w ֤؀ڥͷσʔλಉظ
    w σʔλͷϦετΞ͢Δॱ൪ͰMicroservices͕յΕΔ
    w ͣͬͱݹ͍··ͷσʔλ
    w ͦͷ··ϦετΞͰ͖ͳ͍σʔλͷѻ͍
    w ڞ༗؀ڥͰҙਤ͠ͳ͍topic branchͰಈ͖ଓ͚Δ
    30

    View Slide

  31. ϧʔςΟϯά
    Α ͠ ͳ ʹ ܨ ͙
    w ϞόΠϧνʔϜͱαʔόʔαΠυνʔϜͰڞಉ։ൃ͢Δࡍ
    w Topic branch Ͱ֬ೝ͠ͳ͕Β։ൃ͍ͨ͠
    w ಛఆͷϧʔςΟϯά͢Δज़͕ͳ͍ͨΊɺ master Ͱৗʹ֬ೝ͍ͯ͠Δςελʔ͞Μͱ࿈ܞ
    ͯ͠ڞ༗؀ڥʹͩͯ֬͠ೝ͢Δ͔͠ͳ͍
    w A service -> B sevice -> C service ͱ͍͏ϚΠΫϩαʔϏε͕ؒ͋ͬͨ࣌
    w B ͷ։ൃ͕೉͗͢͠Δ
    w ϦΫΤετΛ࠶ݱ͢Δ؆୯ͳํ๏͸ͳ͍͔ʁ
    31

    View Slide

  32. ϧʔςΟϯά
    ω ο τ ϫ ʔ Ϋ ى Ҽ ͷ ϋ ϯ υ Ϧ ϯ ά ͸ Ͳ ͜ Ͱ ΍ Δ ʁ
    w ΞʔΩςΫνϟͷมߋ΍αʔϏε෼ׂ͢Δࡍ
    w Ͳͷ͘Β͍ΠϯύΫτ͕͋Δʁ
    w Θ͔Βͣාͯ͘ग़͠ʹ͍͘ɺΧφϦʔσϓϩΠͯ͠ཉ͍͠ͱ͖͕͋Δ
    w A/B ؀ڥͰνΣοΫ͍ͨ͠
    w inͱout͕ಉ͡Ͱ͋Δ͜ͱΛอূΛνΣοΫ͢Δ࢓૊Έ͕΄͍͠
    w timeout ͱ retry Λશ෦ͷΞϓϦέʔγϣϯʹॻ͍͍ͯ͘ͷ͸͍͔ͭഁ୼͢Δ
    w αʔϏεΛ෼ׂ͢Δࡍʹ path ϕʔεͰ෼͚ΔͷΛ nginx.conf Ͱ΍Δͷ͸͍͔ͭݶք
    w API Gatewayʹ೚͍͖͍ͤͯͨ
    32

    View Slide

  33. ՝୊͔Β໨తͱखஈΛߟ͑Δ
    ม Խ ʹ ڧ ͍ Π ϯ ϑ ϥ ͷ ࣍ ͷ ε ς ο ϓ ΁
    SREνʔϜʹվ໊ʁ
    ໾ׂΛڱΊΔ͜ͱʹͳΒͳ
    ͍Α͏ݱࡏͷ໊લ͸ҡ࣋ɻ
    SREͷྖҬʹରͯ͠ɺίϛο
    τͰ͖ΔΑ͏αʔόʔαΠυ
    ͷΤϯδχΞΛܴ͑ͯۀ຿ͷ
    தʹSREۀΛ૊ΈࠐΉɻࣄ
    ۀΛ௒͑ͯΠϯϑϥج൫ɺ࢓
    ૊ΈɺΞϓϦͷ͍ΖΜͳํ޲
    ͔Β՝୊ղܾͰ͖ΔΑ͏ʹɻ
    ՝ ୊ ղ ܾ ͸ ɺ πʔϧ ࡞ ੒
    ࠔ ͬͯ ͍ Δ ͜ͱ ͸ ৗ ʹ ҙ ݟ
    ͱͯ͠ د ͤΒ ΕΔɻ ͦ ͷ த
    Ͱɺ ༏ ઌ ॱ Ґ ͕ Ұ ൪ ߴ ͍ ͷ
    Λ ൑ அ ͢Δ ͷ ͸ ೉ ͠ ͍ɻ ੠
    ͷ Ͱ ͔ ͍ ਓ ɺ ࢹ ࠲ ͷ ҧ ͍ ౳
    Ͱ ॱ Ґ ͕ ٯ స ͢Δ ͜ͱ ΋ ͋
    Δɻ ࢥ ͍ ੾ ͬͯ ·ͱ Ίͯ ੔
    ཧ ͯ͠ ΋ Β ͑ ͨ ͜ͱͰ ՝ ୊
    ղ ܾ ͷ ࢹ ք ͕ ޿ ͕ ͬͨɻ
    มԽʹڧ͍ΠϯϑϥΛ࡞Δ͜
    ͱͱɺٻΊΒΕΔ࣍ͷίΞٕ
    ज़ͷ૑ग़͸Ͳͷ͘Β͍ؔ࿈͋
    Δʁ৔͚ͩ༻ҙͯ͠ʮ͋ͱ͸
    ΑΖ͘͠ʯͰίΞٕज़͸೉͠
    ͍ɻऔΓ૊΋͏ͱ͍ͯ͠Δ͜
    ͱɺτϨϯυɺ࿦จɺւ֎ͷ
    ϒϩάͳͲͷهࣄΛҰॹʹݟ
    ͯɺऔΓ૊ΉྖҬʹϑΥʔΧ
    εͰ͖Δج൫Λ੔͑Δɻ
    33

    View Slide

  34. ΠϯϑϥνʔϜ͕໨ࢦ͢ΰʔϧͷมԽ
    มԽʹڧ͍Πϯϑϥ࡞Γ͔Β
    ม Խ ʹ ڧ ͍ Π ϯϑ ϥ ͔ Β ɺ γ ΰ τͰί ί ϩΦ υϧ Λ ૿ ΍ ͢ ج ൫
    Τϯ δχΞͷ ࢓ ࣄ ྔ Λ ૿ ΍ ͢͜ͱ ʹ γ ϑ τ
    αΠτͷ৴པੑ
    S i t e R e l i a b i l i t y
    ΤϯδχΞશମͷੜ࢈ੑ
    D e v e l o p e r P r o d u c t i v i t y
    ίΞٕज़ͱͳΔج൫
    ٕ ज़ ઓ ུ
    34

    View Slide

  35. OKRʹूதͰ͖ΔνʔϜ
    Ұਓ͋ͨΓͷΤϯδχΞͷ࢓ࣄྔΛ૿΍͢
    • OKR(Objectives and Key Results)
    w Oʢ໨ඪʣ
    w KRʢ੒Ռࢦඪʣ
    w Ұਓ͋ͨΓ͕͜ͳͤΔ࢓ࣄྔͷ֦େ
    w Ұਓ͋ͨΓͷػցֶशandࢪࡦͷPDCAͷαΠΫϧ͕ଟ͘ճͤΔ
    w தԝूݖͱ෼ׂ౷࣏ͷׂ߹͕ద੾ͰҰਓͻͱΓ͕νϟϨϯδͰ͖Δ
    w Ұਓ͋ͨΓͷ࢓ࣄྔ͕ݮΔ΋ͷΛ࡟ݮ
    w ໰୊ͷՕॴΛ͙͢ʹಛఆͰ͖Δ
    w ஌Γ͍ͨ৘ใ͕͙͢ʹΘ͔Δ
    w ౷Ұ͢Δ΂͖ͱ͜Ζ͕౷Ұ͞Ε͍ͯΔ
    35

    View Slide

  36. X
    X
    X
    36
    Τ ϯ δ χ Ξ ͔ Β ݟ ͨ ັ ྗ
    ௅ઓͰ͖Δ͜ͱ
    ࣾձతΠϯύΫτ͕͋Δ
    ͍͢͝ਓͨͪͱҰॹʹಇ͚Δ

    View Slide

  37. Ͳ͏ղܾ͍͔ͯ͘͠
    ໨ ඪ ͱ ՝ ୊ ͔ Β
    ώ Ξ Ϧ ϯά΍ ·ͱ Ίͯ ΋ Βͬͨ΋ ͷ Λ ࢀ ߟ ʹ ϓϥ ΠΦϦ ςΟύ εΛ ܾ Ί Δɻ ͦ ͷ த Ͱɺ ಉ
    ࣌ ʹ ಈ ͔ ͤΔ ͷ ͸ ɺ ݱ ν ʔϜͰ ͸ 3 ͭ ͕ ݶ ք ɺ ͦΕͧΕͰ Ұ ൪ ߴ ͍ ΋ ͷ ʹ ू த ͢Δ
    αΠτͷ৴པੑʹͭͳ͕
    ΔऔΓ૊Έ
    S i t e R e l i a b i l i t y
    ΤϯδχΞ͕ࠔ͍ͬͯΔ
    ͜ͱͷղܾ
    D e v e l o p e r P r o d u c t i v i t y
    ະདྷʹඞཁͱͳΔίΞٕ
    ज़ͱͳΔج൫ͮ͘Γ
    ٕ ज़ ઓ ུ
    37

    View Slide

  38. Site Reliability
    α Π τ ͷ ৴ པ ੑ ʹ ͭ ͳ ͕ Δ औ Γ ૊ Έ
    U p t i m e
    ί Ξό Ϧϡʔ ࢮ क
    S c a l a b i l i t y
    Ku b e r n e t e s Ҡ ߦ
    p o s t m o r t e m s
    ਖ਼͘͠໰ ୊Λཧ ղɺ
    ֶ ͼΛ ஝ ੵ ͠ɺ ࠶
    ൃ ๷ ࢭ ͱ ࠜ ຊ ର Ԡ
    D o c s a n d Ta l k
    ؔ ৺ ࣄ ͱ ҉ ໧ ஌ ͷ
    ڞ ༗
    S c a l a b i l i t y ͱ p o s t m o r t e m s
    38

    View Slide

  39. ϙετϞʔςϜ
    ௕ ظ త ੒ ௕ Λ ಷ Խ ͞ ͤ ͳ ͍
    bϙετϞʔςϜ͸ɺΠϯγσϯτͱͦͷΠϯύΫτɺͦͷ؇࿨΍ղফͷͨΊʹߦΘΕͨΞΫ
    γϣϯɺࠜຊݪҼʢ܈ʣɺΠϯγσϯτͷ࠶ൃΛආ͚ΔͨΊͷϑΥϩʔΞοϓͷΞΫγϣϯΛ
    ه࿥͢ΔͨΊʹॻ͔ΕΔ΋ͷͰ͢ɻ`
    ൈਮBetsy BeyerlSREαΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯάz
    w Πϯγσϯτ͔ΒֶͼΛಘΔͨΊͷఆࣜԽ͞Εͨϓϩηε͕ͳ͚Ε͹ɺಉ͡Α͏ͳΠϯγσ
    ϯτ͕ແݶʹ܁Γฦ͠ى͜Δ
    w ໺์͠ͷ··ʹͳͬͯ͠·͑͹ɺΠϯγσϯτͷෳࡶ͞͸Ճ଎౓తʹ૿͠ɺ͋Δ͍͸ੵΈ
    ॏͳͬͯγεςϜͷରॲ͕Ͱ͖ͳ͍
    w ࣦഊ͔ΒֶͿͨΊʹৼΓฦΓͷϙετϞʔςϜΛॻ͘
    39

    View Slide

  40. Developer Productivity
    ΤϯδχΞ͕ࠔ͍ͬͯΔ͜ͱͷղܾࡦ
    To o l
    શ Τϯ δχΞ͕ ۀ
    ຿ Λ ޮ ཰ త ʹ ߦ ͏
    F l u e n t d
    ܾ ·ͬͨ ϩ ά ܗ ࣜ
    Λ ૹ Δ
    D i s t r i b u t e d Tr a c i n g
    ӈ ͷ ਤ ɺ ϚΠΫ ϩ
    αʔϏε ؒ ͷτϨʔ
    εΛ ࣮ ݱ ͢Δ
    Service mesh
    ϚΠΫ ϩ α ʔ Ϗε
    ͷ ωο τϫ ʔ Ϋ ؒ
    Λ ੔ ͑Δ
    O p e n C e n s u s , S e r v i c e M e s h , F l u e n t d e t c
    40

    View Slide

  41. OpenTracing? OpenCensus?
    Distributed tracing ͷΫϥΠΞϯτϥΠϒϥϦ
    OpenTracing
    ෼ࢄτϨʔγϯάΛ࣮ݱ͢ΔͨΊͷڞ௨
    ͷϓϩτίϧͱAPI͕ఆٛ
    ֤ࣾͰ࣮૷ɻZipkin͸Twitterࣾɻ
    طʹ࢖ΘΕ͍ͯΔͱ͜Ζ͕ଟ͘ɺ
    OpenCensus͸ޙൃͰɺOpenTracingͷ
    ΄͏͕ྺ࢙͕͋Δɻͨͩɺ࢓্༷ࣗ෼ͨ
    ͪͰ࣮૷͍ͯ͘͠ඞཁ͕͋Δɻ
    OpenCensus
    metrics΍tracesΛऩूɾ഑৴͢ΔͨΊ
    ͷ֦ுੑͷ͋Δ౷߹తͳΠϯλʔϑΣʔ
    εΛఏڙ͢ΔϑϨʔϜϫʔΫ
    Googleͷ census library ͷ OSS ൛
    ݴޠͱϕϯμʔͷࣗ༝͞Λҡ͍࣋ͯ͠Δ
    Go, Ruby, Python, Java, C++, Erlang,
    PHP
    StackDriver, Datadog APM,
    AWS X-RAY
    41

    View Slide

  42. ٕज़ઓུ
    ະདྷʹඞཁʹͳΔίΞٕज़ͱͳΔج൫ͮ͘Γ
    ܧ ଓ త ͳ ධ Ձ
    Φϑ ϥ Π ϯ ධ Ձ
    D a t a S t o r e
    σʔ λ ͷ ࠷ ద Խ
    Development
    A m a z o n
    S a g e M a ke r
    C I / C D
    ϓϩ μΫ τ· Ͱ ͷ
    σϦό Ϧ
    Ͱ͖Δ ٕ ज़ ྖ Ҭ Λ ૿ ΍ ͢
    42

    View Slide

  43. ٕज़ઓུ
    keyword
    w ࠓޙΛ୲͏ίΞٕज़ΛϓϩμΫτνʔϜ΍$50ɺϦʔυΤϯδχΞͱҰॹʹઓུΛཱͯͯɺ
    ج൫ͱ࢓૊Έͱ࣮૷
    w ίϛϡχέʔγϣϯ
    w ྠߨձ
    w ւ֎ࣄྫ
    • Tech blog
    w ώΞϦϯάௐࠪ
    w ૊৫શମ͔ΒݟͯӨڹྗ͕େ͖͍෼໺Ͱ΋ଞνʔϜͰऔΓ૊Έʹ͍͘΋ͷ
    w σʔλετΞͷMicroservices
    w ։ൃͰ͖Δ΋ͷͷϨϕϧΛ্͛Δ
    w ͍͘Βߴ଎ʹͰ͖ͯ΋ɺ։ൃͰ͖Δ΋ͷʹݶΓ͕͋Ε͹ͦΕҎ্ʹͳΔ͜ͱ͸ͳ͍
    43

    View Slide

  44. Summary
    Site ReliabilityɾDeveloper Productivityɾٕज़ઓུͷऔΓ૊Έ
    44
    ೥͔ΒมԽʹڧ͍ΠϯϑϥΛ໨ࢦ͖ͯͨ͜͠ͱʹΑΔޮՌͱ՝୊
    ม Խ ʹ ڧ ͍ Π ϯϑ ϥ ࡞ Γ ͷ ݁ Ռ
    ຊདྷΠϯϑϥνʔϜͷՌͨ͢΂͖՝୊ղܾ͸ͳʹ͔ʁ
    ໨ ࢦ ͢ ΰʔϧ ͷ ม Խ
    มԽʹڧ͍Πϯϑϥ͸ڧྗͳ෢ث͕ͩͬͨಉ࣌ʹ೉͍͠՝୊΋ੜΜͩ
    ϚΠΫ ϩ α ʔ Ϗε ͷ ՝ ୊
    Ͳ͏ ղ ܾ ͯ͠ ͍ ͘ ͔
    ͍·ݱ࣌఺ͰऔΓ૊ΜͰ͍Δ΋ͷ΍ղܾ͍ͯ͠Δ΋ͷͳʹ͔

    View Slide

  45. X
    X
    X
    45
    Τ ϯ δ χ Ξ ͔ Β ݟ ͨ ັ ྗ
    ௅ઓͰ͖Δ͜ͱ
    ࣾձతΠϯύΫτ͕͋Δ
    ͍͢͝ਓͨͪͱҰॹʹಇ͚Δ

    View Slide

  46. X
    X
    X
    46
    W e a r e H i r i n g !
    https://www.wantedly.com/companies/wantedly

    View Slide