Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Envoy externals and ideas (builderscon 2018)

Yosuke Tomita
September 06, 2018

Envoy externals and ideas (builderscon 2018)

builderscon 2018での発表資料になります。
資料からのリンクは以下にまとまっています。
https://qiita.com/seikoudoku2000/items/62aabc1dc28062d195ad

Yosuke Tomita

September 06, 2018
Tweet

Other Decks in Technology

Transcript

  1. Envoy externals and ideas
    builderscon 2018
    ෌ాཅհ

    View full-size slide

  2. © ChatWork
    ࣗݾ঺հ
    ▸ ෌ాཅհ (@seikoudoku2000)
    ▸ “Dev”͔Β”Ops” ΁ (྆ํͷؾ͕࣋ͪ෼͔Δͭ΋Γ)
    ▸ Navitime (ϩάղੳνʔϜ / ࣾ಺ڞ௨APIͷ։ൃ)
    ▸ Gengo (όοΫΤϯυ։ൃ / AWS, chef, Postgresql)
    ▸ ChatWork (ΠϯϑϥϚωδϝϯτ෦ → SRE෦)
    ▸ ࠷ۙͷςʔϚɿ૊৫ͰγεςϜΛ”͍͍ײ͡”ʹ։ൃɾӡ༻͍ͨ͠

    View full-size slide

  3. © ChatWork
    EnvoyͷଘࡏΛ஌Δ

    View full-size slide

  4. © ChatWork
    ౓؊Λൈ͔Εͨ
    ▸ Microservices ͷ՝୊ʹιϑτ΢ΣΞΛॻ͍ͯਅͬ޲͔Βཱͪ޲͔͏
    ▸ ୯ͳΔPoCͰ͸ͳ͘ɺ͕ͬͭΓಋೖ͢Δॴ·Ͱ΍Γ੾Δ
    ▸ > 100 service / > 10,000 hosts / All service to service traffic
    ▸ ൃ૝͕ҧ͍͗͢Δ / ΋ͷੌ͍Ϩϕϧࠩ…
    ▸ e.g. ELB/ALB ࢖ΘͣɺόϥϯαʔΛࣗલͰ࣮૷ʂʁ
    ▸ ϨϕϧࠩΛײ͡ΕΔΑ͏ʹͳͬͨ…? / ϛυϧ΢ΣΞ։ൃ΁ͷڵຯ

    View full-size slide

  5. © ChatWork
    ࠓ೔ͷ໨ඪ
    ɾEnvoy ੌ͍ʂ͍ͬͯ͏ײಈͷڞ༗
    ɾDev / OpsΛ໰Θͣʹɺࣗ෼ͷ૊৫΍
    γεςϜͷڥքઢΛߟ͑Δ͖͔͚ͬʹ

    View full-size slide

  6. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍
    ▸ Envoyͱ͍͏”ΞΠσΞ”
    ▸ ChatWorkͷະདྷ༧૝ਤ
    ▸ Microservices ͷπϥϛ ͦͷ̎
    ▸ Kubernetesͱ͍͏”ΞΠσΞ”
    ▸ ·ͱΊ

    View full-size slide

  7. © ChatWork
    ࠓ೔࿩͞(ͤ)ͳ͍͜ͱ
    ▸ Envoy deep dive / ࡉ͔ͳ༻ޠఆٛ
    ▸ Matt͞ΜͷൃදΛฉ͜͏
    ▸ ChatWorkʹ͓͚Δಋೖࣄྫ
    ▸ It’s a loooong way…
    ▸ ຊ৔LyftͷൃදɺcookpadͷൃදΛฉ͜͏

    View full-size slide

  8. © ChatWork
    ૝ఆλʔήοτ
    ▸ ΤϯδχΞશൠʂ

    View full-size slide

  9. © ChatWork
    “͍͍ײ͡” ʹ։ൃɾӡ༻͢Δͱ͸ʁ

    View full-size slide

  10. © ChatWork
    “͍͍ײ͡”ʁ
    ▸ ੹೚ͱݖݶ͕ద੾ʹ෼ׂ͞Ε͍ͯΔ / ద੾ͳڥքઢ͕͋Δ
    ▸ ࣗ෼ͷέπΛࣗ෼Ͱ࣋ͬͯνϟϨϯδͰ͖Δ / εϐʔυײ
    ▸ ਓͷέπΛ࣋ͭͷ΋ҧ͏ / ݖҖతʹͳΒͳ͍(ͳΓͨ͘ͳ͍)
    ▸ ໎ͬͨΒ݈શͳํ by mirakui͞Μ
    ▸ devops, 2 pizza rule, microservices

    View full-size slide

  11. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ
    ▸ Envoyͱ͍͏”ΞΠσΞ”
    ▸ ChatWorkͷະདྷ༧૝ਤ
    ▸ Microservices ͷπϥϛ / ϜζΧγϛ ͦͷ̎
    ▸ Kubernetesͱ͍͏”ΞΠσΞ”
    ▸ ·ͱΊ

    View full-size slide

  12. ▸ $MPVE5SFOETCZ"ESJBO$PDLDSPGU
    ▸ l*U`TTUJMMBQSPCMFNCVUBOFBTJFS
    QSPCMFNUPTPMWFz
    ▸ &BTJFSQSPCMFNʹͳΒͳ͍ͳΒߟ͑ͳͯ͘
    ྑ͍ ʁ

    ▸ αʔϏε͕ͲΜͲΜେ͖͘ͳ͍ͬͯ͘ͳΒɺ
    ͍ͣΕඞཁʹͳΔɺͱ͍͏લఏͰ࿩ΛਐΊ
    ͍͖ͯ·͢
    © ChatWork
    Microservicesඞཁʁ

    View full-size slide

  13. © ChatWork
    Microservicesͷಛ௃
    ▸ Fault Tolerant / Resilient
    ▸ Ͳ͔͜མͪͯ΋શ෦͸མͪͳ͍Αʂ
    ▸ Կ͔͋ͬͯ΋͍͍ײ͡ʹճ෮͍ͯ͘͠Αʂ
    ▸ Autonomy
    ▸ ֤αʔϏε͸খ͍͞ͷͰࣗ཯తʹͲΜͲΜಈ͚ΔΑʂ

    View full-size slide

  14. © ChatWork
    Microservices

    View full-size slide

  15. © ChatWork
    Fault Tolerant
    Ͳ͔͜ͷαʔϏε͕ෆௐʹͳͬͯ
    ΋ɺӨڹൣғ͸ִ཭͞Ε͍ͯΔ

    View full-size slide

  16. © ChatWork
    Resilient
    ͕࣌ؒܦͭͱ෮چ (ԿΒ͔ͷ࢓૊Έɺ
    ࡞ΓࠐΈ͸ඞཁ)

    View full-size slide

  17. © ChatWork
    Autonomy
    ֤αʔϏε͸ಠཱ͓ͯ͠Γɺ
    αʔϏεؒͷInterface͚ͩఆٛ͠ɺ
    ͦͷதͰࣗ཯తʹࣗ༝ʹਐԽ

    View full-size slide

  18. © ChatWork
    ͱݴΘΕ͍ͯΔ͕ɺɺɺ

    View full-size slide

  19. © ChatWork
    MicroservicesͬΆ͍΍ͭ
    ▸ Vulnerable (Distributed SPOF by Sam Newman)
    ▸ Ͳ͔͜མͪΔͱશ෦མͪΔʂ
    ▸ ऑ఺͋ͬͨΒΨϯΨϯ߈ΊࠐΜͰඃ֐֦େʂ
    ▸ Unhealthy dependency
    ▸ “ͳ͔ͥ”ܨ͕Βͳ͍
    ▸ Կ͕ى͖ͯΔ͔Α͘෼͔Βͣʹٙ৺҉َʹɻɻ

    View full-size slide

  20. © ChatWork
    Vulnerable

    View full-size slide

  21. © ChatWork
    Vulnerable

    View full-size slide

  22. © ChatWork
    Vulnerable

    View full-size slide

  23. © ChatWork
    (MicroservicesʹݶΒͳ͍)

    View full-size slide

  24. © ChatWork
    ґଘαʔϏεͷSLA͕શ෦99.9%ͱ͢Δͱɺ
    (0.999)^6 ≒ 99.4% → 1೔8෼ɺ݄ʹ4࣌ؒ20෼ͷμ΢ϯ
    Microservices શମͱͯ͠͸αʔϏε਺ͷྦྷ৐ͱͳΔ

    View full-size slide

  25. © ChatWork
    Unhealthy dependency
    ͜͜ͷ΍ΓऔΓͰΤϥʔ͕ग़ͨɻ

    View full-size slide

  26. © ChatWork
    Unhealthy dependency
    Կ͔Α͘෼͔Γ
    ·ͤΜ͕ɺϨεϙϯεฦͬͯ
    ͜ͳͯ͘ΤϥʔʹͳͬͨΜͰௐ
    ΂͍ͯͩ͘͞ʙ
    Τϥʔग़ͯͳ͍͠ɺΦϨΒ
    ͷ͍ͤ͡Όͳ͍Αɻ“Πϯϑϥ”
    ͷ͍ͤͰ͠ΐ

    View full-size slide

  27. © ChatWork
    Unhealthy dependency
    ▸ શମΛ೺ѲͰ͖ͳ͍ / ϒϥοΫϘοΫεͷ෦෼͕͋Δ
    ▸ શମ͕ݟ౉ͤΔΑ͏ͳϩάج൫/μογϡϘʔυ͕͋Δͱ͍͍͕ɺɺ
    ▸ ͓ޓ͍ͷڥքͰकΔ΂͖ϧʔϧ͕ᐆດ
    ▸ શͯΛ਺஋ԽͰ͖ͳ͍ͱఆٛͰ͖ͳ͍
    ▸ 1ݸͷΤϥʔͰ΋ௐ΂Δʁ 5ݸʁ10ݸʁ

    View full-size slide

  28. ▸ #VJMEJOH.JDSPTFSWJDFT
    ▸ 1SPEVDUJPO3FBEZ.JDSPTFSWJDFT
    ▸ #VJMEJOH'BVMU5PMFSBOU.JDSPTFSWJDFT
    ▸ RJJUB·ͱΊ CZࢲ

    © ChatWork
    Microservices࣮૷/ӡ༻ϊ΢ϋ΢

    View full-size slide

  29. © ChatWork
    Microservices࣮૷/ӡ༻keyword
    ▸ timeout
    ▸ ઃఆͳ͠/௕͗ͩ͢ͱɺ͙͢ʹThread pool৯͍ͭͿ͢ͷͰઃఆ͢Δ
    ▸ (࠷ద஋͸৬ਓͷצ…ʁ)
    ▸ circuit breakers
    ▸ ͓͔͘͠ͳͬͨαʔϏε΁ͷΞΫηε͸Ұ୴ࢭΊΔ
    ▸ ෮چޙɺաෛՙঢ়ଶʹ͍ؕΔ͜ͱͰͷඃ֐ͷ֦େΛ๷͙

    View full-size slide

  30. © ChatWork
    Microservices࣮૷/ӡ༻keyword
    ▸ bulkheads
    ▸ Thread poolΛݺͼग़͠αʔϏε͝ͱʹ෼͚Δ
    ▸ ̍ͭͷαʔϏεݺͼग़͠ͰThread poolΛ࢖͍੾ΔͷΛ๷͙
    ▸ monitoring / obserbavility
    ▸ Կ͕ى͖ͯΔ͔ͪΌΜͱ෼͔ΔΑ͏ʹ͢Δ

    View full-size slide

  31. © ChatWork
    ͦΜͳ೉͘͠ͳ͘ͳ͍ʁ

    View full-size slide

  32. © ChatWork
    ৭Μͳݴޠ / ϑϨʔϜϫʔΫ
    ֤αʔϏε͸ࣗ཯తͳͷͰɺ޷͖ͳݴޠ/
    ޷͖ͳϑϨʔϜϫʔΫΛબ୒͍ͨ͠

    View full-size slide

  33. © ChatWork
    ৭ΜͳϥΠϒϥϦ

    View full-size slide

  34. ୭͕ରԠ/୲อ͢Δʁ
    ֤αʔϏεͷDev͕ػೳ։ൃ
    ͷεϐʔυΛམͱͯ͠ɺ“໓ଟʹى͖ͳ͍”
    ͜ͱʹରͯ͠උ͑Δʁ
    Ops͕શݴޠɺશϥΠϒϥϦͷରԠΛ
    ߦ͍ɺϝϯς͠ଓ͚Δɺɺɺʁ

    View full-size slide

  35. © ChatWork
    Observability

    View full-size slide

  36. © ChatWork
    Ұ؀ͨ͠τϨʔγϯά͕ࠔ೉

    View full-size slide

  37. © ChatWork
    Microservicesͷπϥϛ
    ▸ ֤ϊ΢ϋ΢ͷ1ͭ1ͭ͸೉͍֓͠೦Ͱ͸ͳ͍͕ɺશͯͷ࣮૷ݴޠ / ϥΠϒϥ
    ϦͰશͯͷ௨৴ʹڧ੍͠ɺߋ৽͠ଓ͚Δͷ͸૬౰ʹେม
    ▸ ো֐͕ى͖ͯॳΊͯؾͮ͘͜ͱ͕ଟ͍
    ▸ ڥքઢͷ໰୊
    ▸ શମͱͯ͠ͷ҆ఆੑ (Ops) ͕ɺ֤αʔϏεͷ࣮૷(Dev)ʹڧ͘ґଘ
    ▸ Dev͕ػೳ։ൃͷεϐʔυΛ஗Ίͯ·Ͱίετ͔͚ͯରԠʁ
    ▸ OpsͰશαʔϏε΍ΕͬͯݴΘΕͯ΋ݫ͍͠

    View full-size slide

  38. © ChatWork
    Microservicesͷπϥϛ
    ▸ ௨৴ܦ࿏͕૿͑Ε͹૿͑Δ΄Ͳɺো֐ൃੜ࣌ͷσόοά͕೉͘͠ͳΔ
    ▸ Load Balancer΍৭ʑͳϛυϧ΢ΣΞΛ௨ա͍ͯ͘͠
    ▸ IaaS͕ఏڙ͢ΔBalancer(e.g. ALB / ELB) ʹؔͯ͠͸ɺݶΒΕͨϩά͠
    ͔ݟΕͳ͍

    View full-size slide

  39. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ (ࡁ)
    ▸ Envoyͱ͍͏”ΞΠσΞ”
    ▸ ChatWorkͷະདྷ༧૝ਤ
    ▸ Microservices ͷπϥϛ ͦͷ̎
    ▸ Kubernetesͱ͍͏”ΞΠσΞ”
    ▸ ·ͱΊ

    View full-size slide

  40. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ
    ▸ Envoyͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔEnvoyͷΞϓϩʔν
    ▸ ChatWorkͷະདྷ༧૝ਤ
    ▸ Microservices ͷπϥϛ ͦͷ̎
    ▸ Kubernetesͱ͍͏”ΞΠσΞ”
    ▸ ·ͱΊ

    View full-size slide

  41. © ChatWork
    rٶຊໜ
    “ΞΠσΞͱ͍͏ͷ͸ෳ਺ͷ໰୊ΛҰؾʹղܾ͢Δ΋ͷͰ͋Δ”

    View full-size slide

  42. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ
    ▸ Envoyͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔEnvoyͷΞϓϩʔν
    ▸ ChatWorkͷະདྷ༧૝ਤ
    ▸ Microservices ͷπϥϛ / ϜζΧγϛ ͦͷ̎
    ▸ Kubernetesͱ͍͏”ΞΠσΞ”
    ▸ ·ͱΊ

    View full-size slide

  43. © ChatWork
    Envoy
    The network should be transparent to applications.
    When network and application problems do occur ,
    it should be easy to determine the source of the problem.

    View full-size slide

  44. © ChatWork
    Microservices

    View full-size slide

  45. © ChatWork
    1ͭͷαʔϏε಺Ͱ΋

    View full-size slide

  46. © ChatWork
    Dev͸ػೳ։ൃʹूத
    microservicesͷ҆ఆੑʹؔΘΔ
    ͱ͜Ζ͸Ops͕EnvoyͰҰݩ؅ཧ
    Envoy͕service୯ମͱmicroservicesͷ
    ҆ఆӡ༻ͷ”ڥքઢ”ͱͳΔ

    View full-size slide

  47. © ChatWork
    Envoyͷkeyword
    ▸ Service Mesh
    ▸ Sidecar ύλʔϯ
    ▸ Control plane ͱ Data plane
    ▸ Service mesh data plane vs. control plane
    ▸ ݴ༿͕ࠞಉ͞Ε͍ͯΔهࣄ౳΋ଟ͍

    View full-size slide

  48. © ChatWork
    Service Mesh
    ▸ Envoy ͷΑ͏ͳ࢓૊ΈΛந৅Խͨ͠ݴ༿
    ▸ ۩ମྫ͔Βೖͬͨํ͕෼͔Γ΍͍͢ (ͱࢥ͏)

    View full-size slide

  49. ▸ ڞ௨͢ΔपลλεΫΛಠཱͨ͠ίϯϙʔωϯτ
    ʹू໿͠ɺΞϓϦέʔγϣϯͱซઃͯ͠ಈ͔͢
    ▸ Α͋͘ΔྫɿSFWFSTFQSPYZͱͯ͠ͷ/(*/9
    ▸ ΞϓϦέʔγϣϯ 1)1 QZUIPOFUD
    Λ·
    ͍ͨͰ࢖͑ΔQSPYZػೳΛ/HJOYʹ࣋ͨͤΔ
    ▸ &OWPZ.JDSPTFSWJDFTӡ༻ʹඞཁͳػೳΛू
    ΊͨQSPYZ
    © ChatWork
    Sidecarύλʔϯ
    Ұॹʹಈ͘ΞϓϦέʔγϣϯͷ͜ͱ͸
    ؾʹ͠ͳͯ͘ྑ͍

    View full-size slide

  50. © ChatWork
    Data Plane
    ▸ Sidecar proxy = data planeͦͷ΋ͷ
    ▸ Envoy, NGINX, HAProxyͳͲ (LinkerdͰ͸Controle Plane͕౷߹͞Εͨ)
    ▸ microservices ͰඞཁͳػೳΛఏڙ
    ▸ service discovery
    ▸ health checking
    ▸ routing (timeout, circuit breaking)
    ▸ distributed tracing

    View full-size slide

  51. © ChatWork
    Control Plane
    ▸ εςʔτϨεͳdata planeͷू·ΓΛ෼ࢄγεςϜ΁ͱมԽͤ͞Δଘࡏ
    ▸ ಈతͳઃఆ৘ใͷఏڙ (e.g. service discovery)
    ▸ Envoyͷ৔߹͸ɺxDS ͱݺ͹ΕΔAPI͕ఆٛ͞Ε͍ͯΔ (ޙड़)
    ▸ ਓ͕ؒͦͷ໾ׂΛ͢Δ৔߹΋͋Δ (ਓྗ Controle Plane)
    ▸ Istio, Nelson, Envoy Manager (Lyft಺੡), cookpad΍booking.com΋ࣗ࡞
    ▸ (Linkerd͸Control planeΛ಺แ)

    View full-size slide

  52. © ChatWork
    Envoyͷઃఆ֓ཁ
    ▸ ੩తͳconfigurationͱɺAPIʹΑΔಈతͳconfiguration
    ▸ API͸xDS (LDS, RDS, CDS, EDS) ͱݺ͹Ε͍ͯΔ
    ▸ v1(deprecated) ͔Β v2΁ (ωοτ্ͷ৘ใ͕ࠞࡏɻɻ)
    ▸ Protcol BufferͰఆٛ͞Εͯͯ໌֬ͩ͠ɺAPIυΩϡϝϯτ΋ॆ࣮͍ͯ͠
    Δ͕ɺશମͷ֓೦Λઆ໌ͨ͠υΩϡϝϯτݟ͚ͭΒΕͣɻɻ (͜ΕΛ·ͱΊΔ
    ͱਓؾऀʹͳΕΔ͔΋͠Εͳ͍)

    View full-size slide

  53. © ChatWork
    EnvoyͷػೳΛԿݸ͔
    ▸ Timeout / Retry
    ▸ Circuit Breaking
    ▸ Fault Injection
    ▸ Observability / Distributed Tracing

    View full-size slide

  54. © ChatWork
    Timeout / Retry
    clusters:
    - name: service2
    connect_timeout: 1s
    "routes": [
    {
    "cluster": "service2",
    "prefix": "/",
    "timeout_ms": 3000
    "retry_policy" {
    "retry_on": 5xx,
    "num_retries": 3
    "per_try_timeout_ms": 300,
    }
    },

    View full-size slide

  55. © ChatWork
    Circuit breaking
    clusters:
    - name: service2
    connect_timeout: 1s
    circuit_breakers:
    thresholds:
    max_connections: 100
    max_pending_requests: 100
    ᮢ஋Ͱͷopen/closeͷΈɻhalf-openͳ
    ঢ়ଶ͸ଘࡏ͠ͳ͍

    View full-size slide

  56. © ChatWork
    Fault Injection
    clusters:
    - name: service2
    connect_timeout: 1s
    http_filters:
    - name: envoy.fault
    config:
    delay: {fixed_delay: 20s, percent: 50}
    upstream_cluster: service2
    abort: {percent: 20, http_status: 503}
    upstream_cluster: service2
    ଱ো֐ςετɺো֐࠶ݱࢼݧ͕
    ༰қʹߦ͑Δʂ

    View full-size slide

  57. © ChatWork
    Observability / Distributed tracing
    Envoy͕distributed tracingͷγ
    εςϜͷ΍ΓऔΓΛ΍ͬͯ͘ΕΔ
    ௥੻༻ͷheaderΛservice಺Ͱઃఆ

    View full-size slide

  58. © ChatWork
    Observability / Distributed tracing
    ※ NewRelicͱdatadog ΋૬͍࣍Ͱ֘౰ͷػೳΛϦϦʔεɻ
    ετϨʔδ؅ཧɺAPM΍طଘͷmonitoring/alertΛߟ͑Δͱɺ
    ɹ͕ͬͪ͜༗ྗ͔΋ʁ
    Lyft, cookpad ͷࣄྫΛࢀর͍ͯͩ͘͠͞

    View full-size slide

  59. © ChatWork
    Envoy͕΋ͨΒ͢΋ͷ
    ▸ sidecarͱ͍͏৽͍͠ڥքઢ
    ▸ ௨৴ΛѲΔ͜ͱͰɺmicroservicesʹඞཁͳ༷ʑͳػೳΛ౷Ұతʹఏڙ
    ▸ Timeout / retry / circuit breaking etc…
    ▸ Distributed Tracing
    ▸ Dev ͸serviceͷػೳ։ൃʹ஫ྗ
    ▸ ͨͩ͠ɺద੾ͳΤϥʔϋϯυϦϯά͸ඞਢʂ
    ▸ Ops͸sidecarΛ௨ͯ͠Microservicesͱͯ͠ͷ҆ఆੑʹ஫ྗ

    View full-size slide

  60. © ChatWork
    ͍͍ײ͡++ʂ

    View full-size slide

  61. © ChatWork
    ޙ͔ΒΈΔͱɺͦΓΌͦ͏ͩͬͯײ͋Δ͕
    ͦΕ͕ͦ͜”ΞΠσΞ”ͳͷ͔΋͠Εͳ͍

    View full-size slide

  62. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ (ࡁ)
    ▸ Envoyͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔEnvoyͷΞϓϩʔν (ࡁ)
    ▸ ChatWorkͷະདྷ༧૝ਤ
    ▸ Microservices ͷπϥϛ / ϜζΧγϛ ͦͷ̎
    ▸ Kubernetesͱ͍͏”ΞΠσΞ”
    ▸ ·ͱΊ

    View full-size slide

  63. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ
    ▸ Envoyͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔEnvoyͷΞϓϩʔν
    ▸ ChatWorkͷະདྷ༧૝ਤ → ݱঢ়ͱಋೖ΁ͷΞϓϩʔν
    ▸ Microservices ͷπϥϛ / ϜζΧγϛ ͦͷ̎
    ▸ Kubernetesͱ͍͏”ΞΠσΞ”
    ▸ ·ͱΊ

    View full-size slide

  64. © ChatWork
    ͬ͘͟Γ·ͱΊΔͱ
    ▸ ݱঢ়
    ▸ EC2্ͷPHPΞϓϦέʔγϣϯ + k8s্ͷscalaΞϓϦέʔγϣϯ
    ▸ ਐߦத
    ▸ PHPΞϓϦέʔγϣϯͷk8sԽ / ΞϓϦͷ࣮ߦج൫Λk8sʹ౷Ұ
    ▸ ͦͷ࣍
    ▸ service mesh(Envoy) ಋೖ

    View full-size slide

  65. © ChatWork
    ݱঢ়
    ▸ Kubernetes(scala)ͱɺEC2ʹ௚઀provisioning(PHP)ͷซ༻
    ▸ ChatWorkͷ৽ϝοηʔδϯάγεςϜΛࢧ͑Δٕज़
    ▸ 2016೥ͷ೥຤͔Βซ༻͕࢝·ͬͨ
    ▸ Ҏޙͷ৽نΞϓϦ͸k8s্Ͱಈ͍͍ͯΔ
    ▸ ChatWorkʹ͓͚ΔKubernetes on AWS
    ▸ Kubernetesӡ༻վળ / ϊ΢ϋ΢஝ੵ

    View full-size slide

  66. © ChatWork
    Dockerfile Dockerfile

    View full-size slide

  67. © ChatWork
    ਐߦத
    ▸ PHPΞϓϦέʔγϣϯΛKubernetes্Ͱಈ͔͢
    ▸ αʔόӡ༻ͷҰݩԽʹΑΔӡ༻ίετ࡟ݮ
    ▸ ϦϦʔεγεςϜͷվળ
    ▸ DevͱOpsͷڥքઢͷ࠶ઃఆ

    View full-size slide

  68. © ChatWork
    Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile

    View full-size slide

  69. © ChatWork
    ͦͷ࣍
    ▸ Envoy / Istio (Service mesh) ಋೖ
    ▸ ͞ΒͳΔαʔϏεͷ҆ఆੑ
    ▸ ػೳ։ൃεϐʔυ޲্
    ▸ ࠓޙͷαʔϏε૿ʹ଱͑͏Δج൫

    View full-size slide

  70. © ChatWork
    Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile

    View full-size slide

  71. © ChatWork
    Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile
    Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile

    View full-size slide

  72. © ChatWork
    ΤϯδχΞืूத
    http://corp.chatwork.com/ja/recruit/
    ▸ ओମੑΛ࣋ͪɺࣗΒߦಈͰ͖Δ
    ▸ ଞऀΛೝΊɺଚॏͰ͖Δ
    ▸ ৘ใΛूΊɺڞ༗Ͱ͖Δ
    ͱ͍͏ํΛ׻ܴ͠·͢ʂ

    View full-size slide

  73. © ChatWork
    ઌʹEnvoy΍Μͳ͍ͷʁ
    ▸ ݱঢ়ͷΞϓϦέʔγϣϯΠϯϑϥͷ̎ॏ؅ཧ͕େม
    ▸ k8sʹ౷Ұͯ͠ӡ༻ίετ࡟ݮ / service meshಋೖίετ࡟ݮ
    ▸ ͳͥkubernetesʹدͤΔʁ
    ▸ kubernetes΋։ൃ/ӡ༻Λ”͍͍ײ͡” ʹͯ͘͠ΕΔπʔϧͰ͋Δ
    ▸ ৄࡉ͸࣍ͷΞδΣϯμʹͯ
    ▸ Istioͱ͍͏kubernetesͱͷ૬ੑͷ͍͍control plane͕ઈࢍਐԽத

    View full-size slide

  74. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ (ࡁ)
    ▸ Envoyͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔEnvoyͷΞϓϩʔν (ࡁ)
    ▸ ChatWorkͷະདྷ༧૝ਤ → ݱঢ়ͱಋೖ΁ͷΞϓϩʔν (ࡁ)
    ▸ Microservices ͷπϥϛ / ϜζΧγϛ ͦͷ̎
    ▸ Kubernetesͱ͍͏”ΞΠσΞ”
    ▸ ·ͱΊ

    View full-size slide

  75. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ (ࡁ)
    ▸ Envoyͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔEnvoyͷΞϓϩʔν (ࡁ)
    ▸ ChatWorkͷະདྷ༧૝ਤ → ݱঢ়ͱಋೖ΁ͷΞϓϩʔν (ࡁ)
    ▸ Microservices ͷπϥϛ ͦͷ̎ → Kubernetes͕ඞཁͱͳΔഎܠ
    ▸ Kubernetesͱ͍͏”ΞΠσΞ”
    ▸ ·ͱΊ

    View full-size slide

  76. © ChatWork
    Microservicesͷߏஙྫ(k8sલ)

    View full-size slide

  77. © ChatWork
    ໰୊ͦͷ̍
    Dev / Opsͷڥքઢʁ

    View full-size slide

  78. © ChatWork
    Infrastructure as code
    ▸ ཧ૝ → શͯίʔυԽ͞Ε͍ͯΔͷͰɺDev / OpsڞʹมߋՄೳ
    ▸ devops࣮ݱʂ
    ▸ ݱ࣮ → Infrastructure as codeͷ؅ཧ͕Opsʹूத
    ▸ Ansible / Chef ౳ͷprovisioning toolͷֶशίετ΍؀ڥߏஙίετ
    ▸ ϞχλϦϯά / ؂ࢹ / ϩάసૹ౳ɺશମͷӨڹൣғͷ೺Ѳ͕೉͍͠
    (஫ɿ͋͘·Ͱࢲͷܦݧ/஌͍ͬͯΔൣғͷ࿩Ͱ͢ʣ

    View full-size slide

  79. © ChatWork
    Ops :
    AWSͷϦιʔε؅ཧ
    Infrastructure as codeͷ source code
    Dev :
    ΞϓϦέʔγϣϯͷSource Code

    View full-size slide

  80. © ChatWork
    αʔό্Ͱ
    ີ݁߹

    View full-size slide

  81. © ChatWork
    Dev / Opsͷڥքઢʁ
    ▸ Dev : ৽͍͠ͷ࢖͍͍ͨɺɺ/ αοͱόʔδϣϯΞοϓ͍ͨ͠ɺɺ
    ▸ ΍Γ͍ͨ͜ͱʹରͯ͠ͷίετ͕͔ͳΓߴ͍
    ▸ Ops: ݴޠɺϑϨʔϜϫʔΫ૿͑ΔΜ͔ʙɺɺ / ҆ఆͯ͠ಈ͘ͷʁ
    ▸ ਓͷέπΛ࣋ͨ͞ΕͯΔײ
    ▸ ୭Ͱ΋৮ΕΔΑ͏ʹ͢Δॴ·Ͱͷίετ΋தʑʹେ͖͍

    View full-size slide

  82. © ChatWork
    ໰୊ͦͷ̎
    αʔόӡ༻ͬͯ͜ΕͰϕετʁ

    View full-size slide

  83. © ChatWork
    ૬৐Γ͍͚ͤͨ͞Ͳɺɺ

    View full-size slide

  84. © ChatWork
    αʔόӡ༻ͬͯ͜ΕͰϕετʁ
    ▸ ΞϓϦέʔγϣϯͷ૬৐Γ͕೉͍͠
    ▸ ̍instance = 1ΞϓϦέʔγϣϯ
    ▸ ෳ਺ͷΞϓϦέʔγϣϯ༻ͷprovisioningΛ1 instanceʹྲྀͯ͠ɺશͯ
    ਖ਼ৗʹಈ͘͜ͱΛอূ͢Δͷ͕೉͍͠
    ▸ αʔϏε૿͑Δͱ૊Έ߹Θͤ਺͕രൃ͠ɺ؅ཧͰ͖ͳ͍
    ▸ ΍ͬͨͱ͢Δͱɺ·͢·͢Dev͔Β͸৮ΓͮΒ͍΋ͷʹɻɻ

    View full-size slide

  85. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ (ࡁ)
    ▸ Envoyͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔEnvoyͷΞϓϩʔν (ࡁ)
    ▸ ChatWorkͷະདྷ༧૝ਤ → ݱঢ়ͱಋೖ΁ͷΞϓϩʔν (ࡁ)
    ▸ Microservices ͷπϥϛ ͦͷ̎ → Kubernetes͕ඞཁͱͳΔഎܠ (ࡁ)
    ▸ Kubernetesͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔKubernetesͷར఺
    ▸ ·ͱΊ

    View full-size slide

  86. © ChatWork
    ▸ ஫ɿࢲͷ͜Ε·Ͱͷܦݧ ͔Βɺkubernetesͱ͍͏୯ޠΛར༻͍ͯ͠·͢
    ͕ɺίϯςφΦʔέʔετϨʔγϣϯπʔϧ Ұൠʹ౰ͯ͸·Δ(͸ͣͷ)࿩
    Ͱ͢ɻ

    View full-size slide

  87. © ChatWork
    Microservicesͷߏஙྫ(k8s)

    View full-size slide

  88. © ChatWork
    ίϯςφͱ͍͏ϙʔλϏϦςΟͷߴִ͍཭؀ڥͷ͓͔͛Ͱɺڥքઢ͕໌
    ֬ʹɻϞχλϦϯάͱ͔͸ؾʹͤͣʹɺΞϓϦέʔγϣϯ࣮ߦʹඞཁͳ؀ڥʹؔ͢Δఆ͚ٛͩ
    ͋Ε͹ྑ͍ͷͰɺϋʔυϧ͕௿͍ɻ
    Dockerfile Dockerfile Dockerfile

    View full-size slide

  89. © ChatWork
    Ops͸Workerʹڞ௨ͷϞχλϦϯά΍ϩάసૹͷઃఆΛηοτ͢Ε͹ྑ
    ͍ɻݸʑͷWorkerͷଘࡏΛҙࣝ͢Δ͜ͱ͕΄ͱΜͲແ͘ɺར༻ϦιʔεʹԠͨ͡ΞϓϦέʔ
    γϣϯ(pod)഑ஔ΍ɺϓϩηεͷҡ࣋͸kubernetes͕ٓ͘͠΍ͬͯ͘ΕΔɻ

    View full-size slide

  90. © ChatWork
    ڥքઢ͕͍͍ײ͡ʂ
    ▸ ΞϓϦέʔγϣϯ༻ͷ؀ڥߏங͕׬શʹDevͷकඋൣғʹ
    ▸ Dockerfileͷֶशίετ͸͋Δ͕ɺؾʹ͢Δ͜ͱ͕͙ͬͱগͳ͘ͳͬͨ
    ▸ Ops͸clusterͷ؅ཧɺmonitoring / loggingసૹ౳ͷڞ௨ج൫ʹूத
    ▸ 12 factor app ͱ͍͏ΞϓϦέʔγϣϯͷڞ௨ϧʔϧ
    ▸ pod਺ͷҡ࣋΍ɺϦιʔεঢ়گʹԠͨ͡ίϯςφ഑ஔ͸kubernetes͕
    ΍ͬͯ͘ΕΔ

    View full-size slide

  91. © ChatWork
    αʔόӡ༻͕͍͍ײ͡ʂ
    ▸ Pet ͔Β Cattle (PHPαʔόɺPythonαʔό…. → k8s Worker)
    ▸ Ϧιʔεͷ༗ޮ׆༻
    ▸ ̍αʔό্Ͱͷෳ਺ΞϓϦέʔγϣϯͷ૬৐Γ

    View full-size slide

  92. © ChatWork
    ͍͍ײ͡++ʂ

    View full-size slide

  93. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ (ࡁ)
    ▸ Envoyͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔEnvoyͷΞϓϩʔν (ࡁ)
    ▸ ChatWorkͷະདྷ༧૝ਤ → ݱঢ়ͱಋೖ΁ͷΞϓϩʔν (ࡁ)
    ▸ Microservices ͷπϥϛ ͦͷ̎ → Kubernetes͕ඞཁͱͳΔഎܠ (ࡁ)
    ▸ Kubernetesͱ͍͏”ΞΠσΞ” → πϥϛʹର͢Δkubernetesͷར఺(ࡁ)
    ▸ ·ͱΊ

    View full-size slide

  94. © ChatWork
    ΞδΣϯμ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ (ࡁ)
    ▸ Envoyͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔEnvoyͷΞϓϩʔν (ࡁ)
    ▸ ChatWorkͷະདྷ༧૝ਤ → ݱঢ়ͱಋೖ΁ͷΞϓϩʔν (ࡁ)
    ▸ Microservices ͷπϥϛ ͦͷ̎ → Kubernetes͕ඞཁͱͳΔഎܠ (ࡁ)
    ▸ Kubernetesͱ͍͏”ΞΠσΞ” → πϥϛʹର͢Δkubernetesͷར఺(ࡁ)
    ▸ ·ͱΊ

    View full-size slide

  95. © ChatWork
    ࠓ೔࿩ͨ͜͠ͱ
    ▸ Microservices ͷπϥϛ ͦͷ̍ → Envoy͕ඞཁͱͳΔഎܠ
    ▸ Envoyͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔEnvoyͷΞϓϩʔν
    ▸ ChatWorkͷະདྷ༧૝ਤ → ݱঢ়ͱಋೖ΁ͷΞϓϩʔν
    ▸ Microservices ͷπϥϛ ͦͷ̎ → Kubernetes͕ඞཁͱͳΔഎܠ
    ▸ Kubernetesͱ͍͏”ΞΠσΞ” → πϥϛʹର͢ΔKubernetesͷར఺

    View full-size slide

  96. © ChatWork
    ࠓ೔ͷ໨ඪ
    ɾEnvoy ੌ͍ʂ͍ͬͯ͏ײಈͷڞ༗
    ɾDev / OpsΛ໰Θͣʹɺࣗ෼ͷ૊৫΍
    γεςϜͷڥքઢΛߟ͑Δ͖͔͚ͬʹ

    View full-size slide

  97. © ChatWork
    Feedback ͓ئ͍͠·͢ʂʂ
    from ωʔϜΧʔυͷQRίʔυ

    View full-size slide