AbemaTVでネイティブエンジニア
からサーバーエンジニアに挑戦した話

Transcript

1. 1.

   AbemaTVͰωΠςΟϒΤϯδχΞ
 ͔ΒαʔόʔΤϯδχΞʹ௅ઓͨ͠࿩ CyberAgent, Inc. AbemaTV։ൃہ αʔόʔαΠυΤϯδχΞ LE VAN NGHIA (ΪΞ)

2. 2.

   • nghialv2607 • nghialv About me • 2014/11: CyberAgent 15৽ଔೖࣾ

   • ~ 2015/6: AmebaΞϓϦ - iOS • ~ 2016/7: FRESH!ੜ์ૹ - iOS • ~ ࠓ : AbemaTV - Server Side ID ڵຯ • Φʔϓϯιʔε։ൃ ϓϩδΣΫτܦྺ • ΪΞ • ؔ਺ܕϓϩάϥϛϯά Hakuba, Future, MaterialKit…
3. 3.

   ”20୅ͷ൴Β͕ͲΜͳ͜ͱʹʮ ௅ઓʯ͍ͯ͠Δͷ͔” ຊ೔ͷTalk BattleͷςʔϚ͸

4. 4.

   ωΠςΟϒΤϯδχΞ αʔόʔΤϯδχΞ ➡

5. 5.

   Ҡಈͷཧ༝ • ஌ࣝΛ෯޿͍ͨ͘͠ • Ϋϥ΢υɺωοτϫʔΫ • େྔσʔλॲཧ • ෼ࢄγεςϜɺϚΠΫϩαʔϏε

6. 6.

   AbemaTVαʔόʔͷϞχλϦϯάγεςϜΛ࡞੒ ॳΊͯͷλεΫ͸

7. 7.

   1 repository 35 repositories native server AbemaTVͷίʔυ

8. 8.

   AbemaTVͷαʔόαΠυ Cloud DNS Cloud Load Balancer Cloud Router Cloud CDN

   Pub/Sub BigQuery Cloud Storage Transcoder Kubernetes Wowza HLS MPEG-DASH DRM gRPC Microservices Protocol Buffers Varnish Alpine Linux Terraform Packer MongoDB Redis Cluster Elasticsearch Firebase
9. 9.

   • ຊ • ϓϩδΣΫτͷࢿྉ • ݸਓGoogle CloudͷΞΧ΢ϯτ࡞੒ɼ ৭ʑ༡ΜͰΈΔ • AbemaTVίʔυΛಡΉ

   • etcd, kubernetes…ͷίʔυΛಡΉ
10. 10.

    GCP + Kubernetes + gRPC Google Cloud Platform Kubernetes Service

    Service Service gRPC gRPC
11. 11.

    AbemaTVͷΞʔΩςΫνϟ streaming-server gateway Media User Comment Share Question Purchase Firebase

    Watchman Transcoder Ad system encoder Live Transcoder varnish api access live-gateway Compute
 Engine MongoDB Redis Cluster Container
 Engine Cloud Storage Firebase media access live admin access stateful stateless
12. 12.

    Ϩϕϧ ྫ ΞϓϦέʔγϣϯ http, grpcͷΤϥʔɼϨΠςϯγɼ಺෦ঢ়ଶ ίϯςφ ίϯςφϦιʔεঢ়ଶ ΦʔέετϨʔγϣϯ ΫϥελʔͷϦιʔεɼεέδϡʔϦϯά ϗετ

    (OS, Hardware) ϗετͷϦιʔεɼhardwareΤϥʔ ωοτϫʔΫ routers, switches ऩू͢Δσʔλ
13. 13.

    ࣌ܥྻDBͷબ୒ • DalmatinerDB • InfluxDB • Prometheus • OpenTSDB •

    Graphite • … “top10 time series databases”ͷϒϩάʹΑΓ
14. 14.

    • Prometheus͸Kubernetesͱ࿈ܞ͍ͯ͠Δ • αʔϏεσΟεΧόϦʔ: Ϋϥ΢υͷΑ͏ͳμΠφϛοΫͳ؀ڥʹendpoints͕มߋ͞ΕΔ • ίϯςφ, ΦʔέετϨʔγϣϯ ͷϝτϦΫεΛऔಘ͠΍͍͢ •

    Prometheusͷੑೳ • ύϑΥʔϚϯεɿ • ҰͭͷPrometheusαʔό͸800K samples/s • Federation • ѹॖ཰: 12.3ഒ (ѹॖͳ͠ɿ16 bytes/sample, Prometheus: 1.3 bytes/sample) • Prometheus͸Grafanaͱ࿈ܞ͍ͯ͠Δ • Prometheus͸GolangͰॻ͔Ε͍ͯΔ PrometheusΛબ୒ͨ͠ཧ༝
15. 15.

    AbemaTVʹ͓͚ΔPrometheus

16. 16.

    ੒Ռ

17. 17.

    None
18. 18.

    None
19. 19.

    AlertManager Slack࿈ܞ͢ΔBot - ࣾ֎΍ϞόΠϧͰ΋ίϚϯυͰ
 γεςϜͷϝτϦΫε͕ݟ͑Δ - Τϥʔ཰, latency percentileͳͲ ͕ߴ͘ͳΔͱ௨஌͕དྷΔ

20. 20.

    • ωΠςΟϒΤϯδχΞ͔ΒαʔόʔΤϯδχΞʹ௅ઓͷ࿩ • AbemaTVͷϞχλϦϯάγεςϜͷৄ͍͠৘ใ • “AbemaTVʹPrometheusͱ͍͏ϞχλϦϯάγεςϜΛಋೖͨ͠࿩”ͷϒϩάϔ • αʔόʔ΋΍Γ͍ͨωΠςΟϒΤϯδχΞ΁ • ݴޠ͡Όͳͯ͘linux,

    network, cloudؔ͢Δ஌ࣝͷ४උΛΦεεϝ • ·ͩ20୅ͳͷͰ৭ʑ௅ઓ͠·͠ΐ͏ʂ ·ͱΊ