AbemaTVでネイティブエンジニア
からサーバーエンジニアに挑戦した話

Transcript

1. AbemaTVͰωΠςΟϒΤϯδχΞ
 ͔ΒαʔόʔΤϯδχΞʹ௅ઓͨ͠࿩ CyberAgent, Inc. AbemaTV։ൃہ αʔόʔαΠυΤϯδχΞ LE VAN NGHIA (ΪΞ)

2. • nghialv2607 • nghialv About me • 2014/11: CyberAgent 15৽ଔೖࣾ

   • ~ 2015/6: AmebaΞϓϦ - iOS • ~ 2016/7: FRESH!ੜ์ૹ - iOS • ~ ࠓ : AbemaTV - Server Side ID ڵຯ • Φʔϓϯιʔε։ൃ ϓϩδΣΫτܦྺ • ΪΞ • ؔ਺ܕϓϩάϥϛϯά Hakuba, Future, MaterialKit…

3. ”20୅ͷ൴Β͕ͲΜͳ͜ͱʹʮ ௅ઓʯ͍ͯ͠Δͷ͔” ຊ೔ͷTalk BattleͷςʔϚ͸

4. ωΠςΟϒΤϯδχΞ αʔόʔΤϯδχΞ ➡

5. Ҡಈͷཧ༝ • ஌ࣝΛ෯޿͍ͨ͘͠ • Ϋϥ΢υɺωοτϫʔΫ • େྔσʔλॲཧ • ෼ࢄγεςϜɺϚΠΫϩαʔϏε

6. AbemaTVαʔόʔͷϞχλϦϯάγεςϜΛ࡞੒ ॳΊͯͷλεΫ͸

7. 1 repository 35 repositories native server AbemaTVͷίʔυ

8. AbemaTVͷαʔόαΠυ Cloud DNS Cloud Load Balancer Cloud Router Cloud CDN

   Pub/Sub BigQuery Cloud Storage Transcoder Kubernetes Wowza HLS MPEG-DASH DRM gRPC Microservices Protocol Buffers Varnish Alpine Linux Terraform Packer MongoDB Redis Cluster Elasticsearch Firebase

9. • ຊ • ϓϩδΣΫτͷࢿྉ • ݸਓGoogle CloudͷΞΧ΢ϯτ࡞੒ɼ ৭ʑ༡ΜͰΈΔ • AbemaTVίʔυΛಡΉ

   • etcd, kubernetes…ͷίʔυΛಡΉ

10. GCP + Kubernetes + gRPC Google Cloud Platform Kubernetes Service

    Service Service gRPC gRPC

11. AbemaTVͷΞʔΩςΫνϟ streaming-server gateway Media User Comment Share Question Purchase Firebase

    Watchman Transcoder Ad system encoder Live Transcoder varnish api access live-gateway Compute
 Engine MongoDB Redis Cluster Container
 Engine Cloud Storage Firebase media access live admin access stateful stateless

12. Ϩϕϧ ྫ ΞϓϦέʔγϣϯ http, grpcͷΤϥʔɼϨΠςϯγɼ಺෦ঢ়ଶ ίϯςφ ίϯςφϦιʔεঢ়ଶ ΦʔέετϨʔγϣϯ ΫϥελʔͷϦιʔεɼεέδϡʔϦϯά ϗετ

    (OS, Hardware) ϗετͷϦιʔεɼhardwareΤϥʔ ωοτϫʔΫ routers, switches ऩू͢Δσʔλ

13. ࣌ܥྻDBͷબ୒ • DalmatinerDB • InfluxDB • Prometheus • OpenTSDB •

    Graphite • … “top10 time series databases”ͷϒϩάʹΑΓ

14. • Prometheus͸Kubernetesͱ࿈ܞ͍ͯ͠Δ • αʔϏεσΟεΧόϦʔ: Ϋϥ΢υͷΑ͏ͳμΠφϛοΫͳ؀ڥʹendpoints͕มߋ͞ΕΔ • ίϯςφ, ΦʔέετϨʔγϣϯ ͷϝτϦΫεΛऔಘ͠΍͍͢ •

    Prometheusͷੑೳ • ύϑΥʔϚϯεɿ • ҰͭͷPrometheusαʔό͸800K samples/s • Federation • ѹॖ཰: 12.3ഒ (ѹॖͳ͠ɿ16 bytes/sample, Prometheus: 1.3 bytes/sample) • Prometheus͸Grafanaͱ࿈ܞ͍ͯ͠Δ • Prometheus͸GolangͰॻ͔Ε͍ͯΔ PrometheusΛબ୒ͨ͠ཧ༝

15. AbemaTVʹ͓͚ΔPrometheus

16. ੒Ռ

17. None
18. None

19. AlertManager Slack࿈ܞ͢ΔBot - ࣾ֎΍ϞόΠϧͰ΋ίϚϯυͰ
 γεςϜͷϝτϦΫε͕ݟ͑Δ - Τϥʔ཰, latency percentileͳͲ ͕ߴ͘ͳΔͱ௨஌͕དྷΔ

20. • ωΠςΟϒΤϯδχΞ͔ΒαʔόʔΤϯδχΞʹ௅ઓͷ࿩ • AbemaTVͷϞχλϦϯάγεςϜͷৄ͍͠৘ใ • “AbemaTVʹPrometheusͱ͍͏ϞχλϦϯάγεςϜΛಋೖͨ͠࿩”ͷϒϩάϔ • αʔόʔ΋΍Γ͍ͨωΠςΟϒΤϯδχΞ΁ • ݴޠ͡Όͳͯ͘linux,

    network, cloudؔ͢Δ஌ࣝͷ४උΛΦεεϝ • ·ͩ20୅ͳͷͰ৭ʑ௅ઓ͠·͠ΐ͏ʂ ·ͱΊ