Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How mixi2 Uses TiDB for SNS Scalability and Per...
Search
kanmo
February 14, 2025
Programming
42
33k
How mixi2 Uses TiDB for SNS Scalability and Performance
Developers Summit 2025 登壇資料
kanmo
February 14, 2025
Tweet
Share
More Decks by kanmo
See All by kanmo
Keep Phoenix App Productivity
kanmo
2
1.2k
ExAws
kanmo
1
560
Other Decks in Programming
See All in Programming
GoのGenericsによるslice操作との付き合い方
syumai
3
710
GitHub Copilot and GitHub Codespaces Hands-on
ymd65536
1
130
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
takuyakatsusa
2
460
なぜ適用するか、移行して理解するClean Architecture 〜構造を超えて設計を継承する〜 / Why Apply, Migrate and Understand Clean Architecture - Inherit Design Beyond Structure
seike460
PRO
1
720
童醫院敏捷轉型的實踐經驗
cclai999
0
210
Composerが「依存解決」のためにどんな工夫をしているか #phpcon
o0h
PRO
1
250
PHPで始める振る舞い駆動開発(Behaviour-Driven Development)
ohmori_yusuke
2
240
PicoRuby on Rails
makicamel
2
120
Kotlin エンジニアへ送る:Swift 案件に参加させられる日に備えて~似てるけど色々違う Swift の仕様 / from Kotlin to Swift
lovee
1
260
git worktree × Claude Code × MCP ~生成AI時代の並列開発フロー~
hisuzuya
1
520
datadog dash 2025 LLM observability for reliability and stability
ivry_presentationmaterials
0
410
Google Agent Development Kit でLINE Botを作ってみた
ymd65536
2
210
Featured
See All Featured
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.4k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
The Cult of Friendly URLs
andyhume
79
6.5k
How to Think Like a Performance Engineer
csswizardry
24
1.7k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
The World Runs on Bad Software
bkeepers
PRO
69
11k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
5
230
Stop Working from a Prison Cell
hatefulcrawdad
270
20k
Side Projects
sachag
455
42k
Statistics for Hackers
jakevdp
799
220k
Transcript
.*9* גࣜձࣾ.*9*ᇙ໌ल %FWFMPQFST4VNNJU ʮNJYJͷཪʯ 5J%#Ͱ࣮ݱ͢Δ4/4ͷεέʔ ϥϏϦςΟͱύϑΥʔϚϯε
mixi2ʹ͍ͭͯ • จςΩετSNS • ʮϑΥϩʔʯλΠϜϥΠϯͷଞɺSNS mixiʹ͋ͬͨίϛϡχςΟػೳΑΓ ϦΞϧλΠϜͳίϛϡχέʔγϣϯ ͚ͷΠϕϯτػೳ • αʔόGoɺϞόΠϧΫϥΠΞϯτ
FlutterΛ࠾༻ • ॳظͷػೳ։ൃ4ਓ(ϦϦʔε࣌7ਓ)
mixi2 architecture
DBͷબఆ SNSΛ։ൃ͢Δͱ͖ʹٻΊͨཁ݅ • ϦΫΤετɺϢʔβʔͷ૿ՃʹରԠͰ͖ΔεέʔϥϏϦςΟ • োൃੜ࣌ͷϑΣΠϧΦʔόʔ • ʢαʔϏεͱͯ͠ڐ༰Ͱ͖ΔʣϨΠςϯγʔ • τϥϯβΫγϣϯ
• ϝϯςφϯεΦϖϨʔγϣϯͷ༰қ͞ • ίετ໘ NewSQLʹ͍ͭͯಋೖݕ౼Λͨ͠
NewSQLͱ SQLΠϯλʔϑΣʔεΛ࣋ͪ ैདྷͷϦϨʔγϣφϧσʔλϕʔε͕࣋ͭτϥϯβΫγϣϯͱ ڧ͍߹ੑΛอূ͢Δࢄσʔλϕʔε TiDB, Spanner, CockroachDB, YugaByteDBͳͲ
NewSQLͷݕ౼ ͦΕͧΕͷಛੑΛൺֱ͠ɺࠓճTiDBΛ࠾༻ τϥϯβΫ γϣϯ ϨΠςϯγʔ োੑ ӡ༻ίετ Πϯϑϥඅ༻ 3%# 4IBSEJOH
̋ʢ˚ʣ ̋ ˚ ˚ ̋ /FX42- ̋ ˚ ̋ ̋ ˚
TiDBͷಛ • ׂ͝ͱͷίϯϙʔωϯτ͕ࢄஔ͞Ε ͨΞʔΩςΫνϟ • TiDB, TiKV, PD • ֤ίϯϙʔωϯτԽ͢Δ͜ͱʹΑΓ
ύϑΥʔϚϯεͱোੑΛߴΊΒΕΔ • SQL LayerͷTiDBαʔόਫฏํʹ ֦ுՄೳ • ΫϥελΛཧ͢ΔPDαʔόɺ Λ૿͢͜ͱͰՄ༻ੑΛߴΊΒΕΔ ʢ࠷̏ʣ Ҿ༻ݩ: https://docs.pingcap.com/tidb/stable/tidb-architecture
TiDB Storage (TiKV) • σʔλΛ֨ೲ͢ΔͷTiKVίϯϙʔωϯτ • σʔλʢϦʔδϣϯͱ͍͏୯ҐʣΛࣗಈͰෳ ͷϨϓϦΧͰཧ͞ΕΔ • ಉ͡σʔλ͕ࡾͭίϐʔ͞ΕΔ
• ߴՄ༻ੑΛ࣋ͪɺࣗಈϑΣΠϧΦʔόʔΛα ϙʔτ͢Δ • TiKVϊʔυΛΫϥελʹՃ͢Δ͜ͱͰ༰ྔ Λ૿͢͜ͱ͕Ͱ͖Δ • RDBͰߦ͍ͬͯͨσʔλංେԽʹ͏γϟʔ σΟϯάɺϚελʔԽͳͲ͕ෆཁʹͳΔ Ҿ༻ݩ: https://docs.pingcap.com/tidb/stable/tidb-storage
Backup • όοΫΞοϓˍϦετΞ • σʔλόοΫΞοϓฐࣾαʔϏεӡ ༻ʹ͓͍ͯඞਢཁ݅ • εφοϓγϣοτͱ૿ϩά͕औΕΔ • όοΫΞοϓ͔Βղੳ༻σʔλͷߏங
• Dumpͷෛՙ͕ແࢹͰ͖ͳ͍͘Β͍ʹ σʔλ͕૿Ճͨ͠߹ɺόοΫΞοϓ ͔Β෮ݩͨ͠Ϋϥελ͔ΒDWH༻ͷ σʔλ͕औΕΔΑ͏ʹ͢Δ Starting component br: /.tiup/components/br/v8.1.0/br restore full --pd 10.30.2.92:2379 --storage s3://kan-tidb-test-tiup-backup/ snapshot-20240712-3 --send-credentials-to-tikv=false Detail BR log in /tmp/br.log.2024-07-12T08.13.06Z Full Restore <------------------------------------------------------------------- ----------------------------------------------------------------> 100.00%
TiUP • ΫϥελʔͷߏஙɺScale out/Scale inɺBackupଐͷϕϯνπʔϧͳͲ ͯ͢ͷૢ࡞ͰtiupίϚϯυ͕ඇৗʹ ͍͔ͬͨ͢ • υΩϡϝϯτ๛ •
ϦϦʔεޙͷӡ༻ʹߩݙͯ͘͠Εͦ͏ tiup cluster scale-out test-cluster scaleout-tikv.yaml tikv_servers: - host: 10.20.1.206 config: server.labels: rack: r4 1. Ճ͢ΔίϯϙʔωϯτΛyamlʹఆٛ͢Δ 2. tiupίϚϯυΛ࣮ߦ͢Δ
PingCAP TiDBͰࠓճͷαʔϏεཁ݅ΛຬͨͤΔͱஅ͠࠾༻ ։ൃॳظϑΣʔζͰɺPingCAPࣾͱִिͰmtgΛ͠ͳ͕ΒਐΊΔ
Timeline • λΠϜϥΠϯʹɺϢʔβʔͷߘͨ͠λΠ ϜϥΠϯʢϢʔβʔλΠϜϥΠϯʣͱϗʔϜ ը໘ͷλΠϜϥΠϯ͕͋Δ • ϗʔϜλΠϜϥΠϯΛදࣔ͢ΔʹɺPOST ͨ͠ߘΛͰ͖ΔݶΓૣ͘ϑΥϩϫʔͷλΠ ϜϥΠϯʹಧ͚Δඞཁ͕͋Δ
Home Timeline Timeline • ࣄલʹϗʔϜλΠϜϥΠϯΛ࡞͢Δ • ߘऀͷϑΥϩϫʔ/ίϛϡχςΟΛ୳ ͠ɺϑΥϩϫʔຖͷλΠϜϥΠϯςʔ ϒϧʹϙετΛॻ͖ࠐΉ ϙετIDΛ֤Ϣʔβʔ͝ͱʹ৴͍ͯ͠Δ
Post API Redis Redis Redis Find Followers Find Community Members Timeline API
Timeline Tables (Redis) • ϢʔβʔͷϗʔϜλΠϜϥΠϯΛࣄલʹredis ʹ࡞͢Δ • ͜ͷใΛݩʹDBͷใͱΈ߹Θͤͯ ϗʔϜλΠϜϥΠϯΛߏங͢Δ •
ҰఆͷϙετใΛϢʔβʔຖʹอ࣋͢Δ • ࢀর͞ΕΔՄೳੑ͕͘ͳͬͨϙετҰ ఆִؒͰআ͍ͯ͠Δ post_id user: A post_id post_id post_id user: B post_id post_id post_id user: C post_id post_id Redis Sorted Set
Timeline Tables (DB) • TiDBʹશͯͷϙετͱϢʔβʔλΠϜ ϥΠϯͷϙετΛه • ϗʔϜλΠϜϥΠϯDBʹ࡞ΒΕ ͳ͍ •
ϗʔϜλΠϜϥΠϯΛ࡞ΔͨΊͷ ใʢϑΥϩϫʔɺίϛϡχςΟϢʔ βʔʣ͕ଘࡏ͢Δ posts post_id user_id community_id text user_posts post_id user_id community_ users community_id user_id followings following_id user_id
ෛՙςετ ϦϦʔεલʹෛՙݕূΛ࣮ࢪɻAPIͷϨΠςϯγʔͷඪΛ100msʹઃఆ • Writeੑೳ • දతͳRPC͕10000RPSΛग़ͤΔͱ͜Ζ·Ͱݕূ • Readੑೳ • ߘ͕ϑΥϩϫʔͷλΠϜϥΠϯʹදࣔ͞ΕΔ·Ͱͷ͕࣌ؒ5ඵ͔Β10ඵʹͳΔ͜ͱ
Λඪʹௐ • αʔϏε໊͔Β͋ΔఔϢʔβʔ͕དྷΔ͜ͱ͕ఆͰ͖ͨ
Locust - Boomer • ෛՙςετ༻ͷڥΛ༻ҙͯ͠LocustΛ͍ݕূ • ฒྻΛେ͖͍ͨ͘͠ͷͰgoͷworker࣮ ͷboomerΛ༻ͨ͠ • ߈ܸΫϥΠΞϯτͷεέʔϧΞτɺεέʔ
ϧΞοϓΛසൟʹߦ͏ͨΊɺk8s্Ͱಈ͔ͨ͠ • γφϦΦϢʔβʔ࡞͔ΒೝূΛߦ͍ɺϑΥ ϩʔɺϙετɺϥΠΫͳͲͷجຊతͳಈ࡞Λશͯ ࣮ࢪ
ෛՙςετ • ॻ͖ࠐΈੑೳTiDB, TiKVͷεέʔϧΞτͰఆ ௨Γʹੑೳ্ͨ͠ • Ұ෦ɺॏ͍ॲཧʢݕࡧΠϯσοΫεͷॻ͖ࠐ ΈͳͲʣΛඇಉظॲཧʹมߋ • pprofΛ͍ॏ͍ॲཧΛ୳ͨ͠
• ಡΈࠐΈॲཧআॲཧࠓճͷඪΛୡ͢ Δ্Ͱͳ͠ • ߴෛՙ࣌ʹෳͷϑΥϩϫʔ͕ಉ࣌ʹϑΥϩʔ͢ Δɺ͍͍ͶΛ͢ΔέʔεͰසൟϩοΫͪͷλΠϜ Ξτ͕ൃੜ
োςετ Locust͔ΒෛՙΛ͔͚ͳ͕Βtidbͷ֤ίϯϙʔωϯ τΛఀࢭͤ͞ਖ਼͘͠ಈ࡞͢Δ͔ݕূͨ͠ • TiKVͷΠϯελϯεΛఀࢭͤ͞ɺmax-store- down-timeͷ࣌ؒܦաޙʹɺϦʔδϣϯϨϓϦΧ ͕Ϧόϥϯε͢Δ͜ͱ • ΫϥελʔͷTiKVΛݮΒͯ͠Ϧόϥϯε͢Δ ͜ͱ
• Placement DriverͷϦʔμʔมߋ
• ϦϦʔεఆΛ্ճΔϦΫΤετ͕དྷͨ ͕ɺ֤ίϯϙʔωϯτΛεέʔϧΞτ͢Δ ͜ͱͰແࣄՔಇ͠ଓ͚Δ͜ͱ͕Ͱ͖ͨ • ͓ਖ਼݄ͳͲҰ࣌తͳෛՙ͕ݟࠐ·ΕΔ߹ εέʔϧΞτ͢Δ͜ͱͰͳ͘ӡ༻Λଓ ͚Δ͜ͱ͕Ͱ͖͍ͯΔ • ࠓ·ͰαʔϏεແఀࢭͰӡ༻
• TiDBΠϯελϯεͷϊʔυো࣌αʔϏε Өڹͳ͔ͬͨ ϦϦʔε
Slow Query • Secondary IndexΛͬͨΫΤϦͰIndexͷઌ ಄ΧϥϜͰରΛߜΓɺ̎൪ͷΧϥϜͰ ιʔτ͢Δ • In۟ͷཁૉ͕Ұͭͷ߹sort͕ޮ͍͕ͨɺෳ ཁૉࢦఆ͞ΕΔͱsort͕ޮ͔ͳ͍
• ର͕ेສ݅ʹͳΔ͜ͱ͋ΓSlow Queryʹͳͬͯ͠·ͬͨ KEY `idx_1` (`community_id`,`time_series_id`,`user_id`) /* ҎԼͷΑ͏ͳΫΤϦͷ߹ʹkeep order:falseͱͳΔ */ explain analyze select /*+ USE_INDEX(`community_posts`, idx_1) */ * from community_posts where community_id IN (‘aaaa’, ‘bbbb’, ‘cccc’) and time_series_id < 1736935091883343234 order by time_series_id ASC LIMIT 30; +----------------------------------+---------+---------+ | id | estRows | actRows | +----------------------------------+---------+---------+ | TopN_9 | 30.00 | 30 | | └ ─ IndexLookUp_16 | 30.00 | 30 | | ├ ─ TopN_15(Build) | 30.00 | 30 | | │ └ ─ IndexRangeScan_13 | 1317.05 | 1265 | | └ ─ TableRowIDScan_14(Probe) | 30.00 | 30 | +----------------------------------+---------+---------+
Slow Query total_keys: εΩϟϯ͞ΕͨKeyͷ߹ܭɻݹ͍όʔδϣϯΛؚΉ total_process_keys: ॲཧ͞ΕͨKeyɻݹ͍όʔδϣϯΛؚ·ͳ͍ delete_skipped_count: আϚʔΧʔ͕͍ͭͨKeyͷ߹ܭ key_skipped_count: εΩοϓͨ͠Keyͷ߹ܭ
• ظؒͰUPDATEΛ܁Γฦ͔͢DELETE͢Δ έʔε • ෳͷόʔδϣϯ͕ҙਤͤͣੵ͞Εͨ • ಉ͡ςʔϒϧͷϨίʔυΛݕࡧ͢Δͱ͖ ʹSlow Query͕ൃੜ scan_detail: { total_process_keys: 1, total_process_keys_size: 102, total_keys: 26150, get_snapshot_time: 10.5µs, rocksdb: { delete_skipped_count: 179504, key_skipped_count: 205653, . . . }
Operation • ϦϦʔεޙɺTiDB Clusterͷӡ༻tiupίϚϯ υΛ࣮ͬͯࢪ • Scale out/Scale in •
Backup • Dump
ϨϓϦΧ εέδϡʔϧ Մ༻ੑͷ্ͷͨΊɺτϙϩδϥϕϧͷػೳΛ ͍TiKVϊʔυΛҟͳΔϥοΫʹஔ͍ͯ͠Δ EC2 PlacementGroupͰ࡞ͨ͠ύʔςΟγϣ ϯΛPD͕ཧղͰ͖ΔΑ͏ʹ location-labelsͷઃ ఆΛߦ͏ $
tiup ctl:v8.3.0 pd config set location-labels rack -u 10.20.3.13:2379 { "max-replicas": 3, "location-labels": "rack", "strictly-match-label": "false", "enable-placement-rules": "true", "enable-placement-rules-cache": "false", "isolation-level": "" }
Dumpling • tiup dumplingίϚϯυͰDBσʔλΛS3ʹΤ Ϋεϙʔτ(CSV) • ECSͷεέδϡʔϧυλεΫͱͯ͠ఆظత ʹ࣮ߦ͢Δ • Dumpͨ͠σʔλΛղੳ༻DBʢBigQuery)
ʹऔΓࠐΉ $ tiup dumpling -u username -P 4000 -h 10.20.1.244 -- filetype csv -t 8 -o “s3://dump-bucket/20250214" -r 20000 -F 256MiB --csv-output-dialect bigquery --csv-null-value "" --escape-backslash
Backup • tiup brίϚϯυͰϑϧόοΫΞοϓͱϩά όοΫΞοϓΛߦ͏ • ECSͷεέδϡʔϧυλεΫͱͯ͠ఆظత ʹ࣮ߦ͢Δ Ҿ༻ݩ: https://docs.pingcap.com/ja/tidb/stable/br-snapshot-architecture#architecture
Monitoring TiDB Dashboard - Ϋϥελશମͷ࣮ߦεςʔλ εʢQPSɺ࣮ߦ࣌ؒͳͲʣͱSQL࣮ߦ࣌ؒɺε ϩʔΫΤϦΛ֬ೝ͢Δ Grafana - TiDB,
TiKV, PDͷओཁͳϝτϦΫεͷ ࢹ - ϦʔδϣϯɺҟৗͳϦʔδϣϯ - εΩοϓ͞ΕͨΩʔɺআͣΈΩʔͷ
·ͱΊͱࠓޙͷల • গͳ͍ਓͷ։ൃνʔϜ͕ͩɺTiDB ClusterΛ༻ͨ͠αʔϏε։ൃͱӡ༻Λߦ͍͑ͯΔ • TiDBແఀࢭͰӡ༻ΛܧଓɺٸܹͳෛՙʹεέʔϧΞτͰରԠ͢Δ͜ͱ͕Ͱ͖ͨ • υΩϡϝϯτ͕๛ʹ͋Δ͜ͱࢹπʔϧ͕Ұ௨Γଐ͢Δ͜ͱগਓνʔϜʹ ͋Γ͕͍ͨ •
ࠓޙͷల • TiCDCΛ͍νΣϯδΠϕϯτΛϝοηʔδͱͯ͠ૹ৴͠ඇಉظॲཧʹ͍͍ͨʢSQS ͱඇಉظδϣϒͷ෦Λஔ͖͑ΔͨΊʣ
ΞϯέʔτճऀʹϊϕϧςΟΛϓϨθϯτʂ ϐϯΫ ωΠϏʔ άϨʔ ΠΤϩʔ ϒϧʔ άϦʔϯ ※λΦϧϋϯΧνͷΧϥʔͱຕʹݶΓ͕͍͟͝·͢ɻແ͘ͳΓ࣍ୈऴ͍ྃͨ͠·͢ͷͰྃ͝ঝ͍ͩ͘͞ɻ ※ϊϕϧςΟ͓ड͚औΓͷࡍɺճྃը໘ͷදࣔΛ͓ئ͍͠·͢ɻ ·ͨ
+ SCAN HERE!