Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How mixi2 Uses TiDB for SNS Scalability and Per...
Search
kanmo
February 14, 2025
Programming
33
12k
How mixi2 Uses TiDB for SNS Scalability and Performance
Developers Summit 2025 登壇資料
kanmo
February 14, 2025
Tweet
Share
More Decks by kanmo
See All by kanmo
Keep Phoenix App Productivity
kanmo
2
1.1k
ExAws
kanmo
1
540
Other Decks in Programming
See All in Programming
Pulsar2 を雰囲気で使ってみよう
anoken
0
230
sappoRo.R #12 初心者セッション
kosugitti
0
230
2,500万ユーザーを支えるSREチームの6年間のスクラムのカイゼン
honmarkhunt
6
5.1k
[JAWS-UG横浜 #79] re:Invent 2024 の DB アップデートは Multi-Region!
maroon1st
1
140
Amazon Q Developer Proで効率化するAPI開発入門
seike460
PRO
0
110
バックエンドのためのアプリ内課金入門 (サブスク編)
qnighy
8
1.7k
Immutable ActiveRecord
megane42
0
130
CI改善もDatadogとともに
taumu
0
110
ARA Ansible for the teams
kksat
0
150
DROBEの生成AI活用事例 with AWS
ippey
0
130
SRE、開発、QAが協業して挑んだリリースプロセス改革@SRE Kaigi 2025
nealle
3
4.2k
最近のVS Codeで気になるニュース 2025/01
74th
1
250
Featured
See All Featured
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Designing for Performance
lara
604
68k
The Invisible Side of Design
smashingmag
299
50k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
7.1k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
114
50k
Building a Scalable Design System with Sketch
lauravandoore
460
33k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
2.1k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
40
2k
Why Our Code Smells
bkeepers
PRO
335
57k
The Language of Interfaces
destraynor
156
24k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
132
33k
Bootstrapping a Software Product
garrettdimon
PRO
305
110k
Transcript
.*9* גࣜձࣾ.*9*ᇙ໌ल %FWFMPQFST4VNNJU ʮNJYJͷཪʯ 5J%#Ͱ࣮ݱ͢Δ4/4ͷεέʔ ϥϏϦςΟͱύϑΥʔϚϯε
mixi2ʹ͍ͭͯ • จςΩετSNS • ʮϑΥϩʔʯλΠϜϥΠϯͷଞɺSNS mixiʹ͋ͬͨίϛϡχςΟػೳΑΓ ϦΞϧλΠϜͳίϛϡχέʔγϣϯ ͚ͷΠϕϯτػೳ • αʔόGoɺϞόΠϧΫϥΠΞϯτ
FlutterΛ࠾༻ • ॳظͷػೳ։ൃ4ਓ(ϦϦʔε࣌7ਓ)
mixi2 architecture
DBͷબఆ SNSΛ։ൃ͢Δͱ͖ʹٻΊͨཁ݅ • ϦΫΤετɺϢʔβʔͷ૿ՃʹରԠͰ͖ΔεέʔϥϏϦςΟ • োൃੜ࣌ͷϑΣΠϧΦʔόʔ • ʢαʔϏεͱͯ͠ڐ༰Ͱ͖ΔʣϨΠςϯγʔ • τϥϯβΫγϣϯ
• ϝϯςφϯεΦϖϨʔγϣϯͷ༰қ͞ • ίετ໘ NewSQLʹ͍ͭͯಋೖݕ౼Λͨ͠
NewSQLͱ SQLΠϯλʔϑΣʔεΛ࣋ͪ ैདྷͷϦϨʔγϣφϧσʔλϕʔε͕࣋ͭτϥϯβΫγϣϯͱ ڧ͍߹ੑΛอূ͢Δࢄσʔλϕʔε TiDB, Spanner, CockroachDB, YugaByteDBͳͲ
NewSQLͷݕ౼ ͦΕͧΕͷಛੑΛൺֱ͠ɺࠓճTiDBΛ࠾༻ τϥϯβΫ γϣϯ ϨΠςϯγʔ োੑ ӡ༻ίετ Πϯϑϥඅ༻ 3%# 4IBSEJOH
̋ʢ˚ʣ ̋ ˚ ˚ ̋ /FX42- ̋ ˚ ̋ ̋ ˚
TiDBͷಛ • ׂ͝ͱͷίϯϙʔωϯτ͕ࢄஔ͞Ε ͨΞʔΩςΫνϟ • TiDB, TiKV, PD • ֤ίϯϙʔωϯτԽ͢Δ͜ͱʹΑΓ
ύϑΥʔϚϯεͱোੑΛߴΊΒΕΔ • SQL LayerͷTiDBαʔόਫฏํʹ ֦ுՄೳ • ΫϥελΛཧ͢ΔPDαʔόɺ Λ૿͢͜ͱͰՄ༻ੑΛߴΊΒΕΔ ʢ࠷̏ʣ Ҿ༻ݩ: https://docs.pingcap.com/tidb/stable/tidb-architecture
TiDB Storage (TiKV) • σʔλΛ֨ೲ͢ΔͷTiKVίϯϙʔωϯτ • σʔλʢϦʔδϣϯͱ͍͏୯ҐʣΛࣗಈͰෳ ͷϨϓϦΧͰཧ͞ΕΔ • ಉ͡σʔλ͕ࡾͭίϐʔ͞ΕΔ
• ߴՄ༻ੑΛ࣋ͪɺࣗಈϑΣΠϧΦʔόʔΛα ϙʔτ͢Δ • TiKVϊʔυΛΫϥελʹՃ͢Δ͜ͱͰ༰ྔ Λ૿͢͜ͱ͕Ͱ͖Δ • RDBͰߦ͍ͬͯͨσʔλංେԽʹ͏γϟʔ σΟϯάɺϚελʔԽͳͲ͕ෆཁʹͳΔ Ҿ༻ݩ: https://docs.pingcap.com/tidb/stable/tidb-storage
Backup • όοΫΞοϓˍϦετΞ • σʔλόοΫΞοϓฐࣾαʔϏεӡ ༻ʹ͓͍ͯඞਢཁ݅ • εφοϓγϣοτͱ૿ϩά͕औΕΔ • όοΫΞοϓ͔Βղੳ༻σʔλͷߏங
• Dumpͷෛՙ͕ແࢹͰ͖ͳ͍͘Β͍ʹ σʔλ͕૿Ճͨ͠߹ɺόοΫΞοϓ ͔Β෮ݩͨ͠Ϋϥελ͔ΒDWH༻ͷ σʔλ͕औΕΔΑ͏ʹ͢Δ Starting component br: /.tiup/components/br/v8.1.0/br restore full --pd 10.30.2.92:2379 --storage s3://kan-tidb-test-tiup-backup/ snapshot-20240712-3 --send-credentials-to-tikv=false Detail BR log in /tmp/br.log.2024-07-12T08.13.06Z Full Restore <------------------------------------------------------------------- ----------------------------------------------------------------> 100.00%
TiUP • ΫϥελʔͷߏஙɺScale out/Scale inɺBackupଐͷϕϯνπʔϧͳͲ ͯ͢ͷૢ࡞ͰtiupίϚϯυ͕ඇৗʹ ͍͔ͬͨ͢ • υΩϡϝϯτ๛ •
ϦϦʔεޙͷӡ༻ʹߩݙͯ͘͠Εͦ͏ tiup cluster scale-out test-cluster scaleout-tikv.yaml tikv_servers: - host: 10.20.1.206 config: server.labels: rack: r4 1. Ճ͢ΔίϯϙʔωϯτΛyamlʹఆٛ͢Δ 2. tiupίϚϯυΛ࣮ߦ͢Δ
PingCAP TiDBͰࠓճͷαʔϏεཁ݅ΛຬͨͤΔͱஅ͠࠾༻ ։ൃॳظϑΣʔζͰɺPingCAPࣾͱִिͰmtgΛ͠ͳ͕ΒਐΊΔ
Timeline • λΠϜϥΠϯʹɺϢʔβʔͷߘͨ͠λΠ ϜϥΠϯʢϢʔβʔλΠϜϥΠϯʣͱϗʔϜ ը໘ͷλΠϜϥΠϯ͕͋Δ • ϗʔϜλΠϜϥΠϯΛදࣔ͢ΔʹɺPOST ͨ͠ߘΛͰ͖ΔݶΓૣ͘ϑΥϩϫʔͷλΠ ϜϥΠϯʹಧ͚Δඞཁ͕͋Δ
Home Timeline Timeline • ࣄલʹϗʔϜλΠϜϥΠϯΛ࡞͢Δ • ߘऀͷϑΥϩϫʔ/ίϛϡχςΟΛ୳ ͠ɺϑΥϩϫʔຖͷλΠϜϥΠϯςʔ ϒϧʹϙετΛॻ͖ࠐΉ ϙετIDΛ֤Ϣʔβʔ͝ͱʹ৴͍ͯ͠Δ
Post API Redis Redis Redis Find Followers Find Community Members Timeline API
Timeline Tables (Redis) • ϢʔβʔͷϗʔϜλΠϜϥΠϯΛࣄલʹredis ʹ࡞͢Δ • ͜ͷใΛݩʹDBͷใͱΈ߹Θͤͯ ϗʔϜλΠϜϥΠϯΛߏங͢Δ •
ҰఆͷϙετใΛϢʔβʔຖʹอ࣋͢Δ • ࢀর͞ΕΔՄೳੑ͕͘ͳͬͨϙετҰ ఆִؒͰআ͍ͯ͠Δ post_id user: A post_id post_id post_id user: B post_id post_id post_id user: C post_id post_id Redis Sorted Set
Timeline Tables (DB) • TiDBʹશͯͷϙετͱϢʔβʔλΠϜ ϥΠϯͷϙετΛه • ϗʔϜλΠϜϥΠϯDBʹ࡞ΒΕ ͳ͍ •
ϗʔϜλΠϜϥΠϯΛ࡞ΔͨΊͷ ใʢϑΥϩϫʔɺίϛϡχςΟϢʔ βʔʣ͕ଘࡏ͢Δ posts post_id user_id community_id text user_posts post_id user_id community_ users community_id user_id followings following_id user_id
ෛՙςετ ϦϦʔεલʹෛՙݕূΛ࣮ࢪɻAPIͷϨΠςϯγʔͷඪΛ100msʹઃఆ • Writeੑೳ • දతͳRPC͕10000RPSΛग़ͤΔͱ͜Ζ·Ͱݕূ • Readੑೳ • ߘ͕ϑΥϩϫʔͷλΠϜϥΠϯʹදࣔ͞ΕΔ·Ͱͷ͕࣌ؒ5ඵ͔Β10ඵʹͳΔ͜ͱ
Λඪʹௐ • αʔϏε໊͔Β͋ΔఔϢʔβʔ͕དྷΔ͜ͱ͕ఆͰ͖ͨ
Locust - Boomer • ෛՙςετ༻ͷڥΛ༻ҙͯ͠LocustΛ͍ݕূ • ฒྻΛେ͖͍ͨ͘͠ͷͰgoͷworker࣮ ͷboomerΛ༻ͨ͠ • ߈ܸΫϥΠΞϯτͷεέʔϧΞτɺεέʔ
ϧΞοϓΛසൟʹߦ͏ͨΊɺk8s্Ͱಈ͔ͨ͠ • γφϦΦϢʔβʔ࡞͔ΒೝূΛߦ͍ɺϑΥ ϩʔɺϙετɺϥΠΫͳͲͷجຊతͳಈ࡞Λશͯ ࣮ࢪ
ෛՙςετ • ॻ͖ࠐΈੑೳTiDB, TiKVͷεέʔϧΞτͰఆ ௨Γʹੑೳ্ͨ͠ • Ұ෦ɺॏ͍ॲཧʢݕࡧΠϯσοΫεͷॻ͖ࠐ ΈͳͲʣΛඇಉظॲཧʹมߋ • pprofΛ͍ॏ͍ॲཧΛ୳ͨ͠
• ಡΈࠐΈॲཧআॲཧࠓճͷඪΛୡ͢ Δ্Ͱͳ͠ • ߴෛՙ࣌ʹෳͷϑΥϩϫʔ͕ಉ࣌ʹϑΥϩʔ͢ Δɺ͍͍ͶΛ͢ΔέʔεͰසൟϩοΫͪͷλΠϜ Ξτ͕ൃੜ
োςετ Locust͔ΒෛՙΛ͔͚ͳ͕Βtidbͷ֤ίϯϙʔωϯ τΛఀࢭͤ͞ਖ਼͘͠ಈ࡞͢Δ͔ݕূͨ͠ • TiKVͷΠϯελϯεΛఀࢭͤ͞ɺmax-store- down-timeͷ࣌ؒܦաޙʹɺϦʔδϣϯϨϓϦΧ ͕Ϧόϥϯε͢Δ͜ͱ • ΫϥελʔͷTiKVΛݮΒͯ͠Ϧόϥϯε͢Δ ͜ͱ
• Placement DriverͷϦʔμʔมߋ
• ϦϦʔεఆΛ্ճΔϦΫΤετ͕དྷͨ ͕ɺ֤ίϯϙʔωϯτΛεέʔϧΞτ͢Δ ͜ͱͰແࣄՔಇ͠ଓ͚Δ͜ͱ͕Ͱ͖ͨ • ͓ਖ਼݄ͳͲҰ࣌తͳෛՙ͕ݟࠐ·ΕΔ߹ εέʔϧΞτ͢Δ͜ͱͰͳ͘ӡ༻Λଓ ͚Δ͜ͱ͕Ͱ͖͍ͯΔ • ࠓ·ͰαʔϏεແఀࢭͰӡ༻
• TiDBΠϯελϯεͷϊʔυো࣌αʔϏε Өڹͳ͔ͬͨ ϦϦʔε
Slow Query • Secondary IndexΛͬͨΫΤϦͰIndexͷઌ ಄ΧϥϜͰରΛߜΓɺ̎൪ͷΧϥϜͰ ιʔτ͢Δ • In۟ͷཁૉ͕Ұͭͷ߹sort͕ޮ͍͕ͨɺෳ ཁૉࢦఆ͞ΕΔͱsort͕ޮ͔ͳ͍
• ର͕ेສ݅ʹͳΔ͜ͱ͋ΓSlow Queryʹͳͬͯ͠·ͬͨ KEY `idx_1` (`community_id`,`time_series_id`,`user_id`) /* ҎԼͷΑ͏ͳΫΤϦͷ߹ʹkeep order:falseͱͳΔ */ explain analyze select /*+ USE_INDEX(`community_posts`, idx_1) */ * from community_posts where community_id IN (‘aaaa’, ‘bbbb’, ‘cccc’) and time_series_id < 1736935091883343234 order by time_series_id ASC LIMIT 30; +----------------------------------+---------+---------+ | id | estRows | actRows | +----------------------------------+---------+---------+ | TopN_9 | 30.00 | 30 | | └ ─ IndexLookUp_16 | 30.00 | 30 | | ├ ─ TopN_15(Build) | 30.00 | 30 | | │ └ ─ IndexRangeScan_13 | 1317.05 | 1265 | | └ ─ TableRowIDScan_14(Probe) | 30.00 | 30 | +----------------------------------+---------+---------+
Slow Query total_keys: εΩϟϯ͞ΕͨKeyͷ߹ܭɻݹ͍όʔδϣϯΛؚΉ total_process_keys: ॲཧ͞ΕͨKeyɻݹ͍όʔδϣϯΛؚ·ͳ͍ delete_skipped_count: আϚʔΧʔ͕͍ͭͨKeyͷ߹ܭ key_skipped_count: εΩοϓͨ͠Keyͷ߹ܭ
• ظؒͰUPDATEΛ܁Γฦ͔͢DELETE͢Δ έʔε • ෳͷόʔδϣϯ͕ҙਤͤͣੵ͞Εͨ • ಉ͡ςʔϒϧͷϨίʔυΛݕࡧ͢Δͱ͖ ʹSlow Query͕ൃੜ scan_detail: { total_process_keys: 1, total_process_keys_size: 102, total_keys: 26150, get_snapshot_time: 10.5µs, rocksdb: { delete_skipped_count: 179504, key_skipped_count: 205653, . . . }
Operation • ϦϦʔεޙɺTiDB Clusterͷӡ༻tiupίϚϯ υΛ࣮ͬͯࢪ • Scale out/Scale in •
Backup • Dump
ϨϓϦΧ εέδϡʔϧ Մ༻ੑͷ্ͷͨΊɺτϙϩδϥϕϧͷػೳΛ ͍TiKVϊʔυΛҟͳΔϥοΫʹஔ͍ͯ͠Δ EC2 PlacementGroupͰ࡞ͨ͠ύʔςΟγϣ ϯΛPD͕ཧղͰ͖ΔΑ͏ʹ location-labelsͷઃ ఆΛߦ͏ $
tiup ctl:v8.3.0 pd config set location-labels rack -u 10.20.3.13:2379 { "max-replicas": 3, "location-labels": "rack", "strictly-match-label": "false", "enable-placement-rules": "true", "enable-placement-rules-cache": "false", "isolation-level": "" }
Dumpling • tiup dumplingίϚϯυͰDBσʔλΛS3ʹΤ Ϋεϙʔτ(CSV) • ECSͷεέδϡʔϧυλεΫͱͯ͠ఆظత ʹ࣮ߦ͢Δ • Dumpͨ͠σʔλΛղੳ༻DBʢBigQuery)
ʹऔΓࠐΉ $ tiup dumpling -u username -P 4000 -h 10.20.1.244 -- filetype csv -t 8 -o “s3://dump-bucket/20250214" -r 20000 -F 256MiB --csv-output-dialect bigquery --csv-null-value "" --escape-backslash
Backup • tiup brίϚϯυͰϑϧόοΫΞοϓͱϩά όοΫΞοϓΛߦ͏ • ECSͷεέδϡʔϧυλεΫͱͯ͠ఆظత ʹ࣮ߦ͢Δ Ҿ༻ݩ: https://docs.pingcap.com/ja/tidb/stable/br-snapshot-architecture#architecture
Monitoring TiDB Dashboard - Ϋϥελશମͷ࣮ߦεςʔλ εʢQPSɺ࣮ߦ࣌ؒͳͲʣͱSQL࣮ߦ࣌ؒɺε ϩʔΫΤϦΛ֬ೝ͢Δ Grafana - TiDB,
TiKV, PDͷओཁͳϝτϦΫεͷ ࢹ - ϦʔδϣϯɺҟৗͳϦʔδϣϯ - εΩοϓ͞ΕͨΩʔɺআͣΈΩʔͷ
·ͱΊͱࠓޙͷల • গͳ͍ਓͷ։ൃνʔϜ͕ͩɺTiDB ClusterΛ༻ͨ͠αʔϏε։ൃͱӡ༻Λߦ͍͑ͯΔ • TiDBແఀࢭͰӡ༻ΛܧଓɺٸܹͳෛՙʹεέʔϧΞτͰରԠ͢Δ͜ͱ͕Ͱ͖ͨ • υΩϡϝϯτ͕๛ʹ͋Δ͜ͱࢹπʔϧ͕Ұ௨Γଐ͢Δ͜ͱগਓνʔϜʹ ͋Γ͕͍ͨ •
ࠓޙͷల • TiCDCΛ͍νΣϯδΠϕϯτΛϝοηʔδͱͯ͠ૹ৴͠ඇಉظॲཧʹ͍͍ͨʢSQS ͱඇಉظδϣϒͷ෦Λஔ͖͑ΔͨΊʣ
ΞϯέʔτճऀʹϊϕϧςΟΛϓϨθϯτʂ ϐϯΫ ωΠϏʔ άϨʔ ΠΤϩʔ ϒϧʔ άϦʔϯ ※λΦϧϋϯΧνͷΧϥʔͱຕʹݶΓ͕͍͟͝·͢ɻແ͘ͳΓ࣍ୈऴ͍ྃͨ͠·͢ͷͰྃ͝ঝ͍ͩ͘͞ɻ ※ϊϕϧςΟ͓ड͚औΓͷࡍɺճྃը໘ͷදࣔΛ͓ئ͍͠·͢ɻ ·ͨ
+ SCAN HERE!