Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How mixi2 Uses TiDB for SNS Scalability and Per...
Search
kanmo
February 14, 2025
Programming
43
34k
How mixi2 Uses TiDB for SNS Scalability and Performance
Developers Summit 2025 登壇資料
kanmo
February 14, 2025
Tweet
Share
More Decks by kanmo
See All by kanmo
Keep Phoenix App Productivity
kanmo
2
1.2k
ExAws
kanmo
1
570
Other Decks in Programming
See All in Programming
go test -json そして testing.T.Attr / Kyoto.go #63
utgwkk
1
190
マイコンでもRustのtestがしたい その2/KernelVM Tokyo 18
tnishinaga
2
2.4k
Ruby Parser progress report 2025
yui_knk
1
250
More Approvers for Greater OSS and Japan Community
tkikuc
1
100
MCPとデザインシステムに立脚したデザインと実装の融合
yukukotani
1
740
Zendeskのチケットを Amazon Bedrockで 解析した
ryokosuge
3
230
TDD 実践ミニトーク
contour_gara
1
260
AIコーディングAgentとの向き合い方
eycjur
0
250
Nuances on Kubernetes - RubyConf Taiwan 2025
envek
0
220
AIレビュアーをスケールさせるには / Scaling AI Reviewers
technuma
2
230
RDoc meets YARD
okuramasafumi
4
160
A Gopher's Guide to Vibe Coding
danicat
0
190
Featured
See All Featured
Java REST API Framework Comparison - PWX 2021
mraible
33
8.8k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.5k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Building Applications with DynamoDB
mza
96
6.6k
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.4k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
110
20k
Navigating Team Friction
lara
189
15k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.7k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Fireside Chat
paigeccino
39
3.6k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
6.1k
Transcript
.*9* גࣜձࣾ.*9*ᇙ໌ल %FWFMPQFST4VNNJU ʮNJYJͷཪʯ 5J%#Ͱ࣮ݱ͢Δ4/4ͷεέʔ ϥϏϦςΟͱύϑΥʔϚϯε
mixi2ʹ͍ͭͯ • จςΩετSNS • ʮϑΥϩʔʯλΠϜϥΠϯͷଞɺSNS mixiʹ͋ͬͨίϛϡχςΟػೳΑΓ ϦΞϧλΠϜͳίϛϡχέʔγϣϯ ͚ͷΠϕϯτػೳ • αʔόGoɺϞόΠϧΫϥΠΞϯτ
FlutterΛ࠾༻ • ॳظͷػೳ։ൃ4ਓ(ϦϦʔε࣌7ਓ)
mixi2 architecture
DBͷબఆ SNSΛ։ൃ͢Δͱ͖ʹٻΊͨཁ݅ • ϦΫΤετɺϢʔβʔͷ૿ՃʹରԠͰ͖ΔεέʔϥϏϦςΟ • োൃੜ࣌ͷϑΣΠϧΦʔόʔ • ʢαʔϏεͱͯ͠ڐ༰Ͱ͖ΔʣϨΠςϯγʔ • τϥϯβΫγϣϯ
• ϝϯςφϯεΦϖϨʔγϣϯͷ༰қ͞ • ίετ໘ NewSQLʹ͍ͭͯಋೖݕ౼Λͨ͠
NewSQLͱ SQLΠϯλʔϑΣʔεΛ࣋ͪ ैདྷͷϦϨʔγϣφϧσʔλϕʔε͕࣋ͭτϥϯβΫγϣϯͱ ڧ͍߹ੑΛอূ͢Δࢄσʔλϕʔε TiDB, Spanner, CockroachDB, YugaByteDBͳͲ
NewSQLͷݕ౼ ͦΕͧΕͷಛੑΛൺֱ͠ɺࠓճTiDBΛ࠾༻ τϥϯβΫ γϣϯ ϨΠςϯγʔ োੑ ӡ༻ίετ Πϯϑϥඅ༻ 3%# 4IBSEJOH
̋ʢ˚ʣ ̋ ˚ ˚ ̋ /FX42- ̋ ˚ ̋ ̋ ˚
TiDBͷಛ • ׂ͝ͱͷίϯϙʔωϯτ͕ࢄஔ͞Ε ͨΞʔΩςΫνϟ • TiDB, TiKV, PD • ֤ίϯϙʔωϯτԽ͢Δ͜ͱʹΑΓ
ύϑΥʔϚϯεͱোੑΛߴΊΒΕΔ • SQL LayerͷTiDBαʔόਫฏํʹ ֦ுՄೳ • ΫϥελΛཧ͢ΔPDαʔόɺ Λ૿͢͜ͱͰՄ༻ੑΛߴΊΒΕΔ ʢ࠷̏ʣ Ҿ༻ݩ: https://docs.pingcap.com/tidb/stable/tidb-architecture
TiDB Storage (TiKV) • σʔλΛ֨ೲ͢ΔͷTiKVίϯϙʔωϯτ • σʔλʢϦʔδϣϯͱ͍͏୯ҐʣΛࣗಈͰෳ ͷϨϓϦΧͰཧ͞ΕΔ • ಉ͡σʔλ͕ࡾͭίϐʔ͞ΕΔ
• ߴՄ༻ੑΛ࣋ͪɺࣗಈϑΣΠϧΦʔόʔΛα ϙʔτ͢Δ • TiKVϊʔυΛΫϥελʹՃ͢Δ͜ͱͰ༰ྔ Λ૿͢͜ͱ͕Ͱ͖Δ • RDBͰߦ͍ͬͯͨσʔλංେԽʹ͏γϟʔ σΟϯάɺϚελʔԽͳͲ͕ෆཁʹͳΔ Ҿ༻ݩ: https://docs.pingcap.com/tidb/stable/tidb-storage
Backup • όοΫΞοϓˍϦετΞ • σʔλόοΫΞοϓฐࣾαʔϏεӡ ༻ʹ͓͍ͯඞਢཁ݅ • εφοϓγϣοτͱ૿ϩά͕औΕΔ • όοΫΞοϓ͔Βղੳ༻σʔλͷߏங
• Dumpͷෛՙ͕ແࢹͰ͖ͳ͍͘Β͍ʹ σʔλ͕૿Ճͨ͠߹ɺόοΫΞοϓ ͔Β෮ݩͨ͠Ϋϥελ͔ΒDWH༻ͷ σʔλ͕औΕΔΑ͏ʹ͢Δ Starting component br: /.tiup/components/br/v8.1.0/br restore full --pd 10.30.2.92:2379 --storage s3://kan-tidb-test-tiup-backup/ snapshot-20240712-3 --send-credentials-to-tikv=false Detail BR log in /tmp/br.log.2024-07-12T08.13.06Z Full Restore <------------------------------------------------------------------- ----------------------------------------------------------------> 100.00%
TiUP • ΫϥελʔͷߏஙɺScale out/Scale inɺBackupଐͷϕϯνπʔϧͳͲ ͯ͢ͷૢ࡞ͰtiupίϚϯυ͕ඇৗʹ ͍͔ͬͨ͢ • υΩϡϝϯτ๛ •
ϦϦʔεޙͷӡ༻ʹߩݙͯ͘͠Εͦ͏ tiup cluster scale-out test-cluster scaleout-tikv.yaml tikv_servers: - host: 10.20.1.206 config: server.labels: rack: r4 1. Ճ͢ΔίϯϙʔωϯτΛyamlʹఆٛ͢Δ 2. tiupίϚϯυΛ࣮ߦ͢Δ
PingCAP TiDBͰࠓճͷαʔϏεཁ݅ΛຬͨͤΔͱஅ͠࠾༻ ։ൃॳظϑΣʔζͰɺPingCAPࣾͱִिͰmtgΛ͠ͳ͕ΒਐΊΔ
Timeline • λΠϜϥΠϯʹɺϢʔβʔͷߘͨ͠λΠ ϜϥΠϯʢϢʔβʔλΠϜϥΠϯʣͱϗʔϜ ը໘ͷλΠϜϥΠϯ͕͋Δ • ϗʔϜλΠϜϥΠϯΛදࣔ͢ΔʹɺPOST ͨ͠ߘΛͰ͖ΔݶΓૣ͘ϑΥϩϫʔͷλΠ ϜϥΠϯʹಧ͚Δඞཁ͕͋Δ
Home Timeline Timeline • ࣄલʹϗʔϜλΠϜϥΠϯΛ࡞͢Δ • ߘऀͷϑΥϩϫʔ/ίϛϡχςΟΛ୳ ͠ɺϑΥϩϫʔຖͷλΠϜϥΠϯςʔ ϒϧʹϙετΛॻ͖ࠐΉ ϙετIDΛ֤Ϣʔβʔ͝ͱʹ৴͍ͯ͠Δ
Post API Redis Redis Redis Find Followers Find Community Members Timeline API
Timeline Tables (Redis) • ϢʔβʔͷϗʔϜλΠϜϥΠϯΛࣄલʹredis ʹ࡞͢Δ • ͜ͷใΛݩʹDBͷใͱΈ߹Θͤͯ ϗʔϜλΠϜϥΠϯΛߏங͢Δ •
ҰఆͷϙετใΛϢʔβʔຖʹอ࣋͢Δ • ࢀর͞ΕΔՄೳੑ͕͘ͳͬͨϙετҰ ఆִؒͰআ͍ͯ͠Δ post_id user: A post_id post_id post_id user: B post_id post_id post_id user: C post_id post_id Redis Sorted Set
Timeline Tables (DB) • TiDBʹશͯͷϙετͱϢʔβʔλΠϜ ϥΠϯͷϙετΛه • ϗʔϜλΠϜϥΠϯDBʹ࡞ΒΕ ͳ͍ •
ϗʔϜλΠϜϥΠϯΛ࡞ΔͨΊͷ ใʢϑΥϩϫʔɺίϛϡχςΟϢʔ βʔʣ͕ଘࡏ͢Δ posts post_id user_id community_id text user_posts post_id user_id community_ users community_id user_id followings following_id user_id
ෛՙςετ ϦϦʔεલʹෛՙݕূΛ࣮ࢪɻAPIͷϨΠςϯγʔͷඪΛ100msʹઃఆ • Writeੑೳ • දతͳRPC͕10000RPSΛग़ͤΔͱ͜Ζ·Ͱݕূ • Readੑೳ • ߘ͕ϑΥϩϫʔͷλΠϜϥΠϯʹදࣔ͞ΕΔ·Ͱͷ͕࣌ؒ5ඵ͔Β10ඵʹͳΔ͜ͱ
Λඪʹௐ • αʔϏε໊͔Β͋ΔఔϢʔβʔ͕དྷΔ͜ͱ͕ఆͰ͖ͨ
Locust - Boomer • ෛՙςετ༻ͷڥΛ༻ҙͯ͠LocustΛ͍ݕূ • ฒྻΛେ͖͍ͨ͘͠ͷͰgoͷworker࣮ ͷboomerΛ༻ͨ͠ • ߈ܸΫϥΠΞϯτͷεέʔϧΞτɺεέʔ
ϧΞοϓΛසൟʹߦ͏ͨΊɺk8s্Ͱಈ͔ͨ͠ • γφϦΦϢʔβʔ࡞͔ΒೝূΛߦ͍ɺϑΥ ϩʔɺϙετɺϥΠΫͳͲͷجຊతͳಈ࡞Λશͯ ࣮ࢪ
ෛՙςετ • ॻ͖ࠐΈੑೳTiDB, TiKVͷεέʔϧΞτͰఆ ௨Γʹੑೳ্ͨ͠ • Ұ෦ɺॏ͍ॲཧʢݕࡧΠϯσοΫεͷॻ͖ࠐ ΈͳͲʣΛඇಉظॲཧʹมߋ • pprofΛ͍ॏ͍ॲཧΛ୳ͨ͠
• ಡΈࠐΈॲཧআॲཧࠓճͷඪΛୡ͢ Δ্Ͱͳ͠ • ߴෛՙ࣌ʹෳͷϑΥϩϫʔ͕ಉ࣌ʹϑΥϩʔ͢ Δɺ͍͍ͶΛ͢ΔέʔεͰසൟϩοΫͪͷλΠϜ Ξτ͕ൃੜ
োςετ Locust͔ΒෛՙΛ͔͚ͳ͕Βtidbͷ֤ίϯϙʔωϯ τΛఀࢭͤ͞ਖ਼͘͠ಈ࡞͢Δ͔ݕূͨ͠ • TiKVͷΠϯελϯεΛఀࢭͤ͞ɺmax-store- down-timeͷ࣌ؒܦաޙʹɺϦʔδϣϯϨϓϦΧ ͕Ϧόϥϯε͢Δ͜ͱ • ΫϥελʔͷTiKVΛݮΒͯ͠Ϧόϥϯε͢Δ ͜ͱ
• Placement DriverͷϦʔμʔมߋ
• ϦϦʔεఆΛ্ճΔϦΫΤετ͕དྷͨ ͕ɺ֤ίϯϙʔωϯτΛεέʔϧΞτ͢Δ ͜ͱͰແࣄՔಇ͠ଓ͚Δ͜ͱ͕Ͱ͖ͨ • ͓ਖ਼݄ͳͲҰ࣌తͳෛՙ͕ݟࠐ·ΕΔ߹ εέʔϧΞτ͢Δ͜ͱͰͳ͘ӡ༻Λଓ ͚Δ͜ͱ͕Ͱ͖͍ͯΔ • ࠓ·ͰαʔϏεແఀࢭͰӡ༻
• TiDBΠϯελϯεͷϊʔυো࣌αʔϏε Өڹͳ͔ͬͨ ϦϦʔε
Slow Query • Secondary IndexΛͬͨΫΤϦͰIndexͷઌ ಄ΧϥϜͰରΛߜΓɺ̎൪ͷΧϥϜͰ ιʔτ͢Δ • In۟ͷཁૉ͕Ұͭͷ߹sort͕ޮ͍͕ͨɺෳ ཁૉࢦఆ͞ΕΔͱsort͕ޮ͔ͳ͍
• ର͕ेສ݅ʹͳΔ͜ͱ͋ΓSlow Queryʹͳͬͯ͠·ͬͨ KEY `idx_1` (`community_id`,`time_series_id`,`user_id`) /* ҎԼͷΑ͏ͳΫΤϦͷ߹ʹkeep order:falseͱͳΔ */ explain analyze select /*+ USE_INDEX(`community_posts`, idx_1) */ * from community_posts where community_id IN (‘aaaa’, ‘bbbb’, ‘cccc’) and time_series_id < 1736935091883343234 order by time_series_id ASC LIMIT 30; +----------------------------------+---------+---------+ | id | estRows | actRows | +----------------------------------+---------+---------+ | TopN_9 | 30.00 | 30 | | └ ─ IndexLookUp_16 | 30.00 | 30 | | ├ ─ TopN_15(Build) | 30.00 | 30 | | │ └ ─ IndexRangeScan_13 | 1317.05 | 1265 | | └ ─ TableRowIDScan_14(Probe) | 30.00 | 30 | +----------------------------------+---------+---------+
Slow Query total_keys: εΩϟϯ͞ΕͨKeyͷ߹ܭɻݹ͍όʔδϣϯΛؚΉ total_process_keys: ॲཧ͞ΕͨKeyɻݹ͍όʔδϣϯΛؚ·ͳ͍ delete_skipped_count: আϚʔΧʔ͕͍ͭͨKeyͷ߹ܭ key_skipped_count: εΩοϓͨ͠Keyͷ߹ܭ
• ظؒͰUPDATEΛ܁Γฦ͔͢DELETE͢Δ έʔε • ෳͷόʔδϣϯ͕ҙਤͤͣੵ͞Εͨ • ಉ͡ςʔϒϧͷϨίʔυΛݕࡧ͢Δͱ͖ ʹSlow Query͕ൃੜ scan_detail: { total_process_keys: 1, total_process_keys_size: 102, total_keys: 26150, get_snapshot_time: 10.5µs, rocksdb: { delete_skipped_count: 179504, key_skipped_count: 205653, . . . }
Operation • ϦϦʔεޙɺTiDB Clusterͷӡ༻tiupίϚϯ υΛ࣮ͬͯࢪ • Scale out/Scale in •
Backup • Dump
ϨϓϦΧ εέδϡʔϧ Մ༻ੑͷ্ͷͨΊɺτϙϩδϥϕϧͷػೳΛ ͍TiKVϊʔυΛҟͳΔϥοΫʹஔ͍ͯ͠Δ EC2 PlacementGroupͰ࡞ͨ͠ύʔςΟγϣ ϯΛPD͕ཧղͰ͖ΔΑ͏ʹ location-labelsͷઃ ఆΛߦ͏ $
tiup ctl:v8.3.0 pd config set location-labels rack -u 10.20.3.13:2379 { "max-replicas": 3, "location-labels": "rack", "strictly-match-label": "false", "enable-placement-rules": "true", "enable-placement-rules-cache": "false", "isolation-level": "" }
Dumpling • tiup dumplingίϚϯυͰDBσʔλΛS3ʹΤ Ϋεϙʔτ(CSV) • ECSͷεέδϡʔϧυλεΫͱͯ͠ఆظత ʹ࣮ߦ͢Δ • Dumpͨ͠σʔλΛղੳ༻DBʢBigQuery)
ʹऔΓࠐΉ $ tiup dumpling -u username -P 4000 -h 10.20.1.244 -- filetype csv -t 8 -o “s3://dump-bucket/20250214" -r 20000 -F 256MiB --csv-output-dialect bigquery --csv-null-value "" --escape-backslash
Backup • tiup brίϚϯυͰϑϧόοΫΞοϓͱϩά όοΫΞοϓΛߦ͏ • ECSͷεέδϡʔϧυλεΫͱͯ͠ఆظత ʹ࣮ߦ͢Δ Ҿ༻ݩ: https://docs.pingcap.com/ja/tidb/stable/br-snapshot-architecture#architecture
Monitoring TiDB Dashboard - Ϋϥελશମͷ࣮ߦεςʔλ εʢQPSɺ࣮ߦ࣌ؒͳͲʣͱSQL࣮ߦ࣌ؒɺε ϩʔΫΤϦΛ֬ೝ͢Δ Grafana - TiDB,
TiKV, PDͷओཁͳϝτϦΫεͷ ࢹ - ϦʔδϣϯɺҟৗͳϦʔδϣϯ - εΩοϓ͞ΕͨΩʔɺআͣΈΩʔͷ
·ͱΊͱࠓޙͷల • গͳ͍ਓͷ։ൃνʔϜ͕ͩɺTiDB ClusterΛ༻ͨ͠αʔϏε։ൃͱӡ༻Λߦ͍͑ͯΔ • TiDBແఀࢭͰӡ༻ΛܧଓɺٸܹͳෛՙʹεέʔϧΞτͰରԠ͢Δ͜ͱ͕Ͱ͖ͨ • υΩϡϝϯτ͕๛ʹ͋Δ͜ͱࢹπʔϧ͕Ұ௨Γଐ͢Δ͜ͱগਓνʔϜʹ ͋Γ͕͍ͨ •
ࠓޙͷల • TiCDCΛ͍νΣϯδΠϕϯτΛϝοηʔδͱͯ͠ૹ৴͠ඇಉظॲཧʹ͍͍ͨʢSQS ͱඇಉظδϣϒͷ෦Λஔ͖͑ΔͨΊʣ
ΞϯέʔτճऀʹϊϕϧςΟΛϓϨθϯτʂ ϐϯΫ ωΠϏʔ άϨʔ ΠΤϩʔ ϒϧʔ άϦʔϯ ※λΦϧϋϯΧνͷΧϥʔͱຕʹݶΓ͕͍͟͝·͢ɻແ͘ͳΓ࣍ୈऴ͍ྃͨ͠·͢ͷͰྃ͝ঝ͍ͩ͘͞ɻ ※ϊϕϧςΟ͓ड͚औΓͷࡍɺճྃը໘ͷදࣔΛ͓ئ͍͠·͢ɻ ·ͨ
+ SCAN HERE!