Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How mixi2 Uses TiDB for SNS Scalability and Per...
Search
kanmo
February 14, 2025
Programming
38
15k
How mixi2 Uses TiDB for SNS Scalability and Performance
Developers Summit 2025 登壇資料
kanmo
February 14, 2025
Tweet
Share
More Decks by kanmo
See All by kanmo
Keep Phoenix App Productivity
kanmo
2
1.1k
ExAws
kanmo
1
540
Other Decks in Programming
See All in Programming
Multi Step Form, Decentralized Autonomous Organization
pumpkiinbell
1
760
PRレビューのお供にDanger
stoticdev
1
130
一休.com のログイン体験を支える技術 〜Web Components x Vue.js 活用事例と最適化について〜
atsumim
0
570
XStateを用いた堅牢なReact Components設計~複雑なClient Stateをシンプルに~ @React Tokyo ミートアップ #2
kfurusho
1
920
Flutter × Firebase Genkit で加速する生成 AI アプリ開発
coborinai
0
160
Immutable ActiveRecord
megane42
0
140
Grafana Loki によるサーバログのコスト削減
mot_techtalk
1
130
第3回 Snowflake 中部ユーザ会- dbt × Snowflake ハンズオン
hoto17296
4
370
Lottieアニメーションをカスタマイズしてみた
tahia910
0
130
ファインディの テックブログ爆誕までの軌跡
starfish719
2
1.1k
Honoとフロントエンドの 型安全性について
yodaka
7
1.3k
Rubyで始める関数型ドメインモデリング
shogo_tksk
0
120
Featured
See All Featured
Docker and Python
trallard
44
3.3k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
366
25k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Why Our Code Smells
bkeepers
PRO
336
57k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
193
16k
Build The Right Thing And Hit Your Dates
maggiecrowley
34
2.5k
Building Flexible Design Systems
yeseniaperezcruz
328
38k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.7k
Java REST API Framework Comparison - PWX 2021
mraible
28
8.4k
GraphQLの誤解/rethinking-graphql
sonatard
68
10k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Transcript
.*9* גࣜձࣾ.*9*ᇙ໌ल %FWFMPQFST4VNNJU ʮNJYJͷཪʯ 5J%#Ͱ࣮ݱ͢Δ4/4ͷεέʔ ϥϏϦςΟͱύϑΥʔϚϯε
mixi2ʹ͍ͭͯ • จςΩετSNS • ʮϑΥϩʔʯλΠϜϥΠϯͷଞɺSNS mixiʹ͋ͬͨίϛϡχςΟػೳΑΓ ϦΞϧλΠϜͳίϛϡχέʔγϣϯ ͚ͷΠϕϯτػೳ • αʔόGoɺϞόΠϧΫϥΠΞϯτ
FlutterΛ࠾༻ • ॳظͷػೳ։ൃ4ਓ(ϦϦʔε࣌7ਓ)
mixi2 architecture
DBͷબఆ SNSΛ։ൃ͢Δͱ͖ʹٻΊͨཁ݅ • ϦΫΤετɺϢʔβʔͷ૿ՃʹରԠͰ͖ΔεέʔϥϏϦςΟ • োൃੜ࣌ͷϑΣΠϧΦʔόʔ • ʢαʔϏεͱͯ͠ڐ༰Ͱ͖ΔʣϨΠςϯγʔ • τϥϯβΫγϣϯ
• ϝϯςφϯεΦϖϨʔγϣϯͷ༰қ͞ • ίετ໘ NewSQLʹ͍ͭͯಋೖݕ౼Λͨ͠
NewSQLͱ SQLΠϯλʔϑΣʔεΛ࣋ͪ ैདྷͷϦϨʔγϣφϧσʔλϕʔε͕࣋ͭτϥϯβΫγϣϯͱ ڧ͍߹ੑΛอূ͢Δࢄσʔλϕʔε TiDB, Spanner, CockroachDB, YugaByteDBͳͲ
NewSQLͷݕ౼ ͦΕͧΕͷಛੑΛൺֱ͠ɺࠓճTiDBΛ࠾༻ τϥϯβΫ γϣϯ ϨΠςϯγʔ োੑ ӡ༻ίετ Πϯϑϥඅ༻ 3%# 4IBSEJOH
̋ʢ˚ʣ ̋ ˚ ˚ ̋ /FX42- ̋ ˚ ̋ ̋ ˚
TiDBͷಛ • ׂ͝ͱͷίϯϙʔωϯτ͕ࢄஔ͞Ε ͨΞʔΩςΫνϟ • TiDB, TiKV, PD • ֤ίϯϙʔωϯτԽ͢Δ͜ͱʹΑΓ
ύϑΥʔϚϯεͱোੑΛߴΊΒΕΔ • SQL LayerͷTiDBαʔόਫฏํʹ ֦ுՄೳ • ΫϥελΛཧ͢ΔPDαʔόɺ Λ૿͢͜ͱͰՄ༻ੑΛߴΊΒΕΔ ʢ࠷̏ʣ Ҿ༻ݩ: https://docs.pingcap.com/tidb/stable/tidb-architecture
TiDB Storage (TiKV) • σʔλΛ֨ೲ͢ΔͷTiKVίϯϙʔωϯτ • σʔλʢϦʔδϣϯͱ͍͏୯ҐʣΛࣗಈͰෳ ͷϨϓϦΧͰཧ͞ΕΔ • ಉ͡σʔλ͕ࡾͭίϐʔ͞ΕΔ
• ߴՄ༻ੑΛ࣋ͪɺࣗಈϑΣΠϧΦʔόʔΛα ϙʔτ͢Δ • TiKVϊʔυΛΫϥελʹՃ͢Δ͜ͱͰ༰ྔ Λ૿͢͜ͱ͕Ͱ͖Δ • RDBͰߦ͍ͬͯͨσʔλංେԽʹ͏γϟʔ σΟϯάɺϚελʔԽͳͲ͕ෆཁʹͳΔ Ҿ༻ݩ: https://docs.pingcap.com/tidb/stable/tidb-storage
Backup • όοΫΞοϓˍϦετΞ • σʔλόοΫΞοϓฐࣾαʔϏεӡ ༻ʹ͓͍ͯඞਢཁ݅ • εφοϓγϣοτͱ૿ϩά͕औΕΔ • όοΫΞοϓ͔Βղੳ༻σʔλͷߏங
• Dumpͷෛՙ͕ແࢹͰ͖ͳ͍͘Β͍ʹ σʔλ͕૿Ճͨ͠߹ɺόοΫΞοϓ ͔Β෮ݩͨ͠Ϋϥελ͔ΒDWH༻ͷ σʔλ͕औΕΔΑ͏ʹ͢Δ Starting component br: /.tiup/components/br/v8.1.0/br restore full --pd 10.30.2.92:2379 --storage s3://kan-tidb-test-tiup-backup/ snapshot-20240712-3 --send-credentials-to-tikv=false Detail BR log in /tmp/br.log.2024-07-12T08.13.06Z Full Restore <------------------------------------------------------------------- ----------------------------------------------------------------> 100.00%
TiUP • ΫϥελʔͷߏஙɺScale out/Scale inɺBackupଐͷϕϯνπʔϧͳͲ ͯ͢ͷૢ࡞ͰtiupίϚϯυ͕ඇৗʹ ͍͔ͬͨ͢ • υΩϡϝϯτ๛ •
ϦϦʔεޙͷӡ༻ʹߩݙͯ͘͠Εͦ͏ tiup cluster scale-out test-cluster scaleout-tikv.yaml tikv_servers: - host: 10.20.1.206 config: server.labels: rack: r4 1. Ճ͢ΔίϯϙʔωϯτΛyamlʹఆٛ͢Δ 2. tiupίϚϯυΛ࣮ߦ͢Δ
PingCAP TiDBͰࠓճͷαʔϏεཁ݅ΛຬͨͤΔͱஅ͠࠾༻ ։ൃॳظϑΣʔζͰɺPingCAPࣾͱִिͰmtgΛ͠ͳ͕ΒਐΊΔ
Timeline • λΠϜϥΠϯʹɺϢʔβʔͷߘͨ͠λΠ ϜϥΠϯʢϢʔβʔλΠϜϥΠϯʣͱϗʔϜ ը໘ͷλΠϜϥΠϯ͕͋Δ • ϗʔϜλΠϜϥΠϯΛදࣔ͢ΔʹɺPOST ͨ͠ߘΛͰ͖ΔݶΓૣ͘ϑΥϩϫʔͷλΠ ϜϥΠϯʹಧ͚Δඞཁ͕͋Δ
Home Timeline Timeline • ࣄલʹϗʔϜλΠϜϥΠϯΛ࡞͢Δ • ߘऀͷϑΥϩϫʔ/ίϛϡχςΟΛ୳ ͠ɺϑΥϩϫʔຖͷλΠϜϥΠϯςʔ ϒϧʹϙετΛॻ͖ࠐΉ ϙετIDΛ֤Ϣʔβʔ͝ͱʹ৴͍ͯ͠Δ
Post API Redis Redis Redis Find Followers Find Community Members Timeline API
Timeline Tables (Redis) • ϢʔβʔͷϗʔϜλΠϜϥΠϯΛࣄલʹredis ʹ࡞͢Δ • ͜ͷใΛݩʹDBͷใͱΈ߹Θͤͯ ϗʔϜλΠϜϥΠϯΛߏங͢Δ •
ҰఆͷϙετใΛϢʔβʔຖʹอ࣋͢Δ • ࢀর͞ΕΔՄೳੑ͕͘ͳͬͨϙετҰ ఆִؒͰআ͍ͯ͠Δ post_id user: A post_id post_id post_id user: B post_id post_id post_id user: C post_id post_id Redis Sorted Set
Timeline Tables (DB) • TiDBʹશͯͷϙετͱϢʔβʔλΠϜ ϥΠϯͷϙετΛه • ϗʔϜλΠϜϥΠϯDBʹ࡞ΒΕ ͳ͍ •
ϗʔϜλΠϜϥΠϯΛ࡞ΔͨΊͷ ใʢϑΥϩϫʔɺίϛϡχςΟϢʔ βʔʣ͕ଘࡏ͢Δ posts post_id user_id community_id text user_posts post_id user_id community_ users community_id user_id followings following_id user_id
ෛՙςετ ϦϦʔεલʹෛՙݕূΛ࣮ࢪɻAPIͷϨΠςϯγʔͷඪΛ100msʹઃఆ • Writeੑೳ • දతͳRPC͕10000RPSΛग़ͤΔͱ͜Ζ·Ͱݕূ • Readੑೳ • ߘ͕ϑΥϩϫʔͷλΠϜϥΠϯʹදࣔ͞ΕΔ·Ͱͷ͕࣌ؒ5ඵ͔Β10ඵʹͳΔ͜ͱ
Λඪʹௐ • αʔϏε໊͔Β͋ΔఔϢʔβʔ͕དྷΔ͜ͱ͕ఆͰ͖ͨ
Locust - Boomer • ෛՙςετ༻ͷڥΛ༻ҙͯ͠LocustΛ͍ݕূ • ฒྻΛେ͖͍ͨ͘͠ͷͰgoͷworker࣮ ͷboomerΛ༻ͨ͠ • ߈ܸΫϥΠΞϯτͷεέʔϧΞτɺεέʔ
ϧΞοϓΛසൟʹߦ͏ͨΊɺk8s্Ͱಈ͔ͨ͠ • γφϦΦϢʔβʔ࡞͔ΒೝূΛߦ͍ɺϑΥ ϩʔɺϙετɺϥΠΫͳͲͷجຊతͳಈ࡞Λશͯ ࣮ࢪ
ෛՙςετ • ॻ͖ࠐΈੑೳTiDB, TiKVͷεέʔϧΞτͰఆ ௨Γʹੑೳ্ͨ͠ • Ұ෦ɺॏ͍ॲཧʢݕࡧΠϯσοΫεͷॻ͖ࠐ ΈͳͲʣΛඇಉظॲཧʹมߋ • pprofΛ͍ॏ͍ॲཧΛ୳ͨ͠
• ಡΈࠐΈॲཧআॲཧࠓճͷඪΛୡ͢ Δ্Ͱͳ͠ • ߴෛՙ࣌ʹෳͷϑΥϩϫʔ͕ಉ࣌ʹϑΥϩʔ͢ Δɺ͍͍ͶΛ͢ΔέʔεͰසൟϩοΫͪͷλΠϜ Ξτ͕ൃੜ
োςετ Locust͔ΒෛՙΛ͔͚ͳ͕Βtidbͷ֤ίϯϙʔωϯ τΛఀࢭͤ͞ਖ਼͘͠ಈ࡞͢Δ͔ݕূͨ͠ • TiKVͷΠϯελϯεΛఀࢭͤ͞ɺmax-store- down-timeͷ࣌ؒܦաޙʹɺϦʔδϣϯϨϓϦΧ ͕Ϧόϥϯε͢Δ͜ͱ • ΫϥελʔͷTiKVΛݮΒͯ͠Ϧόϥϯε͢Δ ͜ͱ
• Placement DriverͷϦʔμʔมߋ
• ϦϦʔεఆΛ্ճΔϦΫΤετ͕དྷͨ ͕ɺ֤ίϯϙʔωϯτΛεέʔϧΞτ͢Δ ͜ͱͰແࣄՔಇ͠ଓ͚Δ͜ͱ͕Ͱ͖ͨ • ͓ਖ਼݄ͳͲҰ࣌తͳෛՙ͕ݟࠐ·ΕΔ߹ εέʔϧΞτ͢Δ͜ͱͰͳ͘ӡ༻Λଓ ͚Δ͜ͱ͕Ͱ͖͍ͯΔ • ࠓ·ͰαʔϏεແఀࢭͰӡ༻
• TiDBΠϯελϯεͷϊʔυো࣌αʔϏε Өڹͳ͔ͬͨ ϦϦʔε
Slow Query • Secondary IndexΛͬͨΫΤϦͰIndexͷઌ ಄ΧϥϜͰରΛߜΓɺ̎൪ͷΧϥϜͰ ιʔτ͢Δ • In۟ͷཁૉ͕Ұͭͷ߹sort͕ޮ͍͕ͨɺෳ ཁૉࢦఆ͞ΕΔͱsort͕ޮ͔ͳ͍
• ର͕ेສ݅ʹͳΔ͜ͱ͋ΓSlow Queryʹͳͬͯ͠·ͬͨ KEY `idx_1` (`community_id`,`time_series_id`,`user_id`) /* ҎԼͷΑ͏ͳΫΤϦͷ߹ʹkeep order:falseͱͳΔ */ explain analyze select /*+ USE_INDEX(`community_posts`, idx_1) */ * from community_posts where community_id IN (‘aaaa’, ‘bbbb’, ‘cccc’) and time_series_id < 1736935091883343234 order by time_series_id ASC LIMIT 30; +----------------------------------+---------+---------+ | id | estRows | actRows | +----------------------------------+---------+---------+ | TopN_9 | 30.00 | 30 | | └ ─ IndexLookUp_16 | 30.00 | 30 | | ├ ─ TopN_15(Build) | 30.00 | 30 | | │ └ ─ IndexRangeScan_13 | 1317.05 | 1265 | | └ ─ TableRowIDScan_14(Probe) | 30.00 | 30 | +----------------------------------+---------+---------+
Slow Query total_keys: εΩϟϯ͞ΕͨKeyͷ߹ܭɻݹ͍όʔδϣϯΛؚΉ total_process_keys: ॲཧ͞ΕͨKeyɻݹ͍όʔδϣϯΛؚ·ͳ͍ delete_skipped_count: আϚʔΧʔ͕͍ͭͨKeyͷ߹ܭ key_skipped_count: εΩοϓͨ͠Keyͷ߹ܭ
• ظؒͰUPDATEΛ܁Γฦ͔͢DELETE͢Δ έʔε • ෳͷόʔδϣϯ͕ҙਤͤͣੵ͞Εͨ • ಉ͡ςʔϒϧͷϨίʔυΛݕࡧ͢Δͱ͖ ʹSlow Query͕ൃੜ scan_detail: { total_process_keys: 1, total_process_keys_size: 102, total_keys: 26150, get_snapshot_time: 10.5µs, rocksdb: { delete_skipped_count: 179504, key_skipped_count: 205653, . . . }
Operation • ϦϦʔεޙɺTiDB Clusterͷӡ༻tiupίϚϯ υΛ࣮ͬͯࢪ • Scale out/Scale in •
Backup • Dump
ϨϓϦΧ εέδϡʔϧ Մ༻ੑͷ্ͷͨΊɺτϙϩδϥϕϧͷػೳΛ ͍TiKVϊʔυΛҟͳΔϥοΫʹஔ͍ͯ͠Δ EC2 PlacementGroupͰ࡞ͨ͠ύʔςΟγϣ ϯΛPD͕ཧղͰ͖ΔΑ͏ʹ location-labelsͷઃ ఆΛߦ͏ $
tiup ctl:v8.3.0 pd config set location-labels rack -u 10.20.3.13:2379 { "max-replicas": 3, "location-labels": "rack", "strictly-match-label": "false", "enable-placement-rules": "true", "enable-placement-rules-cache": "false", "isolation-level": "" }
Dumpling • tiup dumplingίϚϯυͰDBσʔλΛS3ʹΤ Ϋεϙʔτ(CSV) • ECSͷεέδϡʔϧυλεΫͱͯ͠ఆظత ʹ࣮ߦ͢Δ • Dumpͨ͠σʔλΛղੳ༻DBʢBigQuery)
ʹऔΓࠐΉ $ tiup dumpling -u username -P 4000 -h 10.20.1.244 -- filetype csv -t 8 -o “s3://dump-bucket/20250214" -r 20000 -F 256MiB --csv-output-dialect bigquery --csv-null-value "" --escape-backslash
Backup • tiup brίϚϯυͰϑϧόοΫΞοϓͱϩά όοΫΞοϓΛߦ͏ • ECSͷεέδϡʔϧυλεΫͱͯ͠ఆظత ʹ࣮ߦ͢Δ Ҿ༻ݩ: https://docs.pingcap.com/ja/tidb/stable/br-snapshot-architecture#architecture
Monitoring TiDB Dashboard - Ϋϥελશମͷ࣮ߦεςʔλ εʢQPSɺ࣮ߦ࣌ؒͳͲʣͱSQL࣮ߦ࣌ؒɺε ϩʔΫΤϦΛ֬ೝ͢Δ Grafana - TiDB,
TiKV, PDͷओཁͳϝτϦΫεͷ ࢹ - ϦʔδϣϯɺҟৗͳϦʔδϣϯ - εΩοϓ͞ΕͨΩʔɺআͣΈΩʔͷ
·ͱΊͱࠓޙͷల • গͳ͍ਓͷ։ൃνʔϜ͕ͩɺTiDB ClusterΛ༻ͨ͠αʔϏε։ൃͱӡ༻Λߦ͍͑ͯΔ • TiDBແఀࢭͰӡ༻ΛܧଓɺٸܹͳෛՙʹεέʔϧΞτͰରԠ͢Δ͜ͱ͕Ͱ͖ͨ • υΩϡϝϯτ͕๛ʹ͋Δ͜ͱࢹπʔϧ͕Ұ௨Γଐ͢Δ͜ͱগਓνʔϜʹ ͋Γ͕͍ͨ •
ࠓޙͷల • TiCDCΛ͍νΣϯδΠϕϯτΛϝοηʔδͱͯ͠ૹ৴͠ඇಉظॲཧʹ͍͍ͨʢSQS ͱඇಉظδϣϒͷ෦Λஔ͖͑ΔͨΊʣ
ΞϯέʔτճऀʹϊϕϧςΟΛϓϨθϯτʂ ϐϯΫ ωΠϏʔ άϨʔ ΠΤϩʔ ϒϧʔ άϦʔϯ ※λΦϧϋϯΧνͷΧϥʔͱຕʹݶΓ͕͍͟͝·͢ɻແ͘ͳΓ࣍ୈऴ͍ྃͨ͠·͢ͷͰྃ͝ঝ͍ͩ͘͞ɻ ※ϊϕϧςΟ͓ड͚औΓͷࡍɺճྃը໘ͷදࣔΛ͓ئ͍͠·͢ɻ ·ͨ
+ SCAN HERE!