Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
추천 API 최적화하기
Search
RIDI
August 31, 2018
Technology
4
1.9k
추천 API 최적화하기
RIDI
August 31, 2018
Tweet
Share
More Decks by RIDI
See All by RIDI
원격 근무 팀 운영 경험 공유
ridi
0
2.1k
SVG Icon Design Guide
ridi
2
3.4k
빠르게 훑어보는 리디페이 백엔드 개발기
ridi
2
4.9k
Next.js는 정말 zero config였다.
ridi
0
2k
3일 걸릴 것 같던 구매목록 다운로드는 왜 3주가 걸렸을까?
ridi
0
380
원격으로 한 달 일해보니
ridi
0
1.4k
리디북스 앱에 S Pen Remote 연동하기
ridi
2
2.6k
UI 라이브러리 개발기
ridi
1
2.4k
테스트 환경 개선하기
ridi
8
3.3k
Other Decks in Technology
See All in Technology
スクラムチームを立ち上げる〜チーム開発で得られたもの・得られなかったもの〜
ohnoeight
2
350
IBC 2024 動画技術関連レポート / IBC 2024 Report
cyberagentdevelopers
PRO
0
110
テストコード品質を高めるためにMutation Testingライブラリ・Strykerを実戦導入してみた話
ysknsid25
7
2.6k
AWS Media Services 最新サービスアップデート 2024
eijikominami
0
200
Shopifyアプリ開発における Shopifyの機能活用
sonatard
4
250
AWS Lambda のトラブルシュートをしていて思うこと
kazzpapa3
2
170
[CV勉強会@関東 ECCV2024 読み会] オンラインマッピング x トラッキング MapTracker: Tracking with Strided Memory Fusion for Consistent Vector HD Mapping (Chen+, ECCV24)
abemii
0
220
複雑なState管理からの脱却
sansantech
PRO
1
140
透過型SMTPプロキシによる送信メールの可観測性向上: Update Edition / Improved observability of outgoing emails with transparent smtp proxy: Update edition
linyows
2
210
スクラム成熟度セルフチェックツールを作って得た学びとその活用法
coincheck_recruit
1
140
Incident Response Practices: Waroom's Features and Future Challenges
rrreeeyyy
0
160
【若手エンジニア応援LT会】ソフトウェアを学んできた私がインフラエンジニアを目指した理由
kazushi_ohata
0
150
Featured
See All Featured
Practical Orchestrator
shlominoach
186
10k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
38
1.8k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
33k
Build The Right Thing And Hit Your Dates
maggiecrowley
33
2.4k
For a Future-Friendly Web
brad_frost
175
9.4k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
0
89
Building Adaptive Systems
keathley
38
2.3k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
665
120k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
25
1.8k
Transcript
୶ୌ API ୭ച ೞӝ 2018֙ 8ਘ 31ੌ ਬള © RIDI
Corp.
1. ୶ୌ API ࣗѐ • Book-to-book ୶ୌ • Book-to-user ୶ୌ
• API ҳઑ 2. ୶ୌ API ୭ച ೞӝ • উੋ ࢲࡂਸ ਤೠ ࣗ ӝ • ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ • API दр ӝ 2 ݾର ୶ୌ API ୭ച ೞӝ
3 ܻ٣࠘झ Book-to-book ୶ୌ ୶ୌ API ࣗѐ
4 بࢲр ਬࢎب द ୶ୌ API ࣗѐ 0.11651 0.07987
5 بࢲр ਬࢎب ҅ ୶ୌ API ࣗѐ 1. ҳݒݾ۾ •
ଲ: ా۸ Ӗॳӝ • അ୍: ా۸ Ӗॳӝ, যڌѱ Ѫੋо • ࢳ: যڌѱ Ѫੋо 2. بࢲӝળ ҳݒݾ۾ • ా۸ Ӗॳӝ: ଲ, അ୍ • যڌѱ Ѫੋо: അ୍, ࢳ 3. ਬࢎب (ా۸, যڌѱ) = |{അ୍}| / |{ଲ, അ୍, ࢳ}| ≈ 0.33 ?
6 ܻ٣࠘झ Book-to-user ୶ୌ ୶ୌ API ࣗѐ
بࢲр ਬࢎب৬ ਬ ۽ੌਸ ࢎਊ • അח ҳݒ৬ ୭Ӕ ઑഥೠ
بࢲٜ۽ ۽ੌ ࢤࢿ • ۽ੌҗ ਬࢎبо ֫ بࢲٜਸ ୶ୌ 7 ܻ٣࠘झ Book-to-user ਗܻ ୶ୌ API ࣗѐ
recommend-api data-api.ridibooks.com Classic Load Balancer Store Web Server Client Store
Web Elasticache (Redis) proxy-azure.data.ridi.io hbase.data.ridi.io HAProxy 192.168.0.200:33309 HAProxy 192.168.0.200:33319 HAProxy aurora-db.data.ridi.io aurora-db.ro.data.ridi.io Aurora HBase Cluster HTTP Request Thrift RPC MySQL CRUD Cache MySQL Write MySQL Read Plain Requests Load Balancing ridibooks.com HTTP Request 8 API ҳઑ ୶ୌ API ࣗѐ
1. উੋ ࢲࡂਸ ਤೠ ࣗ ӝ 2. ୶ୌ ઁ৻ ؘఠ߬झ
ܻ࠙ೞӝ 3. API दр ӝ 9 ୶ୌ API ୭ച ೞӝ ୶ୌ API ୭ച ೞӝ
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
11 ୶ୌ ؘఠ উੋ ࢲࡂਸ ਤೠ ࣗ ӝ • Bulk
write৬ real-time read • ড 150݅Ѥ book-to-book ୶ୌҗ ড 90݅Ѥ book-to-user ୶ୌਸ पदрਵ۽ ࢲࡂೞҊ ח ࣗী স۽٘ • Key-value • Key: b_id/u_idx • Value: بࢲ ୶ୌ ܻझ JSON String
After • Scala, Python • Multi-machine batch • ড 2दр
• MongoDBী স۽٘ Before • C++ • Single-machine batch • ড 10दр • MariaDBী স۽٘ 12 ۞झఠ بੑ /റ উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
13 Bulk Loadী ஂডೠ MongoDB উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
• ড 1.6রѐ rowܳ bulk loadೡ ٸ, load average ҃Ҋ ߊࢤ • Bulk loadܳ ೡ ٸ݅ ۽ࣁझ ࣻܳ ৈࢲ प೯ => ৈ ޙઁ ߊࢤ!
14 HBase بੑ উੋ ࢲࡂਸ ਤೠ ࣗ ӝ 1. Bulk
loadী 2. ೞن ীदझమীࢲ ࢎਊ оמ • औѱ بੑೡ ࣻ • ഛࢿ ڪযթ
15 ࠛউೠ HBase উੋ ࢲࡂਸ ਤೠ ࣗ ӝ •
दр оՔঀ ח അ࢚ਸ ߊѼ
16 উੋ ࢲࡂਸ ਤೠ ࣗ ӝ • HBase ࢲ࠺झܳ प೯ೞ
ঋח machineীࢲ݅ ܲ batch স प೯غب۾ ࢸ 1. ܲ Batch স ೱਸ ߉Ҋ ঋਸө? => ৈ ޙઁ ߊࢤ! ޙઁ ױ 1
17 উੋ ࢲࡂਸ ਤೠ ࣗ ӝ • VCNC৬ IBM о٘
١ਸ ଵҊೞৈ ౚ: 2. HBase ౚ ޙઁ ইקө? => ઑӘ աইӟ ೮݅, ৈ ޙઁ ߊࢤ! ޙઁ ױ 2 * http://engineering.vcnc.co.kr/2013/04/hbase-configuration/ * https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.0/ com.ibm.swg.im.infosphere.biginsights.analyze.doc/doc/bigsql_TuneHbase.html ড 20ѐ ۄఠܳ Ѩషೠ റ, 5ѐ ۄఠܳ ౚ. ) ৻ࠗ ਃਸ ܻೞח ؘ ࢎਊೡ Thread ࣻܳ ૐо
18 ޙઁ ױ 3 উੋ ࢲࡂਸ ਤೠ ࣗ ӝ 3.
HappyBase (Python-HBase connector) ޙઁо ইקө? • Locust۽ ۽٘ పझܳ ೮ਸ ٸ, गо হ • HappyBase৬ Django৬ ഐജࢿ ब => ೞ݅, Thrift RPC ࢲߡীࢲ response time ۽Ӓ ߂ ठ۽ ௪ܻ ࠙ࢳ ١ਸ ೡ ࣻ হ.
19 1. ElastiCache + Redis উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
ҙܻഋ ੋ ݫݽܻ ؘఠ झషয ࢲ࠺झ • • Key-value ߑधী • ࡅܲ ࣘب • ױ • ࠺ਊ; DB ਊ݅ఀ ݫݽܻо ਃ • ӝ: 60GB (୭ࣗ cache.m4.4xlarge ਃ) => 1֙ ডী $3100. replica setਸ ҳࢿೞҊ, ਊ טযաݶ…?
20 2. DynamoDB উੋ ࢲࡂਸ ਤೠ ࣗ ӝ ৮߷ೞѱ ҙܻغח
NoSQL ؘఠ߬झ ࢲ࠺झ • • Key-value ߑधী • ױ • ࠺ਊ; ೠ ۨ٘ ӝо ਸ ٸ݅ ਬܻ • ӝ: 60GB (ೠ ۨ٘ ӝ: 100KB) • ੍ӝ/ॳӝ പࣻ: ୡ 40/100ഥ => ೠ ׳ী $5782.80
21 3. Elasticsearch উੋ ࢲࡂਸ ਤೠ ࣗ ӝ ࠙ഋ RESTful
Ѩ࢝ ߂ ࠙ࢳ ূ • • Ѩ࢝ APIীࢲ উਵ۽ ੜ ࢎਊೞҊ • ױ • Bulk loadী ೞ ঋ; load average ޙઁ ߊࢤ • Key-value ߑधী ೞ ঋ
द MongoDB۽… উੋ ࢲࡂਸ ਤೠ ࣗ ӝ • Үള: ܻ
࢚ടী MongoDBо ઁੌ ੜ ݏח Ѫ э. • HBaseࠁ উੋ ࢲࡂ • ࠺Ү ۴ೠ ࠺ਊ (ElastiCache + Redis, DynamoDB, …) • Bulk loadೡ ٸ ޙઁ… • Bulk loadೡ ٸ݃ ࢜ instance۽ Үೞ! 22
1. উੋ ࢲࡂਸ ਤೠ ࣗ ӝ 2. ୶ୌ ઁ৻ ؘఠ߬झ
ܻ࠙ೞӝ 3. API दр ӝ 23 ୶ୌ API ୭ച ೞӝ ୶ୌ API ୭ച ೞӝ
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
25 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ AI ୶ୌ ୶ୌ ઁ৻
26 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ DB ҳઑ (୶ୌ ઑഥद)
27 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ DB ҳઑ (୶ୌઁ৻ ١۾द) Single
Point of Failure!!
SPoF ೱ ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ • ؘఠࣃఠ ਗ
গ۽ ੋ೧ ؘఠ ݃झఠ DB • DB ࢲߡ ࠂҳदө ୶ୌઁ৻ ӝמ ࢎਊࠛо • DB ࢲߡ ݫݽܻ ૐࢸਸ ਤ೧ ࢲߡ ࠗ ਃ • ࠗೞח زউ ୶ୌઁ৻ ӝמ ࢎਊࠛо
SPoF ೱ ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ ୶ୌઁ৻ ప࠶݅ ࡅݶ...
ࢿѺ ܲ ప࠶ٜ ҕઓ 30 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ •
୶ୌઁ৻ ప࠶ • 24दр ղղ INSERT ߊࢤ • Ҋё ࢲ࠺झ৬ ೠ ҙ۲ • ా҅ ప࠶ • ࢜߷ ߓ प೯दী݅ INSERT ߊࢤ • ഥࢎ ղ ࢲ࠺झী݅ ҙ۲ => ߹ب ੋۄী فחѱ જѷ!
୶ୌઁ৻ ప࠶ਸ Aurora۽ ܻ࠙ೞӝ۽ Ѿ 31 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
• ҙܻഋ ࢲ࠺झ ಞܻೣ • ഛࢿ, উࢿ ഛࠁ • 10ms ݅ ܻܻா࣌ ۑ
Aurora ܻ࠙ റ 32 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ • ӝઓ
DB ࢲߡח ઁٚ ࠗ оמ • ୶ୌઁ৻ ӝמী Րӣহ ݫݽܻ ૐࢸী ࢿҕ • DB উࢿ ૐо • ܲ ࢲ࠺झী ೱ߉ ঋ • ݃झఠо ইצ ठۨ࠳ীࢲ ੍ب۾ ߸҃ • যٚ ܻܻா࣌ ۑ ؋࠙ী оמ
թ ࣼઁ 33 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ • Auroraীח JOINী
ਃೠ ప࠶ হ • (അ) যܻா࣌ ױ҅ীࢲ JOIN ࣻ೯ • ܻܻா࣌ ࢸ • APIܳ ࢎਊೠ ӝੋ সؘ
1. উੋ ࢲࡂਸ ਤೠ ࣗ ӝ 2. ୶ୌ ઁ৻ ؘఠ߬झ
ܻ࠙ೞӝ 3. API दр ӝ 34 ୶ୌ API ୭ച ೞӝ ୶ୌ API ୭ച ೞӝ
API दр ӝ
36 ޙઁ ੋध API दр ӝ
37 ޙઁ ഛੋ API दр ӝ 2׳ р 3ୡ ࢚
दрਸ о AI ୶ୌ API ਃ = ড 4݅Ѥ ( ਃ 0.1359%)
38 ޙઁ ױ 1 API दр ӝ Django ORM ࢤࢿೠ
ҳݒݾ۾ ઑഥ ௪ܻ: EXPLAIN SELECT `tb_user_book`.`pk_id`, `tb_user_book`.`u_id`,… => tb_user_book ઑഥо बػ.
39 ޙઁ ױ 2 API दр ӝ • Ѩૐ •
ؘఠ MariaDBо ಣࣗ ې (࠙ ୭ 600ߣ ઑഥ) ਸ Ѽ٣ח ഛੋ • Ѿҗ • ҳݒݾ۾ ઑഥীࢲ ठ۽ ௪ܻ ߊࢤ! • ؘఠ DBח நएо ѱ غҊ “ࢲ DBীח ۠ ޙઁо হ؍ Ѫ эؘ…?”
40 API दр ӝ ୶ୌ APIীࢲ ҳݒݾ۾ ઑഥܳ হগ!
ࢎਊ ۽ੌ - ҳݒ - ୭Ӕ ઑഥ 41 API दр
ӝ بࢲр ਬࢎب Batch API AI ୶ୌ ୶ୌ ܻझ ࢤࢿ җ
ࢎਊ ۽ੌ - ҳݒ - ୭Ӕ ઑഥ 42 API दр
ӝ بࢲр ਬࢎب Batch API AI ୶ୌ ୶ୌ ܻझ ࢤࢿ җ
43 ҳݒݾ۾ ӝ߈ ୶ୌ ܻझܳ ܻ ҅ೞ! API दр ӝ
بࢲр ਬࢎب Batch API ҳݒ ۽ੌ ҳݒӝ߈ ୶ୌ ઑഥ ۽ੌ ઑഥӝ߈ ୶ୌ AI ୶ୌ
44 ೧Ѿ଼ Ѩૐ API दр ӝ 3ୡ ࢚ दрਸ о
AI ୶ୌ API ਃ = ѐࢶ 1/8 ( ਃ 0.0176%) * MongoDBܳ ࢎਊೞݶ ࠗ࠙ب ѐࢶؼ Ѫਸ ࢚פ.
45 Ѿۿ • ו՛ • ݣ૽ ੜ ࢲࡂೞח Ѥ
য۰ ੌ • ݽܰҊ ࢎਊೞח Ѫҗ ঌҊ ࢎਊೞח Ѫ ܰ • খਵ۽ ೡ ੌ • ؘఠ ࢎ౭झ ࡳӝ • അ ࢚ടী উೞӝ ঋӝ ୶ୌ API ୭ച ೞӝ
46 THANK YOU хࢎפ