Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
추천 API 최적화하기
Search
RIDI
August 31, 2018
Technology
4
2k
추천 API 최적화하기
RIDI
August 31, 2018
Tweet
Share
More Decks by RIDI
See All by RIDI
원격 근무 팀 운영 경험 공유
ridi
0
2.1k
SVG Icon Design Guide
ridi
2
3.5k
빠르게 훑어보는 리디페이 백엔드 개발기
ridi
2
4.9k
Next.js는 정말 zero config였다.
ridi
0
2k
3일 걸릴 것 같던 구매목록 다운로드는 왜 3주가 걸렸을까?
ridi
0
390
원격으로 한 달 일해보니
ridi
0
1.4k
리디북스 앱에 S Pen Remote 연동하기
ridi
2
2.6k
UI 라이브러리 개발기
ridi
1
2.5k
테스트 환경 개선하기
ridi
8
3.3k
Other Decks in Technology
See All in Technology
ハイテク休憩
sat
PRO
2
140
Wantedly での Datadog 活用事例
bgpat
1
430
バクラクのドキュメント解析技術と実データにおける課題 / layerx-ccc-winter-2024
shimacos
2
1k
マイクロサービスにおける容易なトランザクション管理に向けて
scalar
0
110
CustomCopを使ってMongoidのコーディングルールを整えてみた
jinoketani
0
220
1等無人航空機操縦士一発試験 合格までの道のり ドローンミートアップ@大阪 2024/12/18
excdinc
0
150
Snowflake女子会#3 Snowpipeの良さを5分で語るよ
lana2548
0
220
Jetpack Composeで始めるServer Cache State
ogaclejapan
2
170
あの日俺達が夢見たサーバレスアーキテクチャ/the-serverless-architecture-we-dreamed-of
tomoki10
0
430
AIのコンプラは何故しんどい?
shujisado
1
190
kargoの魅力について伝える
magisystem0408
0
200
AWS re:Invent 2024で発表された コードを書く開発者向け機能について
maruto
0
190
Featured
See All Featured
Building Adaptive Systems
keathley
38
2.3k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
330
21k
Designing for humans not robots
tammielis
250
25k
4 Signs Your Business is Dying
shpigford
181
21k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.3k
Into the Great Unknown - MozCon
thekraken
33
1.5k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
111
49k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Fireside Chat
paigeccino
34
3.1k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
28
8.3k
Site-Speed That Sticks
csswizardry
2
190
Transcript
୶ୌ API ୭ച ೞӝ 2018֙ 8ਘ 31ੌ ਬള © RIDI
Corp.
1. ୶ୌ API ࣗѐ • Book-to-book ୶ୌ • Book-to-user ୶ୌ
• API ҳઑ 2. ୶ୌ API ୭ച ೞӝ • উੋ ࢲࡂਸ ਤೠ ࣗ ӝ • ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ • API दр ӝ 2 ݾର ୶ୌ API ୭ച ೞӝ
3 ܻ٣࠘झ Book-to-book ୶ୌ ୶ୌ API ࣗѐ
4 بࢲр ਬࢎب द ୶ୌ API ࣗѐ 0.11651 0.07987
5 بࢲр ਬࢎب ҅ ୶ୌ API ࣗѐ 1. ҳݒݾ۾ •
ଲ: ా۸ Ӗॳӝ • അ୍: ా۸ Ӗॳӝ, যڌѱ Ѫੋо • ࢳ: যڌѱ Ѫੋо 2. بࢲӝળ ҳݒݾ۾ • ా۸ Ӗॳӝ: ଲ, അ୍ • যڌѱ Ѫੋо: അ୍, ࢳ 3. ਬࢎب (ా۸, যڌѱ) = |{അ୍}| / |{ଲ, അ୍, ࢳ}| ≈ 0.33 ?
6 ܻ٣࠘झ Book-to-user ୶ୌ ୶ୌ API ࣗѐ
بࢲр ਬࢎب৬ ਬ ۽ੌਸ ࢎਊ • അח ҳݒ৬ ୭Ӕ ઑഥೠ
بࢲٜ۽ ۽ੌ ࢤࢿ • ۽ੌҗ ਬࢎبо ֫ بࢲٜਸ ୶ୌ 7 ܻ٣࠘झ Book-to-user ਗܻ ୶ୌ API ࣗѐ
recommend-api data-api.ridibooks.com Classic Load Balancer Store Web Server Client Store
Web Elasticache (Redis) proxy-azure.data.ridi.io hbase.data.ridi.io HAProxy 192.168.0.200:33309 HAProxy 192.168.0.200:33319 HAProxy aurora-db.data.ridi.io aurora-db.ro.data.ridi.io Aurora HBase Cluster HTTP Request Thrift RPC MySQL CRUD Cache MySQL Write MySQL Read Plain Requests Load Balancing ridibooks.com HTTP Request 8 API ҳઑ ୶ୌ API ࣗѐ
1. উੋ ࢲࡂਸ ਤೠ ࣗ ӝ 2. ୶ୌ ઁ৻ ؘఠ߬झ
ܻ࠙ೞӝ 3. API दр ӝ 9 ୶ୌ API ୭ച ೞӝ ୶ୌ API ୭ച ೞӝ
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
11 ୶ୌ ؘఠ উੋ ࢲࡂਸ ਤೠ ࣗ ӝ • Bulk
write৬ real-time read • ড 150݅Ѥ book-to-book ୶ୌҗ ড 90݅Ѥ book-to-user ୶ୌਸ पदрਵ۽ ࢲࡂೞҊ ח ࣗী স۽٘ • Key-value • Key: b_id/u_idx • Value: بࢲ ୶ୌ ܻझ JSON String
After • Scala, Python • Multi-machine batch • ড 2दр
• MongoDBী স۽٘ Before • C++ • Single-machine batch • ড 10दр • MariaDBী স۽٘ 12 ۞झఠ بੑ /റ উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
13 Bulk Loadী ஂডೠ MongoDB উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
• ড 1.6রѐ rowܳ bulk loadೡ ٸ, load average ҃Ҋ ߊࢤ • Bulk loadܳ ೡ ٸ݅ ۽ࣁझ ࣻܳ ৈࢲ प೯ => ৈ ޙઁ ߊࢤ!
14 HBase بੑ উੋ ࢲࡂਸ ਤೠ ࣗ ӝ 1. Bulk
loadী 2. ೞن ীदझమীࢲ ࢎਊ оמ • औѱ بੑೡ ࣻ • ഛࢿ ڪযթ
15 ࠛউೠ HBase উੋ ࢲࡂਸ ਤೠ ࣗ ӝ •
दр оՔঀ ח അ࢚ਸ ߊѼ
16 উੋ ࢲࡂਸ ਤೠ ࣗ ӝ • HBase ࢲ࠺झܳ प೯ೞ
ঋח machineীࢲ݅ ܲ batch স प೯غب۾ ࢸ 1. ܲ Batch স ೱਸ ߉Ҋ ঋਸө? => ৈ ޙઁ ߊࢤ! ޙઁ ױ 1
17 উੋ ࢲࡂਸ ਤೠ ࣗ ӝ • VCNC৬ IBM о٘
١ਸ ଵҊೞৈ ౚ: 2. HBase ౚ ޙઁ ইקө? => ઑӘ աইӟ ೮݅, ৈ ޙઁ ߊࢤ! ޙઁ ױ 2 * http://engineering.vcnc.co.kr/2013/04/hbase-configuration/ * https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.0/ com.ibm.swg.im.infosphere.biginsights.analyze.doc/doc/bigsql_TuneHbase.html ড 20ѐ ۄఠܳ Ѩషೠ റ, 5ѐ ۄఠܳ ౚ. ) ৻ࠗ ਃਸ ܻೞח ؘ ࢎਊೡ Thread ࣻܳ ૐо
18 ޙઁ ױ 3 উੋ ࢲࡂਸ ਤೠ ࣗ ӝ 3.
HappyBase (Python-HBase connector) ޙઁо ইקө? • Locust۽ ۽٘ పझܳ ೮ਸ ٸ, गо হ • HappyBase৬ Django৬ ഐജࢿ ब => ೞ݅, Thrift RPC ࢲߡীࢲ response time ۽Ӓ ߂ ठ۽ ௪ܻ ࠙ࢳ ١ਸ ೡ ࣻ হ.
19 1. ElastiCache + Redis উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
ҙܻഋ ੋ ݫݽܻ ؘఠ झషয ࢲ࠺झ • • Key-value ߑधী • ࡅܲ ࣘب • ױ • ࠺ਊ; DB ਊ݅ఀ ݫݽܻо ਃ • ӝ: 60GB (୭ࣗ cache.m4.4xlarge ਃ) => 1֙ ডী $3100. replica setਸ ҳࢿೞҊ, ਊ טযաݶ…?
20 2. DynamoDB উੋ ࢲࡂਸ ਤೠ ࣗ ӝ ৮߷ೞѱ ҙܻغח
NoSQL ؘఠ߬झ ࢲ࠺झ • • Key-value ߑधী • ױ • ࠺ਊ; ೠ ۨ٘ ӝо ਸ ٸ݅ ਬܻ • ӝ: 60GB (ೠ ۨ٘ ӝ: 100KB) • ੍ӝ/ॳӝ പࣻ: ୡ 40/100ഥ => ೠ ׳ী $5782.80
21 3. Elasticsearch উੋ ࢲࡂਸ ਤೠ ࣗ ӝ ࠙ഋ RESTful
Ѩ࢝ ߂ ࠙ࢳ ূ • • Ѩ࢝ APIীࢲ উਵ۽ ੜ ࢎਊೞҊ • ױ • Bulk loadী ೞ ঋ; load average ޙઁ ߊࢤ • Key-value ߑधী ೞ ঋ
द MongoDB۽… উੋ ࢲࡂਸ ਤೠ ࣗ ӝ • Үള: ܻ
࢚ടী MongoDBо ઁੌ ੜ ݏח Ѫ э. • HBaseࠁ উੋ ࢲࡂ • ࠺Ү ۴ೠ ࠺ਊ (ElastiCache + Redis, DynamoDB, …) • Bulk loadೡ ٸ ޙઁ… • Bulk loadೡ ٸ݃ ࢜ instance۽ Үೞ! 22
1. উੋ ࢲࡂਸ ਤೠ ࣗ ӝ 2. ୶ୌ ઁ৻ ؘఠ߬झ
ܻ࠙ೞӝ 3. API दр ӝ 23 ୶ୌ API ୭ച ೞӝ ୶ୌ API ୭ച ೞӝ
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
25 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ AI ୶ୌ ୶ୌ ઁ৻
26 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ DB ҳઑ (୶ୌ ઑഥद)
27 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ DB ҳઑ (୶ୌઁ৻ ١۾द) Single
Point of Failure!!
SPoF ೱ ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ • ؘఠࣃఠ ਗ
গ۽ ੋ೧ ؘఠ ݃झఠ DB • DB ࢲߡ ࠂҳदө ୶ୌઁ৻ ӝמ ࢎਊࠛо • DB ࢲߡ ݫݽܻ ૐࢸਸ ਤ೧ ࢲߡ ࠗ ਃ • ࠗೞח زউ ୶ୌઁ৻ ӝמ ࢎਊࠛо
SPoF ೱ ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ ୶ୌઁ৻ ప࠶݅ ࡅݶ...
ࢿѺ ܲ ప࠶ٜ ҕઓ 30 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ •
୶ୌઁ৻ ప࠶ • 24दр ղղ INSERT ߊࢤ • Ҋё ࢲ࠺झ৬ ೠ ҙ۲ • ా҅ ప࠶ • ࢜߷ ߓ प೯दী݅ INSERT ߊࢤ • ഥࢎ ղ ࢲ࠺झী݅ ҙ۲ => ߹ب ੋۄী فחѱ જѷ!
୶ୌઁ৻ ప࠶ਸ Aurora۽ ܻ࠙ೞӝ۽ Ѿ 31 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
• ҙܻഋ ࢲ࠺झ ಞܻೣ • ഛࢿ, উࢿ ഛࠁ • 10ms ݅ ܻܻா࣌ ۑ
Aurora ܻ࠙ റ 32 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ • ӝઓ
DB ࢲߡח ઁٚ ࠗ оמ • ୶ୌઁ৻ ӝמী Րӣহ ݫݽܻ ૐࢸী ࢿҕ • DB উࢿ ૐо • ܲ ࢲ࠺झী ೱ߉ ঋ • ݃झఠо ইצ ठۨ࠳ীࢲ ੍ب۾ ߸҃ • যٚ ܻܻா࣌ ۑ ؋࠙ী оמ
թ ࣼઁ 33 ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ • Auroraীח JOINী
ਃೠ ప࠶ হ • (അ) যܻா࣌ ױ҅ীࢲ JOIN ࣻ೯ • ܻܻா࣌ ࢸ • APIܳ ࢎਊೠ ӝੋ সؘ
1. উੋ ࢲࡂਸ ਤೠ ࣗ ӝ 2. ୶ୌ ઁ৻ ؘఠ߬झ
ܻ࠙ೞӝ 3. API दр ӝ 34 ୶ୌ API ୭ച ೞӝ ୶ୌ API ୭ച ೞӝ
API दр ӝ
36 ޙઁ ੋध API दр ӝ
37 ޙઁ ഛੋ API दр ӝ 2׳ р 3ୡ ࢚
दрਸ о AI ୶ୌ API ਃ = ড 4݅Ѥ ( ਃ 0.1359%)
38 ޙઁ ױ 1 API दр ӝ Django ORM ࢤࢿೠ
ҳݒݾ۾ ઑഥ ௪ܻ: EXPLAIN SELECT `tb_user_book`.`pk_id`, `tb_user_book`.`u_id`,… => tb_user_book ઑഥо बػ.
39 ޙઁ ױ 2 API दр ӝ • Ѩૐ •
ؘఠ MariaDBо ಣࣗ ې (࠙ ୭ 600ߣ ઑഥ) ਸ Ѽ٣ח ഛੋ • Ѿҗ • ҳݒݾ۾ ઑഥীࢲ ठ۽ ௪ܻ ߊࢤ! • ؘఠ DBח நएо ѱ غҊ “ࢲ DBীח ۠ ޙઁо হ؍ Ѫ эؘ…?”
40 API दр ӝ ୶ୌ APIীࢲ ҳݒݾ۾ ઑഥܳ হগ!
ࢎਊ ۽ੌ - ҳݒ - ୭Ӕ ઑഥ 41 API दр
ӝ بࢲр ਬࢎب Batch API AI ୶ୌ ୶ୌ ܻझ ࢤࢿ җ
ࢎਊ ۽ੌ - ҳݒ - ୭Ӕ ઑഥ 42 API दр
ӝ بࢲр ਬࢎب Batch API AI ୶ୌ ୶ୌ ܻझ ࢤࢿ җ
43 ҳݒݾ۾ ӝ߈ ୶ୌ ܻझܳ ܻ ҅ೞ! API दр ӝ
بࢲр ਬࢎب Batch API ҳݒ ۽ੌ ҳݒӝ߈ ୶ୌ ઑഥ ۽ੌ ઑഥӝ߈ ୶ୌ AI ୶ୌ
44 ೧Ѿ଼ Ѩૐ API दр ӝ 3ୡ ࢚ दрਸ о
AI ୶ୌ API ਃ = ѐࢶ 1/8 ( ਃ 0.0176%) * MongoDBܳ ࢎਊೞݶ ࠗ࠙ب ѐࢶؼ Ѫਸ ࢚פ.
45 Ѿۿ • ו՛ • ݣ૽ ੜ ࢲࡂೞח Ѥ
য۰ ੌ • ݽܰҊ ࢎਊೞח Ѫҗ ঌҊ ࢎਊೞח Ѫ ܰ • খਵ۽ ೡ ੌ • ؘఠ ࢎ౭झ ࡳӝ • അ ࢚ടী উೞӝ ঋӝ ୶ୌ API ୭ച ೞӝ
46 THANK YOU хࢎפ