Slide 1
Slide 1 text
୶ୌ API
୭ച ೞӝ
2018֙ 8ਘ 31ੌ ਬള
© RIDI Corp.
Slide 2
Slide 2 text
1. ୶ୌ API ࣗѐ
• Book-to-book ୶ୌ
• Book-to-user ୶ୌ
• API ҳઑ
2. ୶ୌ API ୭ച ೞӝ
• উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
• ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
• API दр ӝ
2
ݾର
୶ୌ API ୭ച ೞӝ
Slide 3
Slide 3 text
3
ܻ٣࠘झ Book-to-book ୶ୌ
୶ୌ API ࣗѐ
Slide 4
Slide 4 text
4
بࢲр ਬࢎب द
୶ୌ API ࣗѐ
0.11651 0.07987
Slide 5
Slide 5 text
5
بࢲр ਬࢎب ҅
୶ୌ API ࣗѐ
1. ҳݒݾ۾
• ଲ: ా۸ Ӗॳӝ
• അ୍: ా۸ Ӗॳӝ, যڌѱ Ѫੋо
• ࢳ: যڌѱ Ѫੋо
2. بࢲӝળ ҳݒݾ۾
• ా۸ Ӗॳӝ: ଲ, അ୍
• যڌѱ Ѫੋо: അ୍, ࢳ
3. ਬࢎب (ా۸, যڌѱ)
= |{അ୍}| / |{ଲ, അ୍, ࢳ}| ≈ 0.33
?
Slide 6
Slide 6 text
6
ܻ٣࠘झ Book-to-user ୶ୌ
୶ୌ API ࣗѐ
Slide 7
Slide 7 text
بࢲр ਬࢎب৬ ਬ ۽ੌਸ ࢎਊ
• അח ҳݒ৬ ୭Ӕ ઑഥೠ بࢲٜ۽ ۽ੌ ࢤࢿ
• ۽ੌҗ ਬࢎبо ֫ بࢲٜਸ ୶ୌ
7
ܻ٣࠘झ Book-to-user ਗܻ
୶ୌ API ࣗѐ
Slide 8
Slide 8 text
recommend-api
data-api.ridibooks.com
Classic
Load
Balancer
Store Web
Server
Client
Store Web
Elasticache
(Redis)
proxy-azure.data.ridi.io
hbase.data.ridi.io
HAProxy
192.168.0.200:33309
HAProxy
192.168.0.200:33319
HAProxy
aurora-db.data.ridi.io
aurora-db.ro.data.ridi.io
Aurora
HBase Cluster
HTTP
Request
Thrift RPC
MySQL
CRUD
Cache
MySQL
Write
MySQL
Read
Plain
Requests
Load
Balancing
ridibooks.com
HTTP
Request
8
API ҳઑ
୶ୌ API ࣗѐ
Slide 9
Slide 9 text
1. উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
2. ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
3. API दр ӝ
9
୶ୌ API ୭ച ೞӝ
୶ୌ API ୭ച ೞӝ
Slide 10
Slide 10 text
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
Slide 11
Slide 11 text
11
୶ୌ ؘఠ
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
• Bulk write৬ real-time read
• ড 150݅Ѥ book-to-book ୶ୌҗ
ড 90݅Ѥ book-to-user ୶ୌਸ
पदрਵ۽ ࢲࡂೞҊ ח ࣗী স۽٘
• Key-value
• Key: b_id/u_idx
• Value: بࢲ ୶ୌ ܻझ JSON String
Slide 12
Slide 12 text
After
• Scala, Python
• Multi-machine batch
• ড 2दр
• MongoDBী স۽٘
Before
• C++
• Single-machine batch
• ড 10दр
• MariaDBী স۽٘
12
۞झఠ بੑ /റ
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
Slide 13
Slide 13 text
13
Bulk Loadী ஂডೠ MongoDB
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
• ড 1.6রѐ rowܳ bulk loadೡ ٸ, load average ҃Ҋ ߊࢤ
• Bulk loadܳ ೡ ٸ݅ ۽ࣁझ ࣻܳ ৈࢲ प೯
=> ৈ ޙઁ ߊࢤ!
Slide 14
Slide 14 text
14
HBase بੑ
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
1. Bulk loadী
2. ೞن ীदझమীࢲ ࢎਊ оמ
• औѱ بੑೡ ࣻ
• ഛࢿ ڪযթ
Slide 15
Slide 15 text
15
ࠛউೠ HBase
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
• दр оՔঀ ח അ࢚ਸ ߊѼ
Slide 16
Slide 16 text
16
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
• HBase ࢲ࠺झܳ प೯ೞ ঋח machineীࢲ݅
ܲ batch স प೯غب۾ ࢸ
1. ܲ Batch স ೱਸ ߉Ҋ ঋਸө?
=> ৈ ޙઁ ߊࢤ!
ޙઁ ױ 1
Slide 17
Slide 17 text
17
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
• VCNC৬ IBM о٘ ١ਸ ଵҊೞৈ ౚ:
2. HBase ౚ ޙઁ ইקө?
=> ઑӘ աইӟ ೮݅, ৈ ޙઁ ߊࢤ!
ޙઁ ױ 2
* http://engineering.vcnc.co.kr/2013/04/hbase-configuration/
* https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.0/
com.ibm.swg.im.infosphere.biginsights.analyze.doc/doc/bigsql_TuneHbase.html
ড 20ѐ ۄఠܳ Ѩషೠ റ, 5ѐ ۄఠܳ ౚ.
) ৻ࠗ ਃਸ ܻೞח ؘ ࢎਊೡ Thread ࣻܳ ૐо
Slide 18
Slide 18 text
18
ޙઁ ױ 3
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
3. HappyBase (Python-HBase connector) ޙઁо ইקө?
• Locust۽ ۽٘ పझܳ ೮ਸ ٸ, गо হ
• HappyBase৬ Django৬ ഐജࢿ ब
=> ೞ݅, Thrift RPC ࢲߡীࢲ response time ۽Ӓ ߂
ठ۽ ௪ܻ ࠙ࢳ ١ਸ ೡ ࣻ হ.
Slide 19
Slide 19 text
19
1. ElastiCache + Redis
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
ҙܻഋ ੋ ݫݽܻ ؘఠ झషয ࢲ࠺झ
•
• Key-value ߑधী
• ࡅܲ ࣘب
• ױ
• ࠺ਊ; DB ਊ݅ఀ ݫݽܻо ਃ
• ӝ: 60GB (୭ࣗ cache.m4.4xlarge ਃ)
=> 1֙ ডী $3100. replica setਸ ҳࢿೞҊ, ਊ טযաݶ…?
Slide 20
Slide 20 text
20
2. DynamoDB
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
৮߷ೞѱ ҙܻغח NoSQL ؘఠ߬झ ࢲ࠺झ
•
• Key-value ߑधী
• ױ
• ࠺ਊ; ೠ ۨ٘ ӝо ਸ ٸ݅ ਬܻ
• ӝ: 60GB (ೠ ۨ٘ ӝ: 100KB)
• ੍ӝ/ॳӝ പࣻ: ୡ 40/100ഥ
=> ೠ ׳ী $5782.80
Slide 21
Slide 21 text
21
3. Elasticsearch
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
࠙ഋ RESTful Ѩ࢝ ߂ ࠙ࢳ ূ
•
• Ѩ࢝ APIীࢲ উਵ۽ ੜ ࢎਊೞҊ
• ױ
• Bulk loadী ೞ ঋ; load average ޙઁ ߊࢤ
• Key-value ߑधী ೞ ঋ
Slide 22
Slide 22 text
द MongoDB۽…
উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
• Үള: ܻ ࢚ടী MongoDBо ઁੌ ੜ ݏח Ѫ э.
• HBaseࠁ উੋ ࢲࡂ
• ࠺Ү ۴ೠ ࠺ਊ (ElastiCache + Redis, DynamoDB, …)
• Bulk loadೡ ٸ ޙઁ…
• Bulk loadೡ ٸ݃ ࢜ instance۽ Үೞ!
22
Slide 23
Slide 23 text
1. উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
2. ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
3. API दр ӝ
23
୶ୌ API ୭ച ೞӝ
୶ୌ API ୭ച ೞӝ
Slide 24
Slide 24 text
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
Slide 25
Slide 25 text
25
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
AI ୶ୌ ୶ୌ ઁ৻
Slide 26
Slide 26 text
26
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
DB ҳઑ (୶ୌ ઑഥद)
Slide 27
Slide 27 text
27
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
DB ҳઑ (୶ୌઁ৻ ١۾द)
Single Point of Failure!!
Slide 28
Slide 28 text
SPoF ೱ
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
• ؘఠࣃఠ ਗ গ۽ ੋ೧ ؘఠ ݃झఠ DB
• DB ࢲߡ ࠂҳदө ୶ୌઁ৻ ӝמ ࢎਊࠛо
• DB ࢲߡ ݫݽܻ ૐࢸਸ ਤ೧ ࢲߡ ࠗ ਃ
• ࠗೞח زউ ୶ୌઁ৻ ӝמ ࢎਊࠛо
Slide 29
Slide 29 text
SPoF ೱ
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
୶ୌઁ৻ ప࠶݅ ࡅݶ...
Slide 30
Slide 30 text
ࢿѺ ܲ ప࠶ٜ ҕઓ
30
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
• ୶ୌઁ৻ ప࠶
• 24दр ղղ INSERT ߊࢤ
• Ҋё ࢲ࠺झ৬ ೠ ҙ۲
• ా҅ ప࠶
• ࢜߷ ߓ प೯दী݅ INSERT ߊࢤ
• ഥࢎ ղ ࢲ࠺झী݅ ҙ۲
=> ߹ب ੋۄী فחѱ જѷ!
Slide 31
Slide 31 text
୶ୌઁ৻ ప࠶ਸ Aurora۽ ܻ࠙ೞӝ۽ Ѿ
31
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
• ҙܻഋ ࢲ࠺झ ಞܻೣ
• ഛࢿ, উࢿ ഛࠁ
• 10ms ݅ ܻܻா࣌ ۑ
Slide 32
Slide 32 text
Aurora ܻ࠙ റ
32
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
• ӝઓ DB ࢲߡח ઁٚ ࠗ оמ
• ୶ୌઁ৻ ӝמী Րӣহ ݫݽܻ ૐࢸী ࢿҕ
• DB উࢿ ૐо
• ܲ ࢲ࠺झী ೱ߉ ঋ
• ݃झఠо ইצ ठۨ࠳ীࢲ ੍ب۾ ߸҃
• যٚ ܻܻா࣌ ۑ ؋࠙ী оמ
Slide 33
Slide 33 text
թ ࣼઁ
33
୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
• Auroraীח JOINী ਃೠ ప࠶ হ
• (അ) যܻா࣌ ױ҅ীࢲ JOIN ࣻ೯
• ܻܻா࣌ ࢸ
• APIܳ ࢎਊೠ ӝੋ সؘ
Slide 34
Slide 34 text
1. উੋ ࢲࡂਸ ਤೠ ࣗ ӝ
2. ୶ୌ ઁ৻ ؘఠ߬झ ܻ࠙ೞӝ
3. API दр ӝ
34
୶ୌ API ୭ച ೞӝ
୶ୌ API ୭ച ೞӝ
Slide 35
Slide 35 text
API दр ӝ
Slide 36
Slide 36 text
36
ޙઁ ੋध
API दр ӝ
Slide 37
Slide 37 text
37
ޙઁ ഛੋ
API दр ӝ
2׳ р 3ୡ ࢚ दрਸ о
AI ୶ୌ API ਃ
= ড 4݅Ѥ ( ਃ 0.1359%)
Slide 38
Slide 38 text
38
ޙઁ ױ 1
API दр ӝ
Django ORM ࢤࢿೠ ҳݒݾ۾ ઑഥ ௪ܻ:
EXPLAIN SELECT `tb_user_book`.`pk_id`, `tb_user_book`.`u_id`,…
=> tb_user_book ઑഥо बػ.
Slide 39
Slide 39 text
39
ޙઁ ױ 2
API दр ӝ
• Ѩૐ
• ؘఠ MariaDBо ಣࣗ ې (࠙ ୭ 600ߣ ઑഥ)
ਸ Ѽ٣ח ഛੋ
• Ѿҗ
• ҳݒݾ۾ ઑഥীࢲ ठ۽ ௪ܻ ߊࢤ!
• ؘఠ DBח நएо ѱ غҊ
“ࢲ DBীח ۠ ޙઁо হ؍ Ѫ эؘ…?”
Slide 40
Slide 40 text
40
API दр ӝ
୶ୌ APIীࢲ ҳݒݾ۾ ઑഥܳ হগ!
Slide 41
Slide 41 text
ࢎਊ ۽ੌ
- ҳݒ
- ୭Ӕ ઑഥ
41
API दр ӝ
بࢲр ਬࢎب
Batch
API
AI ୶ୌ
୶ୌ ܻझ ࢤࢿ җ
Slide 42
Slide 42 text
ࢎਊ ۽ੌ
- ҳݒ
- ୭Ӕ ઑഥ
42
API दр ӝ
بࢲр ਬࢎب
Batch
API
AI ୶ୌ
୶ୌ ܻझ ࢤࢿ җ
Slide 43
Slide 43 text
43
ҳݒݾ۾ ӝ߈ ୶ୌ ܻझܳ ܻ ҅ೞ!
API दр ӝ
بࢲр ਬࢎب
Batch
API
ҳݒ ۽ੌ ҳݒӝ߈ ୶ୌ
ઑഥ ۽ੌ ઑഥӝ߈ ୶ୌ AI ୶ୌ
Slide 44
Slide 44 text
44
೧Ѿ଼ Ѩૐ
API दр ӝ
3ୡ ࢚ दрਸ о
AI ୶ୌ API ਃ
= ѐࢶ 1/8
( ਃ 0.0176%)
* MongoDBܳ ࢎਊೞݶ ࠗ࠙ب ѐࢶؼ Ѫਸ ࢚פ.
Slide 45
Slide 45 text
45
Ѿۿ
• ו՛
• ݣ૽ ੜ ࢲࡂೞח Ѥ য۰ ੌ
• ݽܰҊ ࢎਊೞח Ѫҗ ঌҊ ࢎਊೞח Ѫ ܰ
• খਵ۽ ೡ ੌ
• ؘఠ ࢎ౭झ ࡳӝ
• അ ࢚ടী উೞӝ ঋӝ
୶ୌ API ୭ച ೞӝ
Slide 46
Slide 46 text
46
THANK YOU
хࢎפ