$30 off During Our Annual Pro Sale. View Details »

추천 API 최적화하기

RIDI
August 31, 2018

추천 API 최적화하기

RIDI

August 31, 2018
Tweet

More Decks by RIDI

Other Decks in Technology

Transcript

  1. ୶ୌ API
    ୭੸ച ೞӝ
    2018֙ 8ਘ 31ੌ ਬ૑ള
    © RIDI Corp.

    View Slide

  2. 1. ୶ୌ API ࣗѐ
    • Book-to-book ୶ୌ
    • Book-to-user ୶ୌ
    • API ҳઑ

    2. ୶ୌ API ୭੸ച ೞӝ
    • উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    • API ਽׹दр ઴੉ӝ
    2
    ݾର
    ୶ୌ API ୭੸ച ೞӝ

    View Slide

  3. 3
    ܻ٣࠘झ੄ Book-to-book ୶ୌ
    ୶ୌ API ࣗѐ

    View Slide

  4. 4
    بࢲр ਬࢎب ৘द
    ୶ୌ API ࣗѐ
    0.11651 0.07987

    View Slide

  5. 5
    بࢲр ਬࢎب ҅࢑
    ୶ୌ API ࣗѐ
    1. ҳݒݾ۾
    • ଲ਋: ؀ా۸੄ Ӗॳӝ
    • അ୍: ؀ా۸੄ Ӗॳӝ, যڌѱ ࢓ Ѫੋо
    • ޹ࢳ: যڌѱ ࢓ Ѫੋо
    2. بࢲӝળ ҳݒݾ۾
    • ؀ా۸੄ Ӗॳӝ: ଲ਋, അ୍
    • যڌѱ ࢓ Ѫੋо: അ୍, ޹ࢳ
    3. ਬࢎب (؀ా۸, যڌѱ)
    = |{അ୍}| / |{ଲ਋, അ୍, ޹ࢳ}| ≈ 0.33
    ?

    View Slide

  6. 6
    ܻ٣࠘झ੄ Book-to-user ୶ୌ
    ୶ୌ API ࣗѐ

    View Slide

  7. بࢲр ਬࢎب৬ ਬ੷ ೐۽౵ੌਸ ࢎਊ
    • അ੤ח ҳݒ৬ ୭Ӕ ઑഥೠ بࢲٜ۽ ೐۽౵ੌ ࢤࢿ
    • ೐۽౵ੌҗ ਬࢎبо ֫਷ بࢲٜਸ ୶ୌ
    7
    ܻ٣࠘झ੄ Book-to-user ਗܻ
    ୶ୌ API ࣗѐ

    View Slide

  8. recommend-api
    data-api.ridibooks.com
    Classic
    Load
    Balancer
    Store Web
    Server
    Client
    Store Web
    Elasticache
    (Redis)
    proxy-azure.data.ridi.io
    hbase.data.ridi.io
    HAProxy
    192.168.0.200:33309
    HAProxy
    192.168.0.200:33319
    HAProxy
    aurora-db.data.ridi.io
    aurora-db.ro.data.ridi.io
    Aurora
    HBase Cluster
    HTTP
    Request
    Thrift RPC
    MySQL
    CRUD
    Cache
    MySQL
    Write
    MySQL
    Read
    Plain
    Requests
    Load
    Balancing
    ridibooks.com
    HTTP
    Request
    8
    API ҳઑ
    ୶ୌ API ࣗѐ

    View Slide

  9. 1. উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    2. ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    3. API ਽׹दр ઴੉ӝ
    9
    ୶ୌ API ୭੸ച ೞӝ
    ୶ୌ API ୭੸ച ೞӝ

    View Slide

  10. উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ

    View Slide

  11. 11
    ୶ୌ ؘ੉ఠ
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • Bulk write৬ real-time read
    • ড 150݅Ѥ੄ book-to-book ୶ୌҗ 

    ড 90݅Ѥ੄ book-to-user ୶ୌਸ 

    पदрਵ۽ ࢲࡂೞҊ ੓ח ੷੢ࣗী স۽٘
    • Key-value
    • Key: b_id/u_idx
    • Value: بࢲ ୶ୌ ܻझ౟੄ JSON String

    View Slide

  12. After
    • Scala, Python
    • Multi-machine batch
    • ড 2दр
    • MongoDBী স۽٘
    Before
    • C++
    • Single-machine batch
    • ড 10दр
    • MariaDBী স۽٘
    12
    ௿۞झఠ بੑ ੹/റ
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ

    View Slide

  13. 13
    Bulk Loadী ஂডೠ MongoDB
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • ড 1.6রѐ੄ rowܳ bulk loadೡ ٸ, load average ҃Ҋ ߊࢤ
    • Bulk loadܳ ೡ ٸ݅ ೐۽ࣁझ੄ ࣻܳ ઴ৈࢲ प೯
    => ৈ੹൤ ޙઁ ߊࢤ!

    View Slide

  14. 14
    HBase بੑ
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    1. Bulk loadী ੸೤
    2. ೞن ী௏दझమীࢲ ࢎਊ оמ
    • औѱ بੑೡ ࣻ ੓਺
    • ഛ੢ࢿ੉ ڪযթ

    View Slide

  15. 15
    ࠛউ੿ೠ HBase
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • ਽׹ दр੉ оՔঀ ౓ח അ࢚ਸ ߊѼ

    View Slide

  16. 16
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • HBase ࢲ࠺झܳ प೯ೞ૑ ঋח machineীࢲ݅ 

    ׮ܲ batch ੘স੉ प೯غب۾ ࢸ੿
    1. ׮ܲ Batch ੘স੄ ৔ೱਸ ߉Ҋ ੓૑ ঋਸө?
    => ৈ੹൤ ޙઁ ߊࢤ!
    ޙઁ ૓ױ 1

    View Slide

  17. 17
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • VCNC৬ IBM੄ о੉٘ ١ਸ ଵҊೞৈ ౚ׬:
    2. HBase ౚ׬੄ ޙઁ ইקө?
    => ઑӘ աই૑ӟ ೮૑݅, ৈ੹൤ ޙઁ ߊࢤ!
    ޙઁ ૓ױ 2
    * http://engineering.vcnc.co.kr/2013/04/hbase-configuration/
    * https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.0/
    com.ibm.swg.im.infosphere.biginsights.analyze.doc/doc/bigsql_TuneHbase.html
    ড 20ѐ੄ ౵ۄ޷ఠܳ Ѩషೠ റ, 5ѐ੄ ౵ۄ޷ఠܳ ౚ׬.

    ৘) ৻ࠗ ਃ୒ਸ ୊ܻೞח ؘ ࢎਊೡ Thread੄ ࣻܳ ૐо

    View Slide

  18. 18
    ޙઁ ૓ױ 3
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    3. HappyBase (Python-HBase connector) ੄ ޙઁо ইקө?
    • Locust۽ ۽٘ పझ౟ܳ ೮ਸ ٸ, ௾ ੉गо হ਺
    • HappyBase৬ Django৬੄ ഐജࢿ ੄ब
    => ೞ૑݅, Thrift RPC ࢲߡীࢲ response time ۽Ӓ ߂ 

    ठ۽਋ ௪ܻ ࠙ࢳ ١ਸ ೡ ࣻ হ਺.

    View Slide

  19. 19
    1. ElastiCache + Redis
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    ҙܻഋ ੋ ݫݽܻ ؘ੉ఠ झషয ࢲ࠺झ
    • ੢੼
    • Key-value ߑधী ੸೤
    • ࡅܲ ਽׹ ࣘب
    • ױ੼
    • ࠺ਊ; DB੄ ਊ۝݅ఀ ݫݽܻо ೙ਃ
    • ௼ӝ: 60GB (୭ࣗ cache.m4.4xlarge ೙ਃ)

    => 1֙ ৘ডী $3100. replica setਸ ҳࢿೞҊ, ਊ۝੉ טযաݶ…?

    View Slide

  20. 20
    2. DynamoDB
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    ৮߷ೞѱ ҙܻغח NoSQL ؘ੉ఠ߬੉झ ࢲ࠺झ
    • ੢੼
    • Key-value ߑधী ੸೤
    • ױ੼
    • ࠺ਊ; ೠ ۨ௏٘੄ ௼ӝо ੘ਸ ٸ݅ ਬܻ
    • ௼ӝ: 60GB (ೠ ۨ௏٘੄ ௼ӝ: 100KB)
    • ੍ӝ/ॳӝ പࣻ: ୡ׼ 40/100ഥ

    => ೠ ׳ী $5782.80

    View Slide

  21. 21
    3. Elasticsearch
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    ࠙࢑ഋ RESTful Ѩ࢝ ߂ ࠙ࢳ ূ૓
    • ੢੼
    • Ѩ࢝ APIীࢲ উ੿੸ਵ۽ ੜ ࢎਊೞҊ ੓਺
    • ױ੼
    • Bulk loadী ੸೤ೞ૑ ঋ਺; load average ޙઁ ߊࢤ
    • Key-value ߑधী ੸೤ೞ૑ ঋ਺

    View Slide

  22. ׮द MongoDB۽…
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • Үള: ਋ܻ ࢚ടী MongoDBо ઁੌ ੜ ݏח Ѫ э׮.
    • HBaseࠁ׮ উ੿੸ੋ ࢲࡂ
    • ࠺Ү੸ ੷۴ೠ ࠺ਊ (ElastiCache + Redis, DynamoDB, …)
    • Bulk loadೡ ٸ ޙઁ੼਷…
    • Bulk loadೡ ٸ݃׮ ࢜ instance۽ Ү୓ೞ੗!
    22

    View Slide

  23. 1. উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    2. ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    3. API ਽׹दр ઴੉ӝ
    23
    ୶ୌ API ୭੸ച ೞӝ
    ୶ୌ API ୭੸ച ೞӝ

    View Slide

  24. ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ

    View Slide

  25. 25
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    AI ୶ୌ੄ ୶ୌ ઁ৻

    View Slide

  26. 26
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    DB ҳઑ (୶ୌ ઑഥद)

    View Slide

  27. 27
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    DB ҳઑ (୶ୌઁ৻ ١۾द)
    Single Point of Failure!!

    View Slide

  28. SPoF ੄ ৔ೱ
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    • ؘ੉ఠࣃఠ ੹ਗ ੢গ۽ ੋ೧ ؘ੉ఠ౱ ݃झఠ DB ׮਍
    • DB ࢲߡ ࠂҳदө૑ ୶ୌઁ৻ ӝמ ࢎਊࠛо
    • DB ࢲߡ ݫݽܻ ૐࢸਸ ਤ೧ ࢲߡ ੤ࠗ౴ ೙ਃ
    • ੤ࠗ౴ೞח زউ ୶ୌઁ৻ ӝמ ࢎਊࠛо

    View Slide

  29. SPoF ੄ ৔ೱ
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    ୶ୌઁ৻ ప੉࠶݅ ࡅ૓׮ݶ...

    View Slide

  30. ࢿѺ੉ ׮ܲ ప੉࠶ٜ੄ ҕઓ
    30
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    • ୶ୌઁ৻ ప੉࠶
    • 24दр ղղ INSERT ߊࢤ
    • ؀Ҋё ࢲ࠺झ৬ ޻੽ೠ ҙ۲
    • ా҅ ప੉࠶
    • ࢜߷ ߓ஖ प೯दী݅ INSERT ߊࢤ
    • ഥࢎ ղ ࢲ࠺झী݅ ҙ۲
    => ߹ب੄ ੋ೐ۄী فחѱ જѷ׮!

    View Slide

  31. ୶ୌઁ৻ ప੉࠶ਸ Aurora۽ ܻ࠙ೞӝ۽ Ѿ੿
    31
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    • ҙܻഋ ࢲ࠺झ੄ ಞܻೣ
    • ഛ੢ࢿ, উ੿ࢿ ഛࠁ
    • 10ms ޷݅੄ ܻ೒ܻா੉࣌ ۑ

    View Slide

  32. Aurora ܻ࠙੉੹ റ
    32
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    • ӝઓ DB ࢲߡח ঱ઁٚ ੤ࠗ౴ оמ
    • ୶ୌઁ৻ ӝמী Րӣহ੉ ݫݽܻ ૐࢸী ࢿҕ
    • DB ਍৔ উ੿ࢿ ૐо
    • ׮ܲ ࢲ࠺झী ৔ೱ߉૑ ঋ਺
    • ݃झఠо ইצ ठۨ੉࠳ীࢲ ੍ب۾ ߸҃
    • ઴যٚ ܻ೒ܻா੉࣌ ۑ ؋࠙ী оמ

    View Slide

  33. թ਷ ࣼઁ
    33
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    • Auroraীח JOINী ೙ਃೠ ప੉࠶੉ হ਺
    • (അ੤) য೒ܻா੉࣌ ױ҅ীࢲ JOIN ࣻ೯
    • ܻ೒ܻா੉࣌ ࢸ੿
    • APIܳ ࢎਊೠ ઱ӝ੸ੋ সؘ੉౟

    View Slide

  34. 1. উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    2. ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    3. API ਽׹दр ઴੉ӝ
    34
    ୶ୌ API ୭੸ച ೞӝ
    ୶ୌ API ୭੸ച ೞӝ

    View Slide

  35. API ਽׹दр ઴੉ӝ

    View Slide

  36. 36
    ޙઁ ੋध
    API ਽׹दр ઴੉ӝ

    View Slide

  37. 37
    ޙઁ ഛੋ
    API ਽׹दр ઴੉ӝ
    2׳ р 3ୡ ੉࢚੄ ਽׹दрਸ о૓
    AI ୶ୌ੄ API ਃ୒
    = ড 4݅Ѥ (੹୓ ਃ୒੄ 0.1359%)

    View Slide

  38. 38
    ޙઁ ૓ױ 1
    API ਽׹दр ઴੉ӝ
    Django ORM੉ ࢤࢿೠ ҳݒݾ۾ ઑഥ ௪ܻ:
    EXPLAIN SELECT `tb_user_book`.`pk_id`, `tb_user_book`.`u_id`,…
    => tb_user_book ઑഥо ੄बػ׮.

    View Slide

  39. 39
    ޙઁ ૓ױ 2
    API ਽׹दр ઴੉ӝ
    • Ѩૐ
    • ؘ੉ఠ౱ MariaDBо ಣࣗ੄ ౟ې೗ (࠙׼ ୭؀ 600ߣ ઑഥ)
    ਸ Ѽ٣ח૑ ഛੋ
    • Ѿҗ
    • ҳݒݾ۾ ઑഥীࢲ ठ۽਋ ௪ܻ ߊࢤ!
    • ؘ੉ఠ౱ DBח நएо ੸ѱ غҊ ੓਺

    “ࢲ੼ DBীח ੉۠ ޙઁо হ঻؍ Ѫ э਷ؘ…?”

    View Slide

  40. 40
    API ਽׹दр ઴੉ӝ
    ୶ୌ APIীࢲ੄ ҳݒݾ۾ ઑഥܳ হগ੗!

    View Slide

  41. ࢎਊ੗ ೐۽౵ੌ
    - ҳݒ
    - ୭Ӕ ઑഥ
    41
    API ਽׹दр ઴੉ӝ
    بࢲр ਬࢎب
    Batch
    API
    AI ୶ୌ
    ୶ୌ ܻझ౟ ࢤࢿ җ੿

    View Slide

  42. ࢎਊ੗ ೐۽౵ੌ
    - ҳݒ
    - ୭Ӕ ઑഥ
    42
    API ਽׹दр ઴੉ӝ
    بࢲр ਬࢎب
    Batch
    API
    AI ୶ୌ
    ୶ୌ ܻझ౟ ࢤࢿ җ੿

    View Slide

  43. 43
    ҳݒݾ۾ ӝ߈ ୶ୌ ܻझ౟ܳ ޷ܻ ҅࢑ೞ੗!
    API ਽׹दр ઴੉ӝ
    بࢲр ਬࢎب
    Batch
    API
    ҳݒ ೐۽౵ੌ ҳݒӝ߈ ୶ୌ
    ઑഥ ೐۽౵ੌ ઑഥӝ߈ ୶ୌ AI ୶ୌ

    View Slide

  44. 44
    ೧Ѿ଼ Ѩૐ
    API ਽׹दр ઴੉ӝ
    3ୡ ੉࢚੄ ਽׹दрਸ о૓
    AI ୶ୌ੄ API ਃ୒
    = ѐࢶ ੹੄ 1/8 

    (੹୓ ਃ୒੄ 0.0176%)
    * MongoDBܳ ࢎਊೞݶ ੉ ࠗ࠙ب ѐࢶؼ Ѫਸ ৘࢚೤פ׮.

    View Slide

  45. 45
    Ѿۿ
    • ו՛ ੼
    • ݣ૽൤ ੜ ࢲࡂೞח Ѥ য۰਍ ੌ
    • ݽܰҊ ࢎਊೞח Ѫҗ ঌҊ ࢎਊೞח Ѫ਷ ׮ܰ׮
    • খਵ۽ ೡ ੌ
    • ؘ੉ఠ ࢎ੉঱౭झ౟ ࡳӝ
    • അ ࢚ടী উ઱ೞӝ ঋӝ
    ୶ୌ API ୭੸ച ೞӝ

    View Slide

  46. 46
    THANK YOU
    хࢎ೤פ׮

    View Slide