추천 API 최적화하기

Transcript

1. ୶ୌ API
   ୭੸ച ೞӝ
   2018֙ 8ਘ 31ੌ ਬ૑ള
   © RIDI Corp.
   

   View Slide

2. 1. ୶ୌ API ࣗѐ
   • Book-to-book ୶ୌ
   • Book-to-user ୶ୌ
   • API ҳઑ

   2. ୶ୌ API ୭੸ച ೞӝ
   • উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
   • ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
   • API ਽׹दр ઴੉ӝ
   2
   ݾର
   ୶ୌ API ୭੸ച ೞӝ
   

   View Slide

3. 3
   ܻ٣࠘झ੄ Book-to-book ୶ୌ
   ୶ୌ API ࣗѐ
   

   View Slide

4. 4
   بࢲр ਬࢎب ৘द
   ୶ୌ API ࣗѐ
   0.11651 0.07987
   

   View Slide

5. 5
   بࢲр ਬࢎب ҅࢑
   ୶ୌ API ࣗѐ
   1. ҳݒݾ۾
   • ଲ਋: ؀ా۸੄ Ӗॳӝ
   • അ୍: ؀ా۸੄ Ӗॳӝ, যڌѱ ࢓ Ѫੋо
   • ޹ࢳ: যڌѱ ࢓ Ѫੋо
   2. بࢲӝળ ҳݒݾ۾
   • ؀ా۸੄ Ӗॳӝ: ଲ਋, അ୍
   • যڌѱ ࢓ Ѫੋо: അ୍, ޹ࢳ
   3. ਬࢎب (؀ా۸, যڌѱ)
   = |{അ୍}| / |{ଲ਋, അ୍, ޹ࢳ}| ≈ 0.33
   ?
   

   View Slide

6. 6
   ܻ٣࠘झ੄ Book-to-user ୶ୌ
   ୶ୌ API ࣗѐ
   

   View Slide

7. بࢲр ਬࢎب৬ ਬ੷ ೐۽౵ੌਸ ࢎਊ
   • അ੤ח ҳݒ৬ ୭Ӕ ઑഥೠ بࢲٜ۽ ೐۽౵ੌ ࢤࢿ
   • ೐۽౵ੌҗ ਬࢎبо ֫਷ بࢲٜਸ ୶ୌ
   7
   ܻ٣࠘झ੄ Book-to-user ਗܻ
   ୶ୌ API ࣗѐ
   

   View Slide

8. recommend-api
   data-api.ridibooks.com
   Classic
   Load
   Balancer
   Store Web
   Server
   Client
   Store Web
   Elasticache
   (Redis)
   proxy-azure.data.ridi.io
   hbase.data.ridi.io
   HAProxy
   192.168.0.200:33309
   HAProxy
   192.168.0.200:33319
   HAProxy
   aurora-db.data.ridi.io
   aurora-db.ro.data.ridi.io
   Aurora
   HBase Cluster
   HTTP
   Request
   Thrift RPC
   MySQL
   CRUD
   Cache
   MySQL
   Write
   MySQL
   Read
   Plain
   Requests
   Load
   Balancing
   ridibooks.com
   HTTP
   Request
   8
   API ҳઑ
   ୶ୌ API ࣗѐ
   

   View Slide

9. 1. উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
   2. ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
   3. API ਽׹दр ઴੉ӝ
   9
   ୶ୌ API ୭੸ച ೞӝ
   ୶ୌ API ୭੸ച ೞӝ
   

   View Slide

10. উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    

    View Slide

11. 11
    ୶ୌ ؘ੉ఠ
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • Bulk write৬ real-time read
    • ড 150݅Ѥ੄ book-to-book ୶ୌҗ 

    ড 90݅Ѥ੄ book-to-user ୶ୌਸ 

    पदрਵ۽ ࢲࡂೞҊ ੓ח ੷੢ࣗী স۽٘
    • Key-value
    • Key: b_id/u_idx
    • Value: بࢲ ୶ୌ ܻझ౟੄ JSON String
    

    View Slide

12. After
    • Scala, Python
    • Multi-machine batch
    • ড 2दр
    • MongoDBী স۽٘
    Before
    • C++
    • Single-machine batch
    • ড 10दр
    • MariaDBী স۽٘
    12
    ௿۞झఠ بੑ ੹/റ
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    

    View Slide

13. 13
    Bulk Loadী ஂডೠ MongoDB
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • ড 1.6রѐ੄ rowܳ bulk loadೡ ٸ, load average ҃Ҋ ߊࢤ
    • Bulk loadܳ ೡ ٸ݅ ೐۽ࣁझ੄ ࣻܳ ઴ৈࢲ प೯
    => ৈ੹൤ ޙઁ ߊࢤ!
    

    View Slide

14. 14
    HBase بੑ
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    1. Bulk loadী ੸೤
    2. ೞن ী௏दझమীࢲ ࢎਊ оמ
    • औѱ بੑೡ ࣻ ੓਺
    • ഛ੢ࢿ੉ ڪযթ
    

    View Slide

15. 15
    ࠛউ੿ೠ HBase
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • ਽׹ दр੉ оՔঀ ౓ח അ࢚ਸ ߊѼ
    

    View Slide

16. 16
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • HBase ࢲ࠺झܳ प೯ೞ૑ ঋח machineীࢲ݅ 

    ׮ܲ batch ੘স੉ प೯غب۾ ࢸ੿
    1. ׮ܲ Batch ੘স੄ ৔ೱਸ ߉Ҋ ੓૑ ঋਸө?
    => ৈ੹൤ ޙઁ ߊࢤ!
    ޙઁ ૓ױ 1
    

    View Slide

17. 17
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • VCNC৬ IBM੄ о੉٘ ١ਸ ଵҊೞৈ ౚ׬:
    2. HBase ౚ׬੄ ޙઁ ইקө?
    => ઑӘ աই૑ӟ ೮૑݅, ৈ੹൤ ޙઁ ߊࢤ!
    ޙઁ ૓ױ 2
    * http://engineering.vcnc.co.kr/2013/04/hbase-configuration/
    * https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.0/
    com.ibm.swg.im.infosphere.biginsights.analyze.doc/doc/bigsql_TuneHbase.html
    ড 20ѐ੄ ౵ۄ޷ఠܳ Ѩషೠ റ, 5ѐ੄ ౵ۄ޷ఠܳ ౚ׬.

    ৘) ৻ࠗ ਃ୒ਸ ୊ܻೞח ؘ ࢎਊೡ Thread੄ ࣻܳ ૐо
    

    View Slide

18. 18
    ޙઁ ૓ױ 3
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    3. HappyBase (Python-HBase connector) ੄ ޙઁо ইקө?
    • Locust۽ ۽٘ పझ౟ܳ ೮ਸ ٸ, ௾ ੉गо হ਺
    • HappyBase৬ Django৬੄ ഐജࢿ ੄ब
    => ೞ૑݅, Thrift RPC ࢲߡীࢲ response time ۽Ӓ ߂ 

    ठ۽਋ ௪ܻ ࠙ࢳ ١ਸ ೡ ࣻ হ਺.
    

    View Slide

19. 19
    1. ElastiCache + Redis
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    ҙܻഋ ੋ ݫݽܻ ؘ੉ఠ झషয ࢲ࠺झ
    • ੢੼
    • Key-value ߑधী ੸೤
    • ࡅܲ ਽׹ ࣘب
    • ױ੼
    • ࠺ਊ; DB੄ ਊ۝݅ఀ ݫݽܻо ೙ਃ
    • ௼ӝ: 60GB (୭ࣗ cache.m4.4xlarge ೙ਃ)

    => 1֙ ৘ডী $3100. replica setਸ ҳࢿೞҊ, ਊ۝੉ טযաݶ…?
    

    View Slide

20. 20
    2. DynamoDB
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    ৮߷ೞѱ ҙܻغח NoSQL ؘ੉ఠ߬੉झ ࢲ࠺झ
    • ੢੼
    • Key-value ߑधী ੸೤
    • ױ੼
    • ࠺ਊ; ೠ ۨ௏٘੄ ௼ӝо ੘ਸ ٸ݅ ਬܻ
    • ௼ӝ: 60GB (ೠ ۨ௏٘੄ ௼ӝ: 100KB)
    • ੍ӝ/ॳӝ പࣻ: ୡ׼ 40/100ഥ

    => ೠ ׳ী $5782.80
    

    View Slide

21. 21
    3. Elasticsearch
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    ࠙࢑ഋ RESTful Ѩ࢝ ߂ ࠙ࢳ ূ૓
    • ੢੼
    • Ѩ࢝ APIীࢲ উ੿੸ਵ۽ ੜ ࢎਊೞҊ ੓਺
    • ױ੼
    • Bulk loadী ੸೤ೞ૑ ঋ਺; load average ޙઁ ߊࢤ
    • Key-value ߑधী ੸೤ೞ૑ ঋ਺
    

    View Slide

22. ׮द MongoDB۽…
    উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    • Үള: ਋ܻ ࢚ടী MongoDBо ઁੌ ੜ ݏח Ѫ э׮.
    • HBaseࠁ׮ উ੿੸ੋ ࢲࡂ
    • ࠺Ү੸ ੷۴ೠ ࠺ਊ (ElastiCache + Redis, DynamoDB, …)
    • Bulk loadೡ ٸ ޙઁ੼਷…
    • Bulk loadೡ ٸ݃׮ ࢜ instance۽ Ү୓ೞ੗!
    22
    

    View Slide

23. 1. উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    2. ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    3. API ਽׹दр ઴੉ӝ
    23
    ୶ୌ API ୭੸ച ೞӝ
    ୶ୌ API ୭੸ച ೞӝ
    

    View Slide

24. ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    

    View Slide

25. 25
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    AI ୶ୌ੄ ୶ୌ ઁ৻
    

    View Slide

26. 26
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    DB ҳઑ (୶ୌ ઑഥद)
    

    View Slide

27. 27
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    DB ҳઑ (୶ୌઁ৻ ١۾द)
    Single Point of Failure!!
    

    View Slide

28. SPoF ੄ ৔ೱ
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    • ؘ੉ఠࣃఠ ੹ਗ ੢গ۽ ੋ೧ ؘ੉ఠ౱ ݃झఠ DB ׮਍
    • DB ࢲߡ ࠂҳदө૑ ୶ୌઁ৻ ӝמ ࢎਊࠛо
    • DB ࢲߡ ݫݽܻ ૐࢸਸ ਤ೧ ࢲߡ ੤ࠗ౴ ೙ਃ
    • ੤ࠗ౴ೞח زউ ୶ୌઁ৻ ӝמ ࢎਊࠛо
    

    View Slide

29. SPoF ੄ ৔ೱ
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    ୶ୌઁ৻ ప੉࠶݅ ࡅ૓׮ݶ...
    

    View Slide

30. ࢿѺ੉ ׮ܲ ప੉࠶ٜ੄ ҕઓ
    30
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    • ୶ୌઁ৻ ప੉࠶
    • 24दр ղղ INSERT ߊࢤ
    • ؀Ҋё ࢲ࠺झ৬ ޻੽ೠ ҙ۲
    • ా҅ ప੉࠶
    • ࢜߷ ߓ஖ प೯दী݅ INSERT ߊࢤ
    • ഥࢎ ղ ࢲ࠺झী݅ ҙ۲
    => ߹ب੄ ੋ೐ۄী فחѱ જѷ׮!
    

    View Slide

31. ୶ୌઁ৻ ప੉࠶ਸ Aurora۽ ܻ࠙ೞӝ۽ Ѿ੿
    31
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    • ҙܻഋ ࢲ࠺झ੄ ಞܻೣ
    • ഛ੢ࢿ, উ੿ࢿ ഛࠁ
    • 10ms ޷݅੄ ܻ೒ܻா੉࣌ ۑ
    

    View Slide

32. Aurora ܻ࠙੉੹ റ
    32
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    • ӝઓ DB ࢲߡח ঱ઁٚ ੤ࠗ౴ оמ
    • ୶ୌઁ৻ ӝמী Րӣহ੉ ݫݽܻ ૐࢸী ࢿҕ
    • DB ਍৔ উ੿ࢿ ૐо
    • ׮ܲ ࢲ࠺झী ৔ೱ߉૑ ঋ਺
    • ݃झఠо ইצ ठۨ੉࠳ীࢲ ੍ب۾ ߸҃
    • ઴যٚ ܻ೒ܻா੉࣌ ۑ ؋࠙ী оמ
    

    View Slide

33. թ਷ ࣼઁ
    33
    ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    • Auroraীח JOINী ೙ਃೠ ప੉࠶੉ হ਺
    • (അ੤) য೒ܻா੉࣌ ױ҅ীࢲ JOIN ࣻ೯
    • ܻ೒ܻா੉࣌ ࢸ੿
    • APIܳ ࢎਊೠ ઱ӝ੸ੋ সؘ੉౟
    

    View Slide

34. 1. উ੿੸ੋ ࢲࡂਸ ਤೠ ੷੢ࣗ ଺ӝ
    2. ୶ୌ ઁ৻ ؘ੉ఠ߬੉झ ܻ࠙ೞӝ
    3. API ਽׹दр ઴੉ӝ
    34
    ୶ୌ API ୭੸ച ೞӝ
    ୶ୌ API ୭੸ച ೞӝ
    

    View Slide

35. API ਽׹दр ઴੉ӝ
    

    View Slide

36. 36
    ޙઁ ੋध
    API ਽׹दр ઴੉ӝ
    

    View Slide

37. 37
    ޙઁ ഛੋ
    API ਽׹दр ઴੉ӝ
    2׳ р 3ୡ ੉࢚੄ ਽׹दрਸ о૓
    AI ୶ୌ੄ API ਃ୒
    = ড 4݅Ѥ (੹୓ ਃ୒੄ 0.1359%)
    

    View Slide

38. 38
    ޙઁ ૓ױ 1
    API ਽׹दр ઴੉ӝ
    Django ORM੉ ࢤࢿೠ ҳݒݾ۾ ઑഥ ௪ܻ:
    EXPLAIN SELECT `tb_user_book`.`pk_id`, `tb_user_book`.`u_id`,…
    => tb_user_book ઑഥо ੄बػ׮.
    

    View Slide

39. 39
    ޙઁ ૓ױ 2
    API ਽׹दр ઴੉ӝ
    • Ѩૐ
    • ؘ੉ఠ౱ MariaDBо ಣࣗ੄ ౟ې೗ (࠙׼ ୭؀ 600ߣ ઑഥ)
    ਸ Ѽ٣ח૑ ഛੋ
    • Ѿҗ
    • ҳݒݾ۾ ઑഥীࢲ ठ۽਋ ௪ܻ ߊࢤ!
    • ؘ੉ఠ౱ DBח நएо ੸ѱ غҊ ੓਺
    
    “ࢲ੼ DBীח ੉۠ ޙઁо হ঻؍ Ѫ э਷ؘ…?”
    

    View Slide

40. 40
    API ਽׹दр ઴੉ӝ
    ୶ୌ APIীࢲ੄ ҳݒݾ۾ ઑഥܳ হগ੗!
    

    View Slide

41. ࢎਊ੗ ೐۽౵ੌ
    - ҳݒ
    - ୭Ӕ ઑഥ
    41
    API ਽׹दр ઴੉ӝ
    بࢲр ਬࢎب
    Batch
    API
    AI ୶ୌ
    ୶ୌ ܻझ౟ ࢤࢿ җ੿
    

    View Slide

42. ࢎਊ੗ ೐۽౵ੌ
    - ҳݒ
    - ୭Ӕ ઑഥ
    42
    API ਽׹दр ઴੉ӝ
    بࢲр ਬࢎب
    Batch
    API
    AI ୶ୌ
    ୶ୌ ܻझ౟ ࢤࢿ җ੿
    

    View Slide

43. 43
    ҳݒݾ۾ ӝ߈ ୶ୌ ܻझ౟ܳ ޷ܻ ҅࢑ೞ੗!
    API ਽׹दр ઴੉ӝ
    بࢲр ਬࢎب
    Batch
    API
    ҳݒ ೐۽౵ੌ ҳݒӝ߈ ୶ୌ
    ઑഥ ೐۽౵ੌ ઑഥӝ߈ ୶ୌ AI ୶ୌ
    

    View Slide

44. 44
    ೧Ѿ଼ Ѩૐ
    API ਽׹दр ઴੉ӝ
    3ୡ ੉࢚੄ ਽׹दрਸ о૓
    AI ୶ୌ੄ API ਃ୒
    = ѐࢶ ੹੄ 1/8 

    (੹୓ ਃ୒੄ 0.0176%)
    * MongoDBܳ ࢎਊೞݶ ੉ ࠗ࠙ب ѐࢶؼ Ѫਸ ৘࢚೤פ׮.
    

    View Slide

45. 45
    Ѿۿ
    • ו՛ ੼
    • ݣ૽൤ ੜ ࢲࡂೞח Ѥ য۰਍ ੌ
    • ݽܰҊ ࢎਊೞח Ѫҗ ঌҊ ࢎਊೞח Ѫ਷ ׮ܰ׮
    • খਵ۽ ೡ ੌ
    • ؘ੉ఠ ࢎ੉঱౭झ౟ ࡳӝ
    • അ ࢚ടী উ઱ೞӝ ঋӝ
    ୶ୌ API ୭੸ച ೞӝ
    

    View Slide

46. 46
    THANK YOU
    хࢎ೤פ׮
    

    View Slide