Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building & Deploying CF

15bd1106c5a327d6564f793a1951f269?s=47 Buzzvil
November 17, 2021

Building & Deploying CF

By Peter Kim

15bd1106c5a327d6564f793a1951f269?s=128

Buzzvil

November 17, 2021
Tweet

Transcript

  1. Building & Deploying CF Improving Buzzvil’s Retargeting & my life

    quality by 10X
  2. About myself • 충성충성 • ML Engineer (~3 yrs) •

    Ad Display Team
  3. About myself • 충성충성 • ML Engineer (~3 yrs) •

    Ad Display Team
  4. How did I spend my time? • 99% Data Processing

    & Ops • 1% Machine Learning
  5. How do I spend my time now? • 99% 90%

    Data Processing & Ops • 1% 10% Machine Learning
  6. 월 부킹 금액 10억+ (>20%의 전체 매출)

  7. Retargeting

  8. Retargeting as a recommendation problem

  9. As with any problem, there are many solutions. • Best

    Selling • Most Recently Viewed / Carted / Purchased • BERT • Collaborative Filtering (CF)
  10. As with any problem, there are many solutions • Best

    Selling • Most Recently Viewed / Carted / Purchased • BERT • Collaborative Filtering (CF)
  11. CF Intuition: 비슷한 유저를 찾아라!

  12. 0 for Not Purchased, 1 for Purchased Product 1 Product

    2 Product 3 Product 4 Product 5 User A 1 0 1 0 1 User B 0 1 0 1 1 User C 1 1 1 0 0 User D 0 1 0 0 1
  13. Q: 유저 D가 산 상품들은? Product 1 Product 2 Product

    3 Product 4 Product 5 User A 1 0 1 0 1 User B 0 1 0 1 1 User C 1 1 1 0 0 User D 0 1 0 0 1
  14. Q: 유저 D가 산 상품들은? Product 1 Product 2 Product

    3 Product 4 Product 5 User A 1 0 1 0 1 User B 0 1 0 1 1 User C 1 1 1 0 0 User D 0 1 0 0 1
  15. Item-to-Item CF: 비슷한 유저 아이템을 찾아라! Product 1 Product 2

    Product 3 Product 4 Product 5 User A 1 0 1 0 1 User B 0 1 0 1 1 User C 1 1 1 0 0 User D 0 1 0 0 1
  16. E.g. Similarity of Product 2 & Product 5? Product 1

    Product 2 Product 3 Product 4 Product 5 User A 1 0 1 0 1 User B 0 1 0 1 1 User C 1 1 1 0 0 User D 0 1 0 0 1
  17. E.g. Similarity of Product 2 & Product 5? Product 1

    Product 2 Product 3 Product 4 Product 5 User A 1 0 1 0 1 User B 0 1 0 1 1 User C 1 1 1 0 0 User D 0 1 0 0 1 Sim(P2, P5) = 2
  18. E.g. Similarity of Product 2 & Product 1? Product 1

    Product 2 Product 3 Product 4 Product 5 User A 1 0 1 0 1 User B 0 1 0 1 1 User C 1 1 1 0 0 User D 0 1 0 0 1 Sim(P2, P1) = 1
  19. Many similarity functions can be used between two vectors •

    • Product 4 0 1 0 0 Product 2 0 1 1 1
  20. So how do we build a recommendation system with this?

  21. 1. Match each of the user’s purchased items to similar

    items 2. Combine them into a recommendation list
  22. 쿠팡

  23. Netflix

  24. Youtube

  25. Amazon

  26. For Buzzvil? • 2차 테스트 중간 결과는 긍정적, 더블체크를 위해

    재 테스트 중 더 궁금하시면, Confluence 문서와 Redash에서 확인: 문서 #1, 문서 #2,
  27. ItemCF was popularized by Amazon, 20 years ago

  28. Scalable algorithm to compute item similarities

  29. 30일치의 SSG 구매 데이터를 가지고 Item Similarities를 Compute해보자 Product 1

    ... Product 400,000 User 1 1 ... 1 ... ... ... ... User 600,000 0 ... 1
  30. Version 1: Python Naive Version

  31. Time complexity = 7200 hours, 300일밖에 안걸리네...흠

  32. Find the bottleneck!

  33. Find the bottleneck!

  34. Q: How to optimize the cosine similarity computation?

  35. Q: How to optimize the cosine similarity computation? Product 4

    0 1 0 0 Product 2 0 1 1 1
  36. Hint: What is the difference between these two vectors? Product

    2 0 1 1 1 Product 1 ... Product 400,000 User 1 1 ... 1 ... ... ... ... User 600,000 0 ... 1 Vs.
  37. Hint: What is the difference between these two vectors? Product

    2 0 1 1 1 Product 1 ... Product 400,000 User 1 1 ... 1 ... ... ... ... User 600,000 0 ... 1 Vs. Sparse!
  38. None
  39. Using sparse vectors reduces both memory and time taken from

    scipy.sparse import csr_matrix from sklearn.metrics.pairwise import cosine_similarity a = csr_matrix(...) b = csr_matrix(...) sim = cosine_similarity(a, b) # 6x faster
  40. Using sparse vectors reduces both memory and time taken Naive

    version now takes 1200 hours (50일) 300일 전역하기 전까지는 끝낼 수 있다 from scipy.sparse import csr_matrix from sklearn.metrics.pairwise import cosine_similarity a = csr_matrix(...) b = csr_matrix(...) sim = cosine_similarity(a, b) # 6x faster
  41. 50 days is still not fast enough for production!

  42. Embarrassingly Parallel Workload

  43. Python Distributed Applications API: import ray ray.init() @ray.remote def compute_partial(...):

    # compute partial similarities table futures = [compute_partial(), compute_partial(), ...] ray.get(futures)
  44. 7200 hours, 1200 hours, 4 hours Python Multiprocessing with Ray

    on r5.12xlarge Instance (48 cores) takes ~4 hours
  45. 7200 hours, 1200 hours, 4 hours Python Multiprocessing with Ray

    on r5.12xlarge Instance (48 cores) takes ~4 hours Now ready for production!
  46. How many EC2 instances do we need? train-ssg train-emart train-hs

    train-ns infer-ssg infer-emart infer-hs infer-ns
  47. How do we manage this? $ ssh $ tmux $

    git pull $ crontab
  48. 이걸 내가 혼자서 다? train-ssg train-emart train-hs train-ns infer-ssg infer-emart

    infer-hs infer-ns $ ssh $ tmux $ git pull $ crontab
  49. 이걸 내가 혼자서 다? train-ssg train-emart train-hs train-ns infer-ssg infer-emart

    infer-hs infer-ns $ ssh $ tmux $ git pull $ crontab
  50. The wheel has been invented but it’s not yet ready

    DY 화이팅!
  51. Until then, let’s make my own.

  52. Inspiration • DAGs & Scheduling from AirFlow ◦ Decoupling scheduling

    and task details • Serverless from Lambda ◦ No more SSH, Tmux, Crontab
  53. Let’s see it in action!

  54. Lessons Learned • Use sparse vectors to save memory and

    time • Optimize algorithms from the single-core level, then multi-core • Deploying & managing ML is hell but most of it can be automated • Building your own tools can increase your life quality by 10X ◦ Python + boto3 makes it very easy. PeterFlow is <200 lines.
  55. 10X Life Quality Improvement