Amazon Personalize - 추천 시스템의 원리와 구축 사례

Agenda
- 추천의 정의
- 추천의 중요성
- 추천 성과 지표
- Amazon Personalize 소개
- Amazon Personalize 적용 아키텍처

Transcript

1. © 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved | 1 © 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved 추천 시스템의 원리와 구축 사례 Sungmin Kim AWS Solutions Architect Amazon Personalize
2. Agenda • 추천의 정의 • 추천의 중요성 • 추천 성과 지표 • Amazon Personalize 소개 • Amazon Personalize 적용 아키텍처
3. “추천” 이란 무엇일까?
4. “추천”은 왜 중요할까? E-Commerce의 중요한 지표 • Retention Rate – 고객 체류 시간 • Churn Rate – 고객 이탈율 • (Purchase) Conversion Rate – (구매) 전환율 추천 검색 결제
5. “추천”은 왜 중요할까? 검색 • 원하는 결과를 빨리 찾기 • 정확 • 신속 추천 • 새로운 경험 • 탐험, 탐색 • 발견의 기쁨
6. 추천 알고리즘의 종류 • Collaborative Filtering • Content-based Filtering • Hybrid = Collaborative Filtering + Content-based Filtering
7. Collaborative Filtering user1 user2 user3 user1 user2 user3
8. Content-based Filtering article read by me Similar Recommend new article Read
9. Collaborative Filtering (+) • Content 를 분석 할 필요 없음. • 사용자 행동 로그만으로 추천 계산 가능. • 다양한 곳에 적용 가능. (-) • Cold Start 문제 발생. Content-based Filtering (-) • Content가 풍부한 곳에 적용 가능. • 적용 범위가 비교적 제한적. (예: Music, Movies 등은 적용이 쉽지 않음) (+) • Content 분석 만으로 추천 가능. • 사용자 행동 로그 없이 추천 계산 가능. • Cold Start 문제 완화. Hybrid = Collaborative + Content-based
10. 어떻게 추천을 적용할까? Personalization Non-Personalization
11. 추천 CF Content -based Hybrid Non- Personalization Personalization Cold-Start Problem User-Item Interactions (e.g. View, Buy, Cart) Content (e.g. Customer reviews, Product Details) 데이터 적용
12. 데이터 준비 요구 사항 정리 추천 모델 개발 배포 및 적용 A/B Test 새로운 Feature 새로운 요구 사항 추천 서비스 개발 및 적용 - 반복적인 작업
13. Deliver high-quality recommendations Deliver personalization in days, not months Real-time Works with any product or content Amazon Personalize amazon.com 에서 사용하는 머신 러닝 기술을 이용한 실시간 개인화 및 추천 서비스
14. Solution (Recipes) Model selection, training, tunning and verification Campaign Model hosting, and inference Amazon Personalize Data Set Group Users Items Interactions Data Sets User events / interactions Item meta data (a.k.a catalog information - optional) User meta data (e.g. demographics – optional) Amazon Personalize How it works • GetRecommendations • GetPersonalizedRanking
15. Setting up Amazon Personalize
16. 어떤 데이터를 준비해야할까? • 3 가지 데이터 • Users • 사용자 메타 데이터 • 연령, 성별, 고객 멤버쉽 등 • Items • Item 메타 데이터 • 가격, SKU(상품 재고 관리 단위), 재고 여부 등 • (User-Item) Interactions • 사용자의 Item에 대한 행동 로그 데이터 • 구매(buy), 장바구니 담기(cart), 상품 보기(view) 등 • Interactions 데이터는 추천 계산에 사용되므로 반드시 필요함 • User, Items는 추천 계산에서 사용할 데이터 제외 및 추천 결과 filtering 용도로 사용 • 입력 데이터는 S3에 CSV 포맷의 저장, 첫번째 row에 컬럼 Header가 필요함
17. Formatting Your Input Data Example 컬럼 Header CSV 포맷
18. Users Items Interactions https://docs.aws.amazon.com/personalize/latest/dg/how-it- works-dataset-schema.html
19. Users Items Interactions https://docs.aws.amazon.com/personalize/latest/dg/how-it- works-dataset-schema.html { "type": "record", "name": "Users | Items | Interactions", "namespace": "com.amazonaws.personalize.schema", "fields": [ { "name": "Field Name", "type": "Data Type" }, …. ], "version": "1.0" }
20. 데이터는 얼마나 준비해야할까? Minimum Suggested Data Volume • More than 50 users. • More than 50 items. • More than 1,500 interactions. ※ https://github.com/aws-samples/amazon-personalize-samples/blob/master/PersonalizeCheatSheet2.0.md
21. User personalization Personalized ranking Similar items Recipes • User-personalization • HRNN, HRNN-Metadata, HRNN-Coldstart(legacy) • Popularity-Count (baseline) Recipe • SIMS Recipe • Personalized-Ranking Use case by Recipes
22. 추천 데이터 Content -based 적용 Non- Personalization Personalization User-Item Interactions (e.g. View, Buy, Cart) Content (e.g. Customer reviews, Product Details) • User-Personalization • Popularity-Count • SIMS • Personalized-Ranking • SIMS • Popularity-Count • User-Personalization • Personalized-Ranking
23. 맛있는 요리처럼, 좋은 추천이란? • Coverage • Relevance (≈ Accuracy) • Mean Reciprocal Rank@K • NDCG@K • Precision@K Coverage Relevance ※ Relevance: https://docs.aws.amazon.com/personalize/latest/dg/working-with-training-metrics.html
24. 정말 더 좋은 추천이란? • Coverage • Relevance (≈ Accuracy) • Mean Reciprocal Rank@K • NDCG@K • Precision@K • Serendipity (≈ Surprise) Serendipity Coverage Relevance ※ Relevance: https://docs.aws.amazon.com/personalize/latest/dg/working-with-training-metrics.html
25. relevance serendipity Why Serendipity? It keeps your customers interested.
26. Amazon Personalize 정말 효과가 있을까? https://aws.amazon.com/ko/solutions/case-studies/brandi/
27. Amazon Personalize를 사용하는 고객들
28. Let’s build it
29. • visit • view • cart • buy Web Server Users Items Transactions (Interactions)
30. Web Server Reco. Server ? response request recommendations • visit • view • cart • buy Users Items Transactions (Interactions)
31. Web Server • visit • view • cart • buy Amazon Personalize Users Items Transactions (Interactions)
32. Amazon Personalize Data Set Group Users Items Interactions Solution (Recipes) Model selection, training, tunning and verification Campaign Model hosting, and inference Amazon Personalize Data Sets Users, Items, and Interactions
33. Dataset Group Dataset Import Web Server S3 Personalize Recipe & Solution Campaign • visit • view • cart • buy 자동화? Users Items Transactions (Interactions)
34. Dataset Group Dataset Import Web Server S3 Personalize Recipe & Solution Campaign • visit • view • cart • buy https://github.com/aws-samples/amazon-personalize-samples/tree/master/next_steps/operations/ml_ops AWS Step Functions 자동화? Users Items Transactions (Interactions)
35. https://github.com/aws-samples/amazon-personalize-samples/tree/master/next_steps/operations/ml_ops Amazon Personalize MLOps using Step Functions
36. Amazon Step Functions
37. Amazon Step Functions
38. #6: Dynamic parallelism #5: Parallel processing #4: Human in the loop #3: Error handling #2: Branching #1: Function orchestration Amazon Step Functions Use Cases
39. Amazon Personalize MLOps using Step Functions
40. Dataset Group Dataset Import Web Server S3 Personalize Recipe & Solution Campaign • visit • view • cart • buy AWS Step Functions Users Items Transactions (Interactions)
41. Web Server S3 Step Functions Personalize Runtime API • visit • view • cart • buy Users Items Transactions (Interactions) 사용자 Interactions 실시간 Update?
42. Amazon Personalize Data Set Group Users Items Interactions Solution (Recipes) Model selection, training, tunning and verification Campaign Model hosting, and inference Amazon Personalize Data Sets Users, Items, and Interactions
43. Amazon Personalize Data Set Group Users Items Interactions Solution (Recipes) Model selection, training, tunning and verification Campaign Model hosting, and inference Events Tracker Data Sets Users, Items, and Interactions Amazon Personalize
44. Amazon Personalize Data Set Group Users Items Interactions Solution (Recipes) Model selection, training, tunning and verification Campaign Model hosting, and inference Events Tracker Data Sets Users, Items, and Interactions Real-Time Events Amazon Personalize
45. Web Server S3 Step Functions Personalize Runtime API • visit • view • cart • buy Users Items Transactions (Interactions) 사용자 Interactions 실시간 Update
46. Web Server Event Tracker S3 Step Functions Personalize Runtime API Users Items Transactions (Interactions) 사용자 Interactions 실시간 Update • visit • view • cart • buy
47. Web Server Event Tracker S3 Step Functions Personalize Runtime API Users Items Transactions (Interactions) • visit • view • cart • buy 사용자 Interactions 실시간 Update
48. Web Server Event Tracker S3 Personalize Runtime API Step Functions Users Items Transactions (Interactions) Kinesis Data Streams 사용자 Interactions 실시간 Update • visit • view • cart • buy
49. Web Server Lambda Event Tracker S3 Personalize Runtime API Step Functions Users Items Transactions (Interactions) Kinesis Data Streams 사용자 Interactions 실시간 Update • visit • view • cart • buy PutEvents
50. Web Server Lambda Event Tracker S3 Kinesis Firehose Personalize Runtime API Step Functions Users Items Transactions (Interactions) Kinesis Data Streams • visit • view • cart • buy 사용자 Interactions 실시간 Update PutEvents
51. • Filter – 학습(training)에 사용할 records제외하기 • EVENT_TYPE – 특정 EVENT_TYPE의 records만 사용하고 싶은 경우 • EVENT_VALUE – 특정 threshold 값 이상의 records만 사용하고 싶은 경우 • (e.g. Review Rating > 2.0, Watch > 5) • Dynamic Filter – 추천 API 호출 시 추천 결과에서 제외 하기 • GetRecommendations • GetPersonalizedRanking Dataset Group Dataset Import Recipe & Solution Campaign Amazon Personalize • GetRecommendations • GetPersonalizedRanking Filtering Recommendations ※ https://docs.aws.amazon.com/personalize/latest/dg/filter.html
52. Dynamic Filter • Filtering by item • EXCLUDE ItemId WHERE items.genre IN ($GENRE) • EXCLUDE ItemId WHERE items.genre IN ("Comedy") • INCLUDE ItemId WHERE items.number_of_downloads < 20 • Filtering by interactions • INCLUDE ItemId WHERE interactions.event_type IN ("*") • EXCLUDE ItemId WHERE interactions.event_type IN (“click”, “stream”) • Filtering by item based on user properties • EXCLUDE ItemId WHERE items.number_of_downloads < 20 IF CurrentUser.age > 18 AND CurrentUser.age < 30 • INCLUDE Item.ID WHERE items.genre IN (“Comedy”) | EXCLUDE ItemID WHERE items.description IN ("classic”) ※ Amazon Personalize now supports dynamic filters for applying business rules to your recommendations on the fly
53. 추천 결과 추천 결과 Web Server Amazon Personalize X X Web Server Amazon Personalize X X X X Lambda X X X X Partially Filled Fully Filled 추천 Filtering을 Business Logic 처럼 적용하기 Amazon Personalize Filtering 사용 Filtering Business Logic 적용
54. Batch Recommendation • 사용 사례 § 많은 수의 사용자에 대한 추천을 한 번에 계산 및 저장 하고 싶은 경우 § 배치 기반 워크플로(예: 이메일 또는 알림 전송)를 통해 추천 결과를 지속적으로 제공하고 싶은 경우 • 비용 § 사용자들에게 동일한 추천 결과를 제공해도 되는 경우 § 배치 처리가 훨씬 더 편리하고 경제적 • 배치 추론 작업 방법 • AWS 웹 콘솔 • API 호출
55. Web Server S3 Kinesis Firehose API GW Personalize Batch Inference Users Items Transactions (Interactions) S3 Dynamo DB Glue Lambda Kinesis Data Streams Batch Recommendation • visit • view • cart • buy
56. 실시간 vs 배치 추천 비용 데이터 양 실시간 추천 배치 추천 비용 ü 학습용 데이터 양 ü 모델 학습 시간 ü 추론 API 호출 시간(TPS)
57. Summary • 추천의 중요성 - Retention Rate, Churn Rate, Conversion Rate 등의 E-Commerce 주요 지표에 영항을 줌 • 추천 평가 기준 - Coverage, Relevance, Serendipity의 조화 • Amazon Personalize 적용 시 필요한 데이터 - Users, Items, Interactions • Amazon Personalize Recipes - User-Personalization, SIMS, Popularity-Count, Personalized-Ranking • Amazon Personalize 적용 순서 - Data Ingestion(Data Set Group 생성 & Data Set Import) → Training (Recipe & Solution 생성) → Inference(Campaign 생성) • AWS Step Functions을 이용한 추천 계산 작업 자동화(MLOps) • 추천 결과 필터링 방법 - 추천 계산 시, Interactions 데이터 필터링 - 추천 API 호출 시, 결과에서 필터링 (Dynamic Filter) - 추천 계산 결과 후처리 (Business Logic 적용 하기)
58. Reference • Amazon Personalize Immersion Day • https://personalization-immersionday.workshop.aws/en/ • Sample Code • https://github.com/aws-samples/amazon-personalize-samples/ • Personalize Cheat Sheet • https://github.com/aws-samples/amazon-personalize- samples/blob/master/PersonalizeCheatSheet2.0.md • AWS 리소스 허브 – AI & 기계학습 • https://kr-resources.awscloud.com/aws-ai-and-machinelearning