No content

Load measures - System modification of bottlenecks for the next campaign. - The previous issue has been resolved, but problem occurred elsewhere. - System burst due to sudden traffic such as a campaign. - As a result, scale out and scale up in a hurry.

Agenda - Service overview - System architecture (old) - System load Issues - Points of system improvement - System architecture (new) - Summary

STORE’s R∞ service overview CRM (Customer Relationship Management) Analysis Visualize your store customer details. • Store’s customer data • Yahoo! JAPAN Shopping Premium member rate • Signs of estrangement, • etc. Campaign Automatic distribution of optimal campaign. Who What When Verify the effects PDCA for customer nurturing.

User STORE’s R∞ service overview CRM (Customer Relationship Management) API Cart pages Product detail pages Yahoo! JAPAN Shopping ValueCommerce

Display the coupons that the customer is applying and the coupons that can be used. Notify customers who are viewing product details about coupons that can be used. STORE’s R∞ service overview CRM (Customer Relationship Management) Product detail pages Cart pages

Yahoo! JAPAN Shopping campaign - Regular campaigns by day of the week, date, etc. - 5 to 10 times more requests (RPS) than usual. - The amount of data such as purchase data is the same.

Thu. Fri. Sat. Sun. Mon. Tue. Wed. Thu. Fri. Sat. Sun. Mon. Tue. Wed. Yahoo! JAPAN Shopping campaign Specific two-week request. Regular campaigns by day of the week. Regular campaigns by date.

Coupon data User data (RDBMS) User Store System Architecture (old) Deliver Process Deliver Server (Java Servlet) Yahoo! JAPAN Shopping

Store Access Log Purchase Log (RDBMS) System Architecture (old) Aggregate Process Deliver Server (Java Servlet) Apache Spark

Delivery process Response deterioration User data update Update delay Analysis data update Aggregation delay Issues

Display the coupons that the customer is applying and the coupons that can be used. Notify customers who are viewing product details about coupons that can be used. Issues Product detail pages Cart pages

Issues Analysis Visualize your store customer details. • Store’s customer data • Yahoo! JAPAN Shopping Premium member rate • Signs of estrangement, • etc. Campaign Automatic distribution of optimal campaign. Who What When Verify the effects PDCA for customer nurturing.

Improvement plan Apache Spark scale out. RDBMS scale out. ⎯ Automatically scale out in advance based on CPU / Memory / Load Average. ⎯ Tuning Thread Count ⎯ Add the number of cluster nodes in advance. ⎯ Augment read-only read replicas. API Server scale out.

Improvement plan Apache Spark scale out. RDBMS scale out. ⎯ Automatically scale out in advance based on CPU / Memory / Load Average. ⎯ Tuning Thread Count ⎯ Add the number of cluster nodes in advance. ⎯ Augment read-only read replicas. API Server scale out. Fundamentally change the system configuration. Change to a system architecture that is resistant to load.

Points of system improvement Deliver server Aggregate process User data

Points of system improvement Deliver server Aggregate process User data

Params: user id and product code. Get user data Get coupon data Return target coupon Acceptance of requests Deliver server processing flow User data: 30-40 million. Coupon data: thousands.

Params: user id and product code. Get user data Get coupon data Return target coupon Acceptance of requests Deliver server processing flow User data: 30-40 million. Coupon data: thousands. Key-Value Store mdbm (memory-mapped key/value store)

Coupon data structure (mdbm) Key: Store ID typedef struct { int64_t client_id; int32_t campaign_num; campaign_data_t campaigns[]; } campaign_list_t; typedef struct { _birth_t birth; _range_t age_range; _prefecture_t prefecture; _rank_t rank; _range_t purc_count_total; _range_t purc_amount; ・・・ } typedef struct { int64_t campaign_id; _targeting_condition_t condition; ・・・ } campaign_data_t;

User Data (redis) Coupon Data (RDBMS) Store mdbm generate batch Yahoo Shopping System Architecture (new) Deliver Process Deliver Server (Apache Module) Coupon Data (mdbm)

Points of system improvement User data Deliver server Aggregate process

Demographic info • Sex • Residential areas • Age • Birth month Action history • Last access • Access to specific products • Purchase of specific products • Non-purchase of specific products • Purchase amount in the past Custom info • Yahoo premium member • SB smartphone user • Ymobile smartphone user • Rank specified by the store User data type

Redis data structure field = "customer" KEY = value = field = "acc" value = field = "rank" value = { "sex": 1, "birth": [1964, 1], "prefecture": 27, ... } { "": [ {"item": "", "time": 1665975527}, {"item": "", "time": 1666000534} ], ... } { "": { "purcHist": [ {"amount": 7251, "item": "", "time": 1666003823, "type": 3679}, {"amount": 2709, "item": "", "time": 1666144923, "type": 3679}, ], "repeat": 103428, "rank": 129283, "lastPurc": 1666144923, "pCount": 1, "pAmount": 9960, "tCount": 1, "tAmount": 9960 }, ... "mTCount": 247, "coupon": [ "Nzg1ZDRmYTAzOGRhZjRlMmRkNTNiMTk4YjA3" ] }

Issue of updating • Process of updating an existing record • Need to atomic User Data (redis) Deliver Server (Apache Module) 1. Get 6. Set 2. Decompression 3. Json parse 4. Update

Issue of updating • Process of updating an existing record • Need to atomic User Data (redis) Deliver Server (Apache Module) 1. Get 6. Set 2. Decompression 3. Json parse 4. Update Redis Module (Introduced from redis 4.x)

VCROO.HMGET Original implemental command ⎯ Same as standard HGET ⎯ Same as standard HMGET VCROO.HGET Redis> VCROO.HGET acc {"17603":[ { "item":”", "time":1666459863 } ], "1273540":[ { "item":”", "time":1666253214 } ], … } VCROO.RECCV ⎯ Update specific product access in “acc” field ⎯ Update specific product purchase in “cv” field VCROO.RECACC ・・・

Points of system improvement Aggregate process Deliver server User data

Customer Rank • Different settings for each store • Aggregated for up to 2 years of data • Re-aggregation is required when settings are changed Rank Aggregate Processing

Data Source (RDBMS) Data Source (RDBMS) Issue of adding node node node node node ・・・・・

Data Source (RDBMS) Data Source (RDBMS) Issue of adding node node node node node node node ・・・・・

Instance Storage Request Infrastructure cost (EMR / RDBMS)

Data (S3) Side Data (RDBMS) Store Side data upload batch System Architecture (new) Deliver Server (Apache Module) node node node

Data (S3) Side Data (RDBMS) Store Side data upload batch System Architecture (new) Deliver Server (Apache Module) node node node

Data (S3) Side Data (RDBMS) Store Side data upload batch System Architecture (new) Deliver Server (Apache Module) node node node

Data (S3) Side Data (RDBMS) Store Side data upload batch System Architecture (new) Deliver Server (Apache Module) node node node

What by change system architecture - Infrastructure cost. (Number of servers during campaign.) - Aggregate processing time. 5-6 hours. 2-3 hours. About 40% reduction

Summary System load and performance issues - Provisional solution In a situation that services continues to grow, it will be difficult to deal only with a provisional solution. - Fundamental architectural change Architecture changes should also be considered based on medium- to long-term maintenance costs and infrastructure costs. - Scalability Bottlenecks newly discovered in the process of service growth.

Thank you