Handling race-condition in LINE Help Indonesia OA

Matthew Rendy Tanudjaja
LINE Indonesia Engineering Lead
https://linedevday.linecorp.com/jp/2019/sessions/S1-11

Transcript

1. 2019 DevDay Handling Race-Condition in LINE Help Indonesia OA > Matthew Rendy Tanudjaja > LINE Indonesia Engineering Lead
2. Building a LINE Help OA @linehelpid
3. System Architecture LIUS* line-help-bot OA /services/oauth2/token /services/apexrest/ticket *) LINE Interactive User Support
4. Endpoint Definition /services/oauth2/token client_id client_secret username password HTTP POST access_token 200 error_code 4xx, 5xx error_message
5. Endpoint Definition /services/apexrest/ticket timestamp 200, 4xx, 5xx message access_token mid content HTTP POST
6. Case: Redis Contains Expired Access Token Submit Inquiry Get Access Token Submit Inquiry Get Access Token Response Save Access Token LIUS LIUS LIUS 401 200 Inquiry Submission
7. LIUS' Access Token Problem > Only one single access token is valid per account • Once a new access token is reissued, the old one expires immediately > It can be expired anytime • No constant expiration period
8. Inquiry Submission Case: Concurrent Submission When Access Token Is Invalid, Redis Omitted Submit Inquiry Get Access Token Get Access Token Submit Inquiry Submit Inquiry Get Access Token Get Access Token S I Get T abc123= mrt244= 401 401 abc123= mrt244= ssh099= rdj309= 401 ssh099= rdj309= LIUS abc123= mrt244= ssh099= Process #1 Process #2
9. Race Condition :(
10. Solution
11. System Architecture LIUS line-help-bot OA *) Elneny = L-nanny = LIUS Nanny, an access token store Elneny* /complaint /accesstoken
12. System Architecture Web - Opens HTTP endpoints - Stores complaints to MySQL Ops - Cleans-up complaints periodically - Retrieves access token from LIUS when necessary
13. System Architecture
14. Endpoint Definition /complaint HTTP POST 2xx, 5xx
15. Endpoint Definition /accesstoken HTTP GET access_token 200 5xx
16. Inquiry Submission Case: Redis Contains Expired Access Token Submit Inquiry Complaint Get Access Token Ask to Retry After n Minutes Elneny LIUS 401 LIUS Elneny
17. Complaint Sweeping Case: Concurrent Submission When Access Token Is Invalid, 0 3.5 7 10.5 14 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 2 3 5 6 0 2 2 3 3 4 7 8 11 14 0 2 ops.crontab: */5 * * * * ops.complaint.threshold: 5 # Of Complaints Time
18. March 2019 Successful Submission Rate 0 10 20 30 40 50 60 70 80 Week 1 Week 2 Week 3 Week 4 28.94 45.69 58.30 67.48
19. March 2019 Average Rate Days Spent Elneny Enabled Copywriting Enabled v1.1.1 31.19% 12 No No v1.1.2 57.54% 3 Yes No v1.1.3 62.66% 16 Yes Yes Performance Statistics
20. ͋Γ͕ͱ͏ ͍͟͝·ͨ͠