Handling race-condition in LINE Help Indonesia OA

Handling race-condition in LINE Help Indonesia OA

Matthew Rendy Tanudjaja
LINE Indonesia Engineering Lead
https://linedevday.linecorp.com/jp/2019/sessions/S1-11

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 2019 DevDay Handling Race-Condition in LINE Help Indonesia OA >

    Matthew Rendy Tanudjaja > LINE Indonesia Engineering Lead
  2. Building a LINE Help OA @linehelpid

  3. System Architecture LIUS* line-help-bot OA /services/oauth2/token /services/apexrest/ticket *) LINE Interactive

    User Support
  4. Endpoint Definition /services/oauth2/token client_id client_secret username password HTTP POST access_token

    200 error_code 4xx, 5xx error_message
  5. Endpoint Definition /services/apexrest/ticket timestamp 200, 4xx, 5xx message access_token mid

    content HTTP POST
  6. Case: Redis Contains Expired Access Token Submit Inquiry Get Access

    Token Submit Inquiry Get Access Token Response Save Access Token LIUS LIUS LIUS 401 200 Inquiry Submission
  7. LIUS' Access Token Problem > Only one single access token

    is valid per account • Once a new access token is reissued, the old one expires immediately > It can be expired anytime • No constant expiration period
  8. Inquiry Submission Case: Concurrent Submission When Access Token Is Invalid,

    Redis Omitted Submit Inquiry Get Access Token Get Access Token Submit Inquiry Submit Inquiry Get Access Token Get Access Token S I Get T abc123= mrt244= 401 401 abc123= mrt244= ssh099= rdj309= 401 ssh099= rdj309= LIUS abc123= mrt244= ssh099= Process #1 Process #2
  9. Race Condition :(

  10. Solution

  11. System Architecture LIUS line-help-bot OA *) Elneny = L-nanny =

    LIUS Nanny, an access token store Elneny* /complaint /accesstoken
  12. System Architecture Web - Opens HTTP endpoints - Stores complaints

    to MySQL Ops - Cleans-up complaints periodically - Retrieves access token from LIUS when necessary
  13. System Architecture

  14. Endpoint Definition /complaint HTTP POST 2xx, 5xx

  15. Endpoint Definition /accesstoken HTTP GET access_token 200 5xx

  16. Inquiry Submission Case: Redis Contains Expired Access Token Submit Inquiry

    Complaint Get Access Token Ask to Retry After n Minutes Elneny LIUS 401 LIUS Elneny
  17. Complaint Sweeping Case: Concurrent Submission When Access Token Is Invalid,

    0 3.5 7 10.5 14 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 2 3 5 6 0 2 2 3 3 4 7 8 11 14 0 2 ops.crontab: */5 * * * * ops.complaint.threshold: 5 # Of Complaints Time
  18. March 2019 Successful Submission Rate 0 10 20 30 40

    50 60 70 80 Week 1 Week 2 Week 3 Week 4 28.94 45.69 58.30 67.48
  19. March 2019 Average Rate Days Spent Elneny Enabled Copywriting Enabled

    v1.1.1 31.19% 12 No No v1.1.2 57.54% 3 Yes No v1.1.3 62.66% 16 Yes Yes Performance Statistics
  20. ͋Γ͕ͱ͏ ͍͟͝·ͨ͠