Share
Embed
Pandemic response questionnaire for 25M users, made in 1 week
Hiromu Ogawa
LINE OA Dev Team1 Software Engineer
https://linedevday.linecorp.com/2020/ja/sessions/9526
https://linedevday.linecorp.com/2020/en/sessions/9526
LINE DevDay 2020
November 26, 2020
Transcript
41 slides
-
Self Introduction › Server-Side Software Engineer of LINE Official Account
Manager › Previously developed Smart Channel, LINE Ads, and LINE Points Ads › Joined to LINE in 2015 as a new graduate -
National Survey for COVID-19 › Conducted 5 times of surveys
› Provided answer data to the Ministry of Health, Labor and Welfare › Surveyed the physical condition and awareness of infection prevention -
Numbers of the Survey Message Recipients 83M Preparation Days 6
Answers 25M As of March 2020 -
Background › In March, the number of positives was increasing
in Japan › Data was needed Open Data by the Ministry of Health, Labor and Welfare https://www.mhlw.go.jp/stf/covid-19/open-data.html 0 50 100 150 200 250 300 350 400 450 2/1 2/8 2/15 2/22 2/29 3/7 3/14 3/21 3/28 4/4 Number of Positives April 7: The State of Emergency was declared March 25: The project was started Asked to stay home on weekends in Tokyo -
What We Can Do? › Collect a huge amount of
data in a short period › Take the initiative in the survey -
Options › Need a lot of time to communicate with
partners Collaborate with other companies Develop a new system from scratch › Can design specifically for the survey Use our existing service (LINE Research) › Does not meet the purpose -
Timeline Day 7 Release Day 5 Start QA Day 2
Start Development Day 6 Finish QA Day 4 Finish Development Day 1 Start Project Only 3 Days -
Development Team › Supported by › Infrastructure engineers › Security
engineers › Data scientists › DBAs › Engineers of collaborating services › And more... Server-side Front-end Planner Core Team -
Related Articles › LINEユーザー8300万人を対象とするアンケートを1週間で開発するには? コロナ禍におけるLINEの施策とフロントエンド開発 › https://logmi.jp/tech/articles/322999 Front-end Development Product
Management › リリースまで6日、LINE『新型コロナ対策のための全国調査』の舞台裏 › https://careerhack.en-japan.com/report/detail/1422 -
Development Challenges Develop rapidly Handle high traffic Never stop the
system due to trouble -
Policy to Develop in 3 Days Develop a simple system
in the minimum specification Develop a system step by step Avoid system troubles by design -
User Experience Message Survey Page Thanks Page Chat Room (Flex
Message) In-App Browser (LIFF Platform) -
First Idea (Not Adopted) › Concerns › Time to implement
› Stability when high traffic › Reduce the amount of code › Reduce time to implementation › Reduce unexpected behaviors Web App Client Master Data Serve Survey Page LB App Server Answer Store Send Answers -
Survey Page / Answer Store First Step nginx Client Serve
Survey Page (Static File) Open Survey Page LB App Server Answer Store -
Sending Answers Message Answer Page Thanks Page Entry Event Submit
Event Answer of First Question All Answers -
Survey Page / Answer Store First Step nginx Client Answer
Store (access log) Send Answers Store Answers Open Survey Page LB App Server -
Access Log Format log_format main "request_id:$request_id" ”¥t" "remote_addr:$remote_addr" "¥t" "real_ip:$http_x_true_userip"
"¥t" "msec:$msec” "¥t" "server_protocol:$server_protocol" "¥t" "method:$request_method" "¥t" "scheme:$scheme" "¥t" "host:$host" "¥t" "path:$request_uri" "¥t" "status:$status” ... "¥t" "request_body:$request_body" ; request_id:b58cbf9c66a55a4a7b79e7f18906badd remote_addr:… real_ip:… msec:1585441263.563 server_protocol:HTTP/1.1 method:POST scheme:https host:covid19.line-apps.com path:/api/survey status:200 … request_body:{¥x22userId¥x22:¥x22ue216a4b17f4946f19e8f47 889830f275¥x22,…,¥x22body¥x22:{¥x22a4¥x22:¥x221¥x22,¥x22 a5¥x22:¥x223¥x22,¥x22a6¥x22:[¥x221¥x22],¥x22a7¥x22:[¥x22 2¥x22],…}} Nginx Config Actual Log Entry Answer JSON as a Field in LTSV -
Need for Log Aggregation access log Aggregate Answer Store access
log App Servers Original data for the final result Monitoring Maximize the number of answers -
Aggregate Answer Logs Second Step nginx access log Transfer access
logs every second (Asynchronously) App Server fluentd MySQL Client LB -
Pipeline in fluentd Source @type tail Filter @type record_transformer Filter
@type parser Match @type mysql_bulk Read access logs and parse LTSV Unescape request_body Parse request_body Insert access logs to MySQL -
Need for Log Verification (Unverified) Answer Store Filter unauthorized answer
logs Filter duplicate answers Verified Answer Store Aggregated data The original data of the final result -
Verify Answer Logs Third Step nginx access log App Server
fluentd MySQL Verifier Batch Server LINE Login Client LB Issue ID Token Send Answers with ID Token -
ID Token LINE Login … Issue ID Token • User
Data • Expiration Date • … • Signature Answer Store Verifier Client Verify ID Token Can be verified the login locally and asynchronously -
Verify Answer Logs Third Step nginx access log App Server
fluentd MySQL Verifier Batch Server LINE Login Client LB 1. Fetch Access Logs 2. Verify ID Tokens 3. Write Results Issue ID Token Send Answers with ID Token -
Message Delivery Spikes Time Fast Delivery Messages Sent Answer Rate
Time Slow Delivery Messages Sent Answer Rate Higher Peak Traffic Lower Peak Traffic -
Pseudo Slow Delivery by Manual Control Time Messages Sent Answer
Rate (Pseudo Slow Delivery) Answer Rate (Fast Delivery) Answer Rate (Slow Delivery) Control the Delivery Pace Reduced Peak Traffic Baseline -
Performance Test › Handle 1K answers / sec by single
instance › Reached to 25K requests / sec in past campaigns › Prepare 50 instances for 50K (=25K * 2) requests / sec nginx access log App Server fluentd MySQL -
Monitoring node_exporter App Server Prometheus nginx_exporter fluent-plugin- prometheus Push Gateway
node_exporter Batch Server Verifier Monitoring Server Notify App MySQL (Replica) Grafana LINE Notify Monitoring App Notify Server Alert Manager Since the 4th Survey -
Changes in the Number of Events 0 1,000 2,000 3,000
4,000 5,000 0 10,000,000 20,000,000 30,000,000 40,000,000 50,000,000 60,000,000 70,000,000 3/31 9:00 3/31 18:00 4/1 3:00 4/1 12:00 4/1 21:00 Events / sec Events Events Answer Events Event Rate 66M Events 25M Answers 5K Events / sec -
ID Token Verification Delay 0 1,000 2,000 3,000 4,000 5,000
0 500,000 1,000,000 1,500,000 2,000,000 2,500,000 3,000,000 3,500,000 3/31 09:00 3/31 12:00 3/31 15:00 3/31 18:00 3/31 21:00 4/1 00:00 Events / sec Events Delay Event Rate 3.5M Events Delay -
Retrospective › Felt comfortable with the development Develop a system
step by step Avoid system troubles by design › Some background processing was delayed › Avoided the impact of processing delays on the users Develop a simple system in the minimum specification › Ensured performance and stability -
Conclusion › The scale of the survey was very large,
but the system was very simple › Handled high traffic stably and prepared in a short period › Conducted the national survey for COVID-19 in 3 days development