MTC2018 - CSAT Score Super Improvements - Just Do It

Speaker: Takako Ohshima

One year ago, we started conducting CSAT (Customer Satisfaction) Surveys to measure the satisfaction rate of our users for the US version of Mercari. This scoring system has undergone a series of discussions across our offices in San Francisco (now Palo Alto), Portland, and Tokyo. Now, it’s grown to the point of being included in our company OKR, and we consider it an indispensable metric. In this session, I'll be talking about how the implementation of this metric evolved in the last 12 months, and how it has affected decision-making in our product development.

Transcript

1. CSAT Score Super Improvements - Just Do It Takako Ohshima Software Engineer (Machine Learning)
2. Software Engineer Machine Learning Takako Ohshima
3. Introduction Takako Ohshima (@hatone) 2017/02 Mercari, Inc. (US - hired in US) Customer Experience Improvement (CXI) Team 2018/05 Transferred to the newly created Machine Learning Team
4. CSAT Score 極度改善(しなさい)
5. Today’s Topics - Teams that support CXI - Problems in our team - What is a CSAT Score? - What we learned and what occurred when developing/operating
6. Customer eXperience Improvement CXI? US@Tokyo Palo Alto Portland
7. We work from Tokyo/Palo Alto/Portland to improve the overall customer experience
8. CXI is mainly in charge of: - Internal tools for Mercari CS - Improvements to overall operations
9. What we worked to improve Quality of response to inquiries Goal setting for each individual CS member Prioritization of feature development
10. Issues we had in 2017 There were no measurements for CS members to improve operations, and goal setting was vague. Quality of inquiry responses Goal setting by each CS member Prioritization of feature development Is the return template we’re using for inquiries appropriate? Were the decisions/communications carried out by each individual CS member correct? There are many requests for improvements to internal tools, how do we prioritize them? Couldn’t we fundamentally review tools related to inquiries that have low satisfaction levels?
11. Create a ruler for improvements Team goals and CS member goal setting Quality of inquiry responses Prioritization of feature development Score the satisfaction level of the reply templates and the work of individual CS members for each response Improve items with low satisfaction and a high number of inquiries Goal setting by each CS member
12. Create a ruler for improvements CSAT Score Customer Satisfaction Score
13. Implementing CSAT Score Measurement Inquiry from app After the inquiry is complete, a survey is emailed within 24 hours Calculate results
14. Current functions of Zendesk vs AskNicely When we tried to measure CSAT using Zendesk, we could only measure 2 items.
15. Infrastructure of CSAT calculations (surveys mailed) mercari-admin batch DB Take the following information from each email address: - UserID - tag - macro - name of agent in charge and save a custom parameter within AskNicely to email a survey.
16. Infrastructure of CSAT calculations (gathering survey results) asknicely2bigquery Cloud Functions BigQuery Other functions Cloud Functions Google App Script
17. When developing in the US, we were careful about: - the 4 (+2) time zones - Emails were sent by 4PM(PDT/PST) ≒ 7PM (EDT/EST) - If the email was going to be sent at a much later time, it was discussed with the PM and then sent on the following day. PST MST CST EST AKST HAST
18. Points of difficulty when developing/operating Unexpectedly low limit of API calls on Zendesk How to QA the data results Update the CS report Surveys that were not sent due to a technical problem
19. 画像や写真 Low limit of API calls on Zendesk Looked at the Retry-After Header at the time of a 503 response and adjusted the timing of the request Found the connector for each service, and researched the request numbers If you go over the Rate Limit, you get a 503 error.
20. How to QA the results Is this data correct? Have the PM/CS leaders who operate the system also look at it
21. Update the report from CS Gave CS the ability to freely update reports Used the webhook function on zapier and went through the https trigger on cloud functions asknicely2bigquery Cloud Functions
22. Unsent surveys due to a technical issue Welcome change (this is important!) Settings/ticket-handling mechanism changes on Zendesk, so it’s important to ensure recovery is relatively simple to do. In order to detect any problems, we now have Slack notifications of the number of messages.
23. After developing and using CSAT Score Scoring formula and definition Easy to recover Easy to operate
24. After the creation of CSAT Score: Customer inquiry responses have improved Old Copy It’s possible your tracking information isn’t updating because the package missed a tracking scan. In most cases, the item is still in transit to the destination, and final delivery is unaffected. If you feel the item has been lost, please contact the carrier with your tracking number, and they should be able to locate your item. If the carrier confirms the item has been lost, please contact us via the Help Center in My Page, and we’ll help to resolve the issue. New Copy Packages occasionally miss a tracking scan in transit. In most cases, the item is still in transit to the destination, and final delivery is unaffected. If you feel that the item was lost, I recommend reaching out to the carrier with your tracking number. They should be able to track down your item. If the carrier confirms the item has been lost, please contact us via the Help Center in My Page, and we’ll help to resolve the issue. SHIP:: GENERAL:: Why is my tracking information not updating? We’ve reviewed over 100 templates
25. After the creation of CSAT Score: CS members’ review process improved Internal campaign using CSAT Score
26. After creation of CSAT Score: Determine priority of feature development (goal CSAT Score - current CSAT Score) x survey responses = impact Many ideas for improvements
27. Expanded development and the change that came with it - How to develop with several people effectively - Google App Script can share code, but cannot share a timer - An increase in information we wanted to see in data form - Facebook/Twitter/telephone inquiries, per team - Migrating from DataStudio to Looker - CSAT Score is used in company OKRs
28. Status of CSAT Score Improvements 3.65 (June 2017) 3.86 (September 2018)
29. Respective responsibilities and respective improvements CXI Team Engineer Team in Charge of CS Project & Product Manager Improve the CSAT Score system itself Improve customer service Consider the priority of the product including the impact on CSAT Score
30. CSAT Score 極度改善(しなさい) Take measurements. Determine the problem that needs to be solved. Improve it.