Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MTC2018 - CSAT Score Super Improvements - Just Do It

mercari
October 04, 2018

MTC2018 - CSAT Score Super Improvements - Just Do It

Speaker: Takako Ohshima

One year ago, we started conducting CSAT (Customer Satisfaction) Surveys to measure the satisfaction rate of our users for the US version of Mercari. This scoring system has undergone a series of discussions across our offices in San Francisco (now Palo Alto), Portland, and Tokyo. Now, it’s grown to the point of being included in our company OKR, and we consider it an indispensable metric. In this session, I'll be talking about how the implementation of this metric evolved in the last 12 months, and how it has affected decision-making in our product development.

mercari

October 04, 2018
Tweet

More Decks by mercari

Other Decks in Technology

Transcript

  1. CSAT Score Super Improvements - Just Do It Takako Ohshima

    Software Engineer (Machine Learning)
  2. Introduction Takako Ohshima (@hatone) 2017/02 Mercari, Inc. (US - hired

    in US) Customer Experience Improvement (CXI) Team 2018/05 Transferred to the newly created Machine Learning Team
  3. Today’s Topics - Teams that support CXI - Problems in

    our team - What is a CSAT Score? - What we learned and what occurred when developing/operating
  4. CXI is mainly in charge of: - Internal tools for

    Mercari CS - Improvements to overall operations
  5. What we worked to improve Quality of response to inquiries

    Goal setting for each individual CS member Prioritization of feature development
  6. Issues we had in 2017 There were no measurements for

    CS members to improve operations, and goal setting was vague. Quality of inquiry responses Goal setting by each CS member Prioritization of feature development Is the return template we’re using for inquiries appropriate? Were the decisions/communications carried out by each individual CS member correct? There are many requests for improvements to internal tools, how do we prioritize them? Couldn’t we fundamentally review tools related to inquiries that have low satisfaction levels?
  7. Create a ruler for improvements Team goals and CS member

    goal setting Quality of inquiry responses Prioritization of feature development Score the satisfaction level of the reply templates and the work of individual CS members for each response Improve items with low satisfaction and a high number of inquiries Goal setting by each CS member
  8. Implementing CSAT Score Measurement Inquiry from app After the inquiry

    is complete, a survey is emailed within 24 hours Calculate results
  9. Current functions of Zendesk vs AskNicely When we tried to

    measure CSAT using Zendesk, we could only measure 2 items.
  10. Infrastructure of CSAT calculations (surveys mailed) mercari-admin batch DB Take

    the following information from each email address: - UserID - tag - macro - name of agent in charge and save a custom parameter within AskNicely to email a survey.
  11. When developing in the US, we were careful about: -

    the 4 (+2) time zones - Emails were sent by 4PM(PDT/PST) ≒ 7PM (EDT/EST) - If the email was going to be sent at a much later time, it was discussed with the PM and then sent on the following day. PST MST CST EST AKST HAST
  12. Points of difficulty when developing/operating Unexpectedly low limit of API

    calls on Zendesk How to QA the data results Update the CS report Surveys that were not sent due to a technical problem
  13. 画像や写真 Low limit of API calls on Zendesk Looked at

    the Retry-After Header at the time of a 503 response and adjusted the timing of the request Found the connector for each service, and researched the request numbers If you go over the Rate Limit, you get a 503 error.
  14. How to QA the results Is this data correct? Have

    the PM/CS leaders who operate the system also look at it
  15. Update the report from CS Gave CS the ability to

    freely update reports Used the webhook function on zapier and went through the https trigger on cloud functions asknicely2bigquery Cloud Functions
  16. Unsent surveys due to a technical issue Welcome change (this

    is important!) Settings/ticket-handling mechanism changes on Zendesk, so it’s important to ensure recovery is relatively simple to do. In order to detect any problems, we now have Slack notifications of the number of messages.
  17. After the creation of CSAT Score: Customer inquiry responses have

    improved Old Copy It’s possible your tracking information isn’t updating because the package missed a tracking scan. In most cases, the item is still in transit to the destination, and final delivery is unaffected. If you feel the item has been lost, please contact the carrier with your tracking number, and they should be able to locate your item. If the carrier confirms the item has been lost, please contact us via the Help Center in My Page, and we’ll help to resolve the issue. New Copy Packages occasionally miss a tracking scan in transit. In most cases, the item is still in transit to the destination, and final delivery is unaffected. If you feel that the item was lost, I recommend reaching out to the carrier with your tracking number. They should be able to track down your item. If the carrier confirms the item has been lost, please contact us via the Help Center in My Page, and we’ll help to resolve the issue. SHIP:: GENERAL:: Why is my tracking information not updating? We’ve reviewed over 100 templates
  18. After the creation of CSAT Score: CS members’ review process

    improved Internal campaign using CSAT Score
  19. After creation of CSAT Score: Determine priority of feature development

    (goal CSAT Score - current CSAT Score) x survey responses = impact Many ideas for improvements
  20. Expanded development and the change that came with it -

    How to develop with several people effectively - Google App Script can share code, but cannot share a timer - An increase in information we wanted to see in data form - Facebook/Twitter/telephone inquiries, per team - Migrating from DataStudio to Looker - CSAT Score is used in company OKRs
  21. Respective responsibilities and respective improvements CXI Team Engineer Team in

    Charge of CS Project & Product Manager Improve the CSAT Score system itself Improve customer service Consider the priority of the product including the impact on CSAT Score