How to troubleshoot Xfinity Home customer escalations?

Hands on tips and strategies to do deep dive cross-platform end-to-end deep dive investigations into Digital Home customer escalations.

Transcript

1. How to troubleshoot Xfinity Home customer escalations? Amit Rao Principal iOS Engineer Digital Home Client Engineering Comcast Confidential
2. – Steve Jobs “You’ve got to start with the customer experience and work back toward the technology - not the other way around.” “It is not the employer who pays the wages. Employers only handle the money … It is the customer who pays the wages.” – Henry Ford “Every day we’re saying, ‘How can we keep this customer happy?’ How can we get ahead in innovation by doing this, because if we don’t, somebody else will.” – Bill Gates
4. Outline • Xfinity Home system/app architecture • Customer feedback touchpoints • Customer escalation processes • Current challenges & opportunities • Debugging & Analysis tools for investigations • Troubleshooting XH Forum posts & • Deep Dive Case Studies (Cameras out of scope) • Summary Comcast Confidential
5. Xfinity Home system architecture Comcast Confidential
6. Basic xHome App Architecture Comcast Confidential
7. Customer feedback touchpoint • App/Play store reviews (#reviews-xhome) • XH forums (#xh-forum-posts) • In-app feedback (Tableau dashboards) • HelpShift in-app agent chat support • Tech support (Phone/Web etc) Comcast Confidential
8. Current escalation engineering process • CARE is first line of support and collects data. Executes run book plays to resolve issue. • XH users reporting customer pain on app store reviews are requested to post to forums. • Issues that are unresolved are escalated to client engineering squads via incident tickets (#xh_qa_ios_android) • We need to consider having an on-call rotation engineer/manager escalation path on each squad. Comcast Confidential
9. Current challenges/opportunities • DH platform incidents and deployments have a direct impact on customer experience. XH has multiple points of failure and complex hardware/software/networking dependencies. App feedback is often related to downstream services/hardware/firmware. • Squads cannot sometimes do deep customer investigations due to lack of domain knowledge. • Many customer issues go unresolved resulting in customer dissatisfaction. They have often tried multiple reinstalls, rebooting the device, kill/relaunch the app, rebooting WiFi router, rebooting Touchscreen. Comcast Confidential
10. Current Challenges/Opportunities • Client engineers do not have actionable logs easily available to debug user problems • Tool limitations (e.g. spunk reliability/log rotation policy etc.). Root cause analysis often is hard as engineers need to use disjointed tools. • Once the customer calls support we are already in trouble (CIR drives operational cost). Truck rolls are sometimes needed and adds further cost. Comcast Confidential
11. Tools for Deep Dive • Splunk (SQL like query language), alerts, dashboards. • Management Portal (iControl) • ESD Tools to look up account #/custGuid etc. • Einstein to see if user is active. • Crash Reports in App Center. • Slack! • In-app XH reviews via Tableau dashboard. Comcast Confidential
12. Troubleshooting XH Forum posts • Configure your comcast.net ID as Comcast employee so you can access custGuid of the user. • If you need additional data request that CARE get it from the user. • Start with basic splunk query and build on it. • Try to reproduce the issue in the mobile app. Come up with a theory and try to validate it with experiments & data. Rinse and repeat. • If logs show the issue is beyond the mobile app, engage the right team via slack. Keep tickets upto date. Comcast Confidential
13. Close the loop • If you have a fix/solution make sure the team interfacing with the user is aware. (Cher/Omid/Rachel) • If your investigation is inconclusive or points to a bug, file a JIRA ticket and provide it to ops/CARE. • If the bug/issue is with a downstream service let CARE know and pass the ticket to that team. • Proactively monitor forums and app store reviews for features that you are a SME on for valuable feedback to improve our products. Comcast Confidential
14. Types of XH customer issues • Login errors • Arm/Disarm errors • Camera related errors (out of scope) • App Crashes • App hang or app unresponsive • Activity events • IoT devices • Rules/XRS errors • Push Notifications (duplicate or not received) • Device Onboarding errors (XH and IoT) • App Startup Performance • Other (e.g. Thermostat schedule, Touchscreen out of sync with app etc) Comcast Confidential
15. Case Studies 1. Login errors 2. No Zen option in Xfinity Home app 3. Push Notifications Comcast Confidential
16. DEMO
17. Proposed Improvements • Collect user logs along with in-app feedback • Developer systems and process to take action on Shake Report defects • Measure/monitor product NPS, feature metrics and app store reviews. • Gather in-app feedback data and post it to a Slack channel for visibility. • Improve documentation as a team on escalation engineering tech docs.
18. Process improvements • App defects that are customer impacting often need to be made high priority. They get lost in the defect backlog. • Shake Report defects need to be triaged and looked into. • Features coming in late with critical defects should be turned off and go into the next release. New features need to have Splunk/Localytics/ Looker dashboards for monitoring. Exceptions to engineering processes need to be escalated upto senior leadership. • Feature work needs to be prioritized by Product using a data driven approach as much as possible. It often is not. • We need to partner with Product to deliver product feature metrics so that they can be correlated with NPS feature metrics. Shipping features is not that hard but shipping an awesome customer experience is hard.
19. Summary • Feel the customer pain as it impacts every company’s bottom line. • Strive to deliver best-in-class customer experience via product engineering, escalation engineering and thinking out of the box • Get users to change their 1-star app reviews to 5-star when possible. • Focus on execution but never stop learning, sharing knowledge and having fun as a team!
20. References 1. Camera Triage Responsibility 2. Camera/Live Video prod support 3. XH Architecture docs 4. https://etwiki.sys.comcast.net/display/DHC/ How+To+Monitor+the+Health+of+the+Apps+in +Production 5. How to get access to MP and Einstein?
21. References • ESD Tools • XH How do I …? (splunk, iControl MP ) • Localytics • XH App Error codes • XH in-app NPS Tableau dashboard