Breaking Bug

Breaking Bug is a post-mortem about how we handled in Chicisimo a bug which went largely undetected and of unknown scale.

We will explain how it did affect us, and which techniques and tools did we use to:

1. Investigate what was going on.
2. Properly diagnose and narrow it to the affected part in the app.
3. Analyze its impact and affected people.
4. Finally isolate it and fix it.

In the end we will give some conclusions about how was the real cost of fixing it, and in which ways the process gave us insight and helped us to understand and improve our quality processes.

Transcript

1. Saúl Díaz González Android Developer @sefford
2. Chicisimo Command Center* Madrid, Spain 21 Jun 2017, 5:29 *Not actual Command Center
3. Seems like it doesn’t have internet, but internet is working! Please help! Since last update you cannot enter the app anymore, everything is white. Please fix it or I will uninstall the app. It was great at the beginning, but now it is not loading. I uninstalled and reinstalled several times, but it does not improve. The app is so good I didn’t want to stop using it. Please improve it. Nothing works, I uninstalled and installed several times, at first I thought it was a problem with my cellphone, but it isn’t. What do I do now? The app is blocked and it does not load, I already updated and nothing.
4. THE FIRST LINE OF DEFENSE Testing + CI TESTING Not perfect, but solid suite Domain JUnit E2E, ~65% coverage UI Junit & Robolectric, ~43% coverage Focused on key components
5. THE FIRST LINE OF DEFENSE MANUAL QA & DOGFOODING The team tests the app We ship early internally on a variety of devices and API Levels Manual Testing
6. THE FIRST LINE OF DEFENSE STAGED ROLLOUTS At least 24h w/ constant monitoring before full rollouts, up to 99% to ensure rollbacking CONTINOUS DELIVERY Shipping is a major asset Staged Rollouts
7. GATHERING INTELLIGENCE CRASHLYTICS GMAIL D’OH
8. Place your screenshot here NETWORKING OkHttp + Retrofit USE CASES Covered by tests PRESENTATION Bus-backed UI ProgressSwitcher
9. Place your screenshot here NETWORKING OkHttp + Retrofit USE CASES Covered by tests PRESENTATION Bus-backed UI ProgressSwitcher
11. Heisenbug In computer jargon, a “heisenbug” is a software bug that seems to disappear or alter its behavior when one attempts to study it. Source: Wikipedia
12. HEISENBUG REPORT HEAT MAP BRAZIL MEXICO COLOMBIA ITALY
13. “The only way to fight a Heisenbug is to be there when it happens”
14. Oct 12 08:21:15 android INFO MainActivity [Data successful] hello world!_ userId: sefford, installation:1234512345 Samsung Sm-j700f 6.0.1(api 23) build:110105_ date logtag comp. info user id installation id device info API level build nº
15. Jul 03 17:31:25 android chicisimo: ERROR MainActivity [ProgressSwitcher] feed arrived successfully with 0 elements Build:109402 Jul 03 17:42:38 android chicisimo: ERROR MainActivity [ProgressSwitcher] feed arrived successfully with 0 elements Build:109402 Jul 03 17:54:44 android chicisimo: ERROR MainActivity [ProgressSwitcher] feed arrived successfully with 0 elements Build:109402 Jul 03 18:02:56 android chicisimo: ERROR MainActivity [ProgressSwitcher] feed arrived successfully with 0 elements Build:109402 Jul 03 18:04:11 android chicisimo: ERROR MainActivity [ProgressSwitcher] feed arrived successfully with 0 elements Build:109402 Jul 03 18:09:21 android chicisimo: ERROR MainActivity [ProgressSwitcher] feed arrived successfully with 0 elements Build:109402 Jul 03 18:23:57 android chicisimo: ERROR MainActivity [ProgressSwitcher] feed arrived successfully with 0 elements Build:109402 Jul 03 18:37:33 android chicisimo: ERROR MainActivity [ProgressSwitcher] feed arrived successfully with 0 elements Build:109402 Jul 03 18:38:03 android chicisimo: ERROR MainActivity [ProgressSwitcher] feed arrived successfully with 0 elements Build:109402 Jul 03 18:45:38 android chicisimo: ERROR MainActivity [ProgressSwitcher] feed arrived successfully with 0 elements Build:109402 Jul 03 18:54:49 android chicisimo: ERROR MainActivity [ProgressSwitcher] feed arrived successfully with 0 elements Build:109402 Build:109402
17. “To know your enemy, you must become the enemy Sun Tzu
18. VIRTUAL HOLIDAYS AT SAO PAULO REMOTE DEVICES PROXIES
19. --- a/app/src/main/java/com/chicisimo/common/injection/ApiModule.java +++ b/app/src/main/java/com/chicisimo/common/injection/ApiModule.java @@ -104,7 +104,7 @@ public class ApiModule { .addNetworkInterceptor(responseInterceptor) - .connectTimeout(60, TimeUnit.SECONDS) + .connectTimeout(00, TimeUnit.SECONDS) .readTimeout(60, TimeUnit.SECONDS); }
21. IMPROVING THE TEST SUITE ▰ Switched Retrofit mocks for real requests to MockWebServer ▰ Segment new connection error conditions like timeouts, SSL handshake errors... ▰ Better understanding on how the app behaves on the networking area
22. Place your screenshot here BACKEND No error reported INFRASTRUCTURE Amazon WS NETWORKING OkHttp + Retrofit
23. VYSOR.IO “I wish there was a way to fly to Brazil and hook their phones to my computer properly debug what is going on”
24. ▰ Install ADB Drivers ▰ Install Vysor app ▰ Install Vysor on Chrome
25. INFO: connection from chicisimo.api failed:SocketTimeoutException: Socket closed after 60000ms INFO: connection from chicisimo.api failed:SocketTimeoutException: Socket closed after 60000ms INFO: connection from chicisimo.api failed:SocketTimeoutException: Socket closed after 60000ms INFO: connection from chicisimo.api failed:SocketTimeoutException: Socket closed after 60000ms INFO: connection from chicisimo.api failed:SocketTimeoutException: Socket closed after 60000ms INFO: connection from chicisimo.api failed:SocketTimeoutException: Socket closed after 60000ms INFO: connection from chicisimo.api failed:SocketTimeoutException: Socket closed after 60000ms INFO: connection from chicisimo.api failed:SocketTimeoutException: Socket closed after 60000ms
26. IMPROVING INFRASTRUCTURE IRELAND N.CALIF. SAO PAULO
27. THE REAL WORK BEGINS STETHO FIREBASE PERFORMANCE PAPERTRAIL
28. Send request Return server UTC time Log the time taken with every request Send server the real time frame
29. # NumSamples = 453938; Min = 0.00; Max = 97885.00 # Mean = 1642.091535; Variance = 18645635.742440; SD = 4318.059256; Median 765.000000 # each ∎ represents a count of 5360 0 - 2000 [402006]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (88.56%) 2000 - 3000 [ 21361]: ∎∎∎ (4.71%) 3000 - 4000 [ 5650]: ∎ (1.24%) 4000 - 5000 [ 2682]: (0.59%) 5000 - 6000 [ 1575]: (0.35%) 6000 - 7000 [ 1514]: (0.33%) 7000 - 8000 [ 931]: (0.21%) 8000 - 9000 [ 620]: (0.14%) 9000 - 10000 [ 356]: (0.08%) 10000 - 20000 [ 1678]: (0.37%) 20000 - 97885 [ 15565]: ∎∎ (3.43%) https://github.com/bitly/data_hacks
32. DRAWING OF THE WEB https://github.com/square/okhttp/issues/270 ▰ Hidden API on OkHttp 3.8.0 for Network Introspection ▰ In discussion since 2013 ▰ Available on OkHttp 3.9.0
33. Jul 17 18:21:48 android chicisimo: INFO [Network] get '/albums/82603' server:255.255.255.255_ dns:253ms ssl:752ms connection:670ms installation:1234512345 user:sefford endpoint server DNS SSL connection
34. “Dude, this alpha you sent me is going like s**t, everything takes a century to load. J. Barroso, Chicisimo’s Designer
35. DEUS-IGNER EX-MACHINA WiFi 3G Josue’s Phone Emulators My phones Tethering
37. Server IPv4 First Server IPv6 First
38. THE ACTUAL FIX public class IpV4PriorizerDns implements Dns { public static final Comparator<InetAddress> INET_ADDRESS_COMPARATOR = new Comparator<InetAddress>() { @Override public int compare(InetAddress left, InetAddress right) { if (left instanceof Inet4Address) { return -1; } else if (right instanceof Inet4Address) { return 1; } return 0; } }; @Override public List<InetAddress> lookup(String hostname) throws UnknownHostException { if (hostname == null) { throw new UnknownHostException( "hostname == null"); } else { final List<InetAddress> inetAddresses = new ArrayList<>( Arrays.asList(InetAddress.getAllByName(hostname))); Collections. sort(inetAddresses, INET_ADDRESS_COMPARATOR); return inetAddresses; } } }
39. 4 weeks of dev time That’s a lot of money 0.03 rating down We only recovered 0,012 since 15,656 net users lost And a lot of users
40. Improved our tests And our UX! Learnt a lot of insight If something happens, we’ll know it Got a lot of tools It increases the strategies we can use!
41. WRAPPING UP ▰ Keep calm and be scientific ▰ Engineering comes from latin “ingenium” ▰ Failin’ fine
42. THANKS! Any questions? You can find me at @sefford & sefford@gmail.com https://speakerdeck.com/sefford/breaking-bug
43. CREDITS Special thanks to all the people who made and released these awesome resources for free: ▰ Presentation template by SlidesCarnival ▰ All the people in the Android community who helped during this endeavor