A Bug's Life: What if `select` is Broken After All?

Your project's issue list lies dormant for months, years even. Users seem happy enough, and you've got other priorities. Suddenly, a flurry of users report a crazy-sounding bug. They're seeing a crash, triggered with specific keystrokes, with a common operating system. You can't reproduce it. What in the world?!

You may have heard the debugging tip from the fantastic book Pragmatic Programmer, "select isn't broken". It's always best to assume that bugs are in *your* code. But... what if this one isn't?

In this talk, you'll re-experience this unusual low-level bug, and take away some lessons for the kinds of debugging we need to do all the time as software developers.

Transcript

1. A Bug's Life What If select Is Broken After All? 8th Light, Inc. Colin Jones @trptcolin https://8thlight.com
4. Debugging @trptcolin
6. Tip: “Don’t panic"
7. Tip: “Failing test before fixing code”
8. Tip: “Don’t assume it - prove it”
9. VERY FAST CYCLES
10. Tip: “select isn’t broken”
11. REPL-y @trptcolin
12. REPL-y
13. REPL-y
14. Addressed pain points • Readline bindings in JLine (v1) were lacking • CTRL-C killed the whole REPL • Tab completion didn’t exist
15. Fun to implement!
16. Glue code reply nrepl sjacket clojure-complete jline2 leiningen boot
17. Dependencies!
18. Glue code reply nrepl sjacket clojure-complete jline2 leiningen boot
19. Upstream bugs/features
20. Upstream bugs/features
21. Upstream bugs/features
22. Upstream bugs/features
23. Eventually things were OK
24. Demotivated “Done”
25. The Bug @trptcolin
26. Demotivated “Done”
28. IOException Recurso temporariamente indisponível java.io.FileInputStream.readBytes (FileInputStream.java:-2) java.io.FileInputStream.read (FileInputStream.java:255) java.io.BufferedInputStream.fill (BufferedInputStream.java:246) java.io.BufferedInputStream.read (BufferedInputStream.java:265) jline.internal.NonBlockingInputStream.run (NonBlockingInputStream.java:294) java.lang.Thread.run (Thread.java:748) Bye for now!
29. IOException Recurso te java.io.FileInputStrea java.io.FileInputStrea java.io.BufferedInputS java.io.BufferedInputS jline.internal.NonBloc java.lang.Thread.run ( Bye for now!
32. Tip: “Don’t panic"
33. Reproduce the issue? vs.
34. Reproduce the issue!
35. Reproduce the issue?!?
36. “Works on my machine”
37. “Works on my machine” “What is different between our setups?"
38. Tip: “Don’t panic"
39. Tip: “select isn’t broken”
40. Tip: “select isn’t broken”
41. Tip: “select isn’t broken”
44. Tip: “Don’t panic"
49. “Ubuntu has recently pushed what appears to be a bad kernel update to Ubuntu 18.04 users.”
50. “After lots of debugging and some brief discussion on the LKML, it appears that this is the result of a bug introduced in Linux on 1 November 2018, and fixed on 9 January 2019.”
54. “have not applied the more recent https:// github.com/torvalds/linux/commit/ d3736d82e8169768218ee0ef687188759180 91a0 patch”
55. Tip: Use search • Use rare words • Use rare combinations of words • Use rare phrases
56. “ Broadly prior to the introduction of this bug (i.e for the last 5 years), the program would echo back console input (once enter is pressed), as expected.”
57. “After the introduction of this bug, the program immediately terminates with "Resource temporarily unavailable".”
58. “I have attached to this email a very short c program, which I believe shows the issue.”
62. Tip: Write great bug reports • Omit everything that’s non-essential • Include everything that is essential
64. “it optimizes fast-path for tty_reopen(), but more importantly it won't interrupt ongoing IO on the tty”
67. REPRODUCE THE ISSUE!
68. REPRODUCE THE ISSUE!
69. What now?!
70. What now?! “Not sure if it’s worth trying to build / release a workaround - update your kernels if this affects you?”
71. Best. Comment. Ever.
73. What did we learn? @trptcolin
74. Tip: “select isn’t broken”
75. Search engines are your friends!
76. Ask for help!
77. We’re stronger together
78. Tip: “Don’t assume it - prove it”
79. Tip: “Don’t panic"
80. Recommended resources • Book by David Thomas & Andrew Hunt: The Pragmatic Programmer • Talk by Stuart Halloway: “Debugging with the Scientific Method” • Blog by Julia Evans: “What does debugging a program look like?” @trptcolin
81. Recommended resources • Leiningen report: https://github.com/technomancy/leiningen/issues/ 2533 • Spigot (Minecraft server) forum: https://www.spigotmc.org/threads/ ubuntu-18-04-bad-kernel-update.357871/ • Ubuntu bug report: https://bugs.launchpad.net/ubuntu/+source/ linux/+bug/1813873 • linux-console bug report: https://marc.info/?l=linux- console&m=154883021430618&w=2 About this bug @trptcolin
82. Thank you! 8th Light, Inc. Colin Jones @trptcolin https://8thlight.com