Spooky Scenarios in Debugging: Whom Do You Trust?

Seattle Test in Production Meetup, hosted by LaunchDarkly. Presenter: Rick Zucker, SRE Manager

Transcript

1. Seattle DevOps Meetup, 2019-10-16 Spooky Scenarios in Debugging: Whom Do You Trust? Rick Zucker
2. Agenda • Introduction • Debugging Scenario #1 • Debugging Scenario #2 • Paranoid View • Summary
3. Introduction
4. Scenario #1
5. • C3I system ◦ POC for military using personal computers in the field • Communication link software ◦ Packet creation, processing, routing • System written in Pascal (UCSD P-System) Scenario #1 - Situation
6. • System wasn't working right Scenario #1 - continued
7. • System wasn't working right • No debugger - lots of print statements Scenario #1 - continued
8. • System wasn't working right • No debugger - lots of print statements • Packets were getting corrupted Scenario #1 - continued
9. • System wasn't working right • No debugger - lots of print statements • Packets were getting corrupted • Packet was corrupted despite no assignment statement to the variable between print statements Scenario #1 - continued
10. • UCSD Pascal has 255 character variable names, but only first eight matter ◦ DataPktTx and DataPktRx are two distinct variables Scenario #1 - solution
11. • UCSD Pascal has 255 character variable names, but only first eight matter ◦ DataPktTx and DataPktRx are two distinct variables • System had a two pass compiler ◦ Pascal compiler Scenario #1 - continued (2)
12. • UCSD Pascal has 255 character variable names, but only first eight matter ◦ DataPktTx and DataPktRx are two distinct variables • System had a two pass compiler ◦ Pascal compiler ◦ Fortran linker Scenario #1 - continued (2)
13. • UCSD Pascal has 255 character variable names, but only first eight matter ◦ DataPktTx and DataPktRx are two distinct variables • System had a two pass compiler ◦ Pascal compiler ◦ Fortran linker • Fortran only allows six character variable names Scenario #1 - continued (2)
14. • UCSD Pascal has 255 character variable names, but only first eight matter ◦ DataPktTx and DataPktRx are two distinct variables • System had a two pass compiler ◦ Pascal compiler ◦ Fortran linker • Fortran only allows six character variable names ◦ Linker mapped the two variables on top of each other Scenario #1 - continued (2)
15. • Be careful about what you take for granted ◦ Eventually you may have to doubt the basics • Be careful when combining two pieces of software • Notice everything Scenario #1 - Lesson
16. Scenario #2
17. • QA of a CPU chip design • Focused on memory sub-system • Stressed memory in unnatural ways • Pre-silicon testing ◦ Run on software simulation ◦ ~10 Hz (very slow!) Scenario #2 - Situation
18. • Write happening to wrong address ◦ Off by three bytes ◦ QA test had deliberately misaligned the data ▪ Not supposed to be on 32 bit boundary Scenario #2 - continued 32 bit words aligned on "normal" four byte boundaries in a 32 byte cache line
19. • Write happening to wrong address ◦ Off by three bytes ◦ QA test had deliberately misaligned the data ▪ Not supposed to be on 32 bit boundary Scenario #2 - continued 32 bit words misaligned
20. Simple CPU Pipeline Diagram Main Memory (simulated) Instruction Fetch Instruction Decode Address Generation Memory Cluster Bus Cluster
21. Simple CPU Pipeline Diagram Main Memory (simulated) Instruction Fetch Instruction Decode Address Generation Memory Cluster Bus Cluster
22. Simple CPU Pipeline Diagram Main Memory (simulated) Instruction Fetch Instruction Decode Address Generation Memory Cluster Bus Cluster
23. Simple CPU Pipeline Diagram Main Memory (simulated) Instruction Fetch Instruction Decode Address Generation Memory Cluster Bus Cluster
24. Simple CPU Pipeline Diagram Main Memory (simulated) Instruction Fetch Instruction Decode Address Generation Memory Cluster Bus Cluster
25. Simple CPU Pipeline Diagram Main Memory (simulated) Instruction Fetch Instruction Decode Address Generation Memory Cluster Bus Cluster
26. • Check object file. Wrong address. Scenario #2 - continued
27. • Check object file. Wrong address. ◦ Assembler generated wrong address. Assembler bug! Scenario #2 - continued
28. • Be careful about what you take for granted • As always, consider corner cases, and behaviors not recommended, but permitted (QA credo!) • Notice everything Scenario #2 - Lesson
29. Security View: Example
30. • @securelyfitz designed device to securely wipe flash drives How paranoid should you be? An example
31. • @securelyfitz designed device to securely wipe flash drives • Device has: ◦ USB interface ◦ Serial interface (for programming) ◦ Onboard micro-controller How paranoid should you be? An example
32. What do you trust? Trust Don't Trust Serial Cable Microcontroller IDE/Compiler
33. What do you trust? Trust Don't Trust Serial Cable X Microcontroller IDE/Compiler
34. What do you trust? Trust Don't Trust Serial Cable X Microcontroller X IDE/Compiler
35. What do you trust? Trust Don't Trust Serial Cable X Microcontroller X IDE/Compiler X Package was not signed and site used http, not https
36. What do you trust? Trust Don't Trust Serial Cable X Microcontroller X IDE/Compiler X Didn't trust IDE/Compiler, so didn't produce device. Ultimate release block.
37. • How much do you trust your supply chain? • If device is to be secure, where might bad actors take over? ◦ What can you control and trust? Lessons Learned
38. • Some problems are in places that you would never suspect. Eventually you may have to go there. • Start normally, but eventually be as skeptical as a security expert • Tools are not perfect and are not totally trustworthy Conclusions
39. Proprietary + Confidential Thank You