Slide 1
Slide 1 text
Seattle DevOps Meetup, 2019-10-16
Spooky Scenarios in Debugging:
Whom Do You Trust?
Rick Zucker
Slide 2
Slide 2 text
Agenda
● Introduction
● Debugging Scenario #1
● Debugging Scenario #2
● Paranoid View
● Summary
Slide 3
Slide 3 text
Introduction
Slide 4
Slide 4 text
Scenario #1
Slide 5
Slide 5 text
● C3I system
○ POC for military using personal computers in the field
● Communication link software
○ Packet creation, processing, routing
● System written in Pascal (UCSD P-System)
Scenario #1 - Situation
Slide 6
Slide 6 text
● System wasn't working right
Scenario #1 - continued
Slide 7
Slide 7 text
● System wasn't working right
● No debugger - lots of print statements
Scenario #1 - continued
Slide 8
Slide 8 text
● System wasn't working right
● No debugger - lots of print statements
● Packets were getting corrupted
Scenario #1 - continued
Slide 9
Slide 9 text
● System wasn't working right
● No debugger - lots of print statements
● Packets were getting corrupted
● Packet was corrupted despite no assignment statement to the
variable between print statements
Scenario #1 - continued
Slide 10
Slide 10 text
● UCSD Pascal has 255 character variable names, but only first
eight matter
○ DataPktTx and DataPktRx are two distinct variables
Scenario #1 - solution
Slide 11
Slide 11 text
● UCSD Pascal has 255 character variable names, but only first
eight matter
○ DataPktTx and DataPktRx are two distinct variables
● System had a two pass compiler
○ Pascal compiler
Scenario #1 - continued (2)
Slide 12
Slide 12 text
● UCSD Pascal has 255 character variable names, but only first
eight matter
○ DataPktTx and DataPktRx are two distinct variables
● System had a two pass compiler
○ Pascal compiler
○ Fortran linker
Scenario #1 - continued (2)
Slide 13
Slide 13 text
● UCSD Pascal has 255 character variable names, but only first
eight matter
○ DataPktTx and DataPktRx are two distinct variables
● System had a two pass compiler
○ Pascal compiler
○ Fortran linker
● Fortran only allows six character variable names
Scenario #1 - continued (2)
Slide 14
Slide 14 text
● UCSD Pascal has 255 character variable names, but only first
eight matter
○ DataPktTx and DataPktRx are two distinct variables
● System had a two pass compiler
○ Pascal compiler
○ Fortran linker
● Fortran only allows six character variable names
○ Linker mapped the two variables on top of each other
Scenario #1 - continued (2)
Slide 15
Slide 15 text
● Be careful about what you take for granted
○ Eventually you may have to doubt the basics
● Be careful when combining two pieces of software
● Notice everything
Scenario #1 - Lesson
Slide 16
Slide 16 text
Scenario #2
Slide 17
Slide 17 text
● QA of a CPU chip design
● Focused on memory sub-system
● Stressed memory in unnatural ways
● Pre-silicon testing
○ Run on software simulation
○ ~10 Hz (very slow!)
Scenario #2 - Situation
Slide 18
Slide 18 text
● Write happening to wrong address
○ Off by three bytes
○ QA test had deliberately misaligned the data
■ Not supposed to be on 32 bit boundary
Scenario #2 - continued
32 bit words aligned on "normal" four byte boundaries in a
32 byte cache line
Slide 19
Slide 19 text
● Write happening to wrong address
○ Off by three bytes
○ QA test had deliberately misaligned the data
■ Not supposed to be on 32 bit boundary
Scenario #2 - continued
32 bit words misaligned
Slide 20
Slide 20 text
Simple CPU Pipeline Diagram
Main Memory (simulated)
Instruction
Fetch
Instruction
Decode
Address
Generation
Memory
Cluster
Bus
Cluster
Slide 21
Slide 21 text
Simple CPU Pipeline Diagram
Main Memory (simulated)
Instruction
Fetch
Instruction
Decode
Address
Generation
Memory
Cluster
Bus
Cluster
Slide 22
Slide 22 text
Simple CPU Pipeline Diagram
Main Memory (simulated)
Instruction
Fetch
Instruction
Decode
Address
Generation
Memory
Cluster
Bus
Cluster
Slide 23
Slide 23 text
Simple CPU Pipeline Diagram
Main Memory (simulated)
Instruction
Fetch
Instruction
Decode
Address
Generation
Memory
Cluster
Bus
Cluster
Slide 24
Slide 24 text
Simple CPU Pipeline Diagram
Main Memory (simulated)
Instruction
Fetch
Instruction
Decode
Address
Generation
Memory
Cluster
Bus
Cluster
Slide 25
Slide 25 text
Simple CPU Pipeline Diagram
Main Memory (simulated)
Instruction
Fetch
Instruction
Decode
Address
Generation
Memory
Cluster
Bus
Cluster
Slide 26
Slide 26 text
● Check object file. Wrong address.
Scenario #2 - continued
Slide 27
Slide 27 text
● Check object file. Wrong address.
○ Assembler generated wrong address. Assembler bug!
Scenario #2 - continued
Slide 28
Slide 28 text
● Be careful about what you take for granted
● As always, consider corner cases, and behaviors not
recommended, but permitted (QA credo!)
● Notice everything
Scenario #2 - Lesson
Slide 29
Slide 29 text
Security View: Example
Slide 30
Slide 30 text
● @securelyfitz designed device to securely wipe flash drives
How paranoid should you be? An example
Slide 31
Slide 31 text
● @securelyfitz designed device to securely wipe flash drives
● Device has:
○ USB interface
○ Serial interface (for programming)
○ Onboard micro-controller
How paranoid should you be? An example
Slide 32
Slide 32 text
What do you trust?
Trust Don't Trust
Serial Cable
Microcontroller
IDE/Compiler
Slide 33
Slide 33 text
What do you trust?
Trust Don't Trust
Serial Cable X
Microcontroller
IDE/Compiler
Slide 34
Slide 34 text
What do you trust?
Trust Don't Trust
Serial Cable X
Microcontroller X
IDE/Compiler
Slide 35
Slide 35 text
What do you trust?
Trust Don't Trust
Serial Cable X
Microcontroller X
IDE/Compiler X
Package was not signed and site used http, not https
Slide 36
Slide 36 text
What do you trust?
Trust Don't Trust
Serial Cable X
Microcontroller X
IDE/Compiler X
Didn't trust IDE/Compiler, so didn't produce device. Ultimate release block.
Slide 37
Slide 37 text
● How much do you trust your supply chain?
● If device is to be secure, where might bad actors take over?
○ What can you control and trust?
Lessons Learned
Slide 38
Slide 38 text
● Some problems are in places that you would never suspect.
Eventually you may have to go there.
● Start normally, but eventually be as skeptical as a security expert
● Tools are not perfect and are not totally trustworthy
Conclusions
Slide 39
Slide 39 text
Proprietary + Confidential
Thank You