From Theory to Practice: Blackbox Testing in an Industrial Telecom Environment

ISSRE25

1. The University of Sydney 1 Darshana Das K Krishnahari P Libna Kuriakose T Parvathy C M Dr. Ezudheen P Abraham Jacob Rahul Gopinath From Theory to Practice: Government Engineering College Thrissur, Kerala, India Mettle Networks, Kerala, India University of Sydney Blackbox Testing in an Industrial Telecom Environment
2. The University of Sydney Why This Research? -- Request by Mettle Networks Testing Virtualised Packet Processing Engine vPPE 2
3. The University of Sydney Why This Research? -- Request by Mettle Networks 3 Testing Virtualised Packet Processing Engine vPPE - Implemented in Software rather than ASIC - Optimized hot-paths - Removed complex processing logic
4. The University of Sydney Why This Research? -- Request by Mettle Networks 4 Testing Virtualised Packet Processing Engine vPPE - Implemented in Software rather than ASIC - Optimized hot-paths - Removed complex validation logic Wanted guarantees on Robustness
5. The University of Sydney High Performance vs High Security – Performance and security often are in conflict in High Performance Systems Choice: ... High security High performance ... 5
6. The University of Sydney Additional Constraints from Mettle Mettle Networks wanted to test the software under: - Blackbox Access (only remote access) - Wanted Statistical Security Guarantees (ideal) - Product could not be instrumented. Fuzzer 6
7. The University of Sydney Fuzzer 7 A Common Constraint in Industrial Settings Blackbox testing is often the only option – Industrial systems internals often can't be inspected due to restrictions – Legal/IP – Operational
8. The University of Sydney Protocol Handling in Edge Packet Processing 8 vPPE Architecture
9. The University of Sydney Protocol Handling in Edge Packet Processing 9 – Session Teardown – Policy Enforcement – PPPoE Authentication RFC 2516 Control Plane Subscriber Session Control
10. The University of Sydney Protocol Handling in Edge Packet Processing 10 – Session Teardown – Policy Enforcement – PPPoE Authentication RFC 2516 Control Plane Data Plane – User Traffic Subscriber Session Control
11. The University of Sydney Fuzzer 11 Blackbox Testing of vPPE
12. 12 The standard spec •Reference Specification was unreliable Blackbox Testing of vPPE
13. 13 The standard spec Buggy Implementation •Reference Specification was unreliable Blackbox Testing of vPPE •Implementation was potentially buggy Potential vulnerabilities Unimplemented parts
14. The University of Sydney Systematic Exploration: Specification Inference with L* 14 Learner Oracle Also called Grammar Inference Unknown ?
15. The University of Sydney Systematic Exploration: Specification Inference with L* 15 Learner Oracle Also called Grammar Inference Hypothesis Unknown
16. The University of Sydney Systematic Exploration: Specification Inference with L* 16 Learner Oracle w ab ✓ ✓ abb ✘ ✘ bb ✓ ✓ aaaa ✓ ✓ bbb ✓ ✘
17. The University of Sydney Systematic Exploration: Specification Inference with L* 17 Learner Oracle w ab ✓ ✓ abb ✘ ✘ bb ✓ ✓ aaaa ✓ ✓ bbb ✓ ✘
18. The University of Sydney Systematic Exploration: Specification Inference with L* 18 Learner Oracle w ab ✓ ✓ abb ✘ ✘ bb ✓ ✓ aaaa ✓ ✓ bbb ✓ ✘
19. The University of Sydney Systematic Exploration: Specification Inference with L* 19 ab ✓ ✓ abb ✘ ✘ bb ✓ ✓ aaaa ✓ ✓ bbb ✘ ✘ aaa ✘ ✘ abab ✘ ✘ Learner Oracle w
20. The University of Sydney Systematic Exploration: Specification Inference with L* 20 Learner Oracle w
21. The University of Sydney Systematic Exploration: Specification Inference with L* 21 Learner Oracle w PAC guarantee:
 within ε of true protocol with 1-δ confidence N
22. The University of Sydney Structure Aware Input Generation 22 •PPPoE used binary packets (IP) •Packets generated with Kaitai Struct Fuzzer
23. The University of Sydney Kaitai Struct 23 •Kaitai Struct provides a means to specify binary structure and constraints declaratively https://kaitai.io/
24. The University of Sydney Example Kaitai Struct File (UDP Packet) 24
25. The University of Sydney Methodology 25 •Defined alphabet (packet structure)
 with Kaitai Struct •Built the initial hypothesis from
 documentation •Explored the given system under
 blackbox conditions •Used mutated known and mutated
 packets as alphabet We were only given an ssh login to a system
 that had vPPE client setup
26. The University of Sydney Protocol Exploration 26 •This gave us the complete
 protocol specification
27. The University of Sydney Obtaining Reliability Guarantees 27 •Validated inferred grammar for accuracy and completeness •PAC guarantee: ε = 0.1, δ = 0.1
 i.e. Within 10% of true protocol with 90% confidence
 •Testing Metrics •Precision = 0.93 •Recall = 0.94 •F1 2*P*R/(P+R) = 0.94 •Structural Coverage: •Protocol coverage: Grammar depth of k-path = 3 •Code coverage: 33.6% for the selected test cases N N>=93
28. The University of Sydney Discussion 28 • Async behaviour: ECHO/PADT packets were unreliable, and hence cannot be captured • State ambiguity: Malformed packets sometimes ignored, sometimes not • Server-initiated requests: Interleaved with client requests, hard to model • Single client limitation: Hard to test non-responsive/concurrent states Protocol issues found: • Non-standard sequences (unexpected behaviour) • Validation gaps in authentication & address handling • Optimization opportunities under abnormal conditions
29. The University of Sydney Summary 29 • Blackbox testing of performance critical telecom system • No source code access or instrumentation • Unreliable documentation • Specification inference with L*,
 and packet generation with KaitaiStruct • Results • PAC guarantee: 10% maximum error, with 90% confidence • F1 score of 0.94 • Discovered several unknown behaviours, reported to mettle, and fixed. • Our experience shows the practical value in blackbox testing with grammar inference and structure aware packet generation
30. The University of Sydney Summary 30