Cloud-based components are an all too common speed bump when installing new gear or software. While not an issue in home networks, outbound connections are shunned by default in regulated environments. Enabling communications between the newly installed technology and its cloud service then involves change control requests, committees, firewall admins, and (worst of all) delays... hardly the high-speed future we were promised.
Product builders: it doesn't have to be this way. Right now in your network one type of traffic almost certainly can exit your network without restriction: DNS. That VOIP network you think is isolated? Pretty good chance it can resolve DNS.
This is the story of how we grew one of the larger DNS overlay networks around using Python Twisted. We built a secure and reliable channel between thousands of appliances (hardware and virtual) and hundreds of servers, over the inherently unreliable DNS.
The talk covers designing and building custom network channels in Twisted, Twisted limitations we bumped into, unexpected DNS behaviours, challenges in scaling the channel, and more. Network knowledge is useful but not necessary to follow along, and while we used Twisted, the lessons are applicable in other frameworks too. If you've got an hankering for network code, then this heady mix of network stacks and Python hacks is for you!