$30 off During Our Annual Pro Sale. View Details »

Kneel And Disconnect: Getting The Fastest Connection Out Of A Hostname

Kneel And Disconnect: Getting The Fastest Connection Out Of A Hostname

Did you know that when you resolve a hostname, you can get multiple addresses back, and pick any one to connect to? Some of these addresses will connect near instantly, while others might take a long time or time out. This talk is about a Twisted endpoint API I built that takes a hostname, and returns the connection that takes the least time to complete, from the list of resolved host addresses.

Ashwini Oruganti

April 11, 2014
Tweet

More Decks by Ashwini Oruganti

Other Decks in Programming

Transcript

  1. Kneel and Disconnect - Ashwini Oruganti @_ashfall_ PyCon 2014

  2. Why Happy Eyeballs?

  3. Endpoint an interface with a single method that takes an

    argument returns*: a listening port / a connected protocol
  4. endpoint = HostnameEndpoint(reactor, "twistedmatrix.com", 80) conn = endpoint.connect( Factory.forProtocol(Protocol))

  5. ... Hostname socket.getaddrinfo() Multiple host addresses

  6. IPv4 32-bit addresses e.g. 1.2.3.4 2^32 possible addresses Not enough

  7. IPv6 128-bit addresses e.g. 1:2::3:4 Trying to switch to this.

  8. Hostname Multiple host addresses Return connection established first

  9. Connections in parallel

  10. Connections in parallel and ...

  11. Connections in parallel and ... 300ms delay

  12. reactor.connectTCP('1.2.3.4', 456, f)

  13. reactor.callLater(5, f)

  14. “There are probably people out there who are smart enough

    not to need [tests], we’re not one of those. ” - PyPy docs
  15. def test_something(self): reactor.callLater(5, f)

  16. def test_something(self): reactor.callLater(5, f) # NOPE!

  17. def test_something(self): clock.callLater(5, f) clock.advance(5)

  18. Name!?

  19. HostnameEndpoint?

  20. TCPEndpoint? HostnameEndpoint

  21. DNSEndpoint? HostnameEndpoint TCPEndpoint

  22. “Naming things for how they work is generally worse than

    naming things for what they do.”
  23. Step 1: Make a list of features it should have.

  24. Step 2: Sort them from simplest to most complex. (the

    failure ones come before the success ones)
  25. Step 3: Write a unit test for the first thing

    on that list.
  26. def _nameResolution(self, host): try: d = self.deferToThread( self._getaddrinfo, host, 0)

    return d except socket.gaierror: defer.fail( error.DNSLookupError("...")) # WRONG
  27. def _nameResolution(self, host): return self._deferToThread( self._getaddrinfo, host, 0) def connect(self,

    protocolFactory): [...] def errbackForGai(obj): return fail(error.DNSLookupError("...")) d = self._nameResolution(self._host) d.addErrback(errbackForGai)
  28. Fail Pass Fail Pass Fail Pass

  29. Fail Error Random Hang Pass Random Hang Error Pass

  30. def test_IPv6IsFaster(self): # ... d = self.endpoint.connect( clientFactory) d.addCallback(checkEndpoint) return

    d
  31. Don't return Deferreds from tests (unless…)

  32. If something is hard to test, it is usually a

    bad idea. Good code is generally easy to test.
  33. def test_IPv6IsFaster(self): # ... d = self.endpoint.connect( clientFactory) d.addCallback(checkEndpoint)

  34. 0 seconds later, please.

  35. “ Free your mind! ”

  36. 0 seconds later is still "some amount of time" later,

    even if it's not a very big amount of time
  37. “Compartmentalize.”

  38. It runs, but is it perfect?

  39. def connect(self, protocolFactory): [...] def _canceller(d): [...] def errbackForGai(failure) [...]

    def _endpoints(gaiResult): [...] def attemptConnection(endpoints): [...] def usedEndpointRemoval(connResult, connAttempt): [...] def afterConnectionAttempt(connResult): [...] def checkDone(): [...] def connectFailed(reason): [...] def iterateEndpoint(): [...]
  40. Finite State Machines!

  41. None
  42. None
  43. Thank You! tm.tl/4859 twitter.com/_ashfall_