Upgrade to Pro — share decks privately, control downloads, hide ads and more …

"High-bandwidth HTTP downloads: unpeeling the onion" by Bruce Merry

7b0645f018c0bddc8ce3900ccc3ba70c?s=47 Pycon ZA
October 08, 2020

"High-bandwidth HTTP downloads: unpeeling the onion" by Bruce Merry

The MeerKAT radio telescope produces massive volumes of data. We provide a data access library for scientists to retrieve the data, but our initial implementation using boto had disappointing performance when used on a high-speed (25 Gb/s) network. On investigation, we found that boto wraps requests wraps urllib3 wraps http.client, and these wrapping layers introduce a lot of overheads that limit bandwidth. I'll walk through all the steps involved in getting data from the socket into a final response, show how this reduces throughput, and describe our solution to achieve bandwidths of multiple gigabytes per second.

7b0645f018c0bddc8ce3900ccc3ba70c?s=128

Pycon ZA

October 08, 2020
Tweet

Transcript

  1. www.ska.ac.za HIGH-BANDWIDTH HTTP PyCon ZA Bruce Merry www.ska.ac.za

  2. www.ska.ac.za What do astronomers care about HTTP?

  3. www.ska.ac.za What do astronomers care about HTTP? Figure: MeerKAT storage

    cluster
  4. www.ska.ac.za What do astronomers care about HTTP? Figure: MeerKAT storage

    cluster PB of storage
  5. www.ska.ac.za What do astronomers care about HTTP? Figure: MeerKAT storage

    cluster PB of storage accessed through S -compatible interface.
  6. www.ska.ac.za What do astronomers care about HTTP? Figure: MeerKAT storage

    cluster PB of storage accessed through S -compatible interface. Python library (katdal) presents high-level interface.
  7. www.ska.ac.za What do astronomers care about HTTP? Figure: MeerKAT storage

    cluster PB of storage accessed through S -compatible interface. Python library (katdal) presents high-level interface. Aim for Gb/s ( GB/s).
  8. www.ska.ac.za Rules of the benchmarking game • Fetch a GB

    file from localhost. • Return the content as a bytes. • Use a single thread. • No TLS, no content encoding, no transfer encoding.
  9. www.ska.ac.za First result (requests) 0 500 1000 1500 2000 2500

    3000 3500 MB/s
  10. www.ska.ac.za First result (requests) requests 0 500 1000 1500 2000

    2500 3000 3500 MB/s 501 Python 3.6.12
  11. www.ska.ac.za Why so slow? Will PyPy save the day? requests

    0 500 1000 1500 2000 2500 3000 3500 MB/s 501 Python 3.6.12
  12. www.ska.ac.za Why so slow? Will PyPy save the day? No.

    requests 0 500 1000 1500 2000 2500 3000 3500 MB/s 501 647 Python 3.6.12 PyPy 7.3.1
  13. www.ska.ac.za Inefficient memory model • bytes is immutable: have to

    copy to change • bytearray is zero-initialized • BytesIO.getvalue() makes a copy
  14. www.ska.ac.za Inefficient library implementations • Sometimes libraries make more copies

    than needed. • Sometimes they work in tiny chunks.
  15. www.ska.ac.za Layers requests urllib http.client socket

  16. www.ska.ac.za Test code def load_requests(url: str) -> bytes: return requests.get(url).content

    def load_urllib3(url: str) -> bytes: return urllib3.PoolManager().request('GET', url).data def load_httpclient(url: str) -> bytes: parts = urllib.parse.urlparse(url) conn = http.client.HTTPConnection(parts.netloc) conn.request('GET', parts.path) resp = conn.getresponse() return resp.read(resp.length) def load_socket_read(url: str) -> bytes: # Code is much too long for a slide ...
  17. www.ska.ac.za Results requests urllib3 httpclient socket-read 0 500 1000 1500

    2000 2500 3000 3500 MB/s 501 1371 896 3219 Python 3.6.12
  18. www.ska.ac.za Requests — unnecessary chunking CONTENT_CHUNK_SIZE = 10 * 1024

    ... self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  19. www.ska.ac.za Requests — unnecessary chunking CONTENT_CHUNK_SIZE = 10 * 1024

    ... self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' • kiB is too small to amortize overheads • bytes.join involves a copy
  20. www.ska.ac.za Requests — unnecessary chunking 10 kiB 1 MiB Chunk

    size 0 500 1000 1500 2000 2500 3000 3500 MB/s 501 1037 647 772 Python 3.6.12 PyPy 7.3.1
  21. www.ska.ac.za Requests — an alternative We can bypass Response.content: with

    requests.get(url, stream=True) as resp: return resp.raw.read()
  22. www.ska.ac.za Requests — an alternative requests requests-stream urllib3 0 500

    1000 1500 2000 2500 3000 3500 MB/s 501 1318 1371 Python 3.6.12
  23. www.ska.ac.za What’s up with http.client? Let’s look inside HTTPResponse.read:

  24. www.ska.ac.za What’s up with http.client? Let’s look inside HTTPResponse.read: if

    amt is not None: # Amount is given, implement using readinto b = bytearray(amt) n = self.readinto(b) return memoryview(b)[:n].tobytes()
  25. www.ska.ac.za What’s up with http.client? Let’s look inside HTTPResponse.read: if

    amt is not None: # Amount is given, implement using readinto b = bytearray(amt) n = self.readinto(b) return memoryview(b)[:n].tobytes() • Allocate some memory, and zero-fill it. • Read the data into that memory. • Make a copy of it.
  26. www.ska.ac.za What if we don’t specify an amount? Then it’s

    implemented with via _safe_read instead: def _safe_read(self, amt): s = [] while amt > 0: chunk = self.fp.read(min(amt, MAXAMOUNT)) if not chunk: raise IncompleteRead(b''.join(s), amt) s.append(chunk) amt -= len(chunk) return b"".join(s)
  27. www.ska.ac.za What if we don’t specify an amount? Then it’s

    implemented with via _safe_read instead: def _safe_read(self, amt): s = [] while amt > 0: chunk = self.fp.read(min(amt, MAXAMOUNT)) if not chunk: raise IncompleteRead(b''.join(s), amt) s.append(chunk) amt -= len(chunk) return b"".join(s) At least MAXAMOUNT = 1048576
  28. www.ska.ac.za What if we don’t specify an amount? httpclient httpclient-na

    0 500 1000 1500 2000 2500 3000 3500 MB/s 896 1334 Python 3.6.12
  29. www.ska.ac.za It’ll be better — one day httpclient httpclient-na 0

    500 1000 1500 2000 2500 3000 3500 MB/s 896 1334 892 3233 3258 3236 Python 3.6.12 3.8.2 master
  30. www.ska.ac.za More results requests requests-stream urllib3 httpclient-na 0 500 1000

    1500 2000 2500 3000 3500 MB/s Python 3.6.12 3.8.2 master PyPy 7.3.1
  31. www.ska.ac.za Other libraries httpx tornado aiohttp 0 500 1000 1500

    2000 2500 3000 3500 MB/s 709 868 996 711 481 1090 Python 3.6.12 PyPy 7.3.1
  32. www.ska.ac.za So what do we do about Python .6? Let’s

    relax the rules and return a numpy array. def load_requests_np(url: str) -> np.ndarray: with requests.get(url, stream=True) as resp: data = np.empty(int(resp.headers['Content-length']), np.uint8) resp.raw.readinto(memoryview(data)) return data This gets us MB/s.
  33. www.ska.ac.za So what do we do about Python .6? Let’s

    relax the rules and return a numpy array. def load_requests_np(url: str) -> np.ndarray: with requests.get(url, stream=True) as resp: data = np.empty(int(resp.headers['Content-length']), np.uint8) resp.raw.readinto(memoryview(data)) return data This gets us 6 MB/s.
  34. www.ska.ac.za So what do we do about Python .6? Let’s

    relax the rules and return a numpy array. def load_requests_np(url: str) -> np.ndarray: with requests.get(url, stream=True) as resp: data = np.empty(int(resp.headers['Content-length']), np.uint8) resp.raw.readinto(memoryview(data)) return data This gets us 6 MB/s. This time it’s urllib : def readinto(self, b): temp = self.read(len(b)) if len(temp) == 0: return 0 else: b[: len(temp)] = temp return len(temp)
  35. www.ska.ac.za Now what? Solution: use response.raw._fp.readinto.

  36. www.ska.ac.za Now what? Solution: use response.raw._fp.readinto. requests-np requests-np-fp 0 500

    1000 1500 2000 2500 3000 3500 MB/s 764 3033 Python 3.6.12
  37. www.ska.ac.za Summary People who write HTTP libraries don’t optimize for

    throughput.
  38. www.ska.ac.za Summary People who write HTTP libraries don’t optimize for

    throughput. But sometimes you can do something about it.
  39. www.ska.ac.za References https://github.com/ska-sa/pyconza2020-httpbench https://bugs.python.org/issue21644 https://bugs.python.org/issue36050 https://bugs.python.org/issue36051 https://bugs.python.org/issue41002 https://github.com/psf/requests/issues/5503 https://github.com/urllib3/urllib3/issues/1540

  40. www.ska.ac.za SARAO, a business unit of the National Research Foundation.

    The South African Radio Astronomy Observatory (SARAO) spearheads South Africa’s activities in the Square Kilometre Array Radio Telescope, commonly known as the SKA, in engineering, science and construction. SARAO is a National Facility managed by the National Research Foundation and incorporates radio astronomy instruments and programmes such as the MeerKAT and KAT- telescopes in the Karoo, the Hartebeesthoek Radio Astronomy Observatory (HartRAO) in Gauteng, the African Very Long Baseline Interferometry (AVN) programme in nine African countries as well as the associated human capital development and commercialisation endeavours. Contact information Bruce Merry Senior Science Processing Developer Email: bmerry@ska.ac.za