Upgrade to Pro — share decks privately, control downloads, hide ads and more …

HTTP Tunneling in Go

HTTP Tunneling in Go

HTTP Tunneling can be used in situations similar to the requirements of using VPN, for example, to be able to access a localhost server from outside. This talk will introduce the specific implementation of HTTP tunneling and give audiences a deep knowledge of TCP and HTTP level implementation in Go.

Specifically, this talk discusses the idea of HTTP Tunneling in Go.

88964b936e864ca7d326272eaa70fa9a?s=128

Kazuki Higashiguchi
PRO

April 16, 2022
Tweet

More Decks by Kazuki Higashiguchi

Other Decks in Technology

Transcript

  1. HTTP Tunneling in Go Kazuki Higashiguchi April 23, 2022 @

    Go Conference 2022 Spring
  2. Agenda HTTP(S) Client WebSocket server WebSocket client HTTP(S) Server 1.

    HTTP Tunneling 2. Establish bidirectional connection using WebSocket 3. HTTP Tunneling over WebSocket
  3. About Me Kazuki Higashiguchi Backend Engineer at Autify. ID: @hgsgtk

  4. Autify Solution No-code AI-powered software testing automation services for Web/Mobile

    applications AI automatically maintenance test scripts No code unlocks automation at scale
  5. Agenda HTTP(S) Client WebSocket server WebSocket client HTTP(S) Server 1.

    HTTP Tunneling 2. Establish bidirectional connection using WebSocket 3. HTTP Tunneling over WebSocket
  6. HTTP tunneling • Create a network link between two computers

    in conditions of restricted network connectivity. e.g. firewalls, NATs, ACLs • Use a protocol of higher level (HTTP) to transport a lower level protocol (TCP)
  7. HTTP CONNECT method • Start bidirectional communications with the requested

    resource e.g. access HTTPS websites, HTTP proxy server HTTP(S) Client HTTP(S) Server Proxy (Gateway)
  8. HTTP tunneling with CONNECT Quoted from HTTP: The Definitive Guide

    / 8.5 Tunnels 1. Client sends CONNECT request 2. Gateway opens TCP connection to the server 3. Gateway returns HTTP ready message to the client 4. Start bidirectional communication of raw packets of data
  9. A case study: elazarl/goproxy elazarl/goproxy: An HTTP proxy library for

    Go $ curl -I -x http://localhost:8080 https://example.com HTTP/1.0 200 OK A basic sample code:
  10. Tunneling implementation in elazarl/goproxy https://github.com/elazarl/goproxy/blob/0bfa7c564b 5ffbece2f0832eb4e714fb971f70f3/https.go#L95

  11. Tunneling implementation in elazarl/goproxy 1 2 3

  12. Step 1: Take over the proxy connection http.Hijacker interface allows

    an HTTP handler to take over the connection.
  13. Step 2: Open TCP connections Opens TCP connection using net.Dial

    Returns HTTP ready message: 200 Connection established
  14. Step 3: Streaming with io.Copy

  15. Streaming: TLS handshake over TCP connection • TLS handshake is

    done using the streaming • Application data is sent and receive after TLS handshake
  16. Key takeaways • HTTP Tunneling uses a protocol of higher

    level (HTTP) to transport a lower level protocol (TCP) • The basic flow of HTTP tunneling using the HTTP Connect method • http.Hijack and io.Copy are key functions to implement HTTP tunneling in Go
  17. Agenda HTTP(S) Client WebSocket server WebSocket client HTTP(S) Server 1.

    HTTP Tunneling 2. Establish bidirectional connection using WebSocket 3. HTTP Tunneling over WebSocket
  18. Establish bidirectional connections WebSocket server WebSocket client Handshake WebSocket Conn

    WebSocket Conn HTTP(S) Client WebSocket server WebSocket client HTTP(S) Server
  19. Bidirectional connections • Long/short polling • WebSockets • Server-Sent Events

    • HTTP/2 Push • etc
  20. WebSockets • A mechanism for low-cost, full-duplex communication on Web

    • Designed to work over HTTP (compatible with the HTTP protocol) • Handshake -> Messages -> Close
  21. WebSocket client and server using gorilla/websocket Client: gorilla/websocket: an WebSocket

    implementation in Go Server:
  22. websocket.Conn implementation websocket.Conn has net.Conn in the unexported field. The

    return value of http.Hijack will be set to the field conn. https://github.com/gorilla/websocket/blob/eeb877e8a1fd26c449b7af3cb1d71879287d724c/server.go#L125
  23. Agenda HTTP(S) Client WebSocket server WebSocket client HTTP(S) Server 1.

    HTTP Tunneling 2. Establish bidirectional connection using WebSocket 3. HTTP Tunneling over WebSocket
  24. HTTP Tunneling over WebSocket HTTP(S) Client HTTP(S) Server WebSocket server

    WebSocket client WebSocket Conn WebSocket Conn TCP Conn TCP Conn $ curl -Lv -x http://websocket-server https://target.local WebSocket server and WebSocket client jointly act as a proxy(gateway). https://target.local http://websocket-server streaming streaming
  25. How HTTP Tunneling over WebSocket

  26. How HTTP Tunneling with WebSocket WebSocket server tells the address

    of destination to WebSocket client. WebSocket client open a TCP connection to the destination server.
  27. How HTTP Tunneling with WebSocket Bidirectional message over an established

    WebSocket connection.
  28. WebSocket server implementation A sample code available on github.com/hgsgtk/go-snippets

  29. WebSocket server implementation 1 2 3

  30. Step 1: Write messages to WebSocket connection Send a TCP

    handshake request with the destination address.
  31. Distinguish several types of messages This design transfers several kinds

    of message over a single WebSocket connection • Request to open TCP connections • Notify to establish TCP connections • Streaming raw packets of data • etc
  32. A case study: Chrome Devtools Protocol CDP defines the JSON

    protocol, which is used to communicate in the WebSocket connection. Request Members: id, method, params Response Members: id, result, (error)
  33. Define own protocol for WebSocket communication A small example of

    methods • Handshake: Request to open TCP connections • Completed: Notify to establish TCP connections • Communication: Streaming raw packets of data
  34. Step 2: Take over the proxy connection 2 Take over

    the connection using http.Hijacker.
  35. Step 3: Streaming 3 Use io.Copy to combine WebSocket connection

    and TCP connection.
  36. Notes on net.Conn returned from http.Hijack • http.Hijacker returns net.Conn

    and bufio.ReadWriter. [Code comment on http.Hijack] The returned bufio.Reader may contain unprocessed buffered data from the client. • Reading net.Conn alone may cause data to be dropped. • Read buffers from bufio.Reader.
  37. Wrap websocket.Conn to use in io.Copy: io.Writer Pack incoming packets

    to defined JSON format ンu��(gT��!"�. ��ҠF��0�Q��Yl/��L�E8/���c<��>���C� ��m$�X�Ax�Y%^��J�v�[�o�U�a���U�t �u��5��k��m��&|Ǜq����)�z7����?�… { "method": 3, "data": "ンu��(gT��!"�. ��ҠF��0�Q��Yl/��L�E8/���c<��>� ��C���m$�X�Ax�Y%^��J�v� [�o�U�a���U�t �u��5��k��m��&|Ǜq����) �z7����?�… " }
  38. Wrap websocket.Conn to use in io.Copy: io.Reader Read packets from

    a decoded JSON message ンu��(gT��!"�. ��ҠF��0�Q��Yl/��L�E8/���c<��>���C� ��m$�X�Ax�Y%^��J�v�[�o�U�a���U�t �u��5��k��m��&|Ǜq����)�z7����?�… { "method": 3, "data": "ンu��(gT��!"�. ��ҠF��0�Q��Yl/��L�E8/���c<��>� ��C���m$�X�Ax�Y%^��J�v� [�o�U�a���U�t �u��5��k��m��&|Ǜq����) �z7����?�… " }
  39. WebSocket client implementation

  40. Open a TCP connection to the given destination Dial to

    the destination address conn, err := net.Dial("tcp", “autify.com:443”)
  41. Key takeaways • HTTP Tunneling over WebSocket • Define own

    protocol for WebSocket communication and create a wrap struct for streaming with io.Copy • Should read both net.Conn and bufio.ReadWriter returned from http.Hijacker
  42. Resources • RFC 6455 The WebSocket Protocol • IETF: Tunneling

    TCP based protocols through Web proxy servers • Mmdn web doc: Proxy servers and tunneling • Mmdn web doc: CONNECT • Wikipedia: HTTP Tunnel • Wikipedia: WebSocket • HTTP: The Definitive Guide / 8.5 Tunnels • Cloudflare: What happens in a TLS handshake? | SSL handshake • Chrome DevTools Protocol
  43. Resources • elazarl/goproxy • gorilla/websocket • github.com/hgsgtk/go-snippets/gocon2022spring/tcpws

  44. Thank you for your watching We are taking demo requests

    https://autify.com/ Autify for Web Autify for Mobile