$30 off During Our Annual Pro Sale. View Details »

HTTP Tunneling in Go

HTTP Tunneling in Go

HTTP Tunneling can be used in situations similar to the requirements of using VPN, for example, to be able to access a localhost server from outside. This talk will introduce the specific implementation of HTTP tunneling and give audiences a deep knowledge of TCP and HTTP level implementation in Go.

Specifically, this talk discusses the idea of HTTP Tunneling in Go.

Kazuki Higashiguchi

April 16, 2022
Tweet

More Decks by Kazuki Higashiguchi

Other Decks in Technology

Transcript

  1. HTTP Tunneling in Go
    Kazuki Higashiguchi
    April 23, 2022 @ Go Conference 2022 Spring

    View Slide

  2. Agenda
    HTTP(S)
    Client
    WebSocket
    server
    WebSocket
    client
    HTTP(S)
    Server
    1. HTTP Tunneling
    2. Establish bidirectional connection using WebSocket
    3. HTTP Tunneling over WebSocket

    View Slide

  3. About Me
    Kazuki Higashiguchi
    Backend Engineer at Autify.
    ID: @hgsgtk

    View Slide

  4. Autify Solution
    No-code AI-powered software testing automation
    services for Web/Mobile applications
    AI automatically maintenance test
    scripts
    No code unlocks automation at
    scale

    View Slide

  5. Agenda
    HTTP(S)
    Client
    WebSocket
    server
    WebSocket
    client
    HTTP(S)
    Server
    1. HTTP Tunneling
    2. Establish bidirectional connection using WebSocket
    3. HTTP Tunneling over WebSocket

    View Slide

  6. HTTP tunneling
    ● Create a network link between two computers in
    conditions of restricted network connectivity.
    e.g. firewalls, NATs, ACLs
    ● Use a protocol of higher level (HTTP) to transport a
    lower level protocol (TCP)

    View Slide

  7. HTTP CONNECT method
    ● Start bidirectional communications with the requested
    resource
    e.g. access HTTPS websites, HTTP proxy server
    HTTP(S)
    Client
    HTTP(S)
    Server
    Proxy
    (Gateway)

    View Slide

  8. HTTP tunneling with CONNECT
    Quoted from HTTP: The Definitive Guide / 8.5 Tunnels
    1. Client sends CONNECT request
    2. Gateway opens TCP connection
    to the server
    3. Gateway returns HTTP ready
    message to the client
    4. Start bidirectional
    communication of raw packets
    of data

    View Slide

  9. A case study: elazarl/goproxy
    elazarl/goproxy: An HTTP proxy library for Go
    $ curl -I -x http://localhost:8080 https://example.com
    HTTP/1.0 200 OK
    A basic sample code:

    View Slide

  10. Tunneling implementation in elazarl/goproxy
    https://github.com/elazarl/goproxy/blob/0bfa7c564b
    5ffbece2f0832eb4e714fb971f70f3/https.go#L95

    View Slide

  11. Tunneling implementation in elazarl/goproxy
    1
    2
    3

    View Slide

  12. Step 1: Take over the proxy connection
    http.Hijacker interface allows an HTTP handler to take over
    the connection.

    View Slide

  13. Step 2: Open TCP connections
    Opens TCP connection using net.Dial
    Returns HTTP ready message: 200 Connection established

    View Slide

  14. Step 3: Streaming with io.Copy

    View Slide

  15. Streaming: TLS handshake over TCP connection
    ● TLS handshake is done
    using the streaming
    ● Application data is sent and
    receive after TLS
    handshake

    View Slide

  16. Key takeaways
    ● HTTP Tunneling uses a protocol of higher level (HTTP)
    to transport a lower level protocol (TCP)
    ● The basic flow of HTTP tunneling using the HTTP
    Connect method
    ● http.Hijack and io.Copy are key functions to implement
    HTTP tunneling in Go

    View Slide

  17. Agenda
    HTTP(S)
    Client
    WebSocket
    server
    WebSocket
    client
    HTTP(S)
    Server
    1. HTTP Tunneling
    2. Establish bidirectional connection using WebSocket
    3. HTTP Tunneling over WebSocket

    View Slide

  18. Establish bidirectional connections
    WebSocket
    server
    WebSocket
    client
    Handshake
    WebSocket
    Conn
    WebSocket
    Conn
    HTTP(S)
    Client
    WebSocket
    server
    WebSocket
    client
    HTTP(S)
    Server

    View Slide

  19. Bidirectional connections
    ● Long/short polling
    ● WebSockets
    ● Server-Sent Events
    ● HTTP/2 Push
    ● etc

    View Slide

  20. WebSockets
    ● A mechanism for low-cost,
    full-duplex communication on Web
    ● Designed to work over HTTP
    (compatible with the HTTP
    protocol)
    ● Handshake -> Messages -> Close

    View Slide

  21. WebSocket client and server using gorilla/websocket
    Client:
    gorilla/websocket: an WebSocket implementation in Go
    Server:

    View Slide

  22. websocket.Conn implementation
    websocket.Conn has net.Conn in the unexported field.
    The return value of http.Hijack will be set to the field conn.
    https://github.com/gorilla/websocket/blob/eeb877e8a1fd26c449b7af3cb1d71879287d724c/server.go#L125

    View Slide

  23. Agenda
    HTTP(S)
    Client
    WebSocket
    server
    WebSocket
    client
    HTTP(S)
    Server
    1. HTTP Tunneling
    2. Establish bidirectional connection using WebSocket
    3. HTTP Tunneling over WebSocket

    View Slide

  24. HTTP Tunneling over WebSocket
    HTTP(S)
    Client
    HTTP(S)
    Server
    WebSocket
    server
    WebSocket
    client
    WebSocket
    Conn
    WebSocket
    Conn
    TCP
    Conn
    TCP
    Conn
    $ curl -Lv -x http://websocket-server https://target.local
    WebSocket server and WebSocket client jointly act as a
    proxy(gateway).
    https://target.local
    http://websocket-server
    streaming streaming

    View Slide

  25. How HTTP Tunneling over WebSocket

    View Slide

  26. How HTTP Tunneling with WebSocket
    WebSocket server tells the address of
    destination to WebSocket client.
    WebSocket client open a TCP connection to
    the destination server.

    View Slide

  27. How HTTP Tunneling with WebSocket
    Bidirectional message over
    an established WebSocket
    connection.

    View Slide

  28. WebSocket server implementation
    A sample code available on
    github.com/hgsgtk/go-snippets

    View Slide

  29. WebSocket server implementation
    1
    2
    3

    View Slide

  30. Step 1: Write messages to WebSocket connection
    Send a TCP handshake request
    with the destination address.

    View Slide

  31. Distinguish several types of messages
    This design transfers several kinds of message over a
    single WebSocket connection
    ● Request to open TCP connections
    ● Notify to establish TCP connections
    ● Streaming raw packets of data
    ● etc

    View Slide

  32. A case study: Chrome Devtools Protocol
    CDP defines the JSON protocol, which is used to communicate in the
    WebSocket connection.
    Request
    Members: id, method, params
    Response
    Members: id, result, (error)

    View Slide

  33. Define own protocol for WebSocket communication
    A small example of methods
    ● Handshake:
    Request to open TCP connections
    ● Completed:
    Notify to establish TCP connections
    ● Communication:
    Streaming raw packets of data

    View Slide

  34. Step 2: Take over the proxy connection
    2
    Take over the connection
    using http.Hijacker.

    View Slide

  35. Step 3: Streaming
    3
    Use io.Copy to combine
    WebSocket connection
    and TCP connection.

    View Slide

  36. Notes on net.Conn returned from http.Hijack
    ● http.Hijacker returns net.Conn and
    bufio.ReadWriter.
    [Code comment on http.Hijack]
    The returned bufio.Reader may contain
    unprocessed buffered data from the
    client.
    ● Reading net.Conn alone may cause data
    to be dropped.
    ● Read buffers from bufio.Reader.

    View Slide

  37. Wrap websocket.Conn to use in io.Copy: io.Writer
    Pack incoming packets to defined JSON format
    ンu��(gT��!"�.
    ��ҠF��0�Q��Yl/��L�E8/���c<��>���C�
    ��m$�X�Ax�Y%^��J�v�[�o�U�a���U�t
    �u��5��k��m��&|Ǜq����)�z7����?�…
    {
    "method": 3,
    "data": "ンu��(gT��!"�.
    ��ҠF��0�Q��Yl/��L�E8/���c<��>�
    ��C���m$�X�Ax�Y%^��J�v�
    [�o�U�a���U�t
    �u��5��k��m��&|Ǜq����)
    �z7����?�…
    "
    }

    View Slide

  38. Wrap websocket.Conn to use in io.Copy: io.Reader
    Read packets from a decoded JSON message
    ンu��(gT��!"�.
    ��ҠF��0�Q��Yl/��L�E8/���c<��>���C�
    ��m$�X�Ax�Y%^��J�v�[�o�U�a���U�t
    �u��5��k��m��&|Ǜq����)�z7����?�…
    {
    "method": 3,
    "data": "ンu��(gT��!"�.
    ��ҠF��0�Q��Yl/��L�E8/���c<��>�
    ��C���m$�X�Ax�Y%^��J�v�
    [�o�U�a���U�t
    �u��5��k��m��&|Ǜq����)
    �z7����?�…
    "
    }

    View Slide

  39. WebSocket client implementation

    View Slide

  40. Open a TCP connection to the given destination
    Dial to the destination address
    conn, err := net.Dial("tcp",
    “autify.com:443”)

    View Slide

  41. Key takeaways
    ● HTTP Tunneling over WebSocket
    ● Define own protocol for WebSocket communication and
    create a wrap struct for streaming with io.Copy
    ● Should read both net.Conn and bufio.ReadWriter
    returned from http.Hijacker

    View Slide

  42. Resources
    ● RFC 6455 The WebSocket Protocol
    ● IETF: Tunneling TCP based protocols through Web proxy servers
    ● Mmdn web doc: Proxy servers and tunneling
    ● Mmdn web doc: CONNECT
    ● Wikipedia: HTTP Tunnel
    ● Wikipedia: WebSocket
    ● HTTP: The Definitive Guide / 8.5 Tunnels
    ● Cloudflare: What happens in a TLS handshake? | SSL handshake
    ● Chrome DevTools Protocol

    View Slide

  43. Resources
    ● elazarl/goproxy
    ● gorilla/websocket
    ● github.com/hgsgtk/go-snippets/gocon2022spring/tcpws

    View Slide

  44. Thank you for your watching
    We are taking demo requests https://autify.com/
    Autify for Web Autify for Mobile

    View Slide