GoSF - Decentralizing the Web (again)

Transcript

1. stream multiplexing (in Go)
2. stream multiplexing (in Go)
3. Decentralizing the Web (again)
4. Why don’t you run your own • mail server (gmail) • file server (box, dropbox) • media streaming server (spotify, netflix) • web server (heroku, aws)
5. • content (netflix/spotify) • cost • operational pain (everything else) • security • backups • updates • scale • deployment
6. Why can’t I ssh into my laptop?
7. - Every home consumer and business circa 2000 “How do I connect more machines to the internet than my ISP gives me IP addresses?”
8. The rise of NAT • ~15 years ago • Not enough IPv4 addresses, IPv6 not ready • Unilateral deployment • Not standardized
9. The Cost • No end-to-end connectivity on the internet • Problems for every peer-to-peer protocol • No ordinary user can run their own mail/file/ media/* server • I can’t ssh into my laptop
10. –Every web developer circa 2014 “How do I develop a service that receives webhooks from {API}?”
11. ngrok • Originally a port of localtunnel to learn Go • Establishes a secure tunnel that proxies traffic from a public endpoint to your local machine • Good for building webhook consumers • ngrok 1234 • http://{random}.ngrok.com -> 127.0.0.1:1234
12. Another use of ngrok • Run your own minecraft server. Instructions: • java -jar minecraft_server.jar • ngrok -proto=tcp 25565
13. Could it be easier? java -jar minecraft_server.jar -public
14. Your own “cloud services” ./webmail -tunnel=provider.com
15. An API for Tunneling (in Go)
16. func Dial(addr net.Addr) *Session! ! type Session {! ! ListenTCP(opts *TCPOptions) (*Tunnel, error)! ! ListenHTTP(opts *HTTPOptions) (*Tunnel, error)! }! ! type Tunnel {! ! Addr() net.Addr! ! Accept (net.Conn, error)! ! Close (error) 
 }
17. go-tunnel l, err := net.Listen(“0.0.0.0:9090”)! ! for {! ! conn, err := l.Accept()! ! handleConn(conn)
 }
18. go-tunnel l, err := tunnel.ListenHTTP()! ! http.Serve(l, nil)
19. go-tunnel sess, err := tunnel.Dial(“v1.airlock.io:443”)! ! l, err := sess.ListenTCP(&proto.TCPOptions{! ! RemotePort: 9090,! }, nil)! ! for {! ! conn, err := l.Accept()! ! handleConn(conn)
 }
20. Live Coding!
21. How does that all work?
22. stream multiplexing (in Go)
23. Stream Multiplexing • Many “logical” connections on a single “physical” connection • Basic idea • split streams into chunks • send chunks with a stream identifier • reassemble on the remote side
24. Stream Multiplexing Hey, nice day out, what’s it like? Oh, that’s great! I brought a picnic lunch. It’s sunny, but a little chilly Also, Knock knock Who’s there? Interrupting . . . Okay, I’ll meet you in the park at noon. Interrupting Cow Cool, I think I’ll join you, I could use some sun. MOOOO
25. Stream Multiplexing • Old, old idea • SSH, WebMUX (HTTPNG), BEEP, SPDY, HTTP2, TMP, SCP • SCTP • non-starter, because NATs
26. Stream Multiplexing • Don’t build, steal: SPDYv3/HTTP2 framing layer 0 1 2 3! 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1! +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+! | Length (16) | Type (8) | Flags (8) |! +-+-------------+---------------+-------------------------------+! |R| Stream Identifier (31) |! +-+-------------------------------------------------------------+! | Frame Payload (0...) ...! +---------------------------------------------------------------+ • Many frame types. So complicated. Much confusion. Remove some. Wow. • DATA, HEADERS, PRIORITY, RST_STREAM, SETTINGS, PUSH_PROMISE, CONTINUATION, GOAWAY, WINDOW_UPDATE • SYN, DATA, RST, GOAWAY, WINDOW_UPDATE
27. Built as a library • muxado (github.com/inconshreveable/muxado) • Generic stream multiplexing (only in Go, for now) • Great for tunneling • Also, great for RPC, it turns out
28. muxado’s interfaces type Session interface {! ! Open() (Stream, error)! ! Accept() (Stream, error)! ! Close() (error)! } type Stream interface {! ! Read(p []byte) (int, error)! ! Write(p []byte) (int, error)! ! Close() (error)! }
29. Aside: RPC • RPC - Open connection, send request, read response • memcache, redis, *SQL, task queues, etc • Parallelism? • Open lots of connections! Put them in a pool!
30. Aside: RPC • Smarter RPC • Open base connection, then open many *streams* • No pooling, logical streams are cheap • Parallels to OS threads vs user-mode threads
31. Aside: RPC sess, err := muxado.Dial(“10.1.1.101:4444”)! ! go func() {! ! stream, err := sess.Open()! ! n, err := stream.Write(“SELECT * FROM user”)! ! resp, err := stream.Read()! ! // do stuff with response! ! ! stream.Close()
 }()! ! go func() {! ! stream, err := sess.Open()! ! n, err := stream.Write(“SHOW PROCESSLIST”)! ! resp, err := stream.Read()! ! ! // do stuff with response! ! ! stream.Close()
 }()
32. :( • Kernel/NIC multitasking is better • No standard (part of some other protocol) • Head-of-line-blocking • QUIC
33. What to do with this power? • Start de-centralizing the web • Build applications that anyone can deploy on any internet-connected machine • Allow users to optionally bind to public tunnel addresses • in addition to binding to local ports, of course
34. srvdir • alpha • first go-tunnel project • SimpleHTTPServer with public URLs • https://www.srvdir.net
35. Demo
36. Forward • Standardize the protocols • C implementation /w bindings to other languages • Build more host-anywhere services
37. Addressing the unaddressable • NAT • Heroku • TOR • Embedded • Mobile • ???
38. • @inconshreveable • github.com/inconshreveable/muxado • github.com/inconshreveable/go-tunnel