SCTP for Gophers – Alexey Naidyonov

Transcript

1. SCTP for Gophers SCTP for Gophers Why You Should Try SCTP In Your Next Golang Back-end Project Why You Should Try SCTP In Your Next Golang Back-end Project Apr 2019 Apr 2019 Alexey Naidyonov Alexey Naidyonov CSA, ITooLabs CSA, ITooLabs
2. What is SCTP? What is SCTP? A transport layer IP protocol (like TCP or UDP), used mostly to deliver telephone network signalling over A transport layer IP protocol (like TCP or UDP), used mostly to deliver telephone network signalling over IP. IP. SCTP is designed to transport Public Switched Telephone Network (PSTN) SCTP is designed to transport Public Switched Telephone Network (PSTN) signaling messages over IP networks, but is capable of broader applications. signaling messages over IP networks, but is capable of broader applications. RFC 4960 "Stream Control Transmission Protocol" RFC 4960 "Stream Control Transmission Protocol" (https://tools.ietf.org/html/rfc4960) (https://tools.ietf.org/html/rfc4960) Message-oriented Message-oriented Reliable Reliable Multi-stream Multi-stream Multi-home Multi-home 2 2
3. Disclaimer Disclaimer This talk does not cover: This talk does not cover: SCTP over UDP SCTP over UDP WebRTC Data Channels WebRTC Data Channels usrsctp usrsctp ... or anything alike ... or anything alike 3 3
4. SCTP Support SCTP Support Linux/FreeBSD/Solaris Linux/FreeBSD/Solaris No (native) Windows support (usrsctp) No (native) Windows support (usrsctp) No (native) Darwin support (usrsctp, kext) No (native) Darwin support (usrsctp, kext) Weak NAT traversal support on consumer devices Weak NAT traversal support on consumer devices Suitable Suitable for for server server side, side, not not so so for for client client applications applications 4 4
5. SCTP Features (1/2) SCTP Features (1/2) Message-oriented Message-oriented Message boundary preservation Message boundary preservation Reliable data transfer w/SACK Reliable data transfer w/SACK Unordered data delivery option Unordered data delivery option Multi-stream support Multi-stream support Message bundling Message bundling Path MTU discovery, message fragmentation Path MTU discovery, message fragmentation 5 5
6. SCTP Features (2/2) SCTP Features (2/2) Congestion control and avoidance Congestion control and avoidance Multi-homing support Multi-homing support Built-in heartbeat (reachability check) Built-in heartbeat (reachability check) 4-way handshake with security cookies to prevent connection flood attach 4-way handshake with security cookies to prevent connection flood attach 3-way shutdown 3-way shutdown One-to-one and one-to-many connections One-to-one and one-to-many connections Extensible Extensible 6 6
7. SCTP Model SCTP Model Endpoint Endpoint: an addressable logical endpoint, represented by single port number and one or more IP : an addressable logical endpoint, represented by single port number and one or more IP addresses addresses Association Association: a logical relationship between two endpoints (≈ connection) : a logical relationship between two endpoints (≈ connection) Stream Stream: a logical unidirectional channel transporting applications messages. : a logical unidirectional channel transporting applications messages. Message Message: a user message delivered over stream : a user message delivered over stream Packet Packet: PDU comprises of header and multiple : PDU comprises of header and multiple chunks chunks Chunk Chunk: a basic data unit containing either protocol control information (control chunk) or user data : a basic data unit containing either protocol control information (control chunk) or user data (data chunk) (data chunk) Transmission Transmission Sequence Sequence Number Number (TSN): 32-bit sequence number attached to every data chunk (TSN): 32-bit sequence number attached to every data chunk Stream Stream Sequence Sequence Number Number (SSN): 16-bit sequence number attached to every message of a stream (SSN): 16-bit sequence number attached to every message of a stream 7 7
8. SCTP Multi-stream support SCTP Multi-stream support Stream 0 Stream 2 Stream 1 Stream 0 Host A Send Queues Receive Queue Association Host B Receive Queues Send Queue 8 8
9. SCTP Multi-stream support SCTP Multi-stream support Every stream is independent (no head-of-line blocking) Every stream is independent (no head-of-line blocking) Messages within a stream are ordered using the SSN Messages within a stream are ordered using the SSN Message fragments have the same SSN Message fragments have the same SSN Message might be unordered Message might be unordered Number of streams are negotiated at INIT Number of streams are negotiated at INIT Number of outbound streams: how many streams we plan to send? Number of outbound streams: how many streams we plan to send? Max number of inbound streams: how many streams we wish to accept? Max number of inbound streams: how many streams we wish to accept? Number of streams might be re-negotiated Number of streams might be re-negotiated 9 9
10. SCTP Multi-homing support SCTP Multi-homing support 10.10.100.10 10.10.200.10 10.10.100.20 10.10.200.20 Client Net 1 Server Net 2 Path1 Path 2 socket socket 10 10
11. SCTP Multi-homing support SCTP Multi-homing support Multi-homing means multiple transport addresses Multi-homing means multiple transport addresses Makes sense when different IP addresses are bound to different network paths Makes sense when different IP addresses are bound to different network paths One path is One path is primary primary, in case of failure SCTP falls back to , in case of failure SCTP falls back to alternate alternate path path API allows to send data to a specific transport address API allows to send data to a specific transport address Heartbeat checks performed over all paths Heartbeat checks performed over all paths Addresses might be added or removed on the fly ( Addresses might be added or removed on the fly (RFC5061 RFC5061 (https://tools.ietf.org/html/rfc5061) (https://tools.ietf.org/html/rfc5061) ) ) 11 11
12. SCTP Association init and shutdown SCTP Association init and shutdown 4-way handshake 4-way handshake Client Server Client Server SYN SYN-ACK ACK INIT INIT-ACK COOKIE-ECHO COOKIE-ACK 3-way shutdown 3-way shutdown No half-closed state No half-closed state 12 12
13. SCTP Flow control SCTP Flow control Guarantees receiver buffer will never overflow Guarantees receiver buffer will never overflow Acknowledges successful receipt of DATA chunks (using SACK chunks) Acknowledges successful receipt of DATA chunks (using SACK chunks) Reports Gaps and Duplicated TSNs Reports Gaps and Duplicated TSNs Guarantees fast recovery Guarantees fast recovery 13 13
14. SCTP API SCTP API Not trivial due to model complexity Not trivial due to model complexity Defined by Defined by RFC 6458 "Socket API Extensions for SCTP" RFC 6458 "Socket API Extensions for SCTP" (https://tools.ietf.org/html/rfc6458) (https://tools.ietf.org/html/rfc6458) One-to-one socket One-to-one socket One-to-many socket One-to-many socket Event notifications Event notifications 14 14
15. SCTP API SCTP API socket socket socket socket socket socket association 1 association 2 association 3 Client 1 Client 2 Client 3 Server socket socket socket socket association 1 association 2 association 3 Client 1 Client 2 Client 3 Server One-to-One One-to- Many sctp_peeloff API branches a single one-to-many association to one-to-one socket sctp_peeloff API branches a single one-to-many association to one-to-one socket SCTP_SENDALL flag forces broadcast to all one-to-many associations SCTP_SENDALL flag forces broadcast to all one-to-many associations 15 15
16. SCTP Go API SCTP Go API Multiple cgo Multiple cgo Native Native github.com/nperez-messagebird/sctp github.com/nperez-messagebird/sctp (https://github.com/nperez-messagebird/sctp) (https://github.com/nperez-messagebird/sctp) Native ITooLabs, not open (yet) Native ITooLabs, not open (yet) 16 16
17. SCTP Go API problems SCTP Go API problems Go native networking is way too native Go native networking is way too native Go Go syscall.RawConn syscall.RawConn (starting go 1.9) (starting go 1.9) type RawConn interface { type RawConn interface { // Control invokes f on the underlying connection's file // Control invokes f on the underlying connection's file // descriptor or handle. // descriptor or handle. Control(f func(fd uintptr)) error Control(f func(fd uintptr)) error // Read invokes f on the underlying connection's file // Read invokes f on the underlying connection's file // descriptor or handle; f is expected to try to read from the // descriptor or handle; f is expected to try to read from the // file descriptor. // file descriptor. Read(f func(fd uintptr) (done bool)) error Read(f func(fd uintptr) (done bool)) error // Write is like Read but for writing. // Write is like Read but for writing. Write(f func(fd uintptr) (done bool)) error Write(f func(fd uintptr) (done bool)) error } } (still not enough) (still not enough) 17 17
18. SCTP Go API: the minimal set of runtime calls required SCTP Go API: the minimal set of runtime calls required //go:linkname sysSocket net.sysSocket //go:linkname sysSocket net.sysSocket func sysSocket(family, sotype, proto int) (int, error) func sysSocket(family, sotype, proto int) (int, error) //go:linkname newNetFD net.newFD //go:linkname newNetFD net.newFD func newNetFD(sysfd, family, sotype int, net string) (unsafe.Pointer, error) func newNetFD(sysfd, family, sotype int, net string) (unsafe.Pointer, error) //go:linkname initNetFD net.(*netFD).init //go:linkname initNetFD net.(*netFD).init func initNetFD(netfd unsafe.Pointer) error func initNetFD(netfd unsafe.Pointer) error //go:linkname setDeadline net.(*netFD).SetDeadline //go:linkname setDeadline net.(*netFD).SetDeadline func setDeadline(netfd unsafe.Pointer, t time.Time) error func setDeadline(netfd unsafe.Pointer, t time.Time) error //go:linkname seReadDeadline net.(*netFD).SetReadDeadline //go:linkname seReadDeadline net.(*netFD).SetReadDeadline func setReadDeadline(netfd unsafe.Pointer, t time.Time) error func setReadDeadline(netfd unsafe.Pointer, t time.Time) error //go:linkname setWriteDeadline net.(*netFD).SetWriteDeadline //go:linkname setWriteDeadline net.(*netFD).SetWriteDeadline func setWriteDeadline(netfd unsafe.Pointer, t time.Time) error func setWriteDeadline(netfd unsafe.Pointer, t time.Time) error //go:linkname setsockopt syscall.setsockopt //go:linkname setsockopt syscall.setsockopt func setsockopt(s int, level int, name int, val unsafe.Pointer, vallen uintptr) (err error) func setsockopt(s int, level int, name int, val unsafe.Pointer, vallen uintptr) (err error) //go:linkname getsockopt syscall.getsockopt //go:linkname getsockopt syscall.getsockopt func getsockopt(s int, level int, name int, val unsafe.Pointer, vallen *_Socklen) (err error) func getsockopt(s int, level int, name int, val unsafe.Pointer, vallen *_Socklen) (err error) 18 18
19. SCTP Go API: SCTP Go API: Waiting for Waiting for #15021 #15021 (https://github.com/golang/go/issues/15021) (https://github.com/golang/go/issues/15021) "Allow registrations for new socket type API" "Allow registrations for new socket type API" 19 19
20. SCTP Go API (github.com/nperez-messagebird/sctp) SCTP Go API (github.com/nperez-messagebird/sctp) // Addr represents SCTP end point address // Addr represents SCTP end point address // // // Single SCTP end point might be bound to multiple IP addresses. // Single SCTP end point might be bound to multiple IP addresses. // // type SCTPAddr struct { type SCTPAddr struct { IPAddrs []net.IPAddr IPAddrs []net.IPAddr Port int Port int } } // Resolves SCTP Address. Multiple IP addresses are separated by "/" // Resolves SCTP Address. Multiple IP addresses are separated by "/" // (i.e. 10.50.1.1/10.50.2.1:2904, 10.50.1.1/[2001:db8::1]:2904) // (i.e. 10.50.1.1/10.50.2.1:2904, 10.50.1.1/[2001:db8::1]:2904) // // func ResolveSCTPAddr(addressFamily SCTPAddressFamily, addrs string) (*SCTPAddr, error) func ResolveSCTPAddr(addressFamily SCTPAddressFamily, addrs string) (*SCTPAddr, error) 20 20
21. SCTP Go API (github.com/nperez-messagebird/sctp) SCTP Go API (github.com/nperez-messagebird/sctp) type SCTPSocketMode int type SCTPSocketMode int const ( const ( OneToOne = SCTPSocketMode(iota) OneToOne = SCTPSocketMode(iota) OneToMany OneToMany ) ) type InitMsg struct { type InitMsg struct { NumOstreams uint16 NumOstreams uint16 MaxInstreams uint16 MaxInstreams uint16 MaxAttempts uint16 MaxAttempts uint16 MaxInitTimeout uint16 MaxInitTimeout uint16 } } func NewSCTPListener(laddr *SCTPAddr, init InitMsg, mode SCTPSocketMode) (*SCTPListener, error) func NewSCTPListener(laddr *SCTPAddr, init InitMsg, mode SCTPSocketMode) (*SCTPListener, error) func NewSCTPConnection(laddr, raddr *SCTPAddr, options InitMsg, mode SCTPSocketMode) (*SCTPConn, error) func NewSCTPConnection(laddr, raddr *SCTPAddr, options InitMsg, mode SCTPSocketMode) (*SCTPConn, error) 21 21
22. SCTP Go API (github.com/nperez-messagebird/sctp) SCTP Go API (github.com/nperez-messagebird/sctp) type SndRcvInfo struct { type SndRcvInfo struct { Stream uint16 // stream id Stream uint16 // stream id SSN uint16 // stream sequence number SSN uint16 // stream sequence number Flags uint16 // flags (i.e. SCTP_UNORDERDED) Flags uint16 // flags (i.e. SCTP_UNORDERDED) _ uint16 _ uint16 PPID uint32 // supplied by peer app PPID uint32 // supplied by peer app Context uint32 // message context (supplied by app) Context uint32 // message context (supplied by app) TTL uint32 // time to live (ms) TTL uint32 // time to live (ms) TSN uint32 // TSN TSN uint32 // TSN CumTSN uint32 // Cumulative TSN CumTSN uint32 // Cumulative TSN AssocID int32 // association id AssocID int32 // association id } } func SCTPRead(b []byte) (int, *OOBMessage, int, error) func SCTPRead(b []byte) (int, *OOBMessage, int, error) func (o *OOBMessage) GetSndRcvInfo() *SndRcvInfo func (o *OOBMessage) GetSndRcvInfo() *SndRcvInfo func SCTPWrite(b []byte, info *SndRcvInfo) (int, error) func SCTPWrite(b []byte, info *SndRcvInfo) (int, error) 22 22
23. SCTP Go API (github.com/nperez-messagebird/sctp) SCTP Go API (github.com/nperez-messagebird/sctp) // read (one-to-one) // read (one-to-one) func (ln *SCTPConn) SCTPRead(b []byte) (n int, oob *OOBMessage, flags int, err error) func (ln *SCTPConn) SCTPRead(b []byte) (n int, oob *OOBMessage, flags int, err error) // write (one-to-one) // write (one-to-one) func (ln *SCTPConn) SCTPWrite(b []byte, info *SndRcvInfo) (n int, err error) func (ln *SCTPConn) SCTPWrite(b []byte, info *SndRcvInfo) (n int, err error) 23 23
24. SCTP Go API (github.com/nperez-messagebird/sctp) SCTP Go API (github.com/nperez-messagebird/sctp) // Accept (one-to-one mode) // Accept (one-to-one mode) func (ln *SCTPListener) AcceptSCTP() (*SCTPConn, error) func (ln *SCTPListener) AcceptSCTP() (*SCTPConn, error) // read (one-to-many mode) // read (one-to-many mode) func (ln *SCTPListener) SCTPRead(b []byte) (n int, oob *OOBMessage, flags int, err error) func (ln *SCTPListener) SCTPRead(b []byte) (n int, oob *OOBMessage, flags int, err error) // write (one-to-many mode) // write (one-to-many mode) func (ln *SCTPListener) SCTPWrite(b []byte, info *SndRcvInfo) (n int, err error) func (ln *SCTPListener) SCTPWrite(b []byte, info *SndRcvInfo) (n int, err error) 24 24
25. SCTP Go API (github.com/nperez-messagebird/sctp) SCTP Go API (github.com/nperez-messagebird/sctp) func SubscribeEvents(int evts) error func SubscribeEvents(int evts) error const ( const ( SCTP_EVENT_DATA_IO = 1 << iota SCTP_EVENT_DATA_IO = 1 << iota SCTP_EVENT_ASSOCIATION SCTP_EVENT_ASSOCIATION SCTP_EVENT_ADDRESS SCTP_EVENT_ADDRESS SCTP_EVENT_SEND_FAILURE SCTP_EVENT_SEND_FAILURE SCTP_EVENT_PEER_ERROR SCTP_EVENT_PEER_ERROR SCTP_EVENT_SHUTDOWN SCTP_EVENT_SHUTDOWN SCTP_EVENT_PARTIAL_DELIVERY SCTP_EVENT_PARTIAL_DELIVERY SCTP_EVENT_ADAPTATION_LAYER SCTP_EVENT_ADAPTATION_LAYER SCTP_EVENT_AUTHENTICATION SCTP_EVENT_AUTHENTICATION SCTP_EVENT_SENDER_DRY SCTP_EVENT_SENDER_DRY ) ) 25 25
26. SCTP Go API (ITooLabs) SCTP Go API (ITooLabs) func NewSCTPSocket(network string, mode SocketMode) (*Socket, error) func NewSCTPSocket(network string, mode SocketMode) (*Socket, error) func NewSCTPListener(network string, laddr string, mode SocketMode) (*Listener, error) func NewSCTPListener(network string, laddr string, mode SocketMode) (*Listener, error) // Conn is an association, not the socket itself // Conn is an association, not the socket itself func DialSCTP(network string, laddr *Addr, mode SocketMode) (*Conn, error) func DialSCTP(network string, laddr *Addr, mode SocketMode) (*Conn, error) func (l *Listener) Accept() (*Conn, error) func (l *Listener) Accept() (*Conn, error) func (c *Conn) PeelOff() error func (c *Conn) PeelOff() error // Stream returns a stream handler // Stream returns a stream handler func (c *Conn) IStream(sid uint16) (*IStream, error) {...} func (c *Conn) IStream(sid uint16) (*IStream, error) {...} func (c *Conn) OStream(sid uint16) (*OStream, error) {...} func (c *Conn) OStream(sid uint16) (*OStream, error) {...} func (s *OStream) WriteMsg([]byte) error func (s *OStream) WriteMsg([]byte) error func (s *IStream) RecvMsg([]byte) error func (s *IStream) RecvMsg([]byte) error 26 26
27. SCTP Go API (Events) SCTP Go API (Events) // Represents SCTP notification event // Represents SCTP notification event type Event interface { type Event interface { Type() uint16 // SCTP event type Type() uint16 // SCTP event type Flags() uint16 // SCTP event flags Flags() uint16 // SCTP event flags Len() uint32 // SCTP event length, including header Len() uint32 // SCTP event length, including header } } // EventPeerAddrChange // EventPeerAddrChange // EventAssocChange // EventAssocChange // AuthenticationEvent // AuthenticationEvent // ShutdownEvent // ShutdownEvent // SenderDryEvent // SenderDryEvent // PartialDeliveryEvent // PartialDeliveryEvent // SendFailureEvent // SendFailureEvent // ... // ... func (s *Socket) SubscribeEvents(events EventKind...) func (s *Socket) SubscribeEvents(events EventKind...) func (s *Socket) Events() <-chan Event func (s *Socket) Events() <-chan Event 27 27
28. Multi-homing demo Multi-homing demo Environment: VirtualBox + Docker + OpenVSWitch Environment: VirtualBox + Docker + OpenVSWitch 2 virtual networks (10.10.100/24, 10.10.200/24) 2 virtual networks (10.10.100/24, 10.10.200/24) 2 containers (client, server) 2 containers (client, server) 10.10.100.10 10.10.200.10 10.10.100.20 10.10.200.20 Client Net 1 Server Net 2 28 28
29. SCTP: Where to use SCTP: Where to use A distributed multi-node multi-service application (seem to be ~80% of all Go applications out there) A distributed multi-node multi-service application (seem to be ~80% of all Go applications out there) Strict requirements for high availability and high resiliency Strict requirements for high availability and high resiliency Controlled multi-path networking environment Controlled multi-path networking environment 29 29
30. Thank you Thank you Apr 2019 Apr 2019 Tags: SCTP, networking Tags: SCTP, networking (#ZgotmplZ) (#ZgotmplZ) Alexey Naidyonov Alexey Naidyonov CSA, ITooLabs CSA, ITooLabs anaidyonov@itoolabs.com anaidyonov@itoolabs.com (mailto:anaidyonov@itoolabs.com) (mailto:anaidyonov@itoolabs.com) https://itoolabs.com/ https://itoolabs.com/ (https://itoolabs.com/) (https://itoolabs.com/) @growler @growler (http://twitter.com/growler) (http://twitter.com/growler)