«Сниффинг RPC-трафика for fun and profit» — Марко Кевац (Badoo)

Выступление на Go 1.10 Release Party @ Badoo 24.02.2018.

Использование бинарного сетевого протокола имеет свои преимущества и недостатки. В качестве одного из недостатков часто упоминают сложность при дебаггинге, т.к. человек не может сходу увидеть содержание запроса или ответа.

Марко расскажет, как мы до сих пор «сниффали» наш бинарный протокол с помощью wireshark и самописного «диссектора», а теперь планируем использовать модный «диссектор» на Go.

В докладе посмотрим, как сниффать сетевой трафик в Go, познакомимся с кишками protocol buffers, проникнемся магией генерации, парсинга Go-кода и пакета reflect.

Transcript

1. Sniffing RPC traffic for fun and profit
2. Marko Kevac Software Engineer in Platform team at Badoo. I love low level stuff, learning new things, watching Coub weekly, yogurt and of course Go language. Hello, my name is marko@kevac.org blog.kevac.org
3. Client Client Service Client Another Service Protocols And one more
4. Client Client Service Client Another Service Observability And one more ? ? ? ? ?
5. • Internal/builtin • Dump (simple) • Distributed tracing (advanced) • External • Debugger (medium, slow) • SystemTap/eBPF (expert) • Traffic sniffing & parsing (could be simple) Approaches
6. Client Client Service Client Another Service Observability And one more ? ? ? ? ? Text • Memcached • JSON Binary • HTTP/2 • Protobuf
7. Dissectors Wireshark Wireshark dissectors know how to parse and present various protocols. There a many builtin ones, but you can write your own. github.com/mkevac/gpbs-dissector
8. Pitfails Wireshark • Clunky • Needs *.so files with proto descriptors • It is difficult to remember how to use it
9. GPBRPC dissector in Go Because I can it’s useful!
10. 1 2 3 Dissector in Go Sniffing and parsing traffic. Same as with Wireshark. Single binary that is easy to use and deploy. Wide possibilities for improvement and custom features.
11. Milestones Sniffing network Just use libpcap… Unfortunately we are using TCP/IP. Parsing and presenting GPBRPC protobuf Should be straightforward, right? It is not when you remember our requirement about “single binary” and ease of deployment. • A lot of services • New services, changing services • Various service versions at the same time
12. Sniffing network and TCP/IP
13. • Sniffing using libpcap • Reading from the network and from files • TCP Reassembly Examples: • github.com/google/gopacket/examples/ bidirectional github.com/google/ gopacket
15. Simple RPC Dissector
16. • Create your own packets • Proxy, NAT, Bridge, etc… github.com/google/ gopacket
17. Parsing and presenting protobuf
18. • type, id, length, content • bytes and sub message look the same • We need schema • *.proto is a schema and protobuf descriptor is a computer friendly *.proto file representation Binary format
20. protoc Plugin/ generator Protoc compiler Descriptor *.proto Source *.pb.go
21. • Some are very dynamic (python), some are very static (c). • Go has two main generators: default and gogo. Protoc compiler
25. Unmarshaling
26. Protocol Buffers Descriptor
27. Possible solutions Create struct on the fly Go is no Python, but we have very powerful reflect package, right?.. Custom unmarshaller/ marshaller I am too lazy and I don’t want to support this thing forever… C-like dynamic libraries This should be possible in theory, but Go modules are very immature and I don’t want to deal with module preparing stuff.
28. Creating struct on the fly
29. Generate struct on the fly Struct proto.Unmarshal( … ) json.Marshal( … ) descriptor
30. Generate struct on the fly Struct proto.Unmarshal( … ) json.Marshal( … ) descriptor service
31. Protocol Buffers Descriptor
32. Protocol Buffers Descriptor
33. protoc Plugin/ generator Protoc compiler Descriptor *.proto Source *.pb.go
34. Plugin/ generator Protoc compiler Descriptor Source go/ast
35. Plugin/ generator Protoc compiler Descriptor Source go/types
36. DOES NOT FIT reflect.StructOf() It accepts reflect.Type, not types.Type… types.Type -> reflect.Type There is no library or code that does this. There is a similar code deep in Go compiler, but not the same.
37. DOES NOT FIT reflect.StructOf() It accepts reflect.Type, not types.Type… types.Type -> reflect.Type There is no library or code that does this. There is a similar code deep in Go compiler, but not the same. Muse Jason. E. Aten https://github.com/glycerine/muse golang-nuts
38. ME Plugin/ generator Descriptor Source muse go/types proto.Unmarshal []byte StructOf types.Type reflect.Type
39. … is not proto.Message: missing method ProtoMessage goroutine 1 [running]: main.test() /home/marko/src/github.com/mkevac/generatestruct_old/main.go:231 +0x456 main.main() /home/marko/src/github.com/mkevac/generatestruct_old/main.go:247 +0x25
40. Interface, not structure
41. Interface, not structure
42. Embedding
44. type Wrapper struct{} func (w Wrapper) Reset() {} func (w Wrapper) String() string { return "" } func (w Wrapper) ProtoMessage() {} sf, _ := m.ConvertStruct(u.Underlying()) wsf := reflect.StructField{ Name: "Wrapper", Type: reflect.TypeOf(Wrapper{}), Anonymous: true, Index: []int{0}, } sf = append([]reflect.StructField{wsf}, sf[0]) rt := reflect.StructOf(sf)
45. ME Plugin/ generator Descriptor Source muse go/types proto.Unmarshal []byte StructOf types.Type reflect.Type Wrapper
46. … is not proto.Message: missing method ProtoMessage goroutine 1 [running]: main.test() /home/marko/src/github.com/mkevac/generatestruct_old/main.go:231 +0x456 main.main() /home/marko/src/github.com/mkevac/generatestruct_old/main.go:247 +0x25
47. Fuck that shit Only for first anon field Ok, my type is first and anonymous… But doesn’t work anyway https://github.com/golang/go/issues/20824 https://github.com/golang/go/issues/16522 https://github.com/golang/go/issues/15924
48. Custom unmarshaller/marshaller Just read the specification
49. • type, id, length, content • Protobuf descriptor • https://developers.google.com/protocol- buffers/docs/encoding Binary format
50. func Unmarshal(pb []byte, desc *descriptor.FileDescriptorProto, name string) ([]byte, error) https://github.com/mkevac/pb2json Custom unmarshalling
51. But what if?
52. Interface, not structure
53. Fork proto package What you gonna do?..
54. Change function to func Unmarshal(buf []byte, pb interface{}) error Fork proto package
57. Possible solutions Create struct on the fly Custom unmarshaller/ marshaller C-like dynamic libraries Create struct on the fly + proto fork
58. Future ideas Protobuf Descriptor in services + Service Discovery Web user interface Filtering
59. Thank you! techblog.badoo.com habrahabr.ru/company/badoo github.com/mkevac blog.kevac.org marko@kevac.org