Slide 1
Slide 1 text
Sniffing RPC traffic for fun and
profit
Slide 2
Slide 2 text
Marko Kevac
Software Engineer in Platform team at Badoo.
I love low level stuff, learning new things, watching
Coub weekly, yogurt and of course Go language.
Hello, my name is
[email protected]
blog.kevac.org
Slide 3
Slide 3 text
Client
Client
Service
Client
Another
Service
Protocols
And one
more
Slide 4
Slide 4 text
Client
Client
Service
Client
Another
Service
Observability
And one
more
?
?
?
?
?
Slide 5
Slide 5 text
• Internal/builtin
• Dump (simple)
• Distributed tracing (advanced)
• External
• Debugger (medium, slow)
• SystemTap/eBPF (expert)
• Traffic sniffing & parsing (could be simple)
Approaches
Slide 6
Slide 6 text
Client
Client
Service
Client
Another
Service
Observability
And one
more
?
?
?
?
?
Text
• Memcached
• JSON
Binary
• HTTP/2
• Protobuf
Slide 7
Slide 7 text
Dissectors
Wireshark
Wireshark dissectors know how to parse
and present various protocols.
There a many builtin ones, but you can
write your own.
github.com/mkevac/gpbs-dissector
Slide 8
Slide 8 text
Pitfails
Wireshark
• Clunky
• Needs *.so files with proto descriptors
• It is difficult to remember how to use it
Slide 9
Slide 9 text
GPBRPC dissector in Go
Because I can it’s useful!
Slide 10
Slide 10 text
1 2 3
Dissector in Go
Sniffing and parsing traffic. Same as
with Wireshark.
Single binary that is easy to use and
deploy.
Wide possibilities for improvement
and custom features.
Slide 11
Slide 11 text
Milestones Sniffing network
Just use libpcap… Unfortunately we are using TCP/IP.
Parsing and presenting GPBRPC protobuf
Should be straightforward, right? It is not when you remember our requirement
about “single binary” and ease of deployment.
• A lot of services
• New services, changing services
• Various service versions at the same time
Slide 12
Slide 12 text
Sniffing network and TCP/IP
Slide 13
Slide 13 text
• Sniffing using libpcap
• Reading from the network and from files
• TCP Reassembly
Examples:
• github.com/google/gopacket/examples/
bidirectional
github.com/google/
gopacket
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
Simple RPC Dissector
Slide 16
Slide 16 text
• Create your own packets
• Proxy, NAT, Bridge, etc…
github.com/google/
gopacket
Slide 17
Slide 17 text
Parsing and presenting protobuf
Slide 18
Slide 18 text
• type, id, length, content
• bytes and sub message look the same
• We need schema
• *.proto is a schema and protobuf
descriptor is a computer friendly *.proto
file representation
Binary format
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
protoc Plugin/
generator
Protoc compiler
Descriptor
*.proto
Source
*.pb.go
Slide 21
Slide 21 text
• Some are very dynamic (python), some
are very static (c).
• Go has two main generators: default and
gogo.
Protoc compiler
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
No content
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
Unmarshaling
Slide 26
Slide 26 text
Protocol Buffers Descriptor
Slide 27
Slide 27 text
Possible solutions
Create struct on the fly
Go is no Python, but we have very powerful reflect
package, right?..
Custom unmarshaller/
marshaller
I am too lazy and I don’t want to support this thing
forever…
C-like dynamic libraries
This should be possible in theory, but Go modules are
very immature and I don’t want to deal with module
preparing stuff.
Slide 28
Slide 28 text
Creating struct on the fly
Slide 29
Slide 29 text
Generate struct on the fly
Struct
proto.Unmarshal( … )
json.Marshal( … )
descriptor
Slide 30
Slide 30 text
Generate struct on the fly
Struct
proto.Unmarshal( … )
json.Marshal( … )
descriptor
service
Slide 31
Slide 31 text
Protocol Buffers Descriptor
Slide 32
Slide 32 text
Protocol Buffers Descriptor
Slide 33
Slide 33 text
protoc Plugin/
generator
Protoc compiler
Descriptor
*.proto
Source
*.pb.go
Slide 34
Slide 34 text
Plugin/
generator
Protoc compiler
Descriptor
Source
go/ast
Slide 35
Slide 35 text
Plugin/
generator
Protoc compiler
Descriptor
Source
go/types
Slide 36
Slide 36 text
DOES NOT FIT
reflect.StructOf()
It accepts reflect.Type, not types.Type…
types.Type -> reflect.Type
There is no library or code that does this.
There is a similar code deep in Go compiler, but not the
same.
Slide 37
Slide 37 text
DOES NOT FIT
reflect.StructOf()
It accepts reflect.Type, not types.Type…
types.Type -> reflect.Type
There is no library or code that does this.
There is a similar code deep in Go compiler, but not the
same.
Muse
Jason. E. Aten
https://github.com/glycerine/muse
golang-nuts
Slide 38
Slide 38 text
ME Plugin/
generator
Descriptor
Source
muse
go/types
proto.Unmarshal
[]byte
StructOf
types.Type
reflect.Type
Slide 39
Slide 39 text
… is not proto.Message: missing method ProtoMessage
goroutine 1 [running]:
main.test()
/home/marko/src/github.com/mkevac/generatestruct_old/main.go:231 +0x456
main.main()
/home/marko/src/github.com/mkevac/generatestruct_old/main.go:247 +0x25
Slide 40
Slide 40 text
Interface, not structure
Slide 41
Slide 41 text
Interface, not structure
Slide 42
Slide 42 text
Embedding
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
type Wrapper struct{}
func (w Wrapper) Reset() {}
func (w Wrapper) String() string { return "" }
func (w Wrapper) ProtoMessage() {}
sf, _ := m.ConvertStruct(u.Underlying())
wsf := reflect.StructField{
Name: "Wrapper",
Type: reflect.TypeOf(Wrapper{}),
Anonymous: true,
Index: []int{0},
}
sf = append([]reflect.StructField{wsf}, sf[0])
rt := reflect.StructOf(sf)
Slide 45
Slide 45 text
ME Plugin/
generator
Descriptor
Source
muse
go/types
proto.Unmarshal
[]byte
StructOf
types.Type
reflect.Type
Wrapper
Slide 46
Slide 46 text
… is not proto.Message: missing method ProtoMessage
goroutine 1 [running]:
main.test()
/home/marko/src/github.com/mkevac/generatestruct_old/main.go:231 +0x456
main.main()
/home/marko/src/github.com/mkevac/generatestruct_old/main.go:247 +0x25
Slide 47
Slide 47 text
Fuck that shit
Only for first anon field
Ok, my type is first and anonymous…
But doesn’t work anyway
https://github.com/golang/go/issues/20824
https://github.com/golang/go/issues/16522
https://github.com/golang/go/issues/15924
Slide 48
Slide 48 text
Custom unmarshaller/marshaller
Just read the specification
Slide 49
Slide 49 text
• type, id, length, content
• Protobuf descriptor
• https://developers.google.com/protocol-
buffers/docs/encoding
Binary format
Slide 50
Slide 50 text
func Unmarshal(pb []byte,
desc *descriptor.FileDescriptorProto,
name string) ([]byte, error)
https://github.com/mkevac/pb2json
Custom unmarshalling
Slide 51
Slide 51 text
But what if?
Slide 52
Slide 52 text
Interface, not structure
Slide 53
Slide 53 text
Fork proto package
What you gonna do?..
Slide 54
Slide 54 text
Change function to
func Unmarshal(buf []byte, pb
interface{}) error
Fork proto package
Slide 55
Slide 55 text
No content
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
Possible solutions
Create struct on the fly
Custom unmarshaller/
marshaller
C-like dynamic libraries
Create struct on the fly + proto
fork
Slide 58
Slide 58 text
Future ideas
Protobuf Descriptor in services
+ Service Discovery
Web user interface
Filtering
Slide 59
Slide 59 text
Thank you!
techblog.badoo.com
habrahabr.ru/company/badoo
github.com/mkevac
blog.kevac.org
[email protected]