Tweet
Tweet

Slide 1

Slide 1 text

Fuzzy generics Several months of using 1.18 features Alexey Palazhchenko FOSDEM 2022-02-05

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Why Go 1.18?

Slide 4

Slide 4 text

Why Go 1.18? Generics

Slide 5

Slide 5 text

Why Go 1.18? Generics Fuzzing

Slide 6

Slide 6 text

Why Go 1.18? Generics Fuzzing

Slide 7

Slide 7 text

Generics

Slide 8

Slide 8 text

Generics • Generics do not work there; let's go back to interfaces

Slide 9

Slide 9 text

Generics • Generics do not work there; let's go back to interfaces • Let me extract a small example for a talk

Slide 10

Slide 10 text

Generics • Generics do not work there; let's go back to interfaces • Let me extract a small example for a talk • Hm, that's not that bad

Slide 11

Slide 11 text

BSON Document

Slide 12

Slide 12 text

BSON Document • Ordered map

Slide 13

Slide 13 text

BSON Document • Ordered map • Keys – string, values – any BSON value, including other documents

Slide 14

Slide 14 text

BSON Document • Ordered map • Keys – string, values – any BSON value, including other documents • The central data structure in FerretDB

Slide 15

Slide 15 text

Interfaces type BSONType interface { bsontype() // sealed } type Int int func (Int) bsontype() {} type String string func (String) bsontype() {}

Slide 16

Slide 16 text

Interfaces type BSONType interface { bsontype() // sealed } type Int int func (Int) bsontype() {} type String string func (String) bsontype() {} type Document struct { m map[string]BSONType keys []string } func (*Document) bsontype() {}

Slide 17

Slide 17 text

Interfaces func (d *Document) Set(key String, value BSONType) { if _, ok := d.m[key]; !ok { d.keys = append(d.keys, key) } d.m[key] = value }

Slide 18

Slide 18 text

Interfaces d.Set("foo", Int(42))

Slide 19

Slide 19 text

Interfaces d.Set("foo", Int(42)) d.Set("qux", nil)

Slide 20

Slide 20 text

Generics type BSONType interface { int | string | *Document }

Slide 21

Slide 21 text

Generics type BSONType interface { int | string | *Document } type Document struct { m map[string]any keys []string }

Slide 22

Slide 22 text

Generics func (d *Document) Set[T BSONType](key string, value T) { if _, ok := d.m[key]; !ok { d.keys = append(d.keys, key) } d.m[key] = value }

Slide 23

Slide 23 text

Generics func (d *Document) Set[T BSONType](key string, value T) { if _, ok := d.m[key]; !ok { d.keys = append(d.keys, key) } d.m[key] = value } syntax error: method must have no type parameters

Slide 24

Slide 24 text

Generics func DocumentSet[T BSONType](d *Document, key string, value T) { if _, ok := d.m[key]; !ok { d.keys = append(d.keys, key) } d.m[key] = value }

Slide 25

Slide 25 text

Generics DocumentSet(d, "foo", 42)

Slide 26

Slide 26 text

Generics DocumentSet(d, "foo", 42) // DocumentSet(d, "qux", nil)

Slide 27

Slide 27 text

Interfaces func MakeDocument(pairs ...BSONType) (*Document, error) { l := len(pairs) if l%2 != 0 { return nil, fmt.Errorf("%d arguments", l)

Slide 28

Slide 28 text

Interfaces

Slide 29

Slide 29 text

Interfaces MakeDocument( String("foo"), Int(42), String("bar"), String("baz"), )

Slide 30

Slide 30 text

Interfaces MakeDocument( String("foo"), Int(42), String("bar"), String("baz"), ) MakeDocument(String("invalid number of arguments"))

Slide 31

Slide 31 text

Interfaces MakeDocument( String("foo"), Int(42), String("bar"), String("baz"), ) MakeDocument(String("invalid number of arguments")) MakeDocument(Int(42), String("invalid key type"))

Slide 32

Slide 32 text

Generics

Slide 33

Slide 33 text

Generics func MakeDocument[T BSONType](key string, value T) *Document

Slide 34

Slide 34 text

Generics func MakeDocument[T BSONType](key string, value T) *Document func MakeDocument2[T1, T2 BSONType](key1 string, value1 T1, key2 string, value2 T2) *Document

Slide 35

Slide 35 text

Generics func MakeDocument[T BSONType](key string, value T) *Document func MakeDocument2[T1, T2 BSONType](key1 string, value1 T1, key2 string, value2 T2) *Document func MakeDocument3[T1, T2, T3 BSONType](key1 string, value1 T1, key2 string, value2 T2, key3 string, value3 T3) *Document

Slide 36

Slide 36 text

Interfaces

Slide 37

Slide 37 text

Interfaces func (d *Document) Get(key String) BSONType { return d.m[key] }

Slide 38

Slide 38 text

Interfaces func (d *Document) Get(key String) BSONType { return d.m[key] } assert.Equal(t, Int(42), d.Get("foo")) assert.Equal(t, nil, d.Get("baz"))

Slide 39

Slide 39 text

Generics

Slide 40

Slide 40 text

Generics func DocumentGet[T BSONType](d *Document, key string) T { v, _ := d.m[key].(T) return v }

Slide 41

Slide 41 text

Generics func DocumentGet[T BSONType](d *Document, key string) T { v, _ := d.m[key].(T) return v } assert.Equal(t, 42, DocumentGet[int](d, "foo")) assert.Equal(t, "", DocumentGet[string](d, "foo"))

Slide 42

Slide 42 text

Generics vs Interfaces

Slide 43

Slide 43 text

Generics vs Interfaces • Better in some aspects

Slide 44

Slide 44 text

Generics vs Interfaces • Better in some aspects • Worse in others

Slide 45

Slide 45 text

Generics vs Interfaces • Better in some aspects • Worse in others

Slide 46

Slide 46 text

Fuzzing

Slide 47

Slide 47 text

Fuzzing • testing/quick on steroids

Slide 48

Slide 48 text

Fuzzing • testing/quick on steroids • MongoDB binary protocol parsing packages

Slide 49

Slide 49 text

Fuzzing • testing/quick on steroids • MongoDB binary protocol parsing packages • FerretDB had fuzzing tests even before unit tests

Slide 50

Slide 50 text

Table-driven tests type testCase struct { name string b []byte v bsontype err error }

Slide 51

Slide 51 text

Table-driven tests type testCase struct { name string b []byte v bsontype err error } • Unmarshal b to (v2, err2) • compare with v or err • Marshal v2 to b2 • compare with b

Slide 52

Slide 52 text

Table-driven tests func TestDocument(t *testing.T) { for _, tc := range testCases { tc := tc t.Run(tc.name, func(t *testing.T) {

Slide 53

Slide 53 text

Fuzzing func FuzzDocument(f *testing.F) { for _, tc := range testCases { f.Add(tc.b) } f.Fuzz(func(t *testing.T, b []byte) {

Slide 54

Slide 54 text

When b2 != b

Slide 55

Slide 55 text

When b2 != b • b might be larger than needed

Slide 56

Slide 56 text

When b2 != b • b might be larger than needed • The unread portion of b should be removed

Slide 57

Slide 57 text

When b2 != b • b might be larger than needed • The unread portion of b should be removed • Some bytes might be insigni fi cant

Slide 58

Slide 58 text

When b2 != b • b might be larger than needed • The unread portion of b should be removed • Some bytes might be insigni fi cant • Use canonization (json.Compact, etc)

Slide 59

Slide 59 text

Fuzzing issues

Slide 60

Slide 60 text

Fuzzing issues • No subtests (testing.F.Run)

Slide 61

Slide 61 text

Fuzzing issues • No subtests (testing.F.Run) • Unnamed seed values

Slide 62

Slide 62 text

Fuzzing issues • No subtests (testing.F.Run) • Unnamed seed values • Hanging detection is unreliable with unset GOMAXPROCS

Slide 63

Slide 63 text

Fuzzing

Slide 64

Slide 64 text

Fuzzing • Found a few bugs in `go test -fuzz`

Slide 65

Slide 65 text

Fuzzing • Found a few bugs in `go test -fuzz` • Found many many bugs in FerretDB

Slide 66

Slide 66 text

Fuzzing • Found a few bugs in `go test -fuzz` • Found many many bugs in FerretDB • De fi nitely use it for parsing

Slide 67

Slide 67 text

Links • https://github.com/AlekSi/ generics-vs-interfaces • https://github.com/akutz/
 go-generics-the-hard-way • https://go.dev/doc/tutorial/ • https://go.dev/doc/fuzz/ • https://www.ferretdb.io • https://github.com/FerretDB ⭐ • https://github.com/AlekSi • @paaleksey