Recap: The Future of JSON in Go

by Shunta Komatsu

Slide 1

Slide 1 text

1 Recap: The Future of JSON in Go Go 1.21 Release Party & GopherCon 2023 Recap Shunta Komatsu

Slide 2

Slide 2 text

2 About me ● Shunta Komatsu ● Backend Engineer at Merpay ○ Developing the payment platform ● Joined Merpay in April 2022

Slide 3

Slide 3 text

3 Agenda 1. Introduction of the session 2. What are the problems with the v1 encoding/json? 3. Introduction of possible solutions of the v2 encoding/json 4. Performance and migrations

Slide 4

Slide 4 text

4 Introduction of the session

Slide 5

Slide 5 text

5 What kind of presentation? https://www.gophercon.com/agenda/session/1160910

Slide 6

Slide 6 text

6 The encoding/json package Have you ever used the encoding/json package?

Slide 7

Slide 7 text

7 The v1 encoding/json package The encoding/json package is the 5th most popular Go package. Marshal (encode) Unmarshal (decode)

Slide 8

Slide 8 text

8 What are the problems with the v1 encoding/json?

Slide 9

Slide 9 text

9 What’s wrong with the v1 encoding/json? Four broad categories: 1. Missing functionality 2. API deﬁciencies 3. Performance limitations 4. Behavioral ﬂaws

Slide 10

Slide 10 text

10 Missing functionality ● 91 open issues that affect many developers

Slide 11

Slide 11 text

11 API deﬁciencies ● You can’t unmarshal from io.Reader correctly ○ You need to create an intermediate decoder json.NewDecoder(r).Decode(v), but it doesn’t reject trailing junk (#36225) ● Marshal, Unmarshal, and custom Marshaler and Unmarshaler do not accept options ● Encoder or Decoder can accept options, but some options depend on bytes.Buffer instead of []byte or io.Writer

Slide 12

Slide 12 text

12 Performance limitations ● MarshalJSON forces the implementation to allocate returned []byte ● MarshalJSON and UnmarshalJSON forces a second parse ● Lack of streaming ○ Even though the Encoder.Encode and Decoder.Decode operate on an io.Writer or io.Reader, they buffer the entire JSON value in memory

Slide 13

Slide 13 text

13 Behavioral ﬂaws ● Improper handling of JSON syntax (RFC 8259) ○ Invalid UTF-8 is allowed ○ Duplicate object member names are allowed ● The MarshalJSON and UnmarshalJSON methods cannot be called if the underlying value is not addressable ● Case-insensitive unmarshaling ● Inconsistent error values

Slide 14

Slide 14 text

14 What’s wrong with the v1 encoding/json? Four broad categories: 1. Missing functionality 2. API deﬁciencies 3. Performance limitations 4. Behavioral ﬂaws

Slide 15

Slide 15 text

15 Introduction of possible solutions of the v2 encoding/json

Slide 16

Slide 16 text

16 Goal of the v2 encoding/json ● Mostly backwards compatible ● More correct ● More performant ● More ﬂexibles ● Easy to use (hard to misuse) ● Avoid unsafe

Slide 17

Slide 17 text

17 Overview https://github.com/golang/go/discussions/63397

Slide 18

Slide 18 text

18 Overview ● Two packages jsontext and json ○ jsontext ■ Syntactic functionality that processes JSON based on its grammar ■ “encode” and “decode” ■ A relatively light dependency tree ○ json ■ Semantic functionality that determines the meaning of JSON values as Go values and vice-versa ■ “marshal” and “unmarshal” ■ Depends on the reflect package

Slide 19

Slide 19 text

19 The package “encoding/json/jsontext” https://github.com/golang/go/discussions/63397

Slide 20

Slide 20 text

20 The package “encoding/json/jsontext” ● Encoder and Decoder write and read JSON tokens and values ○ You can process JSON text in a purely streaming way ● NewEncoder and NewDecoder accept options

Slide 21

Slide 21 text

21 The Token and Value type ● A Token represents the smallest structural unit ○ true, false, {, }, [, ], etc. ● A Value is the raw representation of a single JSON value

Slide 22

Slide 22 text

22 ● You can conﬁgure the behaviors ● The Options interface is under the jsonopts package The Options type https://github.com/go-json-experiment/json/blob/master/jsontext/options.go

Slide 23

Slide 23 text

23 The SyntacticError type ● Errors due to non-compliance with the JSON grammar are reported as SyntacticError.

Slide 24

Slide 24 text

24 The package “encoding/json/v2” https://github.com/golang/go/discussions/63397

Slide 25

Slide 25 text

25 The package “encoding/json/v2” ● Marshal and Unmarshal ○ Mostly match the signature of the v1 functions ● MarshalWriter and UnmarshalRead ○ Accept io.Writer and io.Reader instead of []byte ● MarshalEncode and UnmarshalDecode ○ Accept *jsontext.Encoder and *jsontext.Decoder

Slide 26

Slide 26 text

26 https://github.com/golang/go/discussions/63397 Default behavior changes

Slide 27

Slide 27 text

27 ● Case-sensitive name match Default behavior changes

Slide 28

Slide 28 text

28 ● Do not allow invalid UTF-8 Default behavior changes

Slide 29

Slide 29 text

29 ● Do not allow duplicate names Default behavior changes

Slide 30

Slide 30 text

30 Struct tag options ● v2 also supports struct tags ○ omitzero ○ omitempty ○ string ○ nocase ○ inline ○ unknown ○ format

Slide 31

Slide 31 text

31 ● omitzero ○ The ﬁeld is omitted if the value is zero when marshaling Struct tag options

Slide 32

Slide 32 text

32 ● format ○ Speciﬁes the format when serializing Struct tag options

Slide 33

Slide 33 text

33 The Options type ● You can conﬁgure the behaviors ● The Options interface is under the jsonopts package https://github.com/go-json-experiment/json/blob/master/options.go

Slide 34

Slide 34 text

34 The SemanticError type ● Errors due to the inability to correlate JSON data with Go data are reported as SemanticError

Slide 35

Slide 35 text

35 The package “encoding/json/v2” https://github.com/golang/go/discussions/63397

Slide 36

Slide 36 text

36 Performance and migrations

Slide 37

Slide 37 text

37 Performance https://github.com/go-json-experiment/jsonbench/blob/master/images/benchmark-marshal-concrete.png

Slide 38

Slide 38 text

38 Performance https://github.com/go-json-experiment/jsonbench/blob/master/images/benchmark-unmarshal-concrete.png

Slide 39

Slide 39 text

39 Migration

Slide 40

Slide 40 text

40 Summary

Slide 41

Slide 41 text

41 Impressions of this session ● The proposed v2 package is well thought out, not only performance, API ﬂexibility, security safety, and RFC compliance, but also backward compatibility and migration ﬂexibility ● The idea of splitting syntactic functionality and semantic functionality into two packages, jsontext and json, would be a good idea ● Nevertheless, many changes have been made, so you will have to migrate with each and every option, which will be somewhat nerve-wracking :(

Slide 42

Slide 42 text

42 Summary ● The v2 encoding/json package is a proposal to improve problems that v1 has, such as missing functionality, API deﬁciencies, and performance limitations ● You can see the implementation https://github.com/go-json-experiment/json ● Discussion is now open https://github.com/golang/go/discussions/63397

Slide 43

Slide 43 text

43 Advertisement I’ll talk about ● “Recap: Automatically Instrument Your Go Source Code with Orchestrion” at mercari.go #24 on Nov 1 (tomorrow!) ● “The Future of encoding/json” at Go Conference mini 2023 Winter IN KYOTO on Dec 2

Slide 44

Slide 44 text

44 References ● “The Future of JSON in Go” by Joe Tsai at GopherCon 2023 (link) ● https://github.com/go-json-experiment/json ● https://github.com/golang/go/discussions/63397

Slide 45

Slide 45 text

45 Thanks for your attention! The Go gopher was designed by Renee French