1 Recap: The Future of JSON in Go Go 1.21 Release Party & GopherCon 2023 Recap Shunta Komatsu

2 About me ● Shunta Komatsu ● Backend Engineer at Merpay ○ Developing the payment platform ● Joined Merpay in April 2022

3 Agenda 1. Introduction of the session 2. What are the problems with the v1 encoding/json? 3. Introduction of possible solutions of the v2 encoding/json 4. Performance and migrations

4 Introduction of the session

5 What kind of presentation? https://www.gophercon.com/agenda/session/1160910

6 The encoding/json package Have you ever used the encoding/json package?

7 The v1 encoding/json package The encoding/json package is the 5th most popular Go package. Marshal (encode) Unmarshal (decode)

8 What are the problems with the v1 encoding/json?

9 What’s wrong with the v1 encoding/json? Four broad categories: 1. Missing functionality 2. API deficiencies 3. Performance limitations 4. Behavioral flaws

10 Missing functionality ● 91 open issues that affect many developers

11 API deficiencies ● You can’t unmarshal from io.Reader correctly ○ You need to create an intermediate decoder json.NewDecoder(r).Decode(v), but it doesn’t reject trailing junk (#36225) ● Marshal, Unmarshal, and custom Marshaler and Unmarshaler do not accept options ● Encoder or Decoder can accept options, but some options depend on bytes.Buffer instead of []byte or io.Writer

12 Performance limitations ● MarshalJSON forces the implementation to allocate returned []byte ● MarshalJSON and UnmarshalJSON forces a second parse ● Lack of streaming ○ Even though the Encoder.Encode and Decoder.Decode operate on an io.Writer or io.Reader, they buffer the entire JSON value in memory

13 Behavioral flaws ● Improper handling of JSON syntax (RFC 8259) ○ Invalid UTF-8 is allowed ○ Duplicate object member names are allowed ● The MarshalJSON and UnmarshalJSON methods cannot be called if the underlying value is not addressable ● Case-insensitive unmarshaling ● Inconsistent error values

14 What’s wrong with the v1 encoding/json? Four broad categories: 1. Missing functionality 2. API deficiencies 3. Performance limitations 4. Behavioral flaws

15 Introduction of possible solutions of the v2 encoding/json

16 Goal of the v2 encoding/json ● Mostly backwards compatible ● More correct ● More performant ● More flexibles ● Easy to use (hard to misuse) ● Avoid unsafe

17 Overview https://github.com/golang/go/discussions/63397

18 Overview ● Two packages jsontext and json ○ jsontext ■ Syntactic functionality that processes JSON based on its grammar ■ “encode” and “decode” ■ A relatively light dependency tree ○ json ■ Semantic functionality that determines the meaning of JSON values as Go values and vice-versa ■ “marshal” and “unmarshal” ■ Depends on the reflect package

19 The package “encoding/json/jsontext” https://github.com/golang/go/discussions/63397

20 The package “encoding/json/jsontext” ● Encoder and Decoder write and read JSON tokens and values ○ You can process JSON text in a purely streaming way ● NewEncoder and NewDecoder accept options

21 The Token and Value type ● A Token represents the smallest structural unit ○ true, false, {, }, [, ], etc. ● A Value is the raw representation of a single JSON value

22 ● You can configure the behaviors ● The Options interface is under the jsonopts package The Options type https://github.com/go-json-experiment/json/blob/master/jsontext/options.go

23 The SyntacticError type ● Errors due to non-compliance with the JSON grammar are reported as SyntacticError.

24 The package “encoding/json/v2” https://github.com/golang/go/discussions/63397

25 The package “encoding/json/v2” ● Marshal and Unmarshal ○ Mostly match the signature of the v1 functions ● MarshalWriter and UnmarshalRead ○ Accept io.Writer and io.Reader instead of []byte ● MarshalEncode and UnmarshalDecode ○ Accept *jsontext.Encoder and *jsontext.Decoder

26 https://github.com/golang/go/discussions/63397 Default behavior changes

27 ● Case-sensitive name match Default behavior changes

28 ● Do not allow invalid UTF-8 Default behavior changes

29 ● Do not allow duplicate names Default behavior changes

30 Struct tag options ● v2 also supports struct tags ○ omitzero ○ omitempty ○ string ○ nocase ○ inline ○ unknown ○ format

31 ● omitzero ○ The field is omitted if the value is zero when marshaling Struct tag options

32 ● format ○ Specifies the format when serializing Struct tag options

33 The Options type ● You can configure the behaviors ● The Options interface is under the jsonopts package https://github.com/go-json-experiment/json/blob/master/options.go

34 The SemanticError type ● Errors due to the inability to correlate JSON data with Go data are reported as SemanticError

35 The package “encoding/json/v2” https://github.com/golang/go/discussions/63397

36 Performance and migrations

37 Performance https://github.com/go-json-experiment/jsonbench/blob/master/images/benchmark-marshal-concrete.png

38 Performance https://github.com/go-json-experiment/jsonbench/blob/master/images/benchmark-unmarshal-concrete.png

39 Migration

40 Summary

41 Impressions of this session ● The proposed v2 package is well thought out, not only performance, API flexibility, security safety, and RFC compliance, but also backward compatibility and migration flexibility ● The idea of splitting syntactic functionality and semantic functionality into two packages, jsontext and json, would be a good idea ● Nevertheless, many changes have been made, so you will have to migrate with each and every option, which will be somewhat nerve-wracking :(

42 Summary ● The v2 encoding/json package is a proposal to improve problems that v1 has, such as missing functionality, API deficiencies, and performance limitations ● You can see the implementation https://github.com/go-json-experiment/json ● Discussion is now open https://github.com/golang/go/discussions/63397

43 Advertisement I’ll talk about ● “Recap: Automatically Instrument Your Go Source Code with Orchestrion” at mercari.go #24 on Nov 1 (tomorrow!) ● “The Future of encoding/json” at Go Conference mini 2023 Winter IN KYOTO on Dec 2

44 References ● “The Future of JSON in Go” by Joe Tsai at GopherCon 2023 (link) ● https://github.com/go-json-experiment/json ● https://github.com/golang/go/discussions/63397

45 Thanks for your attention! The Go gopher was designed by Renee French