Recap: The Future of JSON in Go

https://gocon.connpass.com/event/299108/

Transcript

1. 1 Recap: The Future of JSON in Go Go 1.21 Release Party & GopherCon 2023 Recap Shunta Komatsu
2. 2 About me • Shunta Komatsu • Backend Engineer at Merpay ◦ Developing the payment platform • Joined Merpay in April 2022
3. 3 Agenda 1. Introduction of the session 2. What are the problems with the v1 encoding/json? 3. Introduction of possible solutions of the v2 encoding/json 4. Performance and migrations
4. 4 Introduction of the session
5. 5 What kind of presentation? https://www.gophercon.com/agenda/session/1160910
6. 6 The encoding/json package Have you ever used the encoding/json package?
7. 7 The v1 encoding/json package The encoding/json package is the 5th most popular Go package. Marshal (encode) Unmarshal (decode)
8. 8 What are the problems with the v1 encoding/json?
9. 9 What’s wrong with the v1 encoding/json? Four broad categories: 1. Missing functionality 2. API deficiencies 3. Performance limitations 4. Behavioral flaws
10. 10 Missing functionality • 91 open issues that affect many developers
11. 11 API deficiencies • You can’t unmarshal from io.Reader correctly ◦ You need to create an intermediate decoder json.NewDecoder(r).Decode(v), but it doesn’t reject trailing junk (#36225) • Marshal, Unmarshal, and custom Marshaler and Unmarshaler do not accept options • Encoder or Decoder can accept options, but some options depend on bytes.Buffer instead of []byte or io.Writer
12. 12 Performance limitations • MarshalJSON forces the implementation to allocate returned []byte • MarshalJSON and UnmarshalJSON forces a second parse • Lack of streaming ◦ Even though the Encoder.Encode and Decoder.Decode operate on an io.Writer or io.Reader, they buffer the entire JSON value in memory
13. 13 Behavioral flaws • Improper handling of JSON syntax (RFC 8259) ◦ Invalid UTF-8 is allowed ◦ Duplicate object member names are allowed • The MarshalJSON and UnmarshalJSON methods cannot be called if the underlying value is not addressable • Case-insensitive unmarshaling • Inconsistent error values
14. 14 What’s wrong with the v1 encoding/json? Four broad categories: 1. Missing functionality 2. API deficiencies 3. Performance limitations 4. Behavioral flaws
15. 15 Introduction of possible solutions of the v2 encoding/json
16. 16 Goal of the v2 encoding/json • Mostly backwards compatible • More correct • More performant • More flexibles • Easy to use (hard to misuse) • Avoid unsafe
17. 17 Overview https://github.com/golang/go/discussions/63397
18. 18 Overview • Two packages jsontext and json ◦ jsontext ▪ Syntactic functionality that processes JSON based on its grammar ▪ “encode” and “decode” ▪ A relatively light dependency tree ◦ json ▪ Semantic functionality that determines the meaning of JSON values as Go values and vice-versa ▪ “marshal” and “unmarshal” ▪ Depends on the reflect package
19. 19 The package “encoding/json/jsontext” https://github.com/golang/go/discussions/63397
20. 20 The package “encoding/json/jsontext” • Encoder and Decoder write and read JSON tokens and values ◦ You can process JSON text in a purely streaming way • NewEncoder and NewDecoder accept options
21. 21 The Token and Value type • A Token represents the smallest structural unit ◦ true, false, {, }, [, ], etc. • A Value is the raw representation of a single JSON value
22. 22 • You can configure the behaviors • The Options interface is under the jsonopts package The Options type https://github.com/go-json-experiment/json/blob/master/jsontext/options.go
23. 23 The SyntacticError type • Errors due to non-compliance with the JSON grammar are reported as SyntacticError.
24. 24 The package “encoding/json/v2” https://github.com/golang/go/discussions/63397
25. 25 The package “encoding/json/v2” • Marshal and Unmarshal ◦ Mostly match the signature of the v1 functions • MarshalWriter and UnmarshalRead ◦ Accept io.Writer and io.Reader instead of []byte • MarshalEncode and UnmarshalDecode ◦ Accept *jsontext.Encoder and *jsontext.Decoder
26. 26 https://github.com/golang/go/discussions/63397 Default behavior changes
27. 27 • Case-sensitive name match Default behavior changes
28. 28 • Do not allow invalid UTF-8 Default behavior changes
29. 29 • Do not allow duplicate names Default behavior changes
30. 30 Struct tag options • v2 also supports struct tags ◦ omitzero ◦ omitempty ◦ string ◦ nocase ◦ inline ◦ unknown ◦ format
31. 31 • omitzero ◦ The field is omitted if the value is zero when marshaling Struct tag options
32. 32 • format ◦ Specifies the format when serializing Struct tag options
33. 33 The Options type • You can configure the behaviors • The Options interface is under the jsonopts package https://github.com/go-json-experiment/json/blob/master/options.go
34. 34 The SemanticError type • Errors due to the inability to correlate JSON data with Go data are reported as SemanticError
35. 35 The package “encoding/json/v2” https://github.com/golang/go/discussions/63397
36. 36 Performance and migrations
37. 37 Performance https://github.com/go-json-experiment/jsonbench/blob/master/images/benchmark-marshal-concrete.png
38. 38 Performance https://github.com/go-json-experiment/jsonbench/blob/master/images/benchmark-unmarshal-concrete.png
39. 39 Migration
40. 40 Summary
41. 41 Impressions of this session • The proposed v2 package is well thought out, not only performance, API flexibility, security safety, and RFC compliance, but also backward compatibility and migration flexibility • The idea of splitting syntactic functionality and semantic functionality into two packages, jsontext and json, would be a good idea • Nevertheless, many changes have been made, so you will have to migrate with each and every option, which will be somewhat nerve-wracking :(
42. 42 Summary • The v2 encoding/json package is a proposal to improve problems that v1 has, such as missing functionality, API deficiencies, and performance limitations • You can see the implementation https://github.com/go-json-experiment/json • Discussion is now open https://github.com/golang/go/discussions/63397
43. 43 Advertisement I’ll talk about • “Recap: Automatically Instrument Your Go Source Code with Orchestrion” at mercari.go #24 on Nov 1 (tomorrow!) • “The Future of encoding/json” at Go Conference mini 2023 Winter IN KYOTO on Dec 2
44. 44 References • “The Future of JSON in Go” by Joe Tsai at GopherCon 2023 (link) • https://github.com/go-json-experiment/json • https://github.com/golang/go/discussions/63397
45. 45 Thanks for your attention! The Go gopher was designed by Renee French