Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Recap: The Future of JSON in Go

Recap: The Future of JSON in Go

Shunta Komatsu

October 31, 2023

More Decks by Shunta Komatsu

Other Decks in Technology


  1. 1 Recap: The Future of JSON in Go Go 1.21

    Release Party & GopherCon 2023 Recap Shunta Komatsu
  2. 2 About me • Shunta Komatsu • Backend Engineer at

    Merpay ◦ Developing the payment platform • Joined Merpay in April 2022
  3. 3 Agenda 1. Introduction of the session 2. What are

    the problems with the v1 encoding/json? 3. Introduction of possible solutions of the v2 encoding/json 4. Performance and migrations
  4. 7 The v1 encoding/json package The encoding/json package is the

    5th most popular Go package. Marshal (encode) Unmarshal (decode)
  5. 9 What’s wrong with the v1 encoding/json? Four broad categories:

    1. Missing functionality 2. API deficiencies 3. Performance limitations 4. Behavioral flaws
  6. 11 API deficiencies • You can’t unmarshal from io.Reader correctly

    ◦ You need to create an intermediate decoder json.NewDecoder(r).Decode(v), but it doesn’t reject trailing junk (#36225) • Marshal, Unmarshal, and custom Marshaler and Unmarshaler do not accept options • Encoder or Decoder can accept options, but some options depend on bytes.Buffer instead of []byte or io.Writer
  7. 12 Performance limitations • MarshalJSON forces the implementation to allocate

    returned []byte • MarshalJSON and UnmarshalJSON forces a second parse • Lack of streaming ◦ Even though the Encoder.Encode and Decoder.Decode operate on an io.Writer or io.Reader, they buffer the entire JSON value in memory
  8. 13 Behavioral flaws • Improper handling of JSON syntax (RFC

    8259) ◦ Invalid UTF-8 is allowed ◦ Duplicate object member names are allowed • The MarshalJSON and UnmarshalJSON methods cannot be called if the underlying value is not addressable • Case-insensitive unmarshaling • Inconsistent error values
  9. 14 What’s wrong with the v1 encoding/json? Four broad categories:

    1. Missing functionality 2. API deficiencies 3. Performance limitations 4. Behavioral flaws
  10. 16 Goal of the v2 encoding/json • Mostly backwards compatible

    • More correct • More performant • More flexibles • Easy to use (hard to misuse) • Avoid unsafe
  11. 18 Overview • Two packages jsontext and json ◦ jsontext

    ▪ Syntactic functionality that processes JSON based on its grammar ▪ “encode” and “decode” ▪ A relatively light dependency tree ◦ json ▪ Semantic functionality that determines the meaning of JSON values as Go values and vice-versa ▪ “marshal” and “unmarshal” ▪ Depends on the reflect package
  12. 20 The package “encoding/json/jsontext” • Encoder and Decoder write and

    read JSON tokens and values ◦ You can process JSON text in a purely streaming way • NewEncoder and NewDecoder accept options
  13. 21 The Token and Value type • A Token represents

    the smallest structural unit ◦ true, false, {, }, [, ], etc. • A Value is the raw representation of a single JSON value
  14. 22 • You can configure the behaviors • The Options

    interface is under the jsonopts package The Options type https://github.com/go-json-experiment/json/blob/master/jsontext/options.go
  15. 23 The SyntacticError type • Errors due to non-compliance with

    the JSON grammar are reported as SyntacticError.
  16. 25 The package “encoding/json/v2” • Marshal and Unmarshal ◦ Mostly

    match the signature of the v1 functions • MarshalWriter and UnmarshalRead ◦ Accept io.Writer and io.Reader instead of []byte • MarshalEncode and UnmarshalDecode ◦ Accept *jsontext.Encoder and *jsontext.Decoder
  17. 30 Struct tag options • v2 also supports struct tags

    ◦ omitzero ◦ omitempty ◦ string ◦ nocase ◦ inline ◦ unknown ◦ format
  18. 31 • omitzero ◦ The field is omitted if the

    value is zero when marshaling Struct tag options
  19. 33 The Options type • You can configure the behaviors

    • The Options interface is under the jsonopts package https://github.com/go-json-experiment/json/blob/master/options.go
  20. 34 The SemanticError type • Errors due to the inability

    to correlate JSON data with Go data are reported as SemanticError
  21. 41 Impressions of this session • The proposed v2 package

    is well thought out, not only performance, API flexibility, security safety, and RFC compliance, but also backward compatibility and migration flexibility • The idea of splitting syntactic functionality and semantic functionality into two packages, jsontext and json, would be a good idea • Nevertheless, many changes have been made, so you will have to migrate with each and every option, which will be somewhat nerve-wracking :(
  22. 42 Summary • The v2 encoding/json package is a proposal

    to improve problems that v1 has, such as missing functionality, API deficiencies, and performance limitations • You can see the implementation https://github.com/go-json-experiment/json • Discussion is now open https://github.com/golang/go/discussions/63397
  23. 43 Advertisement I’ll talk about • “Recap: Automatically Instrument Your

    Go Source Code with Orchestrion” at mercari.go #24 on Nov 1 (tomorrow!) • “The Future of encoding/json” at Go Conference mini 2023 Winter IN KYOTO on Dec 2
  24. 44 References • “The Future of JSON in Go” by

    Joe Tsai at GopherCon 2023 (link) • https://github.com/go-json-experiment/json • https://github.com/golang/go/discussions/63397