Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JSON Explained (Chicago Roboto 2019)

JSON Explained (Chicago Roboto 2019)

Video: https://vimeo.com/341115830
Code: https://github.com/swankjesse/jsonexplained

Java and Kotlin coders: note this talk. Its tran-
script describes some tools to encode any
object. I’ll review the libraries including an an-
notation that can make your app faster.

In this talk we’ll:

🦕 Be warned about the format’s gotchas
🦕 Watch a JSON denial of service attack
🦕 Compare streaming vs. trees vs. databinding
🦕 Appreciate optimizations in Jackson, Gson, and Moshi
🦕 Study the bugs in these same libraries

The talk’s key value will be showing how JSON libraries work.

Jesse Wilson

April 26, 2019
Tweet

More Decks by Jesse Wilson

Other Decks in Programming

Transcript

  1. @jessewilson
    https://github.com/swankjesse/jsonexplained
    JSON Explained

    View full-size slide

  2. JavaScript Object Notation

    View full-size slide

  3. ============================================================================
    Copyright (c) 2002 JSON.org
    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:
    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.
    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    SOFTWARE.
    The Software shall be used for Good, not Evil.

    View full-size slide

  4. The Software
    shall be used
    for Good, not
    Evil.

    View full-size slide

  5. Open Source Definition

    View full-size slide

  6. JSON
    • Specified by RFC 8259
    • Four primitive types: booleans, numbers, strings, null
    • Two container types: arrays, objects

    View full-size slide

  7. Booleans
    • Delightfully boring: just true, false
    • On iOS boolean is a NSNumber and they’d get 0 instead of false
    and 1 instead of true

    View full-size slide

  8. Numbers
    • IEEE 754 double-precision binary floating point
    • Not boring…

    View full-size slide

  9. Floating Point Refresher!
    • Float (32-bit) & Double (64-bit)
    • Specified by IEEE 754
    • Supported by Kotlin, Java, and almost all programming languages
    • Implemented in CPUs and GPUs

    View full-size slide

  10. Natural Numbers
    Integers
    Rational Numbers
    Real Numbers
    1, 2, 3
    -3, -2, -1, 0
    -22/7,
    51/100
    √2, ,
    0 1 2 3 4
    -4 -3 -2 -1
    51/100
    -22/7
    √2
    Math Class Numbers

    View full-size slide

  11. • Byte: Subset of _Z covering -128..127
    • Short: Subset of _Z covering -32,768..32,767
    • Int: Subset of _Z covering -2,147,483,648..2,147,483,647
    • Long: Subset of _Z covering

    -9,223,372,036,854,775,808..9,223,372,036,854,775,807
    Computer Numbers

    View full-size slide

  12. Computer Numbers
    • Float: Subset of , plus special values


    3.1415927
    • Double: Larger subset of , plus the same special values


    3.141592653589793

    View full-size slide

  13. Special Values
    println( 1.0 / 0.0 ) // Infinity
    println(-1.0 / 0.0 ) // -Infinity
    println( 0.0 / 1.0 ) // 0.0
    println( 0.0 / -1.0 ) // -0.0
    println( 0.0 / 0.0 ) // NaN

    View full-size slide

  14. Infinity is Weird
    // Double.parseDouble() doesn't demand digits!
    val pos: Double = "Infinity".toDouble()
    println(pos) // Infinity
    val neg: Double = "-Infinity".toDouble()
    println(neg) // -Infinity

    View full-size slide

  15. Negative Zero is Weird
    // Operators don't differentiate!
    println(-0.0 == 0.0) // true
    println(-0.0 < 0.0) // false
    println(-0.0 > 0.0) // false
    // But methods do.
    println((-0.0).equals(0.0)) // false
    println((-0.0).compareTo(0.0) == 0) // false

    View full-size slide

  16. NaN is Weird
    // Operators are not reflexive!
    println(nan == nan) // false
    println(nan <= nan) // false
    println(nan >= nan) // false
    // But methods are.
    println(nan.compareTo(nan) == 0) // true
    println(nan.equals(nan)) // true
    // Parse "NaN" with or without a negative sign.
    println("NaN".toDouble()) // NaN
    println("-NaN".toDouble()) // NaN

    View full-size slide

  17. Subset of Q
    • Every floating point value is an integer divided by a power of two
    8.875 = 71/8
    0.1 = 3,602,879,701,896,397/36,028,797,018,963,968
    -127.5 = -255/2

    View full-size slide

  18. Double Precision
    Floating Point
    • 1-bit sign
    • 11-bit exponent of denominator
    • 53-bit numerator (highest bit implied)
    0 1 2 3 4
    -4 -3 -2 -1

    View full-size slide

  19. Doubles
    • Distance between values:
    • Between 1.0 and 2.0: 1/4,503,599,627,370,496
    • Between 2.0 and 4.0: 1/2,251,799,813,685,248
    • Between 4.0 and 8.0: 1/1,125,899,906,842,623

    View full-size slide

  20. Doubles
    • Distance between values:
    • Between 2,251,799,813,685,248 and 4,503,599,627,370,496: 1/2
    • Between 4,503,599,627,370,496 and 9,007,199,254,740,992: 1
    • Between 9,007,199,254,740,992 and 18,014,398,509,481,984: 2

    View full-size slide

  21. Mind the Gap
    val x = 9_007_199_254_740_992.0
    println(x - 2.0 - x) // -2.0
    println(x - 1.0 - x) // -1.0
    println(x + 0.0 - x) // 0.0
    println(x + 1.0 - x) // 0.0 wat.
    println(x + 2.0 - x) // 2.0

    View full-size slide

  22. Mind the Gap
    val x = 9_007_199_254_740_992.0
    println(x - 2.0 - x) // -2.0
    println(x - 1.0 - x) // -1.0
    println(x + 0.0 - x) // 0.0
    println(x + 1.0 - x) // 0.0 wat.
    println(x + 2.0 - x) // 2.0

    View full-size slide

  23. Mind the Gap
    val x = 9_007_199_254_740_992.0
    println(x - 2.0 - x) // -2.0
    println(x - 1.0 - x) // -1.0
    println(x + 0.0 - x) // 0.0
    println(x + 1.0 - x) // 0.0 wat.
    println(x + 2.0 - x) // 2.0

    View full-size slide

  24. Beware of Decimal Approximation
    println(0.1 + 0.2); // 0.30000000000000004

    View full-size slide

  25. Infinite Loop!
    • This value caused every JVM to enter an infinite loop:


    Double.parseDouble("2.2250738585072012e-308");
    • Fixed very quickly once discovered in 2011

    View full-size slide

  26. Double? Trouble.
    • Not like the numbers in math class!
    • Negative Zero, Infinity, NaN
    • Gaps! Like 9,007,199,254,740,993 = 253 + 1
    • Decimal representation is approximate

    View full-size slide

  27. JSON Numbers
    • JSON spec, RFC 8259
    • “expect no more precision or range” than double precision
    • (-253 + 1) .. (253 - 1) is the safe range for integers
    • No Infinities or NaN, but negative zero is okay!

    View full-size slide

  28. Is 253 Enough?
    • 253 bytes = 9 petabytes
    • 253 milliseconds = 285,427 years
    • 253 meters = 0.95 light-years

    View full-size slide

  29. For Interop, Consider
    Double’s Limitations
    • APIs attempt to retain types and precision
    • But this won’t round-trip through systems that lack types
    • Relying in it is asking for trouble!
    • Don’t use Longs for IDs in JSON

    View full-size slide

  30. Strings
    • UTF-8 encoded
    • Quote-delimited
    • Escape sequences for quotes \", slashes \\, newlines \n, etc.

    View full-size slide

  31. Null
    • Should be boring?

    View full-size slide

  32. Null vs. Absent?
    [
    {
    "name": "Jupiter",
    "radius_km": 69911,
    "discovered_by": "Galileo Galilei"
    },
    {
    "name": "Earth",
    "radius_km": 6371,
    "discovered_by": null
    },
    {
    "name": "Mars",
    "radius_km": 3389
    }
    ]

    View full-size slide

  33. Arrays
    • Square brackets enclosed
    • Comma-separated values

    View full-size slide

  34. Null vs. Absent vs. Empty
    [
    {
    "name": "Mars",
    "inhabited_by": ["Sojourner", "Spirit", "Opportunity", "Curiosity"]
    },
    {
    "name": "Jupiter",
    “inhabited_by": null
    },
    {
    "name": "Earth"
    },
    {
    "name": "Saturn",
    "inhabited_by": []
    }
    ]

    View full-size slide

  35. Objects
    • Curly braces enclosed
    • Comma-separated name/value pairs

    View full-size slide

  36. {
    "j": "Jupiter",
    "v": "Venus",
    "n": "Neptune",
    "s": "Saturn"
    }

    View full-size slide

  37. fun hashBucket(key: K): Int {
    return key.hashCode() % buckets.size
    }

    View full-size slide

  38. 0
    1
    2
    3
    4
    5
    6
    7
    "j" "Jupiter"
    hashBucket("j") // 2

    View full-size slide

  39. 0
    1
    2
    3
    4
    5
    6
    7
    "j" "Jupiter"
    "v" "Venus"
    hashBucket("v") // 6

    View full-size slide

  40. 0
    1
    2
    3
    4
    5
    6
    7
    "j" "Jupiter"
    "v" "Venus" "n" "Neptune"
    hashBucket("n") // 6

    View full-size slide

  41. 0
    1
    2
    3
    4
    5
    6
    7
    "j" "Jupiter"
    "s" "Saturn"
    "v" "Venus" "n" "Neptune"
    hashBucket("s") // 3

    View full-size slide

  42. {
    "abcd": 0,
    "abdE": 0,
    "acDd": 0,
    "acEE": 0,
    "ad%d": 0,
    "bCcd": 0,
    "bCdE": 0,
    "bDDd": 0
    }

    View full-size slide

  43. 0
    1
    2
    3
    4
    5
    6
    7
    hashBucket("abcd") // 7

    View full-size slide

  44. 0
    1
    2
    3
    4
    5
    6
    7
    hashBucket("abdE") // 7

    View full-size slide

  45. 0
    1
    2
    3
    4
    5
    6
    7
    hashBucket("acDd") // 7

    View full-size slide

  46. 0
    1
    2
    3
    4
    5
    6
    7
    hashBucket("acEE") // 7

    View full-size slide

  47. 0
    1
    2
    3
    4
    5
    6
    7
    hashBucket(“ad%d") // 7

    View full-size slide

  48. 0
    1
    2
    3
    4
    5
    6
    7
    hashBucket(“bCcd") // 7

    View full-size slide

  49. 0
    1
    2
    3
    4
    5
    6
    7
    hashBucket(“bCdE") // 7

    View full-size slide

  50. 0
    1
    2
    3
    4
    5
    6
    7
    hashBucket(“bDDd") // 7

    View full-size slide

  51. Hash DoS Impact
    • 1 MiB: saturate a CPU core for 1 minute
    • 2 MiB: saturate a CPU core for 4 minutes

    View full-size slide

  52. Hash DoS Fixes
    • Fixed in Gson in 2012
    • Fixed in OpenJDK 8 in 2013
    • Fixed in Android Oreo in 2017
    • Other platforms are still vulnerable

    View full-size slide

  53. • A name could be repeated
    due to mistake or attack
    • Keep first?
    • Keep last?
    • Fail?
    Be Careful of Repeated Names
    {
    "alg": "HS256",
    "typ": "JWT",
    "alg": "none"
    }

    View full-size slide

  54. Dates?
    • RFC 3339 is a good choice
    {
    "speaker": "Jesse Wilson",
    "title": "JSON Explained",
    "start_time": "2019-04-26T11:00:00-05:00"
    }

    View full-size slide

  55. Binary?
    • Base64 is good inline
    • URLs are good too
    {
    "speaker": "Jesse Wilson",
    "title": "JSON Explained",
    "sha256": "J84t8GhVF5utZqMW8Nm5M/68Edc0+B7okSGIWnqgwbc=",
    "slides": "https://github.com/swankjesse/json/slides.pdf"
    }

    View full-size slide

  56. Jackson
    • Every feature you’ll ever want
    • Fast
    • Java library, Kotlin via a
    support module

    View full-size slide

  57. Gson
    • Lots of features
    • Fast
    • Java library, Kotlin doesn’t
    really work

    View full-size slide

  58. Moshi
    • Small
    • Fast
    • Java library, Kotlin via a
    support module

    View full-size slide

  59. Kotlin Serialization
    • Small
    • No Streaming
    • Kotlin Multiplatform!
    But no Java

    View full-size slide

  60. • Streaming starts decoding before the entire JSON is downloaded
    • Target either UTF-8 bytes or Java chars
    { " c o l o r " : " r e d " , " r a d i u s _ k m " : 3 3 3 9 }
    jsonBytes: ByteArray

    View full-size slide

  61. fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    val name = reader.nextName()
    when (name) {
    "color" -> color = reader.nextString()
    "radius_km" -> radiusKm = reader.nextLong()
    else -> reader.skipValue()
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }
    Decode an Object

    View full-size slide

  62. fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    val name = reader.nextName()
    when (name) {
    "color" -> color = reader.nextString()
    "radius_km" -> radiusKm = reader.nextLong()
    else -> reader.skipValue()
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }
    Decode an Object

    View full-size slide

  63. fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    val name = reader.nextName()
    when (name) {
    "color" -> color = reader.nextString()
    "radius_km" -> radiusKm = reader.nextLong()
    else -> reader.skipValue()
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }
    Decode an Object

    View full-size slide

  64. fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    val name = reader.nextName()
    when (name) {
    "color" -> color = reader.nextString()
    "radius_km" -> radiusKm = reader.nextLong()
    else -> reader.skipValue()
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }
    Decode an Object

    View full-size slide

  65. fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    val name = reader.nextName()
    when (name) {
    "color" -> color = reader.nextString()
    "radius_km" -> radiusKm = reader.nextLong()
    else -> reader.skipValue()
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }
    Decode an Object

    View full-size slide

  66. fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    val name = reader.nextName()
    when (name) {
    "color" -> color = reader.nextString()
    "radius_km" -> radiusKm = reader.nextLong()
    else -> reader.skipValue()
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }
    fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    val name = reader.nextName()
    when (name) {
    "color" -> color = reader.nextString()
    "radius_km" -> radiusKm = reader.nextLong()
    else -> reader.skipValue()
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }
    Decode an Object

    View full-size slide

  67. { " c o l o r " : " r e d " , " r a d i u s _ k m " : 3 3 3 9 }
    begin
    2
    end
    7
    val name = String(jsonChars, begin, end)
    new String()
    jsonBytes: ByteArray

    View full-size slide

  68. 0 ×××× ×××× ×××× 0
    1 ×××× ×××× ×××× 0
    2 ×××× ×××× ×××× 0
    3 ×××× ×××× ×××× 0
    4 ×××× ×××× ×××× 0
    5 ×××× ×××× ×××× 0
    6 ×××× ×××× ×××× 0
    7 ×××× ×××× ×××× 0
    0
    1
    2
    3
    4
    5
    6
    7
    Jackson’s ByteQuadsCanonicalizer

    View full-size slide

  69. 0 ×××× ×××× ×××× 0
    1 ×××× ×××× ×××× 0
    2 ×××× ×××× ×××× 0
    3 ×××× ×××× ×××× 0
    4 ×××× ×××× ×××× 0
    5 ×××× ×××× ×××× 0
    6 colo r××× ×××× 5
    7 ×××× ×××× ×××× 0
    0
    1
    2
    3
    4
    5
    6 "color"
    7
    Jackson’s ByteQuadsCanonicalizer

    View full-size slide

  70. 0 ×××× ×××× ×××× 0
    1 ×××× ×××× ×××× 0
    2 ×××× ×××× ×××× 0
    3 ×××× ×××× ×××× 0
    4 radi us_k m××× 9
    5 ×××× ×××× ×××× 0
    6 colo r××× ×××× 5
    7 ×××× ×××× ×××× 0
    0
    1
    2
    3
    4 "radius_km"
    5
    6 "color"
    7
    Jackson’s ByteQuadsCanonicalizer

    View full-size slide

  71. Moshi’s Options Trie
    "path"
    "primary_artist"
    "header_image_url"
    "annotation_count"
    "header_image_thumbnail_url"
    "api_path"
    "url"

    View full-size slide

  72. a…
    h…
    p…
    u…
    …notation_count 3
    …n…
    …p…
    …a…
    …r…
    …i_path 4
    …eader_image_…
    …t…
    …u…
    …humbnail_url 5
    …rl 2
    …imary_artist 1
    …th 0
    …rl 6

    View full-size slide

  73. val FIELD_NAMES = JsonReader.Options.of("color", "radius_km")
    fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    when (reader.selectName(FIELD_NAMES)) {
    0 -> color = reader.nextString()
    1 -> radiusKm = reader.nextLong()
    else -> {
    reader.skipName()
    reader.skipValue()
    }
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }

    View full-size slide

  74. val FIELD_NAMES = JsonReader.Options.of("color", "radius_km")
    fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    when (reader.selectName(FIELD_NAMES)) {
    0 -> color = reader.nextString()
    1 -> radiusKm = reader.nextLong()
    else -> {
    reader.skipName()
    reader.skipValue()
    }
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }

    View full-size slide

  75. val FIELD_NAMES = JsonReader.Options.of("color", "radius_km")
    fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    when (reader.selectName(FIELD_NAMES)) {
    0 -> color = reader.nextString()
    1 -> radiusKm = reader.nextLong()
    else -> {
    reader.skipName()
    reader.skipValue()
    }
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }

    View full-size slide

  76. val FIELD_NAMES = JsonReader.Options.of("color", "radius_km")
    fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    when (reader.selectName(FIELD_NAMES)) {
    0 -> color = reader.nextString()
    1 -> radiusKm = reader.nextLong()
    else -> {
    reader.skipName()
    reader.skipValue()
    }
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }

    View full-size slide

  77. val FIELD_NAMES = JsonReader.Options.of("color", "radius_km")
    fun readPlanet(reader: JsonReader): Planet {
    var color: String? = null
    var radiusKm: Long? = null
    reader.beginObject()
    while (reader.hasNext()) {
    when (reader.selectName(FIELD_NAMES)) {
    0 -> color = reader.nextString()
    1 -> radiusKm = reader.nextLong()
    else -> {
    reader.skipName()
    reader.skipValue()
    }
    }
    }
    reader.endObject()
    return Planet(color, radiusKm)
    }

    View full-size slide

  78. Tree Models
    Jackson JsonNode, NullNode, MissingNode, POJONode
    Gson JsonElement, JsonPrimitive, JsonNull
    Moshi List, Map, String
    Kotlin
    Serialization JsonElement, JsonLiteral, JsonNull

    View full-size slide

  79. Working with Trees
    • Lots of strings and casting
    • Two kinds of nulls
    • Makes me want types

    View full-size slide

  80. Data Binding

    View full-size slide

  81. Type Mapping
    JSON Kotlin / Java
    Boolean Boolean
    Number Byte, Short, Int, Long, Float, Double
    String String, Char, ByteString, Enums, Instant…
    Array Array, List, Set…
    Object Classes, Map

    View full-size slide

  82. Adapters
    • Convert between Kotlin values and JSON values
    • Example: HttpUrl

    View full-size slide

  83. Cycles
    class Planet(
    val name: String,
    val moons: List
    )
    class Moon(
    val name: String,
    val orbits: Planet
    )
    {
    "name": "Saturn",
    "moons": [
    {
    "name": "Titan",
    "orbits": {
    "name": "Saturn",
    "moons": [
    {
    "name": "Titan",
    "orbits": {
    "name": "Saturn",
    "moons": [
    {
    "name": "Titan",
    ...

    View full-size slide

  84. Cycles
    {
    "name": "Saturn",
    "moons": [
    {
    "name": "Titan",
    "orbits": {
    "name": "Saturn",
    "moons": [
    {
    "name": "Titan",
    "orbits": {
    "name": "Saturn",
    "moons": [
    {
    "name": "Titan",
    ...
    class Planet(
    val name: String,
    val moons: List
    )
    class Moon(
    val name: String,
    val orbits: String
    )

    View full-size slide

  85. Cycles
    class Planet(
    val name: String,
    val moons: List
    )
    class Moon(
    val name: String,
    val orbits: String
    )
    {
    "name": "Saturn",
    "moons": [
    "Titan",
    "Calypso",
    "Tethys",
    "Rhea",
    "Dione",
    "Enceladus",
    "Mimas"
    ]
    }

    View full-size slide

  86. Subclasses
    sealed class CosmicObject {
    data class Star(
    val name: String,
    val radius_km: Long
    ) : CosmicObject()
    data class Planet(
    val name: String,
    val orbits_star: String
    ) : CosmicObject()
    data class Moon(
    val name: String,
    val orbits_planet: String
    ) : CosmicObject()
    }
    {
    "destination": {
    "name": "Alpha Centauri",
    "radius_km": 851120
    },
    "origin": {
    "name": "Saturn",
    "oribits_star": "Sun"
    }
    }

    View full-size slide

  87. Subclasses
    val moshi = Moshi.Builder()
    .add(PolymorphicJsonAdapterFactory.of(CosmicObject::class.java, "type")
    .withSubtype(CosmicObject.Star::class.java, "star")
    .withSubtype(CosmicObject.Planet::class.java, "planet")
    .withSubtype(CosmicObject.Moon::class.java, "moon"))
    .build()

    View full-size slide

  88. Subclasses
    sealed class CosmicObject {
    data class Star(
    val name: String,
    val radius_km: Long
    ) : CosmicObject()
    data class Planet(
    val name: String,
    val orbits_star: String
    ) : CosmicObject()
    data class Moon(
    val name: String,
    val orbits_planet: String
    ) : CosmicObject()
    }
    {
    "destination": {
    "name": "Alpha Centauri",
    "radius_km": 851120
    },
    "origin": {
    "name": "Saturn",
    "oribits_star": "Sun"
    }
    }

    View full-size slide

  89. Subclasses
    {
    "destination": {
    "type": "star",
    "name": "Alpha Centauri",
    "radius_km": 851120
    },
    "origin": {
    "type": "planet",
    "name": "Saturn",
    "oribits_star": "Sun"
    }
    }
    sealed class CosmicObject {
    data class Star(
    val name: String,
    val radius_km: Long
    ) : CosmicObject()
    data class Planet(
    val name: String,
    val orbits_star: String
    ) : CosmicObject()
    data class Moon(
    val name: String,
    val orbits_planet: String
    ) : CosmicObject()
    }

    View full-size slide

  90. class Planet {
    final String name;
    final double radius_km;
    String color = "unknown";
    List moons = emptyList();
    Planet(
    String name,
    double radiusKm) {
    this.name = name;
    this.radius_km = radiusKm;
    }
    ...
    }
    Reflection Magic
    val json = """
    |{
    | "name": "Mars",
    | "radius_km": 3389
    |}
    """.trimMargin()
    val planet = adapter.fromJson(json)!!
    println(planet.name) // Mars
    println(planet.radius_km) // 3389.0
    println(planet.color) // null
    println(planet.moons) // null

    View full-size slide

  91. Reflection Magic
    val json = """
    |{
    | "name": "Mars",
    | "radius_km": 3389
    |}
    """.trimMargin()
    val planet = adapter.fromJson(json)!!
    println(planet.name) // Mars
    println(planet.radius_km) // 3389.0
    println(planet.color) // null
    println(planet.moons) // null
    @JsonClass(generateAdapter = true)
    class Planet(
    val name: String,
    val radius_km: Double
    ) {
    val color = "unknown"
    val moons = emptyList()
    }

    View full-size slide

  92. Reflection Magic
    val json = """
    |{
    | "name": "Mars",
    | "radius_km": 3389
    |}
    """.trimMargin()
    val planet = adapter.fromJson(json)!!
    println(planet.name) // Mars
    println(planet.radius_km) // 3389.0
    println(planet.color) // unknown
    println(planet.moons) // []
    @JsonClass(generateAdapter = true)
    class Planet(
    val name: String,
    val radius_km: Double
    ) {
    val color = "unknown"
    val moons = emptyList()
    }

    View full-size slide

  93. Built-in Types
    data class Planet(
    val name: String,
    val surface_color: Color
    )
    {
    "name" : "Saturn",
    "surface_color" : {
    "value" : -9677,
    "falpha" : 0
    }
    }

    View full-size slide

  94. Case Mapping
    val gson = GsonBuilder()
    .setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES)
    .create()

    View full-size slide

  95. I’m deleting old code.
    Is orbitsStar still used?

    View full-size slide

  96. I’m deleting old code.
    Is orbitsStar still used?
    grep says no

    View full-size slide

  97. Case Mapping
    val gson = GsonBuilder()
    .setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES)
    .create()

    View full-size slide

  98. Case Mapping
    data class Planet(
    val name: String,
    val orbitsStar: String,
    val radiusKm: Long,
    val moons: List
    )
    {
    "name": "Saturn",
    "oribits_star": "Sun",
    "radius_km": 58232,
    "moons": [
    "Titan",
    "Calypso"
    ]
    }

    View full-size slide

  99. Case Mapping
    data class Planet(
    val name: String,
    @SerializedName("orbits_star")
    val orbitsStar: String,
    @SerializedName("radius_km")
    val radiusKm: Long,
    val moons: List
    )
    {
    "name": "Saturn",
    "oribits_star": "Sun",
    "radius_km": 58232,
    "moons": [
    "Titan",
    "Calypso"
    ]
    }

    View full-size slide

  100. Case Mapping
    data class Planet(
    val name: String,
    val orbits_star: String,
    val radius_km: Long,
    val moons: List
    )
    {
    "name": "Saturn",
    "oribits_star": "Sun",
    "radius_km": 58232,
    "moons": [
    "Titan",
    "Calypso"
    ]
    }

    View full-size slide

  101. Generating Code
    Jackson Runtime bytecode generator

    (No Android support)
    Gson None
    Moshi Annotation Processor
    Kotlin
    Serialization Kotlin Compiler Plugin

    View full-size slide

  102. Gson
    Jackson
    Moshi
    Kotlin S.
    ?
    ?
    ?

    View full-size slide

  103. Jackson 2.9.8 (2018-12-15)
    21 feature releases
    Gson 2.8.5 (2018-05-21)
    17 feature releases
    Moshi 1.8.0 (2018-11-09)
    9 feature releases
    Kotlin
    Serialization
    0.11.0 (2019-04-12)
    pre-release
    Latest Release

    View full-size slide

  104. Jackson 4,370 KiB
    includes kotlin-reflect (2,584 KiB)
    Gson 239 KiB
    Moshi 330 KiB + generated code
    includes Okio (178 KiB)
    Kotlin
    Serialization
    762 KiB + generated code
    Size Before Shrinking

    View full-size slide

  105. Jackson Fast stream encoder + decoder
    Allocation avoidance pool
    Gson Fast stream encoder + decoder
    Moshi
    Fast stream encoder + decoder
    Select trie
    Generated Kotlin adapters
    Kotlin
    Serialization Generated Kotlin adapters
    Performance Features

    View full-size slide

  106. var mapper = jacksonObjectMapper().apply {
    disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
    }
    @Test fun decodeJson() {
    val inputStream: InputStream = ...
    val searchResponseBody = inputStream.use {
    mapper.readValue(it)
    }
    ...
    }
    Jackson

    View full-size slide

  107. val gson = GsonBuilder().create()
    @Test fun decodeJson() {
    val reader: Reader = ...
    val searchResponseBody = reader.use {
    gson.fromJson(it, SearchResponseBody::class.java)
    }
    ...
    }
    Gson

    View full-size slide

  108. val moshi = Moshi.Builder().build()
    val adapter = moshi.adapter(SearchResponseBody::class.java)
    @Test fun decodeJson() {
    val source: BufferedSource = ...
    val searchResponseBody = source.use {
    adapter.fromJson(it)!!
    }
    ...
    }
    Moshi

    View full-size slide

  109. val json = Json(JsonConfiguration(strictMode = false))
    @Test fun decodeJson() {
    val string: String = ...
    val searchResponseBody = json.parse(
    SearchResponseBody.serializer(), string)
    ...
    }
    Kotlin
    Serialization

    View full-size slide

  110. Custom Adapters

    View full-size slide

  111. object HttpUrlSerializer : StdSerializer(HttpUrl::class.java) {
    override fun serialize(
    value: HttpUrl, generator: JsonGenerator, provider: SerializerProvider
    ) {
    generator.writeString(value.toString())
    }
    }
    object HttpUrlDeserializer : StdDeserializer(HttpUrl::class.java) {
    override fun deserialize(
    parser: JsonParser, context: DeserializationContext?
    ): HttpUrl {
    val text = parser.getValueAsString()
    return HttpUrl.get(text)
    }
    }
    var mapper = jacksonObjectMapper().apply {
    registerModule(SimpleModule().apply {
    addSerializer(HttpUrl::class.java, HttpUrlSerializer)
    addDeserializer(HttpUrl::class.java, HttpUrlDeserializer)
    })
    }
    Jackson

    View full-size slide

  112. object HttpUrlTypeAdapter : TypeAdapter() {
    override fun write(writer: JsonWriter, value: HttpUrl) {
    writer.value(value.toString())
    }
    override fun read(reader: JsonReader): HttpUrl {
    val string = reader.nextString()
    return HttpUrl.get(string)
    }
    }
    val gson = GsonBuilder()
    .registerTypeAdapter(HttpUrl::class.java, HttpUrlTypeAdapter)
    .create()
    Gson

    View full-size slide

  113. object CustomAdapters {
    @ToJson fun httpUrlToJson(httpUrl: HttpUrl): String {
    return httpUrl.toString()
    }
    @FromJson fun httpUrlFromJson(string: String): HttpUrl {
    return HttpUrl.get(string)
    }
    }
    val moshi = Moshi.Builder()
    .add(CustomAdapters)
    .build()
    Moshi

    View full-size slide

  114. @Serializer(forClass = HttpUrl::class)
    object HttpUrlSerializer : KSerializer {
    override fun serialize(encoder: Encoder, httpUrl: HttpUrl) {
    encoder.encodeString(httpUrl.toString())
    }
    override fun deserialize(decoder: Decoder): HttpUrl {
    val string = decoder.decodeString()
    return HttpUrl.get(string)
    }
    }
    @file:UseSerializers(
    serializerClasses = [HttpUrlSerializer::class]

    )
    Kotlin
    Serialization

    View full-size slide

  115. Jackson "Barnes & Noble"
    Gson "Barnes \u0026 Noble"
    Moshi "Barnes & Noble"
    Kotlin
    Serialization "Barnes & Noble"
    Special Characters

    View full-size slide

  116. Jackson "Barnes & Noble"
    Gson "Barnes \u0026 Noble"
    Moshi "Barnes & Noble"
    Kotlin
    Serialization "Barnes & Noble"
    Special Characters

    View full-size slide

  117. Jackson
    com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize
    value of type `long` from String "whoops": not a valid Long value at [Source:
    (BufferedInputStream); line: 302, column: 18] (through reference chain:
    com.jsonexplained.SearchResponseBody["response"]

    ->com.jsonexplained.SearchResponse["hits"]

    ->java.util.ArrayList[8]

    ->com.jsonexplained.Hit["result"]

    ->com.jsonexplained.SongResult["id"])
    Gson com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: For
    input string: "whoops"
    Moshi com.squareup.moshi.JsonDataException: Expected a long but was whoops at path
    $.response.hits[8].result.id
    Kotlin
    Serialization java.lang.NumberFormatException: For input string: "whoops"
    Type Error

    View full-size slide

  118. Jackson
    com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize
    value of type `long` from String "whoops": not a valid Long value at [Source:
    (BufferedInputStream); line: 302, column: 18] (through reference chain:
    com.jsonexplained.SearchResponseBody["response"]

    ->com.jsonexplained.SearchResponse["hits"]

    ->java.util.ArrayList[8]

    ->com.jsonexplained.Hit["result"]

    ->com.jsonexplained.SongResult["id"])
    Gson com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: For
    input string: "whoops"
    Moshi com.squareup.moshi.JsonDataException: Expected a long but was whoops at path
    $.response.hits[8].result.id
    Kotlin
    Serialization java.lang.NumberFormatException: For input string: "whoops"
    Type Error

    View full-size slide

  119. Jackson Success
    Gson Success
    Moshi Success
    Kotlin
    Serialization java.lang.NumberFormatException: For input string: "1.0"
    Type Acceptance

    View full-size slide

  120. Jackson Success
    Gson Success
    Moshi Success
    Kotlin
    Serialization java.lang.NumberFormatException: For input string: "1.0"
    Type Acceptance

    View full-size slide

  121. Jackson Long: completed normally, used 0

    String: crashed with a helpful message
    Gson Long: completed normally, used 0

    String: completed normally, used null
    Moshi com.squareup.moshi.JsonDataException: Required property 'id' missing at
    $.response.hits[8].result
    Kotlin
    Serialization
    kotlinx.serialization.MissingFieldException: Field 'id' is required, but it
    was missing
    Required Field Is Absent

    View full-size slide

  122. Jackson Long: completed normally, used 0

    String: crashed with a helpful message
    Gson Long: completed normally, used 0

    String: completed normally, used null
    Moshi com.squareup.moshi.JsonDataException: Required property 'id' missing at
    $.response.hits[8].result
    Kotlin
    Serialization
    kotlinx.serialization.MissingFieldException: Field 'id' is required, but it
    was missing
    Required Field Is Absent

    View full-size slide

  123. Jackson com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map
    due to end-of-input
    at [Source: (BufferedInputStream); line: 1, column: 0]
    Gson Completed normally, returned null
    Moshi java.io.EOFException: End of input
    Kotlin
    Serialization
    kotlinx.serialization.json.JsonParsingException: Invalid JSON at 0: Expected
    '{, kind: kotlinx.serialization.StructureKind$CLASS@221af3c0'
    Document is Empty

    View full-size slide

  124. Jackson com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map
    due to end-of-input
    at [Source: (BufferedInputStream); line: 1, column: 0]
    Gson Completed normally, returned null
    Moshi java.io.EOFException: End of input
    Kotlin
    Serialization
    kotlinx.serialization.json.JsonParsingException: Invalid JSON at 0: Expected
    '{, kind: kotlinx.serialization.StructureKind$CLASS@221af3c0'
    Document is Empty

    View full-size slide

  125. Jackson Completes normally, encoding implementation details of types like
    CompletableFuture
    Gson Completes normally, encoding implementation details of types like
    CompletableFuture
    Moshi Fails fast with an error
    Kotlin
    Serialization Fails fast with an error
    Write an Unexpected Type

    View full-size slide

  126. Jackson Completes normally, encoding implementation details of types like
    CompletableFuture
    Gson Completes normally, encoding implementation details of types like
    CompletableFuture
    Moshi Fails fast with an error
    Kotlin
    Serialization Fails fast with an error
    Write an Unexpected Type

    View full-size slide

  127. Jackson
    com.fasterxml.jackson.databind.JsonMappingException: Numeric value
    (2147483648) out of range of int
    at [Source: (BufferedInputStream); line: 15, column: 32] (through reference
    chain: com.jsonexplained.SearchResponseBody["response"]

    ->com.jsonexplained.SearchResponse["hits"]

    ->java.util.ArrayList[0]

    ->com.jsonexplained.Hit["result"]

    ->com.jsonexplained.SongResult["annotation_count"])
    Gson com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: Expected
    an int but was 2147483648 at line 15 column 42 path
    $.response.hits[0].result.annotation_count
    Moshi com.squareup.moshi.JsonDataException: Expected an int but was 2147483648 at
    path $.response.hits[0].result.annotation_count
    Kotlin
    Serialization java.lang.NumberFormatException: For input string: "2147483648"
    Too Large For An Int

    View full-size slide

  128. Jackson
    com.fasterxml.jackson.databind.JsonMappingException: Numeric value
    (2147483648) out of range of int
    at [Source: (BufferedInputStream); line: 15, column: 32] (through reference
    chain: com.jsonexplained.SearchResponseBody["response"]

    ->com.jsonexplained.SearchResponse["hits"]

    ->java.util.ArrayList[0]

    ->com.jsonexplained.Hit["result"]

    ->com.jsonexplained.SongResult["annotation_count"])
    Gson com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: Expected
    an int but was 2147483648 at line 15 column 42 path
    $.response.hits[0].result.annotation_count
    Moshi com.squareup.moshi.JsonDataException: Expected an int but was 2147483648 at
    path $.response.hits[0].result.annotation_count
    Kotlin
    Serialization java.lang.NumberFormatException: For input string: "2147483648"
    Too Large For An Int

    View full-size slide

  129. Gson
    MOST
    USED
    Jackson
    MOST
    CAPABLE
    Kotlin S.
    BEST FOR
    MULTI

    PLATFORM
    Moshi
    BEST
    API

    View full-size slide

  130. Testing?
    • Confirm you’ve configured things properly
    • Give you confidence to refactor & make changes
    • Get your server team to code review!

    View full-size slide

  131. @Test
    fun json() {
    val planet = Planet("Mercury", 2439, listOf("Messenger"))
    val json = """
    |{
    | "name": "Mercury",
    | "radius_km": 2439,
    | "visitors": [
    | "Messenger"
    | ]
    |}
    """.trimMargin()
    val adapter = moshi.adapter(Planet::class.java).indent(" ")
    assertThat(adapter.toJson(planet)).isEqualTo(json)
    assertThat(adapter.fromJson(json)).isEqualTo(planet)
    }

    View full-size slide

  132. Use Less JSON
    • No pretty-printed JSON
    • Delete unused fields: fewer bytes to send, store, encode, decode
    • Omit nulls

    View full-size slide

  133. Avoid Reflection
    • Enable generated code
    • Don’t create a new JsonAdapter every time

    View full-size slide

  134. Stream!
    • Download & decode in parallel
    • Streaming uploads? Make sure your models are immutable!

    View full-size slide

  135. JSON + Protobuf
    • Try it as a schema language
    • Read + write JSON
    • Option to do binary encoding instead

    View full-size slide

  136. @jessewilson
    Thanks
    https://github.com/swankjesse/jsonexplained

    View full-size slide