============================================================================
Copyright (c) 2002 JSON.org
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
The Software shall be used for Good, not Evil.
Slide 4
Slide 4 text
The Software
shall be used
for Good, not
Evil.
Slide 5
Slide 5 text
Open Source Definition
Slide 6
Slide 6 text
JSON
• Specified by RFC 8259
• Four primitive types: booleans, numbers, strings, null
• Two container types: arrays, objects
Slide 7
Slide 7 text
Booleans
• Delightfully boring: just true, false
• On iOS boolean is a NSNumber and they’d get 0 instead of false
and 1 instead of true
Slide 8
Slide 8 text
Numbers
• IEEE 754 double-precision binary floating point
• Not boring…
Slide 9
Slide 9 text
Floating Point Refresher!
• Float (32-bit) & Double (64-bit)
• Specified by IEEE 754
• Supported by Kotlin, Java, and almost all programming languages
• Implemented in CPUs and GPUs
NaN is Weird
// Operators are not reflexive!
println(nan == nan) // false
println(nan <= nan) // false
println(nan >= nan) // false
// But methods are.
println(nan.compareTo(nan) == 0) // true
println(nan.equals(nan)) // true
// Parse "NaN" with or without a negative sign.
println("NaN".toDouble()) // NaN
println("-NaN".toDouble()) // NaN
Slide 17
Slide 17 text
Subset of Q
• Every floating point value is an integer divided by a power of two
8.875 = 71/8
0.1 = 3,602,879,701,896,397/36,028,797,018,963,968
-127.5 = -255/2
Slide 18
Slide 18 text
Double Precision
Floating Point
• 1-bit sign
• 11-bit exponent of denominator
• 53-bit numerator (highest bit implied)
0 1 2 3 4
-4 -3 -2 -1
Slide 19
Slide 19 text
Doubles
• Distance between values:
• Between 1.0 and 2.0: 1/4,503,599,627,370,496
• Between 2.0 and 4.0: 1/2,251,799,813,685,248
• Between 4.0 and 8.0: 1/1,125,899,906,842,623
Slide 20
Slide 20 text
Doubles
• Distance between values:
• Between 2,251,799,813,685,248 and 4,503,599,627,370,496: 1/2
• Between 4,503,599,627,370,496 and 9,007,199,254,740,992: 1
• Between 9,007,199,254,740,992 and 18,014,398,509,481,984: 2
Slide 21
Slide 21 text
Mind the Gap
val x = 9_007_199_254_740_992.0
println(x - 2.0 - x) // -2.0
println(x - 1.0 - x) // -1.0
println(x + 0.0 - x) // 0.0
println(x + 1.0 - x) // 0.0 wat.
println(x + 2.0 - x) // 2.0
Slide 22
Slide 22 text
Mind the Gap
val x = 9_007_199_254_740_992.0
println(x - 2.0 - x) // -2.0
println(x - 1.0 - x) // -1.0
println(x + 0.0 - x) // 0.0
println(x + 1.0 - x) // 0.0 wat.
println(x + 2.0 - x) // 2.0
Slide 23
Slide 23 text
Mind the Gap
val x = 9_007_199_254_740_992.0
println(x - 2.0 - x) // -2.0
println(x - 1.0 - x) // -1.0
println(x + 0.0 - x) // 0.0
println(x + 1.0 - x) // 0.0 wat.
println(x + 2.0 - x) // 2.0
Slide 24
Slide 24 text
Beware of Decimal Approximation
println(0.1 + 0.2); // 0.30000000000000004
Slide 25
Slide 25 text
Infinite Loop!
• This value caused every JVM to enter an infinite loop:
Double.parseDouble("2.2250738585072012e-308");
• Fixed very quickly once discovered in 2011
Slide 26
Slide 26 text
Double? Trouble.
• Not like the numbers in math class!
• Negative Zero, Infinity, NaN
• Gaps! Like 9,007,199,254,740,993 = 253 + 1
• Decimal representation is approximate
Slide 27
Slide 27 text
JSON Numbers
• JSON spec, RFC 8259
• “expect no more precision or range” than double precision
• (-253 + 1) .. (253 - 1) is the safe range for integers
• No Infinities or NaN, but negative zero is okay!
For Interop, Consider
Double’s Limitations
• APIs attempt to retain types and precision
• But this won’t round-trip through systems that lack types
• Relying in it is asking for trouble!
• Don’t use Longs for IDs in JSON
Slide 30
Slide 30 text
Strings
• UTF-8 encoded
• Quote-delimited
• Escape sequences for quotes \", slashes \\, newlines \n, etc.
Hash DoS Impact
• 1 MiB: saturate a CPU core for 1 minute
• 2 MiB: saturate a CPU core for 4 minutes
Slide 54
Slide 54 text
Hash DoS Fixes
• Fixed in Gson in 2012
• Fixed in OpenJDK 8 in 2013
• Fixed in Android Oreo in 2017
• Other platforms are still vulnerable
Slide 55
Slide 55 text
• A name could be repeated
due to mistake or attack
• Keep first?
• Keep last?
• Fail?
Be Careful of Repeated Names
{
"alg": "HS256",
"typ": "JWT",
"alg": "none"
}
Slide 56
Slide 56 text
Dates?
• RFC 3339 is a good choice
{
"speaker": "Jesse Wilson",
"title": "JSON Explained",
"start_time": "2019-04-26T11:00:00-05:00"
}
Slide 57
Slide 57 text
Binary?
• Base64 is good inline
• URLs are good too
{
"speaker": "Jesse Wilson",
"title": "JSON Explained",
"sha256": "J84t8GhVF5utZqMW8Nm5M/68Edc0+B7okSGIWnqgwbc=",
"slides": "https://github.com/swankjesse/json/slides.pdf"
}
Slide 58
Slide 58 text
Libraries
Slide 59
Slide 59 text
Jackson
• Every feature you’ll ever want
• Fast
• Java library, Kotlin via a
support module
Slide 60
Slide 60 text
Gson
• Lots of features
• Fast
• Java library, Kotlin doesn’t
really work
Slide 61
Slide 61 text
Moshi
• Small
• Fast
• Java library, Kotlin via a
support module
Slide 62
Slide 62 text
Kotlin Serialization
• Small
• No Streaming
• Kotlin Multiplatform!
But no Java
Slide 63
Slide 63 text
Streams
Slide 64
Slide 64 text
• Streaming starts decoding before the entire JSON is downloaded
• Target either UTF-8 bytes or Java chars
{ " c o l o r " : " r e d " , " r a d i u s _ k m " : 3 3 3 9 }
jsonBytes: ByteArray
Slide 65
Slide 65 text
fun readPlanet(reader: JsonReader): Planet {
var color: String? = null
var radiusKm: Long? = null
reader.beginObject()
while (reader.hasNext()) {
val name = reader.nextName()
when (name) {
"color" -> color = reader.nextString()
"radius_km" -> radiusKm = reader.nextLong()
else -> reader.skipValue()
}
}
reader.endObject()
return Planet(color, radiusKm)
}
Decode an Object
Slide 66
Slide 66 text
fun readPlanet(reader: JsonReader): Planet {
var color: String? = null
var radiusKm: Long? = null
reader.beginObject()
while (reader.hasNext()) {
val name = reader.nextName()
when (name) {
"color" -> color = reader.nextString()
"radius_km" -> radiusKm = reader.nextLong()
else -> reader.skipValue()
}
}
reader.endObject()
return Planet(color, radiusKm)
}
Decode an Object
Slide 67
Slide 67 text
fun readPlanet(reader: JsonReader): Planet {
var color: String? = null
var radiusKm: Long? = null
reader.beginObject()
while (reader.hasNext()) {
val name = reader.nextName()
when (name) {
"color" -> color = reader.nextString()
"radius_km" -> radiusKm = reader.nextLong()
else -> reader.skipValue()
}
}
reader.endObject()
return Planet(color, radiusKm)
}
Decode an Object
Slide 68
Slide 68 text
fun readPlanet(reader: JsonReader): Planet {
var color: String? = null
var radiusKm: Long? = null
reader.beginObject()
while (reader.hasNext()) {
val name = reader.nextName()
when (name) {
"color" -> color = reader.nextString()
"radius_km" -> radiusKm = reader.nextLong()
else -> reader.skipValue()
}
}
reader.endObject()
return Planet(color, radiusKm)
}
Decode an Object
Slide 69
Slide 69 text
fun readPlanet(reader: JsonReader): Planet {
var color: String? = null
var radiusKm: Long? = null
reader.beginObject()
while (reader.hasNext()) {
val name = reader.nextName()
when (name) {
"color" -> color = reader.nextString()
"radius_km" -> radiusKm = reader.nextLong()
else -> reader.skipValue()
}
}
reader.endObject()
return Planet(color, radiusKm)
}
Decode an Object
Slide 70
Slide 70 text
fun readPlanet(reader: JsonReader): Planet {
var color: String? = null
var radiusKm: Long? = null
reader.beginObject()
while (reader.hasNext()) {
val name = reader.nextName()
when (name) {
"color" -> color = reader.nextString()
"radius_km" -> radiusKm = reader.nextLong()
else -> reader.skipValue()
}
}
reader.endObject()
return Planet(color, radiusKm)
}
fun readPlanet(reader: JsonReader): Planet {
var color: String? = null
var radiusKm: Long? = null
reader.beginObject()
while (reader.hasNext()) {
val name = reader.nextName()
when (name) {
"color" -> color = reader.nextString()
"radius_km" -> radiusKm = reader.nextLong()
else -> reader.skipValue()
}
}
reader.endObject()
return Planet(color, radiusKm)
}
Decode an Object
Slide 71
Slide 71 text
{ " c o l o r " : " r e d " , " r a d i u s _ k m " : 3 3 3 9 }
begin
2
end
7
val name = String(jsonChars, begin, end)
new String()
jsonBytes: ByteArray
Adapters
• Convert between Kotlin values and JSON values
• Example: HttpUrl
Slide 88
Slide 88 text
Cycles
class Planet(
val name: String,
val moons: List
)
class Moon(
val name: String,
val orbits: Planet
)
{
"name": "Saturn",
"moons": [
{
"name": "Titan",
"orbits": {
"name": "Saturn",
"moons": [
{
"name": "Titan",
"orbits": {
"name": "Saturn",
"moons": [
{
"name": "Titan",
...
Slide 89
Slide 89 text
Cycles
{
"name": "Saturn",
"moons": [
{
"name": "Titan",
"orbits": {
"name": "Saturn",
"moons": [
{
"name": "Titan",
"orbits": {
"name": "Saturn",
"moons": [
{
"name": "Titan",
...
class Planet(
val name: String,
val moons: List
)
class Moon(
val name: String,
val orbits: String
)
Slide 90
Slide 90 text
Cycles
class Planet(
val name: String,
val moons: List
)
class Moon(
val name: String,
val orbits: String
)
{
"name": "Saturn",
"moons": [
"Titan",
"Calypso",
"Tethys",
"Rhea",
"Dione",
"Enceladus",
"Mimas"
]
}
Slide 91
Slide 91 text
Subclasses
sealed class CosmicObject {
data class Star(
val name: String,
val radius_km: Long
) : CosmicObject()
data class Planet(
val name: String,
val orbits_star: String
) : CosmicObject()
data class Moon(
val name: String,
val orbits_planet: String
) : CosmicObject()
}
{
"destination": {
"name": "Alpha Centauri",
"radius_km": 851120
},
"origin": {
"name": "Saturn",
"oribits_star": "Sun"
}
}
Subclasses
sealed class CosmicObject {
data class Star(
val name: String,
val radius_km: Long
) : CosmicObject()
data class Planet(
val name: String,
val orbits_star: String
) : CosmicObject()
data class Moon(
val name: String,
val orbits_planet: String
) : CosmicObject()
}
{
"destination": {
"name": "Alpha Centauri",
"radius_km": 851120
},
"origin": {
"name": "Saturn",
"oribits_star": "Sun"
}
}
Slide 94
Slide 94 text
Subclasses
{
"destination": {
"type": "star",
"name": "Alpha Centauri",
"radius_km": 851120
},
"origin": {
"type": "planet",
"name": "Saturn",
"oribits_star": "Sun"
}
}
sealed class CosmicObject {
data class Star(
val name: String,
val radius_km: Long
) : CosmicObject()
data class Planet(
val name: String,
val orbits_star: String
) : CosmicObject()
data class Moon(
val name: String,
val orbits_planet: String
) : CosmicObject()
}
Slide 95
Slide 95 text
class Planet {
final String name;
final double radius_km;
String color = "unknown";
List moons = emptyList();
Planet(
String name,
double radiusKm) {
this.name = name;
this.radius_km = radiusKm;
}
...
}
Reflection Magic
val json = """
|{
| "name": "Mars",
| "radius_km": 3389
|}
""".trimMargin()
val planet = adapter.fromJson(json)!!
println(planet.name) // Mars
println(planet.radius_km) // 3389.0
println(planet.color) // null
println(planet.moons) // null
Slide 96
Slide 96 text
Reflection Magic
val json = """
|{
| "name": "Mars",
| "radius_km": 3389
|}
""".trimMargin()
val planet = adapter.fromJson(json)!!
println(planet.name) // Mars
println(planet.radius_km) // 3389.0
println(planet.color) // null
println(planet.moons) // null
@JsonClass(generateAdapter = true)
class Planet(
val name: String,
val radius_km: Double
) {
val color = "unknown"
val moons = emptyList()
}
Slide 97
Slide 97 text
Reflection Magic
val json = """
|{
| "name": "Mars",
| "radius_km": 3389
|}
""".trimMargin()
val planet = adapter.fromJson(json)!!
println(planet.name) // Mars
println(planet.radius_km) // 3389.0
println(planet.color) // unknown
println(planet.moons) // []
@JsonClass(generateAdapter = true)
class Planet(
val name: String,
val radius_km: Double
) {
val color = "unknown"
val moons = emptyList()
}
Slide 98
Slide 98 text
Built-in Types
data class Planet(
val name: String,
val surface_color: Color
)
{
"name" : "Saturn",
"surface_color" : {
"value" : -9677,
"falpha" : 0
}
}
Slide 99
Slide 99 text
Case Mapping
val gson = GsonBuilder()
.setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES)
.create()
Slide 100
Slide 100 text
I’m deleting old code.
Is orbitsStar still used?
Slide 101
Slide 101 text
I’m deleting old code.
Is orbitsStar still used?
grep says no
Slide 102
Slide 102 text
Case Mapping
val gson = GsonBuilder()
.setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES)
.create()
Slide 103
Slide 103 text
Case Mapping
data class Planet(
val name: String,
val orbitsStar: String,
val radiusKm: Long,
val moons: List
)
{
"name": "Saturn",
"oribits_star": "Sun",
"radius_km": 58232,
"moons": [
"Titan",
"Calypso"
]
}
Slide 104
Slide 104 text
Case Mapping
data class Planet(
val name: String,
@SerializedName("orbits_star")
val orbitsStar: String,
@SerializedName("radius_km")
val radiusKm: Long,
val moons: List
)
{
"name": "Saturn",
"oribits_star": "Sun",
"radius_km": 58232,
"moons": [
"Titan",
"Calypso"
]
}
Slide 105
Slide 105 text
Case Mapping
data class Planet(
val name: String,
val orbits_star: String,
val radius_km: Long,
val moons: List
)
{
"name": "Saturn",
"oribits_star": "Sun",
"radius_km": 58232,
"moons": [
"Titan",
"Calypso"
]
}
Slide 106
Slide 106 text
Generating Code
Jackson Runtime bytecode generator
(No Android support)
Gson None
Moshi Annotation Processor
Kotlin
Serialization Kotlin Compiler Plugin
Jackson "Barnes & Noble"
Gson "Barnes \u0026 Noble"
Moshi "Barnes & Noble"
Kotlin
Serialization "Barnes & Noble"
Special Characters
Slide 123
Slide 123 text
Jackson "Barnes & Noble"
Gson "Barnes \u0026 Noble"
Moshi "Barnes & Noble"
Kotlin
Serialization "Barnes & Noble"
Special Characters
Slide 124
Slide 124 text
Jackson
com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize
value of type `long` from String "whoops": not a valid Long value at [Source:
(BufferedInputStream); line: 302, column: 18] (through reference chain:
com.jsonexplained.SearchResponseBody["response"]
->com.jsonexplained.SearchResponse["hits"]
->java.util.ArrayList[8]
->com.jsonexplained.Hit["result"]
->com.jsonexplained.SongResult["id"])
Gson com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: For
input string: "whoops"
Moshi com.squareup.moshi.JsonDataException: Expected a long but was whoops at path
$.response.hits[8].result.id
Kotlin
Serialization java.lang.NumberFormatException: For input string: "whoops"
Type Error
Slide 125
Slide 125 text
Jackson
com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize
value of type `long` from String "whoops": not a valid Long value at [Source:
(BufferedInputStream); line: 302, column: 18] (through reference chain:
com.jsonexplained.SearchResponseBody["response"]
->com.jsonexplained.SearchResponse["hits"]
->java.util.ArrayList[8]
->com.jsonexplained.Hit["result"]
->com.jsonexplained.SongResult["id"])
Gson com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: For
input string: "whoops"
Moshi com.squareup.moshi.JsonDataException: Expected a long but was whoops at path
$.response.hits[8].result.id
Kotlin
Serialization java.lang.NumberFormatException: For input string: "whoops"
Type Error
Slide 126
Slide 126 text
Jackson Success
Gson Success
Moshi Success
Kotlin
Serialization java.lang.NumberFormatException: For input string: "1.0"
Type Acceptance
Slide 127
Slide 127 text
Jackson Success
Gson Success
Moshi Success
Kotlin
Serialization java.lang.NumberFormatException: For input string: "1.0"
Type Acceptance
Slide 128
Slide 128 text
Jackson Long: completed normally, used 0
String: crashed with a helpful message
Gson Long: completed normally, used 0
String: completed normally, used null
Moshi com.squareup.moshi.JsonDataException: Required property 'id' missing at
$.response.hits[8].result
Kotlin
Serialization
kotlinx.serialization.MissingFieldException: Field 'id' is required, but it
was missing
Required Field Is Absent
Slide 129
Slide 129 text
Jackson Long: completed normally, used 0
String: crashed with a helpful message
Gson Long: completed normally, used 0
String: completed normally, used null
Moshi com.squareup.moshi.JsonDataException: Required property 'id' missing at
$.response.hits[8].result
Kotlin
Serialization
kotlinx.serialization.MissingFieldException: Field 'id' is required, but it
was missing
Required Field Is Absent
Slide 130
Slide 130 text
Jackson com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map
due to end-of-input
at [Source: (BufferedInputStream); line: 1, column: 0]
Gson Completed normally, returned null
Moshi java.io.EOFException: End of input
Kotlin
Serialization
kotlinx.serialization.json.JsonParsingException: Invalid JSON at 0: Expected
'{, kind: kotlinx.serialization.StructureKind$CLASS@221af3c0'
Document is Empty
Slide 131
Slide 131 text
Jackson com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map
due to end-of-input
at [Source: (BufferedInputStream); line: 1, column: 0]
Gson Completed normally, returned null
Moshi java.io.EOFException: End of input
Kotlin
Serialization
kotlinx.serialization.json.JsonParsingException: Invalid JSON at 0: Expected
'{, kind: kotlinx.serialization.StructureKind$CLASS@221af3c0'
Document is Empty
Slide 132
Slide 132 text
Jackson Completes normally, encoding implementation details of types like
CompletableFuture
Gson Completes normally, encoding implementation details of types like
CompletableFuture
Moshi Fails fast with an error
Kotlin
Serialization Fails fast with an error
Write an Unexpected Type
Slide 133
Slide 133 text
Jackson Completes normally, encoding implementation details of types like
CompletableFuture
Gson Completes normally, encoding implementation details of types like
CompletableFuture
Moshi Fails fast with an error
Kotlin
Serialization Fails fast with an error
Write an Unexpected Type
Slide 134
Slide 134 text
Jackson
com.fasterxml.jackson.databind.JsonMappingException: Numeric value
(2147483648) out of range of int
at [Source: (BufferedInputStream); line: 15, column: 32] (through reference
chain: com.jsonexplained.SearchResponseBody["response"]
->com.jsonexplained.SearchResponse["hits"]
->java.util.ArrayList[0]
->com.jsonexplained.Hit["result"]
->com.jsonexplained.SongResult["annotation_count"])
Gson com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: Expected
an int but was 2147483648 at line 15 column 42 path
$.response.hits[0].result.annotation_count
Moshi com.squareup.moshi.JsonDataException: Expected an int but was 2147483648 at
path $.response.hits[0].result.annotation_count
Kotlin
Serialization java.lang.NumberFormatException: For input string: "2147483648"
Too Large For An Int
Slide 135
Slide 135 text
Jackson
com.fasterxml.jackson.databind.JsonMappingException: Numeric value
(2147483648) out of range of int
at [Source: (BufferedInputStream); line: 15, column: 32] (through reference
chain: com.jsonexplained.SearchResponseBody["response"]
->com.jsonexplained.SearchResponse["hits"]
->java.util.ArrayList[0]
->com.jsonexplained.Hit["result"]
->com.jsonexplained.SongResult["annotation_count"])
Gson com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: Expected
an int but was 2147483648 at line 15 column 42 path
$.response.hits[0].result.annotation_count
Moshi com.squareup.moshi.JsonDataException: Expected an int but was 2147483648 at
path $.response.hits[0].result.annotation_count
Kotlin
Serialization java.lang.NumberFormatException: For input string: "2147483648"
Too Large For An Int
Slide 136
Slide 136 text
Gson
MOST
USED
Jackson
MOST
CAPABLE
Kotlin S.
BEST FOR
MULTI
PLATFORM
Moshi
BEST
API
Slide 137
Slide 137 text
Testing
Slide 138
Slide 138 text
Testing?
• Confirm you’ve configured things properly
• Give you confidence to refactor & make changes
• Get your server team to code review!
Slide 139
Slide 139 text
@Test
fun json() {
val planet = Planet("Mercury", 2439, listOf("Messenger"))
val json = """
|{
| "name": "Mercury",
| "radius_km": 2439,
| "visitors": [
| "Messenger"
| ]
|}
""".trimMargin()
val adapter = moshi.adapter(Planet::class.java).indent(" ")
assertThat(adapter.toJson(planet)).isEqualTo(json)
assertThat(adapter.fromJson(json)).isEqualTo(planet)
}
Slide 140
Slide 140 text
Optimizing
Slide 141
Slide 141 text
Use Less JSON
• No pretty-printed JSON
• Delete unused fields: fewer bytes to send, store, encode, decode
• Omit nulls
Slide 142
Slide 142 text
Avoid Reflection
• Enable generated code
• Don’t create a new JsonAdapter every time
Slide 143
Slide 143 text
Stream!
• Download & decode in parallel
• Streaming uploads? Make sure your models are immutable!
Slide 144
Slide 144 text
JSON + Protobuf
• Try it as a schema language
• Read + write JSON
• Option to do binary encoding instead