Connect platforms with a Kotlin DSL

Not everyone can jump right into a Kotlin Multiplatform world. Many of us have legacy codebases and need to support our businesses first. Yet we can already use Kotlin to connect our platforms and realize significant value. In this talk we’ll learn to build a DSL that defines a contract between an API and its clients. We’ll make our DSL into a common vocabulary that translates platform idiosyncrasies. Then, by connecting it to CI, we’ll enforce our contract to keep our platforms on speaking terms. The examples in this talk will be from Sprout, an internal library in development at Etsy. The Sprout Kotlin DSL is the seed that generates code for many platforms: Retrofit endpoints, Moshi models, Swift, PHP, and more. We’ll look at how to use Kotlin as a templating language, and how the DSL can even generate itself. Finally we’ll peek at the future potential of these types of DSLs. What does connecting a Figma file look like? What about Jetpack Compose? Let’s dream big and then write a DSL for it.

Transcript

1. Connect Platforms with a Kotlin DSL Patrick Cousins
2. Staff Android Engineer, Etsy Founder, Brooklyn Kotlin Patrick Cousins @ alostpacket
3. Agenda 1 DSLs + Sprout 2 Contracts 3 Type info 4 The future
4. Question How do you tell a client a string contains HTML?
5. Question How do you know if an API change will crash your app?
6. Question How do you experiment on big things like a redesign?
7. 7 DSL Domain Specific Language
8. Domain Specific Language • Simplified mini language • Declarative • Can be data, function, configuration • Perfect for a builder pattern • Lots of tutorials online • Easier than you think
9. Extension.function() + { lambda } DSLs are easier than you think
10. // Extension function fun Type.funcName() { this.stuff() }
11. // Receiver fun Type.funcName() { this.stuff() }
12. // lambda { () -> doStuff() }
13. // lambda with receiver { () -> this.doStuff() }
14. Let’s make a Cat DSL
15. class Cat { var type: String = "" }
16. class Cat { var type: String = "" } fun cat(lambda: Cat.() -> Unit) : Cat { return Cat().apply(lambda) }
17. class Cat { var type: String = "" } fun cat(lambda: Cat.() -> Unit) : Cat { return Cat().apply(lambda) }
18. class Cat { var type: String = "" } fun cat(lambda: Cat.() -> Unit) : Cat { return Cat().apply(lambda) }
19. cat { type = "tuxedo" }
20. class Cat { var type: String = "" } class Pets { var petsList = mutableListOf<Cat>() }
21. fun pets(lambda: Pets.() -> Unit): Pets { return Pets().apply(lambda) }
22. pets { }
23. class Pets { var petsList = mutableListOf<Cat>() }
24. class Pets { var petsList = mutableListOf<Cat>() fun cat(lambda: Cat.() -> Unit) { petsList.add( Cat().apply(lambda) ) } }
25. pets { cat { type = "tuxedo" } cat { type = "tabby" } }
26. class Pets { var petsList = mutableListOf<Cat>() fun cat(lambda: Cat.() -> Unit) { petsList.add( Cat().apply(lambda) ) } }
27. fun Pets.cat(lambda: Cat.() -> Unit) { petsList.add( Cat().apply(lambda) ) }
28. pets { cat { type = "tuxedo" } cat { type = "tabby" } }
30. What is Sprout? An internal library in development at Etsy
31. What is Sprout? A code generation pipeline utilizing a custom Kotlin DSL to represent the nexus between platforms
32. Machine readable meta-data generated code
33. Why Kotlin?
34. Why Kotlin? Laura Kelly, AirBnB Droidcon SF 2018
35. Why Kotlin? Discoverable = Code completion and type safety help users discover what is possible in your DSL
37. Why Kotlin? Easy to read = Syntax of a DSL is basically equivalent to the syntax of JSON
38. {}()+.,=" {}[]:," {JSON}
39. Why Kotlin? Easy separation of logic (private) Extension functions keep the DSL clean
40. fun Pets.cat(lambda: Cat.() -> Unit) { petsList.add( Cat().apply(lambda) ) }
42. What does Sprout look like?
43. Sprout's main parts Seed, Resource, Printer, Template, Command line tool
44. class Listings_Fetch : Seedling { override fun seed(): Api { return api { apps { endpoint("Listings_Fetch") { route { get("/listings/:listing_id") } description = "Fetches a listing" resultType = Resource_ListingFetch() } } } } } } Seed
45. class Listings_Fetch : Seedling { override fun seed(): Api { return api { apps { endpoint("Listings_Fetch") { route { get("/listings/:listing_id") } description = "Fetches a listing" resultType = Resource_ListingFetch() } } } } } } Seed
46. class Resource_ListingCard(name: String = "") : Resource( name = name, info = { type = "Resource_ListingCard*" properties { id("listing_id") listingTitle("title") string("shop_average_rating").optional() } resources( Resource_ListingImage("img"), Resource_Money("price") ) } ) Resource
47. fun Api.printAndroidModels(): AndroidPrintableContent { val endpoint = this.service.endpoint val alreadyGenerated = mutableSetOf<Resource>() val toGenerate = generate(endpoint.resultType, mutableSetOf()) val files = mutableMapOf<String, String>() val typeAliases = mutableSetOf<String>() generate(toGenerate, alreadyGenerated, filesMap, typeAliases) return AndroidPrintableContent(files, typeAliases) } Printer
48. class PropertyFunctionTemplate(val name: String, val type: String):Template { override fun render(): String { return """ |@SproutDslMarker |fun Properties.$name(name: String): DataType { | val prop = $type() | _map[name] = prop | return prop |} """.trimIndent() } } Template
49. Command line tool
50. 50 API <-> client contracts (not Kotlin language contracts)
51. Resource Definition of the structure, type, nullability, and optionality of a REST API request and response
52. “The key abstraction of information in REST is a resource. Any information that can be named can be a resource: a document or image, a temporal service (e.g. “today’s weather in Los Angeles”), a collection of other resources, a non-virtual object (e.g. a person), and so on.” Roy Fielding
53. “A Resource is a unit of that API stuff” Me
54. Resource • REST API request and response • machine readable • properties • String, Int, Long, Boolean • (sub) resources • cardinality • structure, type, nullability, optionality • what your JSON is going to look like
55. class ListingsCompare { public static function spec(ResourceSpec $def) { $def->property('user_id', new DataType_ID); $def->property('notes', new DataType_String); $def->property('comparison_id', new DataType_ID); $def->resource('listings', Resource_Listing::class.'*'); } } API Resource
56. $spec = [ 'user_id' => 'DataType_ID', 'notes' => 'DataType_String', 'comparison_id' => 'DataType_ID', 'listings' => [ ... ] ]; Resource spec
57. spec = { properties { int("user_id") string("notes") int("comparison_id") } resources( Resource_ListingCard("listings") ) } Sprout Resource DSL
58. spec = { properties { id("user_id") string("notes") id("comparison_id") } resources( Resource_ListingCard("listings") ) } Sprout $spec = [ 'user_id' => 'DataType_ID', 'notes' => 'DataType_String', 'comparison_id' => 'DataType_ID', 'listings' => [ ... ] ]; API
59. Sprout output $spec = [ 'user_id' => 'DataType_ID', 'notes' => 'DataType_String', 'comparison_id' => 'DataType_ID', 'listings' => [ ... ] ]; API $spec = [ 'user_id' => 'DataType_ID', 'notes' => 'DataType_String', 'comparison_id' => 'DataType_ID', 'listings' => [ ... ] ];
60. function array_diff_assoc_recursive($array1, $array2) { $difference = array(); foreach ($array1 as $key => $value) { if (is_array($value)) { if (!isset($array2[$key]) || !is_array($array2[$key])) { $difference[$key] = $value; } else { $new_diff = array_diff_assoc_recursive($value, $array2[$key]); if (!empty($new_diff)) $difference[$key] = $new_diff; } } else if (!array_key_exists($key, $array2) || $array2[$key] !== $value) { $difference[$key] = $value; } } return $difference; } diff in both directions
61. Run diffing code on CI API Contract
62. if (!empty($diff)) { echo "Mismatch between specs. The following were found in one spec but not the other. For a guide on how to fix this see http://go/sprout/mismatch"; print_r($diff); exit(1); } else { echo "specs match. here's a cat: ~/ᐠ.ꞈ.ᐟ\~"; exit(0); } Zero for success
63. Why is this important? Surface breaking changes early
65. Putting type info in our pipeline Generate code for Android and iOS
66. With the contract in place on the API, the apps need to adhere to it Why do we do this?
67. JSON is Reductive and "lossy" when there are no analogs in transit Why do we do this?
68. {JSON} string Spannable Html.fromHtml("...") NSHTMLTextDocumentType DataType_HTML API
69. /** * ATTENTION! * If you are parsing strings in a BaseModel subclass (and I bet you are), * for your own safety use this wrapper, don't use JsonParser.valueAsString() * Assume you want all your strings stripped of HTML Escaped characters when delivered by the * API. If you show a string in the UI without cleaning up escaped characters, its going to * look ugly, Mother&#39'in ugly. * * (That's a joke about Mother's Day encoding looking like censored swearing, because it * happened to me, -redacted) */ public String parseString(JsonParser jp) throws IOException { return parseString(jp, false); } What could go wrong?
70. StringUtils.escapeHtml4(str) Apache Commons: helpful or dangerous?
71. What happens with too many object allocations? What could go wrong?
73. Side channel metadata What can we do about it?
74. DataType_HTML HtmlModel (moshi) EtsyHtmlResource API {JSON} string
75. Reading into the DSL How?
76. Sprout DSL API
77. While reading API resources into our DSL, we’ll encounter types we haven’t seen before Self-learning • Turns out our DSL is machine readable • We can generate it and... • Add new types not seen before into the DSL • Then we recompile ❤ Developer friendly Helps adoption
78. @SproutDslMarker fun Properties.html(name: String): DataType { val prop = DataType_Html() _map[name] = prop return prop } @SproutDslMarker fun Properties.id(name: String): DataType { val prop = DataType_ID() _map[name] = prop return prop } Property DSL function
79. spec = { properties { int("user_id") string("notes") int("comparison_id") } resources( Resource_ListingCard("listings") ) } Sprout Resource DSL
80. spec = { properties { id("user_id") html("notes") id("comparison_id") } resources( Resource_ListingCard("listings") ) } Sprout Resource DSL
81. @SproutDslMarker fun Properties.html(name: String): DataType { val prop = DataType_Html() _map[name] = prop return prop } @SproutDslMarker fun Properties.id(name: String): DataType { val prop = DataType_ID() _map[name] = prop return prop } Property DSL function
82. class PropertyFunctionTemplate(val name: String, val type: String):Template { override fun render(): String { return """ |@SproutDslMarker |fun Properties.$name(name: String): DataType { | val prop = $type() | _map[name] = prop | return prop |} """.trimIndent() } } Machine writeable
83. Sprout DSL API
84. DataType_HTML HtmlModel (moshi) EtsyHtmlResource API {JSON} string
85. typealias PositiveInt = Int typealias ID = Long typealias PaymentMethod = String typealias SupportedLanguage = String typealias CurrencyCode = String typealias CurrencySymbol = String typealias Latitude = Float typealias Longitude = Float SproutTypeAliases.kt
86. @JsonClass(generateAdapter = true) data class ImageSize( @Json(name = "width") val width: PositiveInt, @Json(name = "height") val height: PositiveInt, @Json(name = "url") val url: String ) Type aliases used in models
87. IDE hints typealias actual type
88. Type aliases • Can be used on iOS and Android • doesn’t change the structure of the JSON • Assignment compatible • provides hint to the engineer working in native code
89. But wait, there's more
90. Styling information Mo metadata, mo generate
91. Styling makes great metadata • Doesn't change often • Machine readable/writeable • Too big to be experimented on?
92. Figma
93. Figma API
94. Figma API
95. Everyone has an API, SDK, or scripting system nowadays Figma, Sketch, Photoshop
96. Jetpack Compose is going to be a game changer
97. Compose represents styling information in code Even more machine writeable
98. data class MaterialTypography( val h1: TextStyle = TextStyle( fontFamily = FontFamily("Roboto"), fontWeight = FontWeight.w100, fontSize = 96.sp), val h2: TextStyle = TextStyle( fontFamily = FontFamily("Roboto"), fontWeight = FontWeight.w100, fontSize = 60.sp), MaterialTheme.kt data class MaterialColors( val primary: Color = Color(0xFF6200EE.toInt()), val primaryVariant: Color = Color(0xFF3700B3.toInt()), val secondary: Color = Color(0xFF03DAC6.toInt()), val secondaryVariant: Color = Color(0xFF018786.toInt()), val background: Color = Color(0xFFFFFFFF.toInt()), Very machine writeable!
99. @Composable fun MaterialTheme() { Colors.Provider(value = colors) { Typography.Provider(value = typography) { CurrentTextStyleProvider(value = typography.body1) { MaterialRippleTheme { MaterialButtonShapeTheme(children = children) } } } } } MaterialTheme.kt
100. @Composable fun MaterialTheme() { Colors.Provider(value = colors) { Typography.Provider(value = typography) { CurrentTextStyleProvider(value = typography.body1) { if (myFeatureFlag) { SnazzyTheme { } } else { SwankyTheme { } } } } } } MaterialTheme.kt
101. Semantic styling, generated styles, & Jetpack Compose = ❤ Android dev 'bout to get more awesome
102. What about a UI? I'm glad you asked
103. Kweb: All-Kotlin library for rapid web development https://kweb.io/
104. fun main() { Kweb(port = 12345, plugins = listOf(fomanticUIPlugin, PrismPlugin())) { doc.body.new { div(fomantic.ui.container).new { div(fomantic.ui.grid).new { div(fomantic.three.wide.column).new { img(src = "sprout-logo.png", attributes = mapOf( "width" to "179", "height" to "120" )) } div(fomantic.eleven.wide.column.middle.aligned).new { var loadingSpinner:Element? = null val loadingDiv = div(fomantic.ui.hidden.middle.aligned).new { loadingSpinner = i(fomantic.notched.circle.hidden).gone() p().text("Loading...") Kweb
106. Guidelines for developing your own
107. Principles and aspirations • Un-opinionated • Does not impose complexity on existing systems • Easy to use, easy to learn • Designed to grow, learn • Lightweight (JVM, Vanila kotlin, no 3rd party libs or dependencies) • Run-books • go/link-all-the-things • Clear error messages • Easy to recover from errors • Do one thing at a time
108. dream big API Contract SDL iOS Android {JSON} test fixtures Figma React Swagger graph ql it should be easy to connect more platforms
109. Thanks for all the fish