Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Managing gRPC with Wire

Managing gRPC with Wire

At Cash App, mobile and server engineers collaborate on Protobuf and gRPC schemas to define APIs between their services. To improve our experience doing so, we built Wire. From a small in-house protobuf generation library, Wire has today matured into our favourite tool to manage and generate gRPC messages and services. Wire is backed by a versatile Gradle plugin and is used on our Android app, our iOS app, and our microservices at Cash App.
The talk will introduce Wire’s basic features before diving into:

- What Proto 3 support brings to the table in terms of new types and JSON serialisation, and the difference between Wire and protoc,
- How Wire handles options and provides comfortable APIs to consume them,
- Details about Wire Gradle plugin internals, what steps are processed and how to configure them,
- How Wire helps managing protobuf dependencies across microservices.

This presentation will provide simple entry points for people starting with Wire, and actionable improvements for experienced users alike.

Benoît Quenaudon

April 26, 2022
Tweet

More Decks by Benoît Quenaudon

Other Decks in Programming

Transcript

  1. WIRE
    @oldergod

    View full-size slide

  2. // service.proto


    message Request {


    optional Customer customer = 1;


    }


    message Customer {


    required string name = 1;


    optional int32 age = 2;


    }


    service CustomerService {


    rpc getCustomer(Request) returns (Response);


    }


    View full-size slide

  3. - Proto3


    - Options


    - Wire Gradle Plugin


    - Dependency Management
    Agenda

    View full-size slide

  4. Identity if absent

    View full-size slide

  5. syntax = "proto2";


    message Message {


    required string a = 1;


    optional string b = 2;


    }

    View full-size slide

  6. syntax = "proto2";


    message Message {


    required string a = 1;


    optional string b = 2;


    }

    View full-size slide

  7. syntax = "proto3";


    message Message {


    string a = 1;


    }

    View full-size slide

  8. Identity if absent
    Strings → ""
    Bytes → empty bytes
    Bools → false
    Numerics → 0, 0L, 0f, 0.0
    Enums → first defined value (which must be 0)
    Messages → null

    View full-size slide

  9. syntax = "proto3";


    message Message {


    string a = 1;


    }

    View full-size slide

  10. syntax = "proto3";


    message Message {


    string a = 1;


    optional string b = 2;


    }

    View full-size slide

  11. syntax = "proto2";


    message MyMessage {


    required string a = 1;


    optional string b = 2;


    }
    class MyMessage(


    val a: String,


    val b: String? = null


    ) : Message() {}

    View full-size slide

  12. syntax = "proto3";


    message MyMessage {


    string a = 1;


    optional string b = 2;


    }
    class MyMessage(


    val a: String = "",


    val b: String? = null


    ) : Message() {}

    View full-size slide

  13. Any → com.squareup.wire.AnyMessage
    Duration → java.time.Duration
    Timestamp → java.time.Instant
    Struct → map
    Wrappers → Boxed types for primitives (String?)
    Empty → kotlin.Unit
    New Types
    google.protobuf.
    ~ Generic placeholder
    ~ same as java.time.Duration
    ~ same as java.time.Instant
    ~ JSON Object
    ~ JSON representation of primitive types + nullability
    ~ For parameter-less RPCs

    View full-size slide

  14. message Request {


    // Only set when it is a Bitcoin request.


    optional BitcoinData bitcoin_data = 1;


    // Only set when it is a Stock request.


    optional StockData stock_data = 2;


    }

    View full-size slide

  15. message Request {


    // Only set when it is a Bitcoin request.


    optional BitcoinData bitcoin_data = 1;


    // Only set when it is a Stock request.


    optional StockData stock_data = 2;


    }
    message Request {


    // Will be either BitcoinData or StockData.


    optional google.protobuf.Any request_data = 1;


    }

    View full-size slide

  16. Any → com.squareup.wire.AnyMessage
    Duration → java.time.Duration
    Timestamp → java.time.Instant
    Struct → map
    Wrappers → Boxed types for primitives (String?)
    Empty → kotlin.Unit
    New Types
    google.protobuf.

    View full-size slide

  17. Any → com.squareup.wire.AnyMessage
    Duration → java.time.Duration
    Timestamp → java.time.Instant
    Struct → map
    Wrappers → Boxed primitives (e.g. String?)
    Empty → kotlin.Unit
    New Types
    google.protobuf.

    View full-size slide

  18. syntax = "proto3";


    import "google/protobuf/timestamp.proto";


    message EndpointResponse {


    // ...


    google.protobuf.Timestamp valid_until = 6;


    }

    View full-size slide

  19. val response: EndpointResponse


    if (isResponseStale(...)) {


    fetch()


    }


    private fun isResponseStale(validUntil: Instant): Boolean

    View full-size slide

  20. val protocResponse: Message.EndpointResponse


    if (isResponseStale(...)) {


    fetch()


    }


    private fun isResponseStale(validUntil: Instant): Boolean
    Protoc

    View full-size slide

  21. val millis = System.currentTimeMillis()


    val protocResponse: Message.EndpointResponse


    if (isResponseStale(...)) {


    fetch()


    }


    private fun isResponseStale(validUntil: Instant): Boolean
    Protoc

    View full-size slide

  22. val millis = System.currentTimeMillis()


    val protocResponse = Message.EndpointResponse.newBuilder()


    .setValidUntil(


    Timestamp...


    )


    .build()


    if (isResponseStale(...)) {


    fetch()


    }


    private fun isResponseStale(validUntil: Instant): Boolean
    Protoc

    View full-size slide

  23. val millis = System.currentTimeMillis()


    val protocResponse = Message.EndpointResponse.newBuilder()


    .setValidUntil(


    Timestamp.newBuilder()


    .setSeconds(millis / 1000)


    .setNanos(((millis % 1000) * 1000000).toInt())


    .build()


    )


    .build()


    if (isResponseStale(...)) {


    fetch()


    }


    private fun isResponseStale(validUntil: Instant): Boolean
    Protoc

    View full-size slide

  24. val millis = System.currentTimeMillis()


    val protocResponse = Message.EndpointResponse.newBuilder().build()


    if (isResponseStale(...)) {


    fetch()


    }


    private fun isResponseStale(validUntil: Instant): Boolean
    Protoc

    View full-size slide

  25. val millis = System.currentTimeMillis()


    val protocResponse = Message.EndpointResponse.newBuilder().build()


    if (isResponseStale(


    Instant.ofEpochSecond(


    protocResponse.validUntil.seconds,


    protocResponse.validUntil.nanos.toLong()


    )


    )) {


    fetch()


    }


    private fun isResponseStale(validUntil: Instant): Boolean
    Protoc

    View full-size slide

  26. Wire
    val wireResponse: EndpointResponse


    if (isResponseStale(...)) {


    fetch()


    }


    private fun isResponseStale(validUntil: Instant): Boolean

    View full-size slide

  27. Wire
    val millis = System.currentTimeMillis()


    val wireResponse: EndpointResponse


    if (isResponseStale(...)) {


    fetch()


    }


    private fun isResponseStale(validUntil: Instant): Boolean

    View full-size slide

  28. Wire
    val millis = System.currentTimeMillis()


    val wireResponse = EndpointResponse(Instant.ofEpochMilli(millis))


    if (isResponseStale(...)) {


    fetch()


    }


    private fun isResponseStale(validUntil: Instant): Boolean

    View full-size slide

  29. Wire
    val millis = System.currentTimeMillis()


    val wireResponse = EndpointResponse(Instant.ofEpochMilli(millis))


    if (isResponseStale(wireResponse.valid_until!!)) {


    fetch()


    }


    private fun isResponseStale(validUntil: Instant): Boolean

    View full-size slide

  30. JSON
    • Proto2 : Wire 㲗 Protoc communication is dangerous.


    • Proto3 : All good.

    View full-size slide

  31. Proto2 㲗 Proto3


    Compatibility

    View full-size slide

  32. Proto2 㲗 Proto3 Compatibility
    • Cannot reference proto2 enums in proto3 message.


    • Because of identity-if-absent.


    • Anything else is good.

    View full-size slide

  33. Protobuf Options

    View full-size slide

  34. Protobuf Options
    message Money {


    optional int64 amount = 1 [(squareup.redacted) = true];


    optional string title = 2 [deprecated = true];


    optional Currency current = 3 [(whatever.opt) = 3];


    }

    View full-size slide

  35. Protobuf Options
    message Money {


    optional int64 amount = 1 [(squareup.redacted) = true];


    optional string title = 2 [deprecated = true];


    optional Currency current = 3 [(whatever.opt) = 3];


    }

    View full-size slide

  36. Protobuf Options
    • No intrinsic logic embedded


    • Wire handles a few


    • deprecated


    • default


    • json_name


    • packed


    • *.redacted


    • java_package


    • wire_package


    • "How about my option?"

    View full-size slide

  37. extend google.protobuf.MessageOptions {


    optional string documentation_url = 22200;


    }


    message Octagon {


    option (documentation_url) = "https://en.wikipedia.org/wiki/Octagon";


    optional bool stop = 1;


    }

    View full-size slide

  38. wire {


    kotlin {


    emitDeclaredOptions = true


    emitAppliedOptions = true


    }


    }
    extend google.protobuf.MessageOptions {


    optional string documentation_url = 22200;


    }


    message Octagon {


    option (documentation_url) = "https://en.wikipedia.org/wiki/Octagon";


    optional bool stop = 1;


    }

    View full-size slide

  39. // emitDeclaredOptions = true


    @Retention(AnnotationRetention.RUNTIME)


    @Target(AnnotationTarget.CLASS)


    public annotation class DocumentationUrlOption(


    public val value: String


    )
    // emitAppliedOptions = true


    @DocumentationUrlOption("https://en.wikipedia.org/wiki/Octagon")


    public class Octagon(


    ...


    ) : Message(ADAPTER, unknownFields) {}

    View full-size slide

  40. Wire Gradle Plugin

    View full-size slide

  41. Schema
    .proto
    Elements
    Java Kotlin Proto
    Schema
    Swift
    Parse
    Link
    Prune
    Generate

    View full-size slide

  42. Schema
    .proto
    Elements
    Java Kotlin Proto
    Schema
    Swift
    Parse
    Link
    Prune
    Generate
    wire {


    sourcePath {


    srcDir 'src/main/protos'


    }


    sourcePath {


    srcJar 'lib/pizza-protos.jar'


    }


    protoPath {


    srcJar 'com.example.pizza:pizza-protos:1.0.0'


    }


    ...


    }

    View full-size slide

  43. Schema
    .proto
    Elements
    Java Kotlin Proto
    Schema
    Swift
    Parse
    Link
    Prune
    Generate
    wire {


    sourcePath {


    srcDir 'src/main/protos'


    include 'com/example/pizza/pizza_delivery.proto'


    include 'com/example/pizza/pizza.proto'


    }


    }

    View full-size slide

  44. Schema
    .proto
    Elements
    Java Kotlin Proto
    Schema
    Swift
    Parse
    Link
    Prune
    Generate
    wire {


    sourcePath {


    srcDir 'src/main/protos'


    include 'com/example/pizza/pizza_delivery.proto'


    include 'com/example/pizza/pizza.proto'


    }


    }

    View full-size slide

  45. Schema
    .proto
    Elements
    Java Kotlin Proto
    Schema
    Swift
    Parse
    Link
    Prune
    Generate
    wire {


    sourcePath {


    srcDir 'src/main/protos'


    include 'com/example/pizza/pizza_delivery.proto'


    include 'com/example/pizza/pizza.proto'


    }


    }

    View full-size slide

  46. Schema
    .proto
    Elements
    Java Kotlin Proto
    Schema
    Swift
    Parse
    Link
    Prune
    Generate
    wire {


    protoPath {


    srcJar 'com.squareup.protos:all-protos'


    }


    sourcePath {


    srcDir 'src/main/protos'


    include 'com/example/pizza/pizza_delivery.proto'


    include 'com/example/pizza/pizza.proto'


    }


    }

    View full-size slide

  47. Schema
    .proto
    Elements
    Java Kotlin Proto
    Schema
    Swift
    Parse
    Link
    Prune
    Generate
    wire {


    root 'com.example.store.Store'


    }

    View full-size slide

  48. wire {


    root 'com.example.store.Store'


    }

    View full-size slide

  49. wire {


    root 'com.example.store.Store'


    }

    View full-size slide

  50. Schema
    .proto
    Elements
    Java Kotlin Proto
    Schema
    Swift
    Parse
    Link
    Prune
    Generate
    wire {


    prune 'com.example.store.Store'


    prune 'com.example.geo.Country'


    }

    View full-size slide

  51. wire {


    prune 'com.example.store.Store'


    prune 'com.example.geo.Country'


    }

    View full-size slide

  52. wire {


    prune 'com.example.store.Store'


    prune 'com.example.geo.Country'


    }

    View full-size slide

  53. wire {


    prune 'com.example.store.Store'


    prune 'com.example.geo.Country'


    }

    View full-size slide

  54. Schema
    .proto
    Elements
    Java Kotlin Proto
    Schema
    Swift
    Parse
    Link
    Prune
    Generate
    wire {


    kotlin {


    includes = ['com.example.pizza.*']


    excludes = ['com.example.sales.*']


    exclusive = false


    out "${buildDir}/custom"


    }


    }

    View full-size slide

  55. wire {


    ...


    kotlin {


    // Kotlin emits the matched types only.


    includes = [‘com.example.pizza.*']


    exclusive = true


    }


    java {


    // Java gets everything else!


    }


    }

    View full-size slide

  56. Schema
    .proto
    Elements
    Java Kotlin Proto
    Schema
    Swift
    Parse
    Link
    Prune
    Generate
    wire {


    custom {


    schemaHandlerFactoryClass = "MyHandlerFactory"


    }


    }
    Schema
    Handler
    Coming soon™

    View full-size slide

  57. interface SchemaHandler {


    fun handle(schema: Schema, context: Context)


    interface Factory : Serializable {


    fun create(): SchemaHandler


    }


    }
    Coming soon™

    View full-size slide

  58. Dependency management

    View full-size slide

  59. Common


    • common/money.proto


    • common/customer.proto
    Service Trades


    • trade/payment.proto
    Service Bank


    • bank/balance.proto
    Mobile

    View full-size slide

  60. Common


    • common/money.proto


    • common/customer.proto
    // money.proto


    syntax = "proto3";


    package common;


    message Money {


    int64 amount = 1;


    Current currency = 2;


    }


    enum Currency {


    EUR = 0;


    GBP = 1;


    USD = 2;


    CAD = 3;


    }
    // customer.proto


    syntax = "proto3";


    package common;


    message Customer {


    string token = 1;


    string name = 2;


    }

    View full-size slide

  61. Common


    • common/money.proto


    • common/customer.proto
    wire {


    sourcePath {


    srcDir("src/main/proto")


    }


    }

    View full-size slide

  62. Common


    • common/money.proto


    • common/customer.proto
    wire {


    sourcePath {


    srcDir("src/main/proto")


    }


    kotlin {


    }


    }

    View full-size slide

  63. Common


    • common/money.proto


    • common/customer.proto
    wire {


    sourcePath {


    srcDir("src/main/proto")


    }


    kotlin {


    }


    }


    sourceSets {


    val main by getting {


    resources.srcDir(


    project.file(“src/main/proto/")


    )


    }


    }

    View full-size slide

  64. Common


    • common/money.proto


    • common/customer.proto
    wire {


    protoLibrary = true


    sourcePath {


    srcDir("src/main/proto")


    }


    kotlin {


    }


    }

    View full-size slide

  65. Common


    • common/money.proto


    • common/customer.proto
    wire {


    protoLibrary = true


    sourcePath {


    srcDir("src/main/proto")


    }


    kotlin {


    }


    }
    jar published internally to 'app.cash.common:common'

    View full-size slide

  66. Common


    • common/money.proto


    • common/customer.proto
    Service Trades


    • trade/payment.proto
    Service Bank


    • bank/balance.proto
    Mobile

    View full-size slide

  67. Service Trades


    • trade/payment.proto
    syntax = "proto3";


    package trade;


    import "common/customer.proto";


    import "common/money.proto";


    message PaymentRequest {


    string common.Money money = 1;


    string common.Customer recipient = 2;


    }


    message PaymentResponse {}


    service PaymentService {


    rpc Pay(PaymentRequest) returns (PaymentResponse);


    }

    View full-size slide

  68. Service Trades


    • trade/payment.proto
    wire {


    protoLibrary = true


    sourcePath {


    srcDir("src/main/proto")


    }


    kotlin {


    }


    }

    View full-size slide

  69. Service Trades


    • trade/payment.proto
    wire {


    protoLibrary = true


    protoPath {


    srcJar("app.cash.common:common:")


    }


    sourcePath {


    srcDir("src/main/proto")


    }


    kotlin {


    }


    }

    View full-size slide

  70. Service Trades


    • trade/payment.proto
    wire {


    protoLibrary = true


    protoPath {


    srcJar("app.cash.common:common:")


    }


    sourcePath {


    srcDir("src/main/proto")


    }


    kotlin {


    }


    }


    dependencies {


    implementation("app.cash.common:common:")


    }

    View full-size slide

  71. Service Trades


    • trade/payment.proto
    wire {


    protoLibrary = true


    protoPath {


    srcJar("app.cash.common:common:")


    }


    sourcePath {


    srcDir("src/main/proto")


    }


    kotlin {


    }


    }


    dependencies {


    implementation("app.cash.common:common:")


    }
    jar published internally to ‘app.cash.trade:trade'

    View full-size slide

  72. Common


    • common/money.proto


    • common/customer.proto
    Service Trades


    • trade/payment.proto
    Service Bank


    • bank/balance.proto
    Mobile

    View full-size slide

  73. Mobile
    wire {


    kotlin {


    }


    }

    View full-size slide

  74. Mobile
    wire {


    kotlin {


    android = true


    }


    }

    View full-size slide

  75. Mobile
    wire {


    sourcePath { srcJar("app.cash.trade:trade:") }


    sourcePath { srcJar("app.cash.balance:balance:") }


    kotlin {


    android = true


    }


    }

    View full-size slide

  76. Mobile
    wire {


    sourcePath { srcJar("app.cash.common:common:") }


    sourcePath { srcJar("app.cash.trade:trade:") }


    sourcePath { srcJar("app.cash.balance:balance:") }


    kotlin {


    android = true


    }


    }

    View full-size slide

  77. Common


    • common/money.proto


    • common/customer.proto
    Service Trades


    • trade/payment.proto
    Service Bank


    • bank/balance.proto
    Mobile

    View full-size slide

  78. References
    • Protocol Bu
    ff
    ers documentations


    • https://developers.google.com/protocol-bu
    ff
    ers/


    • Wire documentations


    • https://square.github.io/wire/


    • Protogram


    • https://github.com/mattprecious/protogram

    View full-size slide

  79. Protogram
    CLI
    Web

    View full-size slide

  80. WIRE
    FIN
    for real

    View full-size slide