Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Managing gRPC with Wire

Managing gRPC with Wire

At Cash App, mobile and server engineers collaborate on Protobuf and gRPC schemas to define APIs between their services. To improve our experience doing so, we built Wire. From a small in-house protobuf generation library, Wire has today matured into our favourite tool to manage and generate gRPC messages and services. Wire is backed by a versatile Gradle plugin and is used on our Android app, our iOS app, and our microservices at Cash App.
The talk will introduce Wire’s basic features before diving into:

- What Proto 3 support brings to the table in terms of new types and JSON serialisation, and the difference between Wire and protoc,
- How Wire handles options and provides comfortable APIs to consume them,
- Details about Wire Gradle plugin internals, what steps are processed and how to configure them,
- How Wire helps managing protobuf dependencies across microservices.

This presentation will provide simple entry points for people starting with Wire, and actionable improvements for experienced users alike.

05162bc961c3654218bf1839974a4f35?s=128

Benoît Quenaudon

April 26, 2022
Tweet

More Decks by Benoît Quenaudon

Other Decks in Programming

Transcript

  1. WIRE @oldergod

  2. None
  3. // service.proto message Request { optional Customer customer = 1;

    } message Customer { required string name = 1; optional int32 age = 2; } service CustomerService { rpc getCustomer(Request) returns (Response); }
  4. - Proto3 - Options - Wire Gradle Plugin - Dependency

    Management Agenda
  5. None
  6. Identity if absent

  7. syntax = "proto2"; message Message { required string a =

    1; optional string b = 2; }
  8. syntax = "proto2"; message Message { required string a =

    1; optional string b = 2; }
  9. syntax = "proto3"; message Message { string a = 1;

    }
  10. Identity if absent Strings → "" Bytes → empty bytes

    Bools → false Numerics → 0, 0L, 0f, 0.0 Enums → first defined value (which must be 0) Messages → null
  11. syntax = "proto3"; message Message { string a = 1;

    }
  12. syntax = "proto3"; message Message { string a = 1;

    optional string b = 2; }
  13. syntax = "proto2"; message MyMessage { required string a =

    1; optional string b = 2; } class MyMessage( val a: String, val b: String? = null ) : Message<Message, Builder>() {} →
  14. syntax = "proto3"; message MyMessage { string a = 1;

    optional string b = 2; } class MyMessage( val a: String = "", val b: String? = null ) : Message<Message, Builder>() {} →
  15. New Types

  16. Any → com.squareup.wire.AnyMessage Duration → java.time.Duration Timestamp → java.time.Instant Struct

    → map<String, *> Wrappers → Boxed types for primitives (String?) Empty → kotlin.Unit New Types google.protobuf. ~ Generic placeholder ~ same as java.time.Duration ~ same as java.time.Instant ~ JSON Object ~ JSON representation of primitive types + nullability ~ For parameter-less RPCs
  17. message Request { // Only set when it is a

    Bitcoin request. optional BitcoinData bitcoin_data = 1; // Only set when it is a Stock request. optional StockData stock_data = 2; }
  18. message Request { // Only set when it is a

    Bitcoin request. optional BitcoinData bitcoin_data = 1; // Only set when it is a Stock request. optional StockData stock_data = 2; } message Request { // Will be either BitcoinData or StockData. optional google.protobuf.Any request_data = 1; } ↓
  19. Any → com.squareup.wire.AnyMessage Duration → java.time.Duration Timestamp → java.time.Instant Struct

    → map<String, *> Wrappers → Boxed types for primitives (String?) Empty → kotlin.Unit New Types google.protobuf.
  20. Any → com.squareup.wire.AnyMessage Duration → java.time.Duration Timestamp → java.time.Instant Struct

    → map<String, *> Wrappers → Boxed primitives (e.g. String?) Empty → kotlin.Unit New Types google.protobuf.
  21. syntax = "proto3"; import "google/protobuf/timestamp.proto"; message EndpointResponse { // ...

    google.protobuf.Timestamp valid_until = 6; }
  22. val response: EndpointResponse if (isResponseStale(...)) { fetch() } private fun

    isResponseStale(validUntil: Instant): Boolean
  23. val protocResponse: Message.EndpointResponse if (isResponseStale(...)) { fetch() } private fun

    isResponseStale(validUntil: Instant): Boolean Protoc
  24. val millis = System.currentTimeMillis() val protocResponse: Message.EndpointResponse if (isResponseStale(...)) {

    fetch() } private fun isResponseStale(validUntil: Instant): Boolean Protoc
  25. val millis = System.currentTimeMillis() val protocResponse = Message.EndpointResponse.newBuilder() .setValidUntil( Timestamp...

    ) .build() if (isResponseStale(...)) { fetch() } private fun isResponseStale(validUntil: Instant): Boolean Protoc
  26. val millis = System.currentTimeMillis() val protocResponse = Message.EndpointResponse.newBuilder() .setValidUntil( Timestamp.newBuilder()

    .setSeconds(millis / 1000) .setNanos(((millis % 1000) * 1000000).toInt()) .build() ) .build() if (isResponseStale(...)) { fetch() } private fun isResponseStale(validUntil: Instant): Boolean Protoc
  27. val millis = System.currentTimeMillis() val protocResponse = Message.EndpointResponse.newBuilder().build() if (isResponseStale(...))

    { fetch() } private fun isResponseStale(validUntil: Instant): Boolean Protoc
  28. val millis = System.currentTimeMillis() val protocResponse = Message.EndpointResponse.newBuilder().build() if (isResponseStale(

    Instant.ofEpochSecond( protocResponse.validUntil.seconds, protocResponse.validUntil.nanos.toLong() ) )) { fetch() } private fun isResponseStale(validUntil: Instant): Boolean Protoc
  29. Wire val wireResponse: EndpointResponse if (isResponseStale(...)) { fetch() } private

    fun isResponseStale(validUntil: Instant): Boolean
  30. Wire val millis = System.currentTimeMillis() val wireResponse: EndpointResponse if (isResponseStale(...))

    { fetch() } private fun isResponseStale(validUntil: Instant): Boolean
  31. Wire val millis = System.currentTimeMillis() val wireResponse = EndpointResponse(Instant.ofEpochMilli(millis)) if

    (isResponseStale(...)) { fetch() } private fun isResponseStale(validUntil: Instant): Boolean
  32. Wire val millis = System.currentTimeMillis() val wireResponse = EndpointResponse(Instant.ofEpochMilli(millis)) if

    (isResponseStale(wireResponse.valid_until!!)) { fetch() } private fun isResponseStale(validUntil: Instant): Boolean
  33. JSON

  34. Why JSON?

  35. JSON • Proto2 : Wire 㲗 Protoc communication is dangerous.

    • Proto3 : All good.
  36. Proto2 㲗 Proto3 Compatibility

  37. Proto2 㲗 Proto3 Compatibility • Cannot reference proto2 enums in

    proto3 message. • Because of identity-if-absent. • Anything else is good.
  38. Protobuf Options

  39. Protobuf Options message Money { optional int64 amount = 1

    [(squareup.redacted) = true]; optional string title = 2 [deprecated = true]; optional Currency current = 3 [(whatever.opt) = 3]; }
  40. Protobuf Options message Money { optional int64 amount = 1

    [(squareup.redacted) = true]; optional string title = 2 [deprecated = true]; optional Currency current = 3 [(whatever.opt) = 3]; }
  41. Protobuf Options • No intrinsic logic embedded • Wire handles

    a few • deprecated • default • json_name • packed • *.redacted • java_package • wire_package • "How about my option?"
  42. extend google.protobuf.MessageOptions { optional string documentation_url = 22200; } message

    Octagon { option (documentation_url) = "https://en.wikipedia.org/wiki/Octagon"; optional bool stop = 1; }
  43. wire { kotlin { emitDeclaredOptions = true emitAppliedOptions = true

    } } extend google.protobuf.MessageOptions { optional string documentation_url = 22200; } message Octagon { option (documentation_url) = "https://en.wikipedia.org/wiki/Octagon"; optional bool stop = 1; }
  44. // emitDeclaredOptions = true @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.CLASS) public annotation class DocumentationUrlOption(

    public val value: String ) // emitAppliedOptions = true @DocumentationUrlOption("https://en.wikipedia.org/wiki/Octagon") public class Octagon( ... ) : Message<Octagon, Octagon.Builder>(ADAPTER, unknownFields) {}
  45. Wire Gradle Plugin

  46. Schema .proto Elements Java Kotlin Proto Schema Swift Parse Link

    Prune Generate
  47. Schema .proto Elements Java Kotlin Proto Schema Swift Parse Link

    Prune Generate wire { sourcePath { srcDir 'src/main/protos' } sourcePath { srcJar 'lib/pizza-protos.jar' } protoPath { srcJar 'com.example.pizza:pizza-protos:1.0.0' } ... }
  48. Schema .proto Elements Java Kotlin Proto Schema Swift Parse Link

    Prune Generate wire { sourcePath { srcDir 'src/main/protos' include 'com/example/pizza/pizza_delivery.proto' include 'com/example/pizza/pizza.proto' } }
  49. Schema .proto Elements Java Kotlin Proto Schema Swift Parse Link

    Prune Generate wire { sourcePath { srcDir 'src/main/protos' include 'com/example/pizza/pizza_delivery.proto' include 'com/example/pizza/pizza.proto' } }
  50. Schema .proto Elements Java Kotlin Proto Schema Swift Parse Link

    Prune Generate wire { sourcePath { srcDir 'src/main/protos' include 'com/example/pizza/pizza_delivery.proto' include 'com/example/pizza/pizza.proto' } }
  51. Schema .proto Elements Java Kotlin Proto Schema Swift Parse Link

    Prune Generate wire { protoPath { srcJar 'com.squareup.protos:all-protos' } sourcePath { srcDir 'src/main/protos' include 'com/example/pizza/pizza_delivery.proto' include 'com/example/pizza/pizza.proto' } }
  52. None
  53. None
  54. None
  55. None
  56. Schema .proto Elements Java Kotlin Proto Schema Swift Parse Link

    Prune Generate wire { root 'com.example.store.Store' }
  57. wire { root 'com.example.store.Store' }

  58. wire { root 'com.example.store.Store' }

  59. Schema .proto Elements Java Kotlin Proto Schema Swift Parse Link

    Prune Generate wire { prune 'com.example.store.Store' prune 'com.example.geo.Country' }
  60. wire { prune 'com.example.store.Store' prune 'com.example.geo.Country' }

  61. wire { prune 'com.example.store.Store' prune 'com.example.geo.Country' }

  62. wire { prune 'com.example.store.Store' prune 'com.example.geo.Country' }

  63. Schema .proto Elements Java Kotlin Proto Schema Swift Parse Link

    Prune Generate wire { kotlin { includes = ['com.example.pizza.*'] excludes = ['com.example.sales.*'] exclusive = false out "${buildDir}/custom" } }
  64. wire { ... kotlin { // Kotlin emits the matched

    types only. includes = [‘com.example.pizza.*'] exclusive = true } java { // Java gets everything else! } }
  65. Schema .proto Elements Java Kotlin Proto Schema Swift Parse Link

    Prune Generate wire { custom { schemaHandlerFactoryClass = "MyHandlerFactory" } } Schema Handler Coming soon™
  66. interface SchemaHandler { fun handle(schema: Schema, context: Context) interface Factory

    : Serializable { fun create(): SchemaHandler } } Coming soon™
  67. Dependency management

  68. Common • common/money.proto • common/customer.proto Service Trades • trade/payment.proto Service

    Bank • bank/balance.proto Mobile
  69. Common • common/money.proto • common/customer.proto // money.proto syntax = "proto3";

    package common; message Money { int64 amount = 1; Current currency = 2; } enum Currency { EUR = 0; GBP = 1; USD = 2; CAD = 3; } // customer.proto syntax = "proto3"; package common; message Customer { string token = 1; string name = 2; }
  70. Common • common/money.proto • common/customer.proto wire { sourcePath { srcDir("src/main/proto")

    } }
  71. Common • common/money.proto • common/customer.proto wire { sourcePath { srcDir("src/main/proto")

    } kotlin { } }
  72. Common • common/money.proto • common/customer.proto wire { sourcePath { srcDir("src/main/proto")

    } kotlin { } } sourceSets { val main by getting { resources.srcDir( project.file(“src/main/proto/") ) } }
  73. Common • common/money.proto • common/customer.proto wire { protoLibrary = true

    sourcePath { srcDir("src/main/proto") } kotlin { } }
  74. Common • common/money.proto • common/customer.proto wire { protoLibrary = true

    sourcePath { srcDir("src/main/proto") } kotlin { } } jar published internally to 'app.cash.common:common'
  75. Common • common/money.proto • common/customer.proto Service Trades • trade/payment.proto Service

    Bank • bank/balance.proto Mobile
  76. Service Trades • trade/payment.proto syntax = "proto3"; package trade; import

    "common/customer.proto"; import "common/money.proto"; message PaymentRequest { string common.Money money = 1; string common.Customer recipient = 2; } message PaymentResponse {} service PaymentService { rpc Pay(PaymentRequest) returns (PaymentResponse); }
  77. Service Trades • trade/payment.proto wire { protoLibrary = true sourcePath

    { srcDir("src/main/proto") } kotlin { } }
  78. Service Trades • trade/payment.proto wire { protoLibrary = true protoPath

    { srcJar("app.cash.common:common:<version>") } sourcePath { srcDir("src/main/proto") } kotlin { } }
  79. Service Trades • trade/payment.proto wire { protoLibrary = true protoPath

    { srcJar("app.cash.common:common:<version>") } sourcePath { srcDir("src/main/proto") } kotlin { } } dependencies { implementation("app.cash.common:common:<version>") }
  80. Service Trades • trade/payment.proto wire { protoLibrary = true protoPath

    { srcJar("app.cash.common:common:<version>") } sourcePath { srcDir("src/main/proto") } kotlin { } } dependencies { implementation("app.cash.common:common:<version>") } jar published internally to ‘app.cash.trade:trade'
  81. Common • common/money.proto • common/customer.proto Service Trades • trade/payment.proto Service

    Bank • bank/balance.proto Mobile
  82. Mobile wire { kotlin { } }

  83. Mobile wire { kotlin { android = true } }

  84. Mobile wire { sourcePath { srcJar("app.cash.trade:trade:<version>") } sourcePath { srcJar("app.cash.balance:balance:<version>")

    } kotlin { android = true } }
  85. Mobile wire { sourcePath { srcJar("app.cash.common:common:<version>") } sourcePath { srcJar("app.cash.trade:trade:<version>")

    } sourcePath { srcJar("app.cash.balance:balance:<version>") } kotlin { android = true } }
  86. Common • common/money.proto • common/customer.proto Service Trades • trade/payment.proto Service

    Bank • bank/balance.proto Mobile
  87. WIRE FIN

  88. References • Protocol Bu ff ers documentations • https://developers.google.com/protocol-bu ff

    ers/ • Wire documentations • https://square.github.io/wire/ • Protogram • https://github.com/mattprecious/protogram
  89. Protogram CLI Web

  90. WIRE FIN for real