Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Beyond JSON: Improving Inter-app Communication

F04bfa14141dca6713f0d9caa763e26b?s=47 Aaron Quint
September 01, 2015

Beyond JSON: Improving Inter-app Communication

A talk I gave at BaRuCo/Full Stack Fest 2015 about improving inter-app communication by using proctol buffers and non-http protocols.

F04bfa14141dca6713f0d9caa763e26b?s=128

Aaron Quint

September 01, 2015
Tweet

Transcript

  1. BEYOND JSON @aq / Full Stack Fest / Sept 2015

  2. I’m @aq HELLO!

  3. beatsryetypes.com @beatsryetypes

  4. CatskillsConf.com Oct 23-25, 2015

  5. Ruby, JS, Go Performance Consulting quirkey.com/hireme

  6. None
  7. Because there are more apps Inter-app Communication is more important

    than ever
  8. HONEY, I SHRUNK THE SERVICE

  9. To avoid misunderstanding An introduction to terms

  10. The transportation mechanism for delivering information from one application to

    another. Protocol
  11. Shorthand for Serialization Format, the specific way data is encoded

    and structured for delivery between applications. Format
  12. The shape of the data and how it is delivered.

    Format over Protocol
  13. Format over Protocol JSON over HTTP

  14. A very Brief History of Inter-App Communication

  15. Pre-Internet Days

  16. Punchcards over Sneakernet

  17. Early Internet Days

  18. XML over HTTP

  19. Nowish

  20. JSON over HTTP

  21. The New Default JSON over HTTP for ERRRRYTHING

  22. Because. But, WHY?

  23. It’s EZ

  24. Break it down • Human Readable/Writable (Well certain humans) •

    Parsers/Clients for every Language • Relatively Fast • Has a few explicit types (String, #, Bool) • Compared to XML/SOAP, Relatively Compact
  25. Like most things in our world. Well, It’s EZ at

    first.
  26. It always starts small.

  27. { "id": "1234", "title": "Beyond JSON" }

  28. { "id": "1234", "title": "Beyond JSON”, "author": “Aaron Quint”, “created_at":

    “2015-08-01” }
  29. { "id": "1234", "title": "Beyond JSON”, "author": { "name": “Aaron

    Quint”, "title": “Chief Scientist” }, “favorites": [ { "id": “123”, “user”: “Kat Howard” } ], “created_at": “2015-08-01” }
  30. { "id": "1234", "title": "Beyond JSON”, "author": { "name": “Aaron

    Quint”, "title": “Chief Scientist” “company": { "id": "1334”, "name": “Quirkey NYC” } }, “favorites": [ { "id": “123”, “user”: { “id”: “152”, "name": “Kat Howard”, } } ], “created_at": “2015-08-01” }
  31. None
  32. What we want • Works well in many languages •

    Relatively Fast • Relatively Compact • Has explicit types • +Handles Change Well
  33. What if I told you this already exists?

  34. that’s protobufs to you Protocol Buffers!

  35. None
  36. Procotol?! Buffers?!

  37. package app; message Meta { required string key = 1;

    required string value = 2; } message Image { optional string id = 1; optional bytes body = 2; optional string extension = 3; optional uint64 size = 4; optional uint32 width = 5; optional uint32 height = 6; optional string filename = 7; repeated Meta meta = 8; } message Request { required string command = 1; required Image image = 2; repeated Meta transform = 3; optional bool returnbody = 4; } message Response { required string status = 1; required string message = 2; optional Image image = 3; } app.proto app/*.pb.rb app.pb.go protoc —ruby_out protoc —go_out
  38. image = App::Image.new(:id => "123") bytes = image.encode #=> bytes

    image = App::Image.decode(bytes) image.id #=> 123 package app; message Meta { required string key = 1; required string value = 2; } message Image { optional string id = 1; optional bytes body = 2; optional string extension = 3; optional uint64 size = 4; optional uint32 width = 5; optional uint32 height = 6; optional string filename = 7; repeated Meta meta = 8; } message Request { required string command = 1; required Image image = 2; repeated Meta transform = 3; optional bool returnbody = 4; } message Response { required string status = 1; required string message = 2; optional Image image = 3; } image := app.Image{Id: proto.String("123")} bytes, err := proto.Marshal(image) if err != nil { // ... } newImage := &app.Image{} err = proto.Unmarshal(bytes, newImage) if err != nil { // ... } newImage.GetId() #=> "123"
  39. package app; message Meta { required string key = 1;

    required string value = 2; } message Image { optional string id = 1; optional bytes body = 2; optional string extension = 3; optional uint64 size = 4; optional uint32 width = 5; optional uint32 height = 6; optional string filename = 7; repeated Meta meta = 8; } message Request { required string command = 1; required Image image = 2; repeated Meta transform = 3; optional bool returnbody = 4; } message Response { required string status = 1; required string message = 2; optional Image image = 3; } explicit types repeated fields unique tags for backwards and forwards compatibility required fields nested types
  40. ==================== Sizes beefcake: 23b protobuf: 23b json: 71b yajl: 71b

    msgpack: 54b ------------------------------------------------- encode/beefcake 36.915k (± 5.1%) i/s - 184.808k encode/protobuf 34.186k (± 6.7%) i/s - 172.462k encode/json 75.959k (± 8.3%) i/s - 378.367k encode/yajl 81.132k (±11.4%) i/s - 407.199k encode/msgpack 74.240k (±38.1%) i/s - 316.250k Comparison: encode/yajl: 81132.2 i/s encode/json: 75959.2 i/s - 1.07x slower encode/msgpack: 74239.7 i/s - 1.09x slower encode/beefcake: 36915.1 i/s - 2.20x slower encode/protobuf: 34186.1 i/s - 2.37x slower ------------------------------------------------- decode/beefcake 33.048k (± 9.3%) i/s - 164.672k decode/protobuf 45.565k (± 8.7%) i/s - 226.838k decode/json 80.541k (±17.0%) i/s - 390.258k decode/yajl 92.441k (±18.2%) i/s - 442.662k decode/msgpack 126.193k (±12.7%) i/s - 624.618k Comparison: decode/msgpack: 126192.7 i/s decode/yajl: 92441.2 i/s - 1.37x slower decode/json: 80540.7 i/s - 1.57x slower decode/protobuf: 45565.1 i/s - 2.77x slower decode/beefcake: 33047.8 i/s - 3.82x slower https://github.com/quirkey/format_benchmark
  41. Clear wins • Centralized Schema • Easy Serialization into objects

    • Handles removal/addition of fields easily • Real types + nesting • Compact (on the wire)
  42. Downsides • Encoded output is not human readable (does that

    matter?) • Ruby encoding/decoding is not fast (it’s not that slow - wire matters more) • Ruby libraries are not the best (Support for other languages is great, though)
  43. Protobufs over … But what about the Protocol?

  44. Just stuff them in the body and call it a

    day. HTTP?
  45. The simplest thing I could possibly think of within the

    bounds of NIH syndrome. TCPEZ http://github.com/paperlesspost/tcpez http://github.com/paperlesspost/tcpez-client
  46. client server Header Body 4 PING Header Body 4 PONG

  47. client server Header Body n <protobuf> Header Body n <protobuf>

  48. TCPEZ • Client -> Server (Req/Resp) • So far Go

    server + Ruby client • Client is the load balancer • Request and Response encoding/decoding are not part of the protocol. • Each tcpez based project defines its own Request/ Response and these can be in any format • Built in by default: Good logging, Statsd, Protocol Buffers, Pipelined requests!
  49. Multiple Apps, Requests in the 100M since 10/2013 TCPEZ is

    in production at Paperless Post
  50. A quick aside …

  51. http://research.google.com/pubs/pub36356.html Read Google’s Dapper Paper Building a distributed system without

    introspectability is a fools errand
  52. None
  53. Clients for all (incl. Ruby) but server is Java only

    now GRPC = A layer above Protobufs over HTTP/2
  54. New version of protobufs has new features: reserved tags, repeated

    compaction, importing, more … GRPC also relies on proto3
  55. The Moral of the Story

  56. Don’t accept the community defaults without checking out the alternatives

  57. @aq quirkey.com github.com/quirkey THANKS