RSocket как wire протокол

Многие из нас сталкивались с проблемой того, как подружить разные источники данных; многие ощущали боль конфигураций соединений и connection-pool для разных баз и не только; и всем, конечно же, хотелось чтобы большинство этих вещей было "вшито" в то решение, с которым мы работаем...

А можно ли решить все эти проблемы при помощи протокола? В данном докладе мы ознакомимся с экспериментами в области применения RSocket как wire protocol для разных баз и не только. На примере взаимодействия разных баз данных, к примеру, с Kafka, мы увидим насколько проще становится жизнь, если все они общаются по одному принципу

Transcript

1. as wire Protocol? RSocket
2. • Software Engineer focused on distributed systems development, adopting Reactive Manifesto and Reactive Programming techniques • Open source geek, active contributor of Project Reactor / RSocket • Author of the book "Reactive Programming in Spring 5” • Achieved 4-times better performance by tuning Reactor for RSocket Project About Me @OlehDokuka
3. Safe Harbor Everything you will hear is not related to any upcoming futures of storage protocols Everything that I’m going to say is nothing more than experiments or my OWN vision
4. Today’s Outline Wire Protocols Problematics RSocket
5. What is wire protocol? so
6. the mechanism for transmitting data from point a to point b
7. Existing Wire Protocol • MongoDb Wire Protocol • Postgresql Wire Protocol • Kafka Wire Protocol • … • …
8. WHY? Everyone did that
9. Reasons • Historically happened (in case of SQL Datas stores) • Kafka: HTTP libraries in many languages to be surprisingly shabby • Because I can (MongoDB)
10. MongoDB Wire Protocol
11. Mongo: Handshake Authentication Client Server TCP Error Success
12. Mongo: Messaging Message Client Server TCP Message
13. is a simple request-response style protocol MongoDB Wire Protocol
14. Mongo: Message Format 0 0 1 1 0 1 0 1 1 0 1 0 1 0 0 1 0 0 0 1 1 0 1 1 1 1 1 1 0 1 0 1 1 0 0 0 Logical Message Headers Message struct MsgHeader { int32 messageLength; int32 requestID; int32 responseTo; 
 int32 opCode; } struct MsgHeader { int32 messageLength; int32 requestID; int32 responseTo; 
 int32 opCode; } struct MsgHeader { int32 messageLength; int32 requestID; int32 responseTo; 
 int32 opCode; }
15. Mongo: Multiplexing Client Server TCP Message Message Message Message Message Message
16. Mongo: Message Types struct OP_QUERY { MsgHeader header; int32 flags; cstring fullCollectionName ; // "dbname.collectionname" int32 numberToSkip; // number of documents to skip int32 numberToReturn; // number of documents to return // in the first OP_REPLY batch document query; // query object. See below for details. [ document returnFieldsSelector; ] } struct OP_QUERY { MsgHeader header; int32 flags; cstring fullCollectionName ; // "dbname.collectionname" int32 numberToSkip; // number of documents to skip int32 numberToReturn; // number of documents to return // in the first OP_REPLY batch document query; // query object. See below for details. [ document returnFieldsSelector; ] } struct OP_QUERY { MsgHeader header; int32 flags; cstring fullCollectionName ; // "dbname.collectionname" int32 numberToSkip; // number of documents to skip int32 numberToReturn; // number of documents to return // in the first OP_REPLY batch document query; // query object. See below for details. [ document returnFieldsSelector; ] } struct OP_GET_MORE { MsgHeader header; int32 ZERO; cstring fullCollectionName; int32 numberToReturn; int64 cursorID; } struct OP_GET_MORE { MsgHeader header; int32 ZERO; cstring fullCollectionName; int32 numberToReturn; int64 cursorID; }
17. Mongo: Backpressure by Fetching Client Server TCP Cursor Query More More 2 1 3
18. Mongo 4.+: Message Types struct OP_MSG { MsgHeader header; uint32 flagBits; Sections[] sections; optional<uint32> checksum; } struct OP_MSG { MsgHeader header; uint32 flagBits; Sections[] sections; optional<uint32> checksum; }
19. Mongo: Streaming Client Server TCP Cursor Query 2 1 3
20. Mongo: Message Cancelling struct OP_KILL_CURSORS { MsgHeader header; int32 ZERO; int32 numberOfCursorIDs; int64* cursorIDs; }
21. is a simple request-response style protocol MongoDB Wire Protocol is a simple request-response style protocol is a simple request-response style protocol
22. Kafka Wire Protocol
23. Kafka: Message Types Request Header v0 => request_api_key request_api_version correlation_id request_api_key => INT16 request_api_version => INT16 correlation_id => INT32 Fetch Request (Version: 0) => replica_id max_wait_time min_bytes [topics] … … Fetch Response (Version: 0) => [responses] responses => topic [partition_responses] topic => STRING partition_responses => partition_header record_set partition_header => partition error_code high_watermark partition => INT32 error_code => INT16 high_watermark => INT64 record_set => RECORDS
24. Kafka: Throttling Mechanism Produce Request (Version: 0) => acks timeout [topic_data] acks => INT16 timeout => INT32 topic_data => topic [data] topic => STRING data => partition record_set partition => INT32 record_set => RECORDS Produce Response (Version: 1) => [responses] throttle_time_ms responses => topic [partition_responses] topic => STRING partition_responses => partition error_code base_offset partition => INT32 error_code => INT16 base_offset => INT64 throttle_time_ms => INT32 Produce Response (Version: 1) => [responses] throttle_time_ms responses => topic [partition_responses] topic => STRING partition_responses => partition error_code base_offset partition => INT32 error_code => INT16 base_offset => INT64 throttle_time_ms => INT32
25. Kafka: Some downsides
26. is a complex protocol Kafka Wire Protocol is a complex protocol
27. PostgreSql Wire Protocol
28. Postgresql: Summary • No multiplexing • Similar mechanisms for messaging as in Kafka and Mongo • It is own message binary format
29. is a simple request-response style protocol Postgresql Wire Protocol
30. Problems Summary • Wire Protocols are VERY Similar • Most of them are not ready to proper streaming and other todays requirements
31. In Vacuum Wire Protocol
32. Everyone Do: Handshake Storage Selection Authentication Client Server TCP
33. Everyone Do: Data Exchange Command Client Server Result TCP
34. Everyone Do: Client <- Server Backpressure Command Client Server Result TCP More Result More Result
35. Every Wants: Client -> Server Backpressure Client Server TCP Throttling
36. Some Do: Multiplexing Client Server TCP
37. Every Wish: Streaming Client Server TCP Query 2 1 3
38. Every Wish: Transport Variety Client Server TCP / UDP / QUIC / WebSocket
39. Problems Summary • Wire Protocols are VERY Similar • Most of them are not ready to proper streaming and other todays requirements • Most-likely all of them will reinvent the same again in the future
40. What if everyone use
41. What is • Open Source Layer 5/6 communication protocol • Reactive streams semantics • Application-level flow control • Supports both RPC and event-based messaging • Up to 10x faster than HTTP/1.1, using 90% less resources Built by leaders in microservices and cloud computing
42. 42 Binary Messaging
43. 43 Binary Messaging RSocket RSocket Connection
44. 44 Binary Messaging RSocket RSocket Connection
45. 45 Binary Messaging RSocket RSocket Connection
46. Binary Messaging 0 0 1 1 0 1 0 1 1 0 1 0 1 0 0 1 0 0 0 1 1 0 1 1 1 1 1 1 0 1 0 1 1 0 0 0 Logical Message Metadata Data • Data - allows any message format (including custom binary protocol) with no size limitations • Metadata - stands for headers / meta- info • Composite Metadata Extension - provides a way to provide Tracing / Monitoring / Routing / MimeType / Anything Cloud Applications needs today
47. 47 Performance via Multiplexing
48. STREAM BLUE STREAM YELLOW STREAM RED 48 Project Reactor / RxJava / Any RS Performance via Multiplexing
49. 49 WebSocket TCP QUIC Aeron Flexibility via Transport Agnostic • Message’s Framing / Logical Streams identity / all infrastructural information are provided on top if necessary • User may use any Reliable Transport
50. 50 10 9 Resilience via Reactive Streams Spec
51. 51 Resilience via Leasing STREAM BLUE STREAM YELLOW STREAM RED STREAM BLUE STREAM YELLOW STREAM RED Requester Requester STREAM BLUE STREAM YELLOW STREAM RED Responder Requester
52. 52 Resilience via Leasing Responder Requester Requests Count : 10 
 Duration : next 10 secs STREAM BLUE STREAM YELLOW STREAM RED Requests Count : 9 
 Duration : next 10 secs Requests Count : 8 
 Duration : next 10 secs Requests Count : 7 
 Duration : next 10 secs Requests Count : 7 
 Duration : next 9 secs Requests Count : 7 
 Duration : next 8 secs Requests Count : 7 
 Duration : next 7 secs Requests Count : 7 
 Duration : next 0 secs Requests Count : 10 
 Duration : next 10 secs
53. Resilience Advanced • Server sales capacity to its clients • Natural Rate Limiting • Built-in Circuit Breaker
54. • Request - Response • Request - Stream • Request - Channel 
 (aka Stream - Stream)
 
 • Fire - and - Forget 
 (flush out and that is it) Wide Communication Patterns
55. Wide Communication Patterns • Clients is required to establish connection • Real Peer-To-Peer • Both sides are equal and can or cannot implements mentioned communication >>> CLIENT REQUEST STREAM >>> <<< SERVER REQUEST STREAM <<< SERVER CLIENT PEER PEER
57. Summary • Using RSocket you do not have to reinvent wheel • Interoperability between RSocket based protocols
58. So, if everyone use
59. Client Server RSocket Message Message Message Message Message 2 1 3 reqiest(5) 2 1 3
60. Wait I do the same in my web 
 application every day
61. We all are Building wire protocols
63. You can use RSocket too And yes!
64. 64 User Experience • RSocket-RPC (via Protobuf Generators) for all supported languages • RSocket-IPC (plain inter-process communication) • RSocket-GraphQL • Spring-Boot-Starter-RSocket (Spring-Messaging integration)
65. Service A ??? RSocket Setup Frame Metadata Push
66. Introducing Broker Specification + Broker Spec
67. Computer A Computer B Application Transport Internet Internet Internet Internet Network Access Network Access Network Access Network Access Router #1 Router #2 Application Transport Components of TCP/IP Communication
68. API Abstraction Client Library API Abstraction Client Library Service A Service B Broker #1 Broker #2 Components of RSocket Communication Netifi Broker Netifi Broker
69. But this future we overview in another speech
70. Summary • RSocket is a powerful binary Protocol • In can Simplify communication between A and B • All the date stores just expose its own API • ALL the database vendors can benefit from using RSocket
71. What to do next? • http://rsocket.io - official web-page • https://bit.ly/2Fku9VC - video channel • https://community.reactive.foundation/ - community forum @OlehDokuka