Real time app use case from the PMU

Transcript

1. A real time app use case from PMU Benjamin LE-GALL / Loïc DIVAD Data Engineers @ PMU / Publicis Sapient Engineering
2. @Anxiome @loicmdivad @loicmdivad @Anxiome 2 Benjamin LE GALL Data Engineer PMU Loïc DIVAD Data Engineer Publicis Sapient Engineering
3. @Anxiome @loicmdivad The Numbers 3 Source: PMU.fr
4. @Anxiome @loicmdivad The Channels 4 Horse Betting Poker Sports Betting
5. @Anxiome @loicmdivad The Channels 5 Horse Betting Poker Sports Betting Online Online Online
6. @Anxiome @loicmdivad The Channels 6 Horse Betting Poker Sports Betting Online Online Online App
7. @Anxiome @loicmdivad The Channels 7 Horse Betting Poker Sports Betting Online Online Online App SMS
8. @Anxiome @loicmdivad The Channels 8 Horse Betting Poker Sports Betting Online Online Online App SMS Point of Sales
9. @Anxiome @loicmdivad The Channels 9 Horse Betting Poker Sports Betting Online Point of Sales Online Online App SMS
10. @Anxiome @loicmdivad @loicmdivad @Anxiome “LE PARI MUTUEL” A Digital Business 10
11. @Anxiome @loicmdivad Mutualized Betting ♞ Bettor vs Bettor and not Bettor vs Bookmaker ♞ Ods updated in real time and frozen when the race starts ♞ Most of the bets are submitted in the 5 minutes before the race 11
12. @Anxiome @loicmdivad Mutualized Betting ♞ Bettor vs Bettor and not Bettor vs Bookmaker ♞ Ods updated in real time and stopped when the race starts ♞ Most of the bets submitted in the 5 minutes before the race 12
13. @Anxiome @loicmdivad Anonymous Clients ♞ Several millions of bets every day ♞ Most of those bets are anonymous ♘ We have few information on the clients ♘ We still need to be able to reward them 13
14. @Anxiome @loicmdivad Anonymous Customers ♞ Several millions of bets every day ♞ Most of those bets are anonymous ♘ We have little information on the clients ♘ We still need to be able to reward them ♞ Need for a tool allowing us to reward them despite them being anonymous 14
15. @Anxiome @loicmdivad 15 The overly simplified workflow
16. @Anxiome @loicmdivad @loicmdivad @Anxiome 16 How to reward the most secret customers ever?
17. @Anxiome @loicmdivad @loicmdivad @Anxiome THE CONTEXT A Race against your Legacy 17
18. @Anxiome @loicmdivad DISCLAIMER 18
19. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 19 Point of Sales and Racecourse Terminals
20. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 20 Rabbit MQ Point of Sales and Racecourse Terminals
21. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 21 TIBCO EMS TIBCO BusinessWorks™ Rabbit MQ Point of Sales and Racecourse Terminals
22. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 22 TIBCO EMS TIBCO BusinessWorks™ Rabbit MQ Main Information System Point of Sales and Racecourse Terminals
23. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 23 TIBCO EMS TIBCO BusinessWorks™ Rabbit MQ Main Information System Point of Sales and Racecourse Terminals
24. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 24 TIBCO EMS TIBCO BusinessWorks™ Rabbit MQ Main Information System Development Cost Point of Sales and Racecourse Terminals
25. @Anxiome @loicmdivad ITC Frames and their encoding 25 ♞ Different Types of frames ♘ T001: “Démarrage de la Borne” ♘ T031: “Prise de pari” ♘ T033: “Paiement” ♘ T034: “Annulation de pari” ♞ More than 150 types of frames ♞ With a specific encoding READING EBCDIC COBOL
26. @Anxiome @loicmdivad ITC Frames and their encoding 26 ♞ Different Types of frames ♘ T001: “Démarrage de la Borne” ♘ T031: “Prise de pari” ♘ T033: “Paiement” ♘ T034: “Annulation de pari” ♞ More than 150 types of frames ♞ With a specific encoding READING EBCDIC COBOL 011 101 11100100101010101001010101 ... 010 xN
27. @Anxiome @loicmdivad ITC Frames and their encoding 27 ♞ Protocol understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules
28. @Anxiome @loicmdivad ITC Frames and their encoding 28 Source: Wikipédia, EBCDIC ♞ Protocol understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules # COBOL COPY BOOK 01 Reunion PIC X(10) 05 Course PIC X(10) 05 Pari PIC X(20)
29. @Anxiome @loicmdivad ITC Frames and their encoding 29 Source: Wikipédia, EBCDIC ♞ Protocol understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules Array(0xd4, 0xfc, 0xA1, 0xB4, 0x03, 0x01, 0xa5, ...) Array(0xd2, 0xfe, 0x10, 0x02, 0x03, 0x01, ...) # Test Case Byte Frames
30. @Anxiome @loicmdivad ITC Frames and their encoding 30 Source: Wikipédia, EBCDIC ♞ Protocol understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules Array(0xd4, 0xfc, 0xA1, 0xB4, 0x03, 0x01, 0xa5, ...) Array(0xd2, 0xfe, 0x10, 0x02, 0x03, 0x01, ...) Amount € Nb of horses
31. @Anxiome @loicmdivad 31 The overly simplified workflow
32. @Anxiome @loicmdivad @loicmdivad @Anxiome The answer: Stream Processing Applications 32
33. @Anxiome @loicmdivad @loicmdivad @Anxiome CONFLUENT PLATFORM The right horseshoe, for the right races 33
34. @Anxiome @loicmdivad Apache Kafka 34 ♞ One stream, multiple consumers ♞ The very low latency ♘ Distributed ♞ Open source ♞ The team skills But it has a lot of others properties: ♞ Resilience ♞ Scalability Source: confluent.io
35. @Anxiome @loicmdivad Confluent Platform ♞ Kafka Versions ♘ Kafka Connect 0.9.0 ♘ Kafka Streams 0.10.0 ♞ Schema Registry ♞ Centralise the event-log ♘ To better democratize event streams ♘ To give a more flexible access than the Data Lake ♘ It needs to be on a completely separate platform. 35 Source: confluent.io
36. @Anxiome @loicmdivad Confluent Platform OSS 36 Source: confluent.io ♞ Kafka Versions ♘ Kafka Connect 0.9.0 ♘ Kafka Streams 0.10.0 ♞ Schema Registry ♞ Centralise the event-log ♘ To better democratize event streams ♘ To give a more flexible access than the Data Lake ♘ It needs to be on a completely separate platform.
37. @Anxiome @loicmdivad @loicmdivad @Anxiome STREAM PROCESSING Riding at very low latency 37
38. @Anxiome @loicmdivad Stream Processing 101 38 Kafka Streams
39. @Anxiome @loicmdivad @loicmdivad @Anxiome 39 The Tech: Kafka Streams & friends
40. @Anxiome @loicmdivad @loicmdivad @Anxiome The Push PDV* Project * : “Point of sales” 40
41. @Anxiome @loicmdivad Kafka Connect, a bridge to your new centralized event log 41 ♞ Quickly fork the message queues ♘ With no side effect on the main flow ♞ Distributed and Resilient ♞ Fast and integrated to Kafka ♞ Already has a lot of connectors ♞ Lets you develop a connector for your system if: ♘ It doesn't have one yet ♘ It is proprietary software REQUEST RESPONSE MESSAGE QUEUES KAFKA TOPICS
42. @Anxiome @loicmdivad ITC-DECODER, Continuous decoding with Kafka Streams 42 HEADER DECODING HEADER DECODING CORRELATION FULL DECODING KAFKA STREAMS APPLICATION BETS CANCELLATION REQUEST RESPONSE ♞ We only consider the requests matching a response ♞ The full decoding is very CPU intensive ♞ Data Lake and other consumers have a direct access to the decoded frames ♘ Initially received at Day+1 by the big data team ♞ It produces the stream of bets needed by the reward system
43. @Anxiome @loicmdivad ITC-DECODER, Continuous decoding with Kafka Streams 43 HEADER DECODING HEADER DECODING CORRELATION FULL DECODING KAFKA STREAMS APPLICATION BETS CANCELLATION REQUEST RESPONSE ♞ We only consider the requests matching a response ♞ The full decoding is very CPU intensive ♞ Data Lake and other consumers have a direct access to the decoded frames ♘ Initially received at Day+1 by the big data team ♞ It produces the stream of bets needed by the reward system Data Lake
44. @Anxiome @loicmdivad PUSH-PDV, Marketing campaign rules applied at low latency BETS CANCELED BETS ♞ Loads in promotional coupons from the Data Lake and loads them in a KTable ♞ Loads in a promotional campaign containing the constraints of the campaign (date, bet type …) ♞ Confronts the bets to the campaign and dispatches a coupon to the PDV accordingly ♞ Confronts the cancellations to the rewarded bets to cancel the coupon if the related bet is canceled 44 COUPONS KAFKA STREAMS APPLICATION REWARDS CANCELED COUPONS
45. @Anxiome @loicmdivad 45 DEC DEC CORR DEC Data Lake REQUEST RESPONSE MESSAGE QUEUES BETS CANCELS COUPONS PUSH-PDV REWARDS CANCELS ITC-DECODER Data Lake PUSH-PDV, the global pipeline
46. @Anxiome @loicmdivad 46 The overly simplified workflow
47. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 47 TIBCO EMS TIBCO BusinessWorks™ Rabbit MQ Main Information System Point of Sales and Racecourse Terminals Real time applications
48. @Anxiome @loicmdivad @loicmdivad @Anxiome Challenges and Frustrations 48
49. @Anxiome @loicmdivad The operational nightmare! 49 ♞ Rough tooling ♞ Capacity planning ♞ Configuration jungle ♞ Self made monitoring ♞ Completely non-existent alerting
50. @Anxiome @loicmdivad Keep up with the open source 50 Exception in thread "StreamThread-1" ProcessorStateException: task [3_8] Failed to flush state store PUSH-OFFRE-STORE Caused by: org.apache.kafka.streams.errors.ProcessorStateException: Error while executing flush from store ??? at org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:346) at org.apache.kafka.streams.state.internals.RocksDBStore.flush(RocksDBStore.java:337) at org.apache.kafka.streams.state.internals.MeteredKeyValueStore$6.run(MeteredKeyValueStore.java:92) at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188) at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.flush(MeteredKeyValueStore.java:186) at org.apache.kafka.streams.state.internals.CachingKeyValueStore.flush(CachingKeyValueStore.java:112) at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:323) ... 8 more Caused by: org.rocksdb.RocksDBException: V at org.rocksdb.RocksDB.flush(Native Method) at org.rocksdb.RocksDB.flush(RocksDB.java:1642) at org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:344) ... 14 more
51. @Anxiome @loicmdivad Correlation, a Stream to Stream join ♞ Small joining window ♞ Intensive operation ♞ Causes records lag ♘ Which is fatal to this exercise 51
52. @Anxiome @loicmdivad 52 Processor API, the low level access to Kafka Streams PROCESSOR RocksDB Key-Value Store PROCESSOR
53. @Anxiome @loicmdivad @loicmdivad @Anxiome CONCLUSION The finish line 53
54. @Anxiome @loicmdivad 54 End of the POC Push SMS Push PDV in Production Upgrade kafka 1.0 Push PDV is splitted in 3 apps Mid Dec. 2017 Feb. 2018 Jun. 2017 Apr. 2018 Ops team takes the lead on the kafka cluster Soccer World Cup Jun. 2018 May 2018 The Push PDV Timeline
55. @Anxiome @loicmdivad @loicmdivad @Anxiome THANKS 55
56. @Anxiome @loicmdivad A Real Time App Use Case from PMU “Que les meilleurs stream!” @Anxiome @loicmdivad
57. @Anxiome @loicmdivad @loicmdivad @Anxiome QUESTIONS? 57
58. @Anxiome @loicmdivad PICTURES 58 ♞ Photo by Mathew Schwartz on Unsplash ♞ “PMU Carte Blanche” a collection by Kourtney Roy