Share
Embed
Transcript
58 slides
-
A real time app use case from PMU Benjamin LE-GALL
/ Loïc DIVAD Data Engineers @ PMU / Publicis Sapient Engineering -
@Anxiome @loicmdivad @loicmdivad @Anxiome 2 Benjamin LE GALL Data Engineer
PMU Loïc DIVAD Data Engineer Publicis Sapient Engineering -
@Anxiome @loicmdivad The Channels 5 Horse Betting Poker Sports Betting
Online Online Online -
@Anxiome @loicmdivad The Channels 6 Horse Betting Poker Sports Betting
Online Online Online App -
@Anxiome @loicmdivad The Channels 7 Horse Betting Poker Sports Betting
Online Online Online App SMS -
@Anxiome @loicmdivad The Channels 8 Horse Betting Poker Sports Betting
Online Online Online App SMS Point of Sales -
@Anxiome @loicmdivad The Channels 9 Horse Betting Poker Sports Betting
Online Point of Sales Online Online App SMS -
@Anxiome @loicmdivad Mutualized Betting ♞ Bettor vs Bettor and not
Bettor vs Bookmaker ♞ Ods updated in real time and frozen when the race starts ♞ Most of the bets are submitted in the 5 minutes before the race 11 -
@Anxiome @loicmdivad Mutualized Betting ♞ Bettor vs Bettor and not
Bettor vs Bookmaker ♞ Ods updated in real time and stopped when the race starts ♞ Most of the bets submitted in the 5 minutes before the race 12 -
@Anxiome @loicmdivad Anonymous Clients ♞ Several millions of bets every
day ♞ Most of those bets are anonymous ♘ We have few information on the clients ♘ We still need to be able to reward them 13 -
@Anxiome @loicmdivad Anonymous Customers ♞ Several millions of bets every
day ♞ Most of those bets are anonymous ♘ We have little information on the clients ♘ We still need to be able to reward them ♞ Need for a tool allowing us to reward them despite them being anonymous 14 -
@Anxiome @loicmdivad @loicmdivad @Anxiome 16 How to reward the most
secret customers ever? -
@Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 19 Point of Sales
and Racecourse Terminals -
@Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 20 Rabbit MQ Point
of Sales and Racecourse Terminals -
@Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 21 TIBCO EMS TIBCO
BusinessWorks™ Rabbit MQ Point of Sales and Racecourse Terminals -
@Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 22 TIBCO EMS TIBCO
BusinessWorks™ Rabbit MQ Main Information System Point of Sales and Racecourse Terminals -
@Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 23 TIBCO EMS TIBCO
BusinessWorks™ Rabbit MQ Main Information System Point of Sales and Racecourse Terminals -
@Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 24 TIBCO EMS TIBCO
BusinessWorks™ Rabbit MQ Main Information System Development Cost Point of Sales and Racecourse Terminals -
@Anxiome @loicmdivad ITC Frames and their encoding 25 ♞ Different
Types of frames ♘ T001: “Démarrage de la Borne” ♘ T031: “Prise de pari” ♘ T033: “Paiement” ♘ T034: “Annulation de pari” ♞ More than 150 types of frames ♞ With a specific encoding READING EBCDIC COBOL -
@Anxiome @loicmdivad ITC Frames and their encoding 26 ♞ Different
Types of frames ♘ T001: “Démarrage de la Borne” ♘ T031: “Prise de pari” ♘ T033: “Paiement” ♘ T034: “Annulation de pari” ♞ More than 150 types of frames ♞ With a specific encoding READING EBCDIC COBOL 011 101 11100100101010101001010101 ... 010 xN -
@Anxiome @loicmdivad ITC Frames and their encoding 27 ♞ Protocol
understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules -
@Anxiome @loicmdivad ITC Frames and their encoding 28 Source: Wikipédia,
EBCDIC ♞ Protocol understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules # COBOL COPY BOOK 01 Reunion PIC X(10) 05 Course PIC X(10) 05 Pari PIC X(20) -
@Anxiome @loicmdivad ITC Frames and their encoding 29 Source: Wikipédia,
EBCDIC ♞ Protocol understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules Array(0xd4, 0xfc, 0xA1, 0xB4, 0x03, 0x01, 0xa5, ...) Array(0xd2, 0xfe, 0x10, 0x02, 0x03, 0x01, ...) # Test Case Byte Frames -
@Anxiome @loicmdivad ITC Frames and their encoding 30 Source: Wikipédia,
EBCDIC ♞ Protocol understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules Array(0xd4, 0xfc, 0xA1, 0xB4, 0x03, 0x01, 0xa5, ...) Array(0xd2, 0xfe, 0x10, 0x02, 0x03, 0x01, ...) Amount € Nb of horses -
@Anxiome @loicmdivad Apache Kafka 34 ♞ One stream, multiple consumers
♞ The very low latency ♘ Distributed ♞ Open source ♞ The team skills But it has a lot of others properties: ♞ Resilience ♞ Scalability Source: confluent.io -
@Anxiome @loicmdivad Confluent Platform ♞ Kafka Versions ♘ Kafka Connect
0.9.0 ♘ Kafka Streams 0.10.0 ♞ Schema Registry ♞ Centralise the event-log ♘ To better democratize event streams ♘ To give a more flexible access than the Data Lake ♘ It needs to be on a completely separate platform. 35 Source: confluent.io -
@Anxiome @loicmdivad Confluent Platform OSS 36 Source: confluent.io ♞ Kafka
Versions ♘ Kafka Connect 0.9.0 ♘ Kafka Streams 0.10.0 ♞ Schema Registry ♞ Centralise the event-log ♘ To better democratize event streams ♘ To give a more flexible access than the Data Lake ♘ It needs to be on a completely separate platform. -
@Anxiome @loicmdivad @loicmdivad @Anxiome The Push PDV* Project * :
“Point of sales” 40 -
@Anxiome @loicmdivad Kafka Connect, a bridge to your new centralized
event log 41 ♞ Quickly fork the message queues ♘ With no side effect on the main flow ♞ Distributed and Resilient ♞ Fast and integrated to Kafka ♞ Already has a lot of connectors ♞ Lets you develop a connector for your system if: ♘ It doesn't have one yet ♘ It is proprietary software REQUEST RESPONSE MESSAGE QUEUES KAFKA TOPICS -
@Anxiome @loicmdivad ITC-DECODER, Continuous decoding with Kafka Streams 42 HEADER
DECODING HEADER DECODING CORRELATION FULL DECODING KAFKA STREAMS APPLICATION BETS CANCELLATION REQUEST RESPONSE ♞ We only consider the requests matching a response ♞ The full decoding is very CPU intensive ♞ Data Lake and other consumers have a direct access to the decoded frames ♘ Initially received at Day+1 by the big data team ♞ It produces the stream of bets needed by the reward system -
@Anxiome @loicmdivad ITC-DECODER, Continuous decoding with Kafka Streams 43 HEADER
DECODING HEADER DECODING CORRELATION FULL DECODING KAFKA STREAMS APPLICATION BETS CANCELLATION REQUEST RESPONSE ♞ We only consider the requests matching a response ♞ The full decoding is very CPU intensive ♞ Data Lake and other consumers have a direct access to the decoded frames ♘ Initially received at Day+1 by the big data team ♞ It produces the stream of bets needed by the reward system Data Lake -
@Anxiome @loicmdivad PUSH-PDV, Marketing campaign rules applied at low latency
BETS CANCELED BETS ♞ Loads in promotional coupons from the Data Lake and loads them in a KTable ♞ Loads in a promotional campaign containing the constraints of the campaign (date, bet type …) ♞ Confronts the bets to the campaign and dispatches a coupon to the PDV accordingly ♞ Confronts the cancellations to the rewarded bets to cancel the coupon if the related bet is canceled 44 COUPONS KAFKA STREAMS APPLICATION REWARDS CANCELED COUPONS -
@Anxiome @loicmdivad 45 DEC DEC CORR DEC Data Lake REQUEST
RESPONSE MESSAGE QUEUES BETS CANCELS COUPONS PUSH-PDV REWARDS CANCELS ITC-DECODER Data Lake PUSH-PDV, the global pipeline -
@Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 47 TIBCO EMS TIBCO
BusinessWorks™ Rabbit MQ Main Information System Point of Sales and Racecourse Terminals Real time applications -
@Anxiome @loicmdivad The operational nightmare! 49 ♞ Rough tooling ♞
Capacity planning ♞ Configuration jungle ♞ Self made monitoring ♞ Completely non-existent alerting -
@Anxiome @loicmdivad Keep up with the open source 50 Exception
in thread "StreamThread-1" ProcessorStateException: task [3_8] Failed to flush state store PUSH-OFFRE-STORE Caused by: org.apache.kafka.streams.errors.ProcessorStateException: Error while executing flush from store ??? at org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:346) at org.apache.kafka.streams.state.internals.RocksDBStore.flush(RocksDBStore.java:337) at org.apache.kafka.streams.state.internals.MeteredKeyValueStore$6.run(MeteredKeyValueStore.java:92) at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188) at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.flush(MeteredKeyValueStore.java:186) at org.apache.kafka.streams.state.internals.CachingKeyValueStore.flush(CachingKeyValueStore.java:112) at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:323) ... 8 more Caused by: org.rocksdb.RocksDBException: V at org.rocksdb.RocksDB.flush(Native Method) at org.rocksdb.RocksDB.flush(RocksDB.java:1642) at org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:344) ... 14 more -
@Anxiome @loicmdivad Correlation, a Stream to Stream join ♞ Small
joining window ♞ Intensive operation ♞ Causes records lag ♘ Which is fatal to this exercise 51 -
@Anxiome @loicmdivad 52 Processor API, the low level access to
Kafka Streams PROCESSOR RocksDB Key-Value Store PROCESSOR -
@Anxiome @loicmdivad 54 End of the POC Push SMS Push
PDV in Production Upgrade kafka 1.0 Push PDV is splitted in 3 apps Mid Dec. 2017 Feb. 2018 Jun. 2017 Apr. 2018 Ops team takes the lead on the kafka cluster Soccer World Cup Jun. 2018 May 2018 The Push PDV Timeline -
@Anxiome @loicmdivad A Real Time App Use Case from PMU
“Que les meilleurs stream!” @Anxiome @loicmdivad -
@Anxiome @loicmdivad PICTURES 58 ♞ Photo by Mathew Schwartz on
Unsplash ♞ “PMU Carte Blanche” a collection by Kourtney Roy