Real time app use case from the PMU

Ed81876bf33da90cdae47ce9b8df056b?s=47 Loïc DIVAD
November 14, 2019

Real time app use case from the PMU

Ed81876bf33da90cdae47ce9b8df056b?s=128

Loïc DIVAD

November 14, 2019
Tweet

Transcript

  1. A real time app use case from PMU Benjamin LE-GALL

    / Loïc DIVAD Data Engineers @ PMU / Publicis Sapient Engineering
  2. @Anxiome @loicmdivad @loicmdivad @Anxiome 2 Benjamin LE GALL Data Engineer

    PMU Loïc DIVAD Data Engineer Publicis Sapient Engineering
  3. @Anxiome @loicmdivad The Numbers 3 Source: PMU.fr

  4. @Anxiome @loicmdivad The Channels 4 Horse Betting Poker Sports Betting

  5. @Anxiome @loicmdivad The Channels 5 Horse Betting Poker Sports Betting

    Online Online Online
  6. @Anxiome @loicmdivad The Channels 6 Horse Betting Poker Sports Betting

    Online Online Online App
  7. @Anxiome @loicmdivad The Channels 7 Horse Betting Poker Sports Betting

    Online Online Online App SMS
  8. @Anxiome @loicmdivad The Channels 8 Horse Betting Poker Sports Betting

    Online Online Online App SMS Point of Sales
  9. @Anxiome @loicmdivad The Channels 9 Horse Betting Poker Sports Betting

    Online Point of Sales Online Online App SMS
  10. @Anxiome @loicmdivad @loicmdivad @Anxiome “LE PARI MUTUEL” A Digital Business

    10
  11. @Anxiome @loicmdivad Mutualized Betting ♞ Bettor vs Bettor and not

    Bettor vs Bookmaker ♞ Ods updated in real time and frozen when the race starts ♞ Most of the bets are submitted in the 5 minutes before the race 11
  12. @Anxiome @loicmdivad Mutualized Betting ♞ Bettor vs Bettor and not

    Bettor vs Bookmaker ♞ Ods updated in real time and stopped when the race starts ♞ Most of the bets submitted in the 5 minutes before the race 12
  13. @Anxiome @loicmdivad Anonymous Clients ♞ Several millions of bets every

    day ♞ Most of those bets are anonymous ♘ We have few information on the clients ♘ We still need to be able to reward them 13
  14. @Anxiome @loicmdivad Anonymous Customers ♞ Several millions of bets every

    day ♞ Most of those bets are anonymous ♘ We have little information on the clients ♘ We still need to be able to reward them ♞ Need for a tool allowing us to reward them despite them being anonymous 14
  15. @Anxiome @loicmdivad 15 The overly simplified workflow

  16. @Anxiome @loicmdivad @loicmdivad @Anxiome 16 How to reward the most

    secret customers ever?
  17. @Anxiome @loicmdivad @loicmdivad @Anxiome THE CONTEXT A Race against your

    Legacy 17
  18. @Anxiome @loicmdivad DISCLAIMER 18

  19. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 19 Point of Sales

    and Racecourse Terminals
  20. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 20 Rabbit MQ Point

    of Sales and Racecourse Terminals
  21. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 21 TIBCO EMS TIBCO

    BusinessWorks™ Rabbit MQ Point of Sales and Racecourse Terminals
  22. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 22 TIBCO EMS TIBCO

    BusinessWorks™ Rabbit MQ Main Information System Point of Sales and Racecourse Terminals
  23. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 23 TIBCO EMS TIBCO

    BusinessWorks™ Rabbit MQ Main Information System Point of Sales and Racecourse Terminals
  24. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 24 TIBCO EMS TIBCO

    BusinessWorks™ Rabbit MQ Main Information System Development Cost Point of Sales and Racecourse Terminals
  25. @Anxiome @loicmdivad ITC Frames and their encoding 25 ♞ Different

    Types of frames ♘ T001: “Démarrage de la Borne” ♘ T031: “Prise de pari” ♘ T033: “Paiement” ♘ T034: “Annulation de pari” ♞ More than 150 types of frames ♞ With a specific encoding READING EBCDIC COBOL
  26. @Anxiome @loicmdivad ITC Frames and their encoding 26 ♞ Different

    Types of frames ♘ T001: “Démarrage de la Borne” ♘ T031: “Prise de pari” ♘ T033: “Paiement” ♘ T034: “Annulation de pari” ♞ More than 150 types of frames ♞ With a specific encoding READING EBCDIC COBOL 011 101 11100100101010101001010101 ... 010 xN
  27. @Anxiome @loicmdivad ITC Frames and their encoding 27 ♞ Protocol

    understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules
  28. @Anxiome @loicmdivad ITC Frames and their encoding 28 Source: Wikipédia,

    EBCDIC ♞ Protocol understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules # COBOL COPY BOOK 01 Reunion PIC X(10) 05 Course PIC X(10) 05 Pari PIC X(20)
  29. @Anxiome @loicmdivad ITC Frames and their encoding 29 Source: Wikipédia,

    EBCDIC ♞ Protocol understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules Array(0xd4, 0xfc, 0xA1, 0xB4, 0x03, 0x01, 0xa5, ...) Array(0xd2, 0xfe, 0x10, 0x02, 0x03, 0x01, ...) # Test Case Byte Frames
  30. @Anxiome @loicmdivad ITC Frames and their encoding 30 Source: Wikipédia,

    EBCDIC ♞ Protocol understood by the previous mainframe ♞ A way to transfer the informations faster ♞ Additional compression rules Array(0xd4, 0xfc, 0xA1, 0xB4, 0x03, 0x01, 0xa5, ...) Array(0xd2, 0xfe, 0x10, 0x02, 0x03, 0x01, ...) Amount € Nb of horses
  31. @Anxiome @loicmdivad 31 The overly simplified workflow

  32. @Anxiome @loicmdivad @loicmdivad @Anxiome The answer: Stream Processing Applications 32

  33. @Anxiome @loicmdivad @loicmdivad @Anxiome CONFLUENT PLATFORM The right horseshoe, for

    the right races 33
  34. @Anxiome @loicmdivad Apache Kafka 34 ♞ One stream, multiple consumers

    ♞ The very low latency ♘ Distributed ♞ Open source ♞ The team skills But it has a lot of others properties: ♞ Resilience ♞ Scalability Source: confluent.io
  35. @Anxiome @loicmdivad Confluent Platform ♞ Kafka Versions ♘ Kafka Connect

    0.9.0 ♘ Kafka Streams 0.10.0 ♞ Schema Registry ♞ Centralise the event-log ♘ To better democratize event streams ♘ To give a more flexible access than the Data Lake ♘ It needs to be on a completely separate platform. 35 Source: confluent.io
  36. @Anxiome @loicmdivad Confluent Platform OSS 36 Source: confluent.io ♞ Kafka

    Versions ♘ Kafka Connect 0.9.0 ♘ Kafka Streams 0.10.0 ♞ Schema Registry ♞ Centralise the event-log ♘ To better democratize event streams ♘ To give a more flexible access than the Data Lake ♘ It needs to be on a completely separate platform.
  37. @Anxiome @loicmdivad @loicmdivad @Anxiome STREAM PROCESSING Riding at very low

    latency 37
  38. @Anxiome @loicmdivad Stream Processing 101 38 Kafka Streams

  39. @Anxiome @loicmdivad @loicmdivad @Anxiome 39 The Tech: Kafka Streams &

    friends
  40. @Anxiome @loicmdivad @loicmdivad @Anxiome The Push PDV* Project * :

    “Point of sales” 40
  41. @Anxiome @loicmdivad Kafka Connect, a bridge to your new centralized

    event log 41 ♞ Quickly fork the message queues ♘ With no side effect on the main flow ♞ Distributed and Resilient ♞ Fast and integrated to Kafka ♞ Already has a lot of connectors ♞ Lets you develop a connector for your system if: ♘ It doesn't have one yet ♘ It is proprietary software REQUEST RESPONSE MESSAGE QUEUES KAFKA TOPICS
  42. @Anxiome @loicmdivad ITC-DECODER, Continuous decoding with Kafka Streams 42 HEADER

    DECODING HEADER DECODING CORRELATION FULL DECODING KAFKA STREAMS APPLICATION BETS CANCELLATION REQUEST RESPONSE ♞ We only consider the requests matching a response ♞ The full decoding is very CPU intensive ♞ Data Lake and other consumers have a direct access to the decoded frames ♘ Initially received at Day+1 by the big data team ♞ It produces the stream of bets needed by the reward system
  43. @Anxiome @loicmdivad ITC-DECODER, Continuous decoding with Kafka Streams 43 HEADER

    DECODING HEADER DECODING CORRELATION FULL DECODING KAFKA STREAMS APPLICATION BETS CANCELLATION REQUEST RESPONSE ♞ We only consider the requests matching a response ♞ The full decoding is very CPU intensive ♞ Data Lake and other consumers have a direct access to the decoded frames ♘ Initially received at Day+1 by the big data team ♞ It produces the stream of bets needed by the reward system Data Lake
  44. @Anxiome @loicmdivad PUSH-PDV, Marketing campaign rules applied at low latency

    BETS CANCELED BETS ♞ Loads in promotional coupons from the Data Lake and loads them in a KTable ♞ Loads in a promotional campaign containing the constraints of the campaign (date, bet type …) ♞ Confronts the bets to the campaign and dispatches a coupon to the PDV accordingly ♞ Confronts the cancellations to the rewarded bets to cancel the coupon if the related bet is canceled 44 COUPONS KAFKA STREAMS APPLICATION REWARDS CANCELED COUPONS
  45. @Anxiome @loicmdivad 45 DEC DEC CORR DEC Data Lake REQUEST

    RESPONSE MESSAGE QUEUES BETS CANCELS COUPONS PUSH-PDV REWARDS CANCELS ITC-DECODER Data Lake PUSH-PDV, the global pipeline
  46. @Anxiome @loicmdivad 46 The overly simplified workflow

  47. @Anxiome @loicmdivad THE HOLLY BETTING HIGHWAY 47 TIBCO EMS TIBCO

    BusinessWorks™ Rabbit MQ Main Information System Point of Sales and Racecourse Terminals Real time applications
  48. @Anxiome @loicmdivad @loicmdivad @Anxiome Challenges and Frustrations 48

  49. @Anxiome @loicmdivad The operational nightmare! 49 ♞ Rough tooling ♞

    Capacity planning ♞ Configuration jungle ♞ Self made monitoring ♞ Completely non-existent alerting
  50. @Anxiome @loicmdivad Keep up with the open source 50 Exception

    in thread "StreamThread-1" ProcessorStateException: task [3_8] Failed to flush state store PUSH-OFFRE-STORE Caused by: org.apache.kafka.streams.errors.ProcessorStateException: Error while executing flush from store ??? at org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:346) at org.apache.kafka.streams.state.internals.RocksDBStore.flush(RocksDBStore.java:337) at org.apache.kafka.streams.state.internals.MeteredKeyValueStore$6.run(MeteredKeyValueStore.java:92) at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188) at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.flush(MeteredKeyValueStore.java:186) at org.apache.kafka.streams.state.internals.CachingKeyValueStore.flush(CachingKeyValueStore.java:112) at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:323) ... 8 more Caused by: org.rocksdb.RocksDBException: V at org.rocksdb.RocksDB.flush(Native Method) at org.rocksdb.RocksDB.flush(RocksDB.java:1642) at org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:344) ... 14 more
  51. @Anxiome @loicmdivad Correlation, a Stream to Stream join ♞ Small

    joining window ♞ Intensive operation ♞ Causes records lag ♘ Which is fatal to this exercise 51
  52. @Anxiome @loicmdivad 52 Processor API, the low level access to

    Kafka Streams PROCESSOR RocksDB Key-Value Store PROCESSOR
  53. @Anxiome @loicmdivad @loicmdivad @Anxiome CONCLUSION The finish line 53

  54. @Anxiome @loicmdivad 54 End of the POC Push SMS Push

    PDV in Production Upgrade kafka 1.0 Push PDV is splitted in 3 apps Mid Dec. 2017 Feb. 2018 Jun. 2017 Apr. 2018 Ops team takes the lead on the kafka cluster Soccer World Cup Jun. 2018 May 2018 The Push PDV Timeline
  55. @Anxiome @loicmdivad @loicmdivad @Anxiome THANKS 55

  56. @Anxiome @loicmdivad A Real Time App Use Case from PMU

    “Que les meilleurs stream!” @Anxiome @loicmdivad
  57. @Anxiome @loicmdivad @loicmdivad @Anxiome QUESTIONS? 57

  58. @Anxiome @loicmdivad PICTURES 58 ♞ Photo by Mathew Schwartz on

    Unsplash ♞ “PMU Carte Blanche” a collection by Kourtney Roy