Building Reactive Pipelines with AWS

Almost six months ago, our team started the journey to replicate some of our data stored in on-premise MySQL machines to AWS. This included over a billion records stored in multiple tables. This is our story!

Transcript

1. Building Reactive Pipelines with AWS
2. Hello! Nikhil Kuriakose @nikhilkuria Hotel Search
3. Will you book a hotel without images?
4. nope!!
5. trivago DB
6. Locked in!
10. 1 2 3 ID Name Age 101 Bob 54 101 Bob 55 101 Bobby 54 101 Bobby 54
14. • Delivery of data as streams • Easy data manipulation • Running data based tests
16. Sink
17. • Transform to a better schema • Preprocessing • Validation and Transformation
18. Kinesis
20. • Kafka Streams • Source of Truth and Publisher
23. • Kinesis and Lambdas • RDS Aurora instances • EC2 instances • Cloudwatch metrics • Alarms • Slack notifications
24. Such an unhappy sight !!
25. Streaming changes into Kinesis
26. feedback Thank you!!