Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data Analytics with Amazon Kinesis

Data Analytics with Amazon Kinesis

F741e7e29aa3a4476d48a64ac2a73874?s=128

Suman Debnath

January 28, 2020
Tweet

Transcript

  1. © 2020, Amazon Web Services, Inc. or its Affiliates. Suman

    Debnath Principal Developer Advocate Amazon Web Services Data Analytics with Amazon Kinesis
  2. © 2020, Amazon Web Services, Inc. or its Affiliates.

  3. © 2020, Amazon Web Services, Inc. or its Affiliates.

  4. © 2020, Amazon Web Services, Inc. or its Affiliates.

  5. © 2020, Amazon Web Services, Inc. or its Affiliates.

  6. © 2020, Amazon Web Services, Inc. or its Affiliates.

  7. © 2020, Amazon Web Services, Inc. or its Affiliates. What

    is streaming data? Typical characteristics Low-latency Continuous Ordered, incremental High volume
  8. © 2020, Amazon Web Services, Inc. or its Affiliates. Why

    streaming data? Get actionable insights quickly Source: Perishable insights, Mike Gualtieri, Forrester Real time Seconds Minutes Hours Days Months Value of data to decision-making Preventive/Predictive Actionable Reactive Historical Time critical decisions Traditional “batch” business intelligence Information half-life in decision-making
  9. © 2020, Amazon Web Services, Inc. or its Affiliates. real-time

  10. © 2020, Amazon Web Services, Inc. or its Affiliates.

  11. © 2020, Amazon Web Services, Inc. or its Affiliates. Amazon

    Kinesis • Kinesis is a managed alternative to Apache Kafka • Great for application logs, metrics, IoT, clickstreams • Great for “real-time” big data • Great for streaming processing frameworks (Spark, NiFi, etc...) • Data is automatically replicated synchronously to 3 AZ Amazon Kinesis Data Streams Amazon Kinesis Data Firehose Amazon Kinesis Data Analytics Amazon Kinesis Video Streams
  12. © 2020, Amazon Web Services, Inc. or its Affiliates. Amazon

    Kinesis Amazon Kinesis Amazon Kinesis Data Streams Amazon Kinesis Data Analytics Amazon Kinesis Data Firehose Amazon S3 Amazon Redshift Amazon Elasticsearch Service
  13. © 2020, Amazon Web Services, Inc. or its Affiliates. Kinesis

    Streams Overview • Streams are divided in ordered Shards/Partitions • Data retention is 24 hours by default, can go up to 7 days • Ability to reprocess / replay data • Multiple applications can consume the same stream • Once data is inserted in Kinesis, it can’t be deleted (immutability) Shard 1 Shard 2 Shard n Consumer Producer Up to 1 MB or 1000 records per second, per shard Up to 2MB per second, per shard
  14. © 2020, Amazon Web Services, Inc. or its Affiliates. Kinesis

    Streams Shards • One stream is made of many different shards • Billing is per shard provisioned, can have as many shards as you want • Batching available or per message calls. • The number of shards can evolve over time (reshard / merge) • Records are ordered per shard Shard 1 Shard 2 Shard n Consumer Producer
  15. © 2020, Amazon Web Services, Inc. or its Affiliates. Kinesis

    Streams Records • Data Blob – • Data being sent, serialized as bytes. Up to 1 MB. Can represent anything • Record Key – • Sent alongside a record, helps to group records in Shards. Same Key = Same Shard • Use a highly distributed key to avoid the “hot partition” • Sequence Number – • Unique identifier for each records put in shards. Added by Kinesis after ingestion Data Blob (up to 1MB) Bytes Record Key Record Key
  16. © 2020, Amazon Web Services, Inc. or its Affiliates. Kinesis

    Streams Records Shard A Shard B Shard N
  17. © 2020, Amazon Web Services, Inc. or its Affiliates. Kinesis

    Producers AWS SDK Kinesis Producer Library Kinesis Agent Amazon Kinesis Data Stream
  18. © 2020, Amazon Web Services, Inc. or its Affiliates. Kinesis

    Consumers Amazon Kinesis Data Stream AWS Lambda Amazon Kinesis Data Firehose AWS SDK Kinesis Producer Library Kinesis Agent
  19. © 2020, Amazon Web Services, Inc. or its Affiliates. Kinesis

    Data Analytics
  20. © 2020, Amazon Web Services, Inc. or its Affiliates. Demo

    Transaction Rate Alarm
  21. © 2020, Amazon Web Services, Inc. or its Affiliates. Transaction

    Rate Alarm
  22. © 2020, Amazon Web Services, Inc. or its Affiliates. http://ratings.go-aws.com/rate/89

  23. © 2020, Amazon Web Services, Inc. or its Affiliates. Stay

    Connected … /suman-d /_sumand Stay in touch …
  24. © 2020, Amazon Web Services, Inc. or its Affiliates. Thank

    you