New AWS services: Kinesis, CloudTrail

Quick overview of some of new services announced at 2013 aws:reinvent, Kinesis and CloudTrail

Transcript

1. New Services Overview Kinesis, CloudTrail
2. Kinesis service overview
3. What is Kinesis? • Managed service for real-time processing of streaming data. ◦ Think: Streaming/real-time hadoop, map/reduce ◦ Related Software ▪ Kafka ▪ Storm ▪ Spark Streaming
4. Uses Cases • Real-time metrics and analytics ◦ Realtime website traffic reports ◦ Bid optimization for ads ◦ Fraud detection ◦ Stock/financial analytics/decisions • Log/Event processing
5. Kinesis Terminology • Data Record: (seq#, partition key, blob) • Stream: Ordered sequence of data records, distributed into shards.
6. Kinesis Provides • Durable stream storage ◦ replicated 3X • Event ordering • Event partitioning ◦ Each shard has all records with same partition key ◦ One worker per shard ◦ Think: Map/Reduce shuffling
7. Kinesis architecture overview Producer Producer Producer Kinesis App 1 App 2 Dynamo S3 S1 S2 S3 S4 worker1 worker2 worker3 worker4 worker1 worker2 worker3 worker4
8. Kinesis workers • one worker per shard • your own code, running on your own machines (you pick instances) ◦ Can run multiple workers per machine (think of it as a thread) • with client library, designed to work with AutoScaling
9. Kinesis Client Library • Handles shard assignment / re-assignment • Provides checkpointing / resuming progress in event stream • Stores state in Dynamo • Currently java only
10. Kinesis shard assignment Kinesis S1 S2 S3 Machine1 Machine2 Machine3 INFO: Worker 10.0.1.87:fbf0edd8-e6b7-424e-9544-f561774a72cd saw 1 total leases, 1 available leases, 1 workers. Target is 1 leases, I have 0 leases, I will take 1 leases
11. Kinesis checkpointing Kinesis shard1 shard2 Worker Worker Worker
12. IRecordProcessor public void initialize(String shardId) public void shutdown( IRecordProcessorCheckpointer checkpointer, ShutdownReason reason )
13. IRecordProcessor public void processRecords( List<Record> records, IRecordProcessorCheckpointer cp) record.getData() record.getSequenceNumber() record.getPartitionKey()
14. Kinesis Client Dynamo State Table Attribute Example Value leaseKey (Hash Key) "shardId-000000000000" checkpoint "21269322407426698664519200225895317504" leaseCounter 40 leaseOwner "10.0.1.87:59fbf14a-90ae-44d3-ab5e-602e79296874" ownerSwitchesSinceCheckpoint 8 INFO: With failover time 10000ms and epsilon 25ms, LeaseCoordinator will renew leases every 3308ms and take leases every 20050ms
15. Kinesis Integrations • Dynamo • S3 • Redshift • Storm • Spark streaming
16. Shard Limits • Each Shard ◦ ingest ▪ 1MB/second ▪ 1,000 PUT transactions ▪ 50k per PUT ◦ egress ▪ 2MB/second
17. Pricing • $0.015/hour/shard ($0.36/day/shard) • $0.028 per 1,000,000 transactions • Example ◦ 400 gigs/day (10MB/sec) ◦ $4.22/day ($130)
18. Status • Announced at aws:reinvent • Currently in limited Preview ◦ us-east-1 only • GA coming “in single digit months” ◦ all regions
19. Status @ Bizo • Currently prototyping bid optimization system • Would love to share our experience ◦ follows us at http://dev.bizo.com
20. Questions/Discussion? @ogrodnek larry@bizo.com http://dev.bizo.com
21. Amazon CloudTrail service overview
22. What is CloudTrail? • Records AWS API calls and delivers log files to S3 (json). ◦ including API class from Console, SDK, command- line tools, higher-level services • Includes response data for Create, Update, Delete operations (not read) • Includes errors, including auth errors
23. S3 Delivery • Delivered within 15 minutes of activity • Actually partitioned within s3! ◦ /<prefix>/AWSLogs/<AccountID>/CloudTrail/ <region>/<year>/<month>/<day>/<file>.json.gz • SNS notification available on new file deliver
24. Example CloudTrail JSON { "Records": [ { "awsRegion": "us-east-1", " eventName": "DescribeVolumes", " eventSource": "ec2.amazonaws.com", "eventTime": "2013-12-02T00:04:59Z", " eventVersion": "1.0", "requestParameters": { "filterSet": { "items": [ { "name": "tag:application", "valueSet": { "items": [{"value": "app-server"}]} }, ... }, "responseElements": "<responseOmitted>", "sourceIPAddress": "45.121.14.176", "userAgent": "aws-sdk-java/1.6.6 Linux/3.2.0-23-virtual Java_HotSpot(TM)_64-Bit_Server_VM/23.3-b01", "userIdentity": { "accessKeyId": "ABCDEFGHIJKLMNOPQRST", "accountId": "1234567890", "arn": "arn:aws:iam::1234567890:user/app-server-prod", "principalId": "UVWXYZABCDEFHIJK", "type": "IAMUser", "userName": "app-server-prod" }},...
25. Supported Services • us-east-1, us-west-2 only ◦ EC2 ◦ EBS ◦ Redshift ◦ RDS ◦ VPC ◦ IAM ◦ STS
26. Questions/Comments? @ogrodnek larry@bizo.com http://dev.bizo.com