Image Area What is “streaming” ? Definition: - a type of data processing engine that is designed with infinite data sets in mind. (https://www.oreilly.com/ideas/the-world-bey ond-batch-streaming-101)
“Batch” vs “Streaming” - Batch Processing - Process grouped logs at once, and process occasionaly - Streaming Processing - Micro-batch - ex) AWS Lambda with “Batch Size = 2 ~ n” (n is a not-too-large natural number.) - Real Streaming - ex) AWS Lambda with “Batch Size = 1”
Image Area - Event time, which is the time at which events actually occurred. - Processing time, which is the time at which events are observed in the system. Figure: Example time domain mapping from https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 “Event Time” vs “Processing Time”
Kappa Architecture - introduced by Jay Kreps, a co-founder and CEO at Confluent which was acquired by LinkedIn - “Questioning the Lambda Architecture” https://www.oreilly.com/ideas/questioning-the-lambda-architecture - Streaming Layer + Serving Layer
Lambda vs Kappa Architecture Pros Cons Lambda Architecture - Robust to data consistency - Harder to maintain multiple layers Kappa Architecture - Simple implementation - Need extra works to guarantee data onsistency
Apache Hadoop/Storm/Spark/Kafka/Flink Name Speciality Batch Processing Stream Processing Hadoop HDFS(FIle System), YARN, MapReduce O X Storm Topology (Spout + Bolt), Tuple, Task X O Kafka Broker, Producer/Consumer (“Un-Managed Kinesis” ?) X O Spark inspired by Hadoop’s MapReduce engine O O (Spark Streaming) Flink Batch and Streaming in One System, ML Support, DataStram API O O
How to tackle on “Late Logs”? - Discarding - simply discard late logs - Watermarks - “all input data with event times less than X have been observed.” - Triggers - declaring when the output for a window should be materialized - Accumulation - accumulate multiple results that are observed for the same window Read https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102 for more details :)
What is “Watermark” Definition: a special mark contained in electronic documents, pictures, music etc that is used to stop people from copying them Example: audio watermark detection, photo watermark, copyright watermark
How to “Watermark” - A. Save State in any external memory - e.g. RDS, DynamoDB - B. Calculate on-memory - formula … - e.g. late threshold = 10 minutes - use `median(event time)` instead to handle the too-future logs - e.g. mobile devices’ system clock are somehow modified by users incorrectly watermark = max(event time) - late_threshold
Overall Data-flow - Kinesis Stream receives logs from Android clients directly via kinesis:PutRecord - once per 90 sec, from at most 15,000 devices - Lambda polls Kinesis and aggregate as impression by increment - DynamoDB stores incremented records with UpdateItem (ADD) operation - another Lambda(s) polls DynamoDB Streams and aggregate hourly/daily
Aggregation - Aggregate gradually per minute -> per 10 min -> hourly -> daily - Partition usual logs/lagged logs, and update records to the separate tables - NOT discarding (for now) to see how many logs will be discarded - determine whether logs’ timestamp is behind the watermark or not - watermark … the median of all timestamps - because users can change system clocks to the future
Watermark Implementation func (sr *StreamRecords) watermark(eventTimes []EventTime) (median EventTime) { sort.Ints(eventTimes) // 1. sort l := len(eventTimes) // 2. get the median if l%2 == 0 { // when even median = Mean(eventTimes[l/2-1 : l/2 + 1]) } else { // when odd median = EventTime(eventTimes[l/2]) } return median }
Image Area What to “monitor”? The famous Google’s “SRE” book says in Chapter 6 “The Four Golden Signals” section: - Latency - Traffix - Errors - Saturation Monitoring - The Four Golden Signals
Image Area - Latency - How long does it take? - Traffix - How many Get/PutRecords? - Errors - Availability? - How many errors occur? - Saturation - Any delayed data? - IteratorAge? Monitoring - CloudWatch Dashboard
Monitoring - Stream Saturation - Simply calculate the diff between updated records’ timestamp and current time, and compare with the threshold. - timestamp … get from DynamoDB Stream event records - threshold … pass via ENV (currently 10 min) - calculation … - [DynamoDB Stream -> Custom CloudWatch Metrics -> SNS -> Lambda -> Slack] time.Since(timestamp).Minutes() >= threshold
Image Area Cognito - pass role to the Android clients - which can kinesis:PutRecord to the target Kinesis Stream ARN - use unidentified pool - because the Android does not need any login feature - much more secure than embedding API_KEY/CREDENTIAL_KEY to the Android clients
Lambda x Golang 1.x - Officially supported from January 15th, 2018 - https://dev.classmethod.jp/cloud/aws/aws-lambda-supports-go/ - aws/aws-lambda-go - https://github.com/aws/aws-lambda-go - can easily grasp what kind of JSON event records will be available with Type - IMHO - One of the favorite language above other officially supported languages - Type, Runtime Performance, ecosystem, etc. - goroutine/channels have too much overhead for running on Lambda
Lambda x Golang 1.x 1. init() function for declaring the global vars > A single instance of your Lambda function will never handle multiple events simultaneously https://docs.aws.amazon.com/lambda/latest/dg/go-programming-model-handler-types.html
Lambda x Golang 1.x 2. use -ldflags=”-s -w” to reduce the binary size - -s … Omit the symbol table and debug information. - -w … Omit the DWARF symbol table. by https://golang.org/cmd/link/
Serverless Framework - Why Serverless Framework? - Development Speed (easy to configure) - Motivation (never used before at production) - (Might be) easy to migrate to CloudFormation/SAM later - Why not Apex? - Easy to deploy Lambda, but that’s all - Why not SAM? - Writing CloudFormation stacks from the scratch might takes time - However, sam-local is a great tool so might migrate to SAM in the near future
kenju
3150
toricky6
nano72mkn
tanakahisateru
ivargrimstad
atinfinity
gregonnet
sanfrecce_osaka
uhooi
ryos36
akakou
doggy
paulrobertlloyd
addyosmani
jponch
tenderlove
tmm1
caitiem20
eileencodes
3n
garrettdimon
destraynor