Image Area What is “streaming” ? Definition: - a type of data processing engine that is designed with infinite data sets in mind. (https://www.oreilly.com/ideas/the-world-bey ond-batch-streaming-101)
“Batch” vs “Streaming” - Batch Processing - Process grouped logs at once, and process occasionaly - Streaming Processing - Micro-batch - ex) AWS Lambda with “Batch Size = 2 ~ n” (n is a not-too-large natural number.) - Real Streaming - ex) AWS Lambda with “Batch Size = 1”
Image Area - Event time, which is the time at which events actually occurred. - Processing time, which is the time at which events are observed in the system. Figure: Example time domain mapping from https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 “Event Time” vs “Processing Time”
Kappa Architecture - introduced by Jay Kreps, a co-founder and CEO at Confluent which was acquired by LinkedIn - “Questioning the Lambda Architecture” https://www.oreilly.com/ideas/questioning-the-lambda-architecture - Streaming Layer + Serving Layer
Lambda vs Kappa Architecture Pros Cons Lambda Architecture - Robust to data consistency - Harder to maintain multiple layers Kappa Architecture - Simple implementation - Need extra works to guarantee data onsistency
Apache Hadoop/Storm/Spark/Kafka/Flink Name Speciality Batch Processing Stream Processing Hadoop HDFS(FIle System), YARN, MapReduce O X Storm Topology (Spout + Bolt), Tuple, Task X O Kafka Broker, Producer/Consumer (“Un-Managed Kinesis” ?) X O Spark inspired by Hadoop’s MapReduce engine O O (Spark Streaming) Flink Batch and Streaming in One System, ML Support, DataStram API O O
How to tackle on “Late Logs”? - Discarding - simply discard late logs - Watermarks - “all input data with event times less than X have been observed.” - Triggers - declaring when the output for a window should be materialized - Accumulation - accumulate multiple results that are observed for the same window Read https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102 for more details :)
What is “Watermark” Definition: a special mark contained in electronic documents, pictures, music etc that is used to stop people from copying them Example: audio watermark detection, photo watermark, copyright watermark
How to “Watermark” - A. Save State in any external memory - e.g. RDS, DynamoDB - B. Calculate on-memory - formula … - e.g. late threshold = 10 minutes - use `median(event time)` instead to handle the too-future logs - e.g. mobile devices’ system clock are somehow modified by users incorrectly watermark = max(event time) - late_threshold
Overall Data-flow - Kinesis Stream receives logs from Android clients directly via kinesis:PutRecord - once per 90 sec, from at most 15,000 devices - Lambda polls Kinesis and aggregate as impression by increment - DynamoDB stores incremented records with UpdateItem (ADD) operation - another Lambda(s) polls DynamoDB Streams and aggregate hourly/daily
Aggregation - Aggregate gradually per minute -> per 10 min -> hourly -> daily - Partition usual logs/lagged logs, and update records to the separate tables - NOT discarding (for now) to see how many logs will be discarded - determine whether logs’ timestamp is behind the watermark or not - watermark … the median of all timestamps - because users can change system clocks to the future
Watermark Implementation func (sr *StreamRecords) watermark(eventTimes []EventTime) (median EventTime) { sort.Ints(eventTimes) // 1. sort l := len(eventTimes) // 2. get the median if l%2 == 0 { // when even median = Mean(eventTimes[l/2-1 : l/2 + 1]) } else { // when odd median = EventTime(eventTimes[l/2]) } return median }
Image Area What to “monitor”? The famous Google’s “SRE” book says in Chapter 6 “The Four Golden Signals” section: - Latency - Traffix - Errors - Saturation Monitoring - The Four Golden Signals
Image Area - Latency - How long does it take? - Traffix - How many Get/PutRecords? - Errors - Availability? - How many errors occur? - Saturation - Any delayed data? - IteratorAge? Monitoring - CloudWatch Dashboard
Monitoring - Stream Saturation - Simply calculate the diff between updated records’ timestamp and current time, and compare with the threshold. - timestamp … get from DynamoDB Stream event records - threshold … pass via ENV (currently 10 min) - calculation … - [DynamoDB Stream -> Custom CloudWatch Metrics -> SNS -> Lambda -> Slack] time.Since(timestamp).Minutes() >= threshold
Image Area Cognito - pass role to the Android clients - which can kinesis:PutRecord to the target Kinesis Stream ARN - use unidentified pool - because the Android does not need any login feature - much more secure than embedding API_KEY/CREDENTIAL_KEY to the Android clients
Lambda x Golang 1.x - Officially supported from January 15th, 2018 - https://dev.classmethod.jp/cloud/aws/aws-lambda-supports-go/ - aws/aws-lambda-go - https://github.com/aws/aws-lambda-go - can easily grasp what kind of JSON event records will be available with Type - IMHO - One of the favorite language above other officially supported languages - Type, Runtime Performance, ecosystem, etc. - goroutine/channels have too much overhead for running on Lambda
Lambda x Golang 1.x 1. init() function for declaring the global vars > A single instance of your Lambda function will never handle multiple events simultaneously https://docs.aws.amazon.com/lambda/latest/dg/go-programming-model-handler-types.html
Lambda x Golang 1.x 2. use -ldflags=”-s -w” to reduce the binary size - -s … Omit the symbol table and debug information. - -w … Omit the DWARF symbol table. by https://golang.org/cmd/link/
Serverless Framework - Why Serverless Framework? - Development Speed (easy to configure) - Motivation (never used before at production) - (Might be) easy to migrate to CloudFormation/SAM later - Why not Apex? - Easy to deploy Lambda, but that’s all - Why not SAM? - Writing CloudFormation stacks from the scratch might takes time - However, sam-local is a great tool so might migrate to SAM in the near future
kenju
youtalk
pulse1923
mfunaki
skanehira
hazumirr
okaharuna
itsmedreamwalker
subomi
mascii
manfredsteyer
lnit
okashoi
iamctodd
addyosmani
3n
jensimmons
kevinliebholz
afnizarnur
keathley
notwaldorf
jmmastey
trallard
marcelosomers