Stream Data Processing with Kinesis and Go at Timehop

Stream data processing, data pipeline architecture, unified log system, event sourcing, CQRS, Complex Event Processing — these are just some of the names for an approach to system design that emphasizes and embraces that data *flow* like a stream. There’s been a recent surge in discussions about building systems this way, from LinkedIn to Confluent to Yahoo. There are lots of fascinating and inspiring articles, books, and conference talks, but many of them are bold, broad, and fundamental. There’s a dearth of guidance on the nuts and bolts of actually building systems this way. So in the spirit of Go this talk will start to fill that gap a bit, from the bottom up.

I’ll describe an existing queue-based data processing system at Timehop that was starting to break down, and the steps we took to replace it with a stream-based system. We will discuss the overall dataflow of each system and review the Go code used to interface with Kinesis and process the streaming data.

Transcript

1. Stream'Data'Processing'with'Kinesis'and'Go'at' Timehop Avi$Flax June%2015
3. Background • Whenever(a(user(opens(the(app,(the(app(no0fies(the(API • We(do(4(things(with(this(data • (1)(Count(daily(unique(user(opens • (2)(Record(the(last(opened(0me(for(each(user • (3)(Update(user(data • (4)(Archive(the(app(open(event
4. Prior%System
5. Interlude)I:)Jay)Kreps)❤s)Logs • Jan%2011:%Ka+a%released%as%FLOSS • Jul%2011:%Ka+a%entered%Apache%incuba=on • Nov%2012:%Ka+a%graduated%incuba=on • Dec%2013:%Kreps%published%The%Log:%What%every%soGware% engineer%should%know%about%realJ=me%data's%unifying%abstrac=on • Sep%2014:%published%in%book%form:%I%❤%Logs:%Event%Data,%Stream% Processing,%and%Data%Integra=on • Nov%2014:%coJfounded%Confluent
6. Interlude)II:)Unified)Logs)in)a)Nutshell • A#very#specific#approach#to#streaming#data#transport • a#server#providing#“an#append6only,#totally6ordered#sequence#of# records#ordered#by#=me” • Decouples:+producers+&+consumers;+transport+&+processing;+ consumers)&)consumers
7. Interlude)III:)AWS)Kinesis …recently)Amazon)has)offered)a)service)that)is)very)very)similar)to) Ka6a)called)Kinesis…)I)was)pre:y)happy)about)this. —"Jay"Kreps
8. Interlude)IV:)Go • An$object+oriented$systems$language$with$GC
13. The$Workers
15. Producer(/(apicard • We’re&using&Sendgrid’s&go-kinesis&library&to&produce&records& to&Kinesis • We&pre7y&quickly&ran&into&some&issues: • Kinesis&some<mes&returns&500s&(Amazon&considers&this& normal) • Expense&of&sending&records&individually • Need&to&do&batching&and&retrying&in&the&background • Solu<on:&added&batchproducer&to&go-kinesis
16. go-kinesis/batchproducer type Producer interface { Start() error Stop() error Add(data []byte, partitionKey string) error Flush(timeout time.Duration, sendStats bool) (sent int, remaining int, err error) } // BatchingKinesisClient is a subset of KinesisClient to ease mocking. type BatchingKinesisClient interface { PutRecords(args *kinesis.RequestArgs) (resp *kinesis.PutRecordsResp, err error) } func New(client BatchingKinesisClient, streamName string, config Config) (Producer, error)
17. API$Startup AppOpenBatcher, err = batchproducer.New(ksis, appOpenStreamName, config) if err != nil { golog.Fatal(gologID, "Oh noes!") } AppOpenBatcher.Start()
18. App#Opens#API#Resource func enqueueAppOpenEventForStream(event appopen.AppOpen) { avrobytes, err := event.ToAvro() if err != nil { ... return } partitionKey := strconv.Itoa(time.Unix(event.Timestamp, 0).Second()) err = conf.AppOpenBatcher.Add(avrobytes, partitionKey) if err != nil { ... return } }
20. Streams(Repo Four%notable%concepts/packages: • models • package kclmultilang • tasks • cmd
21. package …/streams/models/appopen type AppOpen struct{ Timestamp int64 UserID int64 ... } func FromAvro(record []byte) (*AppOpen, error) func (s AppOpen) ToAvro() ([]byte, error) func (s AppOpen) Validate() error
22. kclmultilang type Config struct { StreamName string WorkerName string ... } func RunWithSingleProcessor(Config, SingleEventProcessor) type SingleEventProcessor func(appopen.AppOpen, log.Logger) error
23. Task%RecordLastOpen // RecordLastOpen updates a certain Redis key for each user that // stores the last time that user opened the Timehop mobile app. func RecordLastOpen( event appopen.AppOpen, redis redis.Pool, logger log.Logger ) error { key := fmt.Sprintf("user:%v:checkpoint", event.UserID) field := fmt.Sprintf("%v_app_open", event.Platform) value := fmt.Sprint(event.Timestamp) _, err := redis.HSet(key, field, value) return err }
24. Worker&lastopens cmd/appopen/lastopens/main.go func main() { resultsRedisURL := env.MandatoryVar("RESULTS_REDIS_URL") streamName := env.MandatoryVar("STREAM_NAME") logFilePath := env.ImportantVar("LOG_PATH", defaultLogFilePath) config := kclmultilang.Config{...} resultsRedisPool := redis.NewPool(resultsRedisURL, redis.DefaultConfig) processor := func(record appopen.AppOpen, logger log.Logger) error { return lastopens.RecordLastOpen(record, resultsRedisPool, logger) } kclmultilang.RunWithSingleProcessor(config, processor) }
26. Lessons&Learned,&Hints,&Tips,&and&Miscellany • Didn’t(need(to(write(kclmultilang(—(could(have((should(have)( used(Niek(Sanders’(gokinesis • Probably(could(have(delayed(adding(the(batch(producer(to(go- kinesis(—(could(have(lived(without(it • Deployment(is(the(other(90%
27. Ques%ons,)Comments,)Sugges%ons?
28. More%Resources • The%Log:%an%epic%so0ware%engineering%ar3cle%by%Bryan%Pendleton • The%three%eras%of%business%data%processing%by%Alex%Dean • Loving%a%LogAOriented%Architecture%by%Andrew%Montalen3 • Stream%Processing,%Event%Sourcing,%Reac3ve,%CEP…%And%Making% Sense%Of%It%All%by%Mar3n%Kleppman And$a$whole$bunch$more$here$including$books$and$videos.
29. Bonus:'AWS'Lambda'and'Go • Ruben'Fonseca:'AWS'Lambda'Func4ons'in'Go
30. AWS$Lambda$Go$Adapter exports.handler = function(event, context, test_config) { var config = test_config || prod_config; var options = { env: config.env, input: JSON.stringify(event) } var result = child_process.spawnSync(config.child_path, [], options); if (result.status !== 0) { return context.fail(new Error(result.stderr.toString())); } context.succeed(); }