Stream'Data'Processing'with'Kinesis'and'Go'at' Timehop Avi$Flax June%2015

No content

Background • Whenever(a(user(opens(the(app,(the(app(no0fies(the(API • We(do(4(things(with(this(data • (1)(Count(daily(unique(user(opens • (2)(Record(the(last(opened(0me(for(each(user • (3)(Update(user(data • (4)(Archive(the(app(open(event

Prior%System

Interlude)I:)Jay)Kreps)❤s)Logs • Jan%2011:%Ka+a%released%as%FLOSS • Jul%2011:%Ka+a%entered%Apache%incuba=on • Nov%2012:%Ka+a%graduated%incuba=on • Dec%2013:%Kreps%published%The%Log:%What%every%soGware% engineer%should%know%about%realJ=me%data's%unifying%abstrac=on • Sep%2014:%published%in%book%form:%I%❤%Logs:%Event%Data,%Stream% Processing,%and%Data%Integra=on • Nov%2014:%coJfounded%Confluent

Interlude)II:)Unified)Logs)in)a)Nutshell • A#very#specific#approach#to#streaming#data#transport • a#server#providing#“an#append6only,#totally6ordered#sequence#of# records#ordered#by#=me” • Decouples:+producers+&+consumers;+transport+&+processing;+ consumers)&)consumers

Interlude)III:)AWS)Kinesis …recently)Amazon)has)offered)a)service)that)is)very)very)similar)to) Ka6a)called)Kinesis…)I)was)pre:y)happy)about)this. —"Jay"Kreps

Interlude)IV:)Go • An$object+oriented$systems$language$with$GC

No content

No content

No content

No content

The$Workers

No content

Producer(/(apicard • We’re&using&Sendgrid’s&go-kinesis&library&to&produce&records& to&Kinesis • We&pre7y&quickly&ran&into&some&issues: • Kinesis&some

go-kinesis/batchproducer type Producer interface { Start() error Stop() error Add(data []byte, partitionKey string) error Flush(timeout time.Duration, sendStats bool) (sent int, remaining int, err error) } // BatchingKinesisClient is a subset of KinesisClient to ease mocking. type BatchingKinesisClient interface { PutRecords(args *kinesis.RequestArgs) (resp *kinesis.PutRecordsResp, err error) } func New(client BatchingKinesisClient, streamName string, config Config) (Producer, error)

API$Startup AppOpenBatcher, err = batchproducer.New(ksis, appOpenStreamName, config) if err != nil { golog.Fatal(gologID, "Oh noes!") } AppOpenBatcher.Start()

App#Opens#API#Resource func enqueueAppOpenEventForStream(event appopen.AppOpen) { avrobytes, err := event.ToAvro() if err != nil { ... return } partitionKey := strconv.Itoa(time.Unix(event.Timestamp, 0).Second()) err = conf.AppOpenBatcher.Add(avrobytes, partitionKey) if err != nil { ... return } }

No content

Streams(Repo Four%notable%concepts/packages: • models • package kclmultilang • tasks • cmd

package …/streams/models/appopen type AppOpen struct{ Timestamp int64 UserID int64 ... } func FromAvro(record []byte) (*AppOpen, error) func (s AppOpen) ToAvro() ([]byte, error) func (s AppOpen) Validate() error

kclmultilang type Config struct { StreamName string WorkerName string ... } func RunWithSingleProcessor(Config, SingleEventProcessor) type SingleEventProcessor func(appopen.AppOpen, log.Logger) error

Task%RecordLastOpen // RecordLastOpen updates a certain Redis key for each user that // stores the last time that user opened the Timehop mobile app. func RecordLastOpen( event appopen.AppOpen, redis redis.Pool, logger log.Logger ) error { key := fmt.Sprintf("user:%v:checkpoint", event.UserID) field := fmt.Sprintf("%v_app_open", event.Platform) value := fmt.Sprint(event.Timestamp) _, err := redis.HSet(key, field, value) return err }

Worker&lastopens cmd/appopen/lastopens/main.go func main() { resultsRedisURL := env.MandatoryVar("RESULTS_REDIS_URL") streamName := env.MandatoryVar("STREAM_NAME") logFilePath := env.ImportantVar("LOG_PATH", defaultLogFilePath) config := kclmultilang.Config{...} resultsRedisPool := redis.NewPool(resultsRedisURL, redis.DefaultConfig) processor := func(record appopen.AppOpen, logger log.Logger) error { return lastopens.RecordLastOpen(record, resultsRedisPool, logger) } kclmultilang.RunWithSingleProcessor(config, processor) }

No content

Lessons&Learned,&Hints,&Tips,&and&Miscellany • Didn’t(need(to(write(kclmultilang(—(could(have((should(have)( used(Niek(Sanders’(gokinesis • Probably(could(have(delayed(adding(the(batch(producer(to(go- kinesis(—(could(have(lived(without(it • Deployment(is(the(other(90%

Ques%ons,)Comments,)Sugges%ons?

More%Resources • The%Log:%an%epic%so0ware%engineering%ar3cle%by%Bryan%Pendleton • The%three%eras%of%business%data%processing%by%Alex%Dean • Loving%a%LogAOriented%Architecture%by%Andrew%Montalen3 • Stream%Processing,%Event%Sourcing,%Reac3ve,%CEP…%And%Making% Sense%Of%It%All%by%Mar3n%Kleppman And$a$whole$bunch$more$here$including$books$and$videos.

Bonus:'AWS'Lambda'and'Go • Ruben'Fonseca:'AWS'Lambda'Func4ons'in'Go

AWS$Lambda$Go$Adapter exports.handler = function(event, context, test_config) { var config = test_config || prod_config; var options = { env: config.env, input: JSON.stringify(event) } var result = child_process.spawnSync(config.child_path, [], options); if (result.status !== 0) { return context.fail(new Error(result.stderr.toString())); } context.succeed(); }