Implementing Feature Flags in Serverless Environments

With feature flags, we can change our software's behavior at runtime without modifying or re-deploying our code. There are many good reasons to use feature flags. Among other things, they allow us to separate the risk of deployment from that of rolling out new functionality. However, implementing feature flags in serverless environments like AWS Lambda poses some challenges. In this presentation, Mathias will show you how to overcome these difficulties by building a serverless flag storage pipeline for the popular feature management platform LaunchDarkly. All using Go, of course.

Transcript

1. Implementing Feature Flags in Serverless Environments - GDG Go & Cloud User Group Hamburg -
2. Hello I'm Mathias Lafeldt Freelance Solutions Architect @mlafeldt
3. Feature Flags 101 The Ultimate Feature Flag Guide ! https://rollout.io/blog/ultimate-feature-flag-guide
4. "A feature flag is a way to change your software's functionality without changing and re-deploying your code."
5. A Very Contrived Example if (feature.isActive("holiday-greeting")) { print("Happy holidays," + user.name + "!"); }
6. Use Cases of Feature Flags • Separate deployment from feature rollout • Canary releases • Production testing • Turning things off with a kill switch • A/B testing, Chaos Engineering, and other experiments • Migrations • Trunk-based development
7. Open Source and SaaS Solutions • Unleash - Enterprise ready feature toggles service (OSS) • Rollout - A secure feature management system for the enterprise • Split - Feature experimentation platform for engineering and product teams • LaunchDarkly - Feature management platform for modern development
11. LaunchDarkly Go SDK Example import ld "gopkg.in/launchdarkly/go-client.v4" ldClient, err := ld.MakeClient("some-sdk-key", 5*time.Second) if err != nil { ... } defer ldClient.Close() ldUser := ld.NewUser("some-user") flagA, _ := ldClient.BoolVariation("some.bool.flag", ldUser, false) if flagA { ... } flagB, _ := ldClient.IntVariation("some.int.flag", ldUser, 2018) doSomething(flagB)
12. Go Serverless, not Flagless: Implementing Feature Flags in Serverless Environments https://blog.launchdarkly.com/go-serveless-not-flagless- implementing-feature-flags-in-serverless-environments (April 14, 2017)
13. "[Lambda functions] will have poor cold start performance because we need to call out to LaunchDarkly whenever a new Execution Context is created."
15. Using the Redis Feature Store import ld "gopkg.in/launchdarkly/go-client.v4" import redis "gopkg.in/launchdarkly/go-client.v4/redis" config := ld.DefaultConfig config.FeatureStore = redis.NewRedisFeatureStoreFromUrl( "redis://localhost:6379", "launchdarkly", 30*time.Second, nil) // Enable daemon mode to only read flags from Redis config.UseLdd = true ldClient, err := ld.MakeCustomClient("some-sdk-key", config, 5*time.Second) if err != nil { ... }
18. "From various reports around the web, [Lambda] cold starts within VPCs could add up to 10 seconds of latency!" https://medium.freecodecamp.org/lambda-vpc-cold-starts-a-latency-killer-5408323278dd
19. If only there were some serverless key-value store that we can easily access from AWS Lambda... !
21. Let's Write a DynamoDB Feature Store! type FeatureStore interface { Get(kind VersionedDataKind, key string) (VersionedData, error) All(kind VersionedDataKind) (map[string]VersionedData, error) Init(map[VersionedDataKind]map[string]VersionedData) error Delete(kind VersionedDataKind, key string, version int) error Upsert(kind VersionedDataKind, item VersionedData) error Initialized() bool } var VersionedDataKinds = [...]VersionedDataKind{ Features, Segments, }
22. Diving into the DynamoDB API and AWS SDK • GetItem • PutItem • Scan • Query • BatchWriteItem • ...
23. $ curl -s -H 'Authorization: some-sdk-key' https://app.launchdarkly.com/sdk/latest-flags | jq . { "some.bool.flag": { "key": "some.bool.flag", "version": 2, "on": true, "prerequisites": [], "salt": "57c59e2cfb724c749ff438332035ccea", "sel": "cedb458484144ead94864dfff58cee51", "targets": [], "rules": [], "fallthrough": { "variation": 0 }, "offVariation": 1, "variations": [ true, false ], "trackEvents": true, "debugEventsUntilDate": null, "deleted": false }, "some.int.flag": ... }
24. dynamodbattribute.MarshalMap/UnmarshalMap type FeatureFlag struct { Key string `json:"key" bson:"key"` Version int `json:"version" bson:"version"` On bool `json:"on" bson:"on"` Prerequisites []Prerequisite `json:"prerequisites" bson:"prerequisites"` Salt string `json:"salt" bson:"salt"` Sel string `json:"sel" bson:"sel"` Targets []Target `json:"targets" bson:"targets"` Rules []Rule `json:"rules" bson:"rules"` Fallthrough VariationOrRollout `json:"fallthrough" bson:"fallthrough"` OffVariation *int `json:"offVariation" bson:"offVariation"` Variations []interface{} `json:"variations" bson:"variations"` TrackEvents bool `json:"trackEvents" bson:"trackEvents"` DebugEventsUntilDate *uint64 `json:"debugEventsUntilDate" bson:"debugEventsUntilDate"` Deleted bool `json:"deleted" bson:"deleted"` }
25. How to Store Everything in the Same DynamoDB Table? • Use the item's kind as the primary partition key, e.g. features or segments • Use the item's key as the primary sort key, e.g. feature.foo • Figure out how to do this without modifying LaunchDarkly's structs • ! Add a namespace attribute after using MarshalMap
26. Making the Test Suite Happy import ldtest "gopkg.in/launchdarkly/go-client.v4/shared_test" func TestDynamoDBFeatureStore(t *testing.T) { table := os.Getenv("LAUNCHDARKLY_DYNAMODB_TABLE") if table == "" { t.Skip("LAUNCHDARKLY_DYNAMODB_TABLE not set in environment") } ldtest.RunFeatureStoreTests(t, func() ld.FeatureStore { store, err := NewDynamoDBFeatureStore(table, nil) if err != nil { t.Fatal(err) } return store }) }
27. Making LaunchDarkly and AWS users happy (and this little puppy)
28. Now Open Source: DynamoDB Store for LaunchDarkly's Go SDK github.com/mlafeldt/launchdarkly-dynamo-store
29. Using the DynamoDB Feature Store import ld "gopkg.in/launchdarkly/go-client.v4" import "github.com/mlafeldt/launchdarkly-dynamo-store/dynamodb" store, err := dynamodb.NewDynamoDBFeatureStore("some-table", nil) if err != nil { ... } config := ld.DefaultConfig config.FeatureStore = store config.UseLdd = true ldClient, err := ld.MakeCustomClient("some-sdk-key", config, 5*time.Second) if err != nil { ... }
30. Wait - we are not done yet. ! How to propagate flag updates?
31. Serverless Flag Storage Pipeline
32. ! Demo !
33. Takeaways • Use feature flags! • Take advantage of existing feature flag tooling • Cache flags in environments sensitive to cold starts • Choose the right key-value store for the job • Use a (serverless) flag storage pipeline to propagate changes
34. Thank you. @mlafeldt mathias.lafeldt@gmail.com github.com/mlafeldt/launchdarkly-dynamo-store