Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Implementing Feature Flags in Serverless Environments

Implementing Feature Flags in Serverless Environments

With feature flags, we can change our software's behavior at runtime without modifying or re-deploying our code. There are many good reasons to use feature flags. Among other things, they allow us to separate the risk of deployment from that of rolling out new functionality. However, implementing feature flags in serverless environments like AWS Lambda poses some challenges. In this presentation, Mathias will show you how to overcome these difficulties by building a serverless flag storage pipeline for the popular feature management platform LaunchDarkly. All using Go, of course.


Mathias Lafeldt

June 26, 2018


  1. Implementing Feature Flags in Serverless Environments - GDG Go &

    Cloud User Group Hamburg -
  2. Hello I'm Mathias Lafeldt Freelance Solutions Architect @mlafeldt

  3. Feature Flags 101 The Ultimate Feature Flag Guide ! https://rollout.io/blog/ultimate-feature-flag-guide

  4. "A feature flag is a way to change your software's

    functionality without changing and re-deploying your code."
  5. A Very Contrived Example if (feature.isActive("holiday-greeting")) { print("Happy holidays," +

    user.name + "!"); }
  6. Use Cases of Feature Flags • Separate deployment from feature

    rollout • Canary releases • Production testing • Turning things off with a kill switch • A/B testing, Chaos Engineering, and other experiments • Migrations • Trunk-based development
  7. Open Source and SaaS Solutions • Unleash - Enterprise ready

    feature toggles service (OSS) • Rollout - A secure feature management system for the enterprise • Split - Feature experimentation platform for engineering and product teams • LaunchDarkly - Feature management platform for modern development
  8. None
  9. None
  10. None
  11. LaunchDarkly Go SDK Example import ld "gopkg.in/launchdarkly/go-client.v4" ldClient, err :=

    ld.MakeClient("some-sdk-key", 5*time.Second) if err != nil { ... } defer ldClient.Close() ldUser := ld.NewUser("some-user") flagA, _ := ldClient.BoolVariation("some.bool.flag", ldUser, false) if flagA { ... } flagB, _ := ldClient.IntVariation("some.int.flag", ldUser, 2018) doSomething(flagB)
  12. Go Serverless, not Flagless: Implementing Feature Flags in Serverless Environments

    https://blog.launchdarkly.com/go-serveless-not-flagless- implementing-feature-flags-in-serverless-environments (April 14, 2017)
  13. "[Lambda functions] will have poor cold start performance because we

    need to call out to LaunchDarkly whenever a new Execution Context is created."
  14. None
  15. Using the Redis Feature Store import ld "gopkg.in/launchdarkly/go-client.v4" import redis

    "gopkg.in/launchdarkly/go-client.v4/redis" config := ld.DefaultConfig config.FeatureStore = redis.NewRedisFeatureStoreFromUrl( "redis://localhost:6379", "launchdarkly", 30*time.Second, nil) // Enable daemon mode to only read flags from Redis config.UseLdd = true ldClient, err := ld.MakeCustomClient("some-sdk-key", config, 5*time.Second) if err != nil { ... }
  16. None
  17. None
  18. "From various reports around the web, [Lambda] cold starts within

    VPCs could add up to 10 seconds of latency!" https://medium.freecodecamp.org/lambda-vpc-cold-starts-a-latency-killer-5408323278dd
  19. If only there were some serverless key-value store that we

    can easily access from AWS Lambda... !
  20. None
  21. Let's Write a DynamoDB Feature Store! type FeatureStore interface {

    Get(kind VersionedDataKind, key string) (VersionedData, error) All(kind VersionedDataKind) (map[string]VersionedData, error) Init(map[VersionedDataKind]map[string]VersionedData) error Delete(kind VersionedDataKind, key string, version int) error Upsert(kind VersionedDataKind, item VersionedData) error Initialized() bool } var VersionedDataKinds = [...]VersionedDataKind{ Features, Segments, }
  22. Diving into the DynamoDB API and AWS SDK • GetItem

    • PutItem • Scan • Query • BatchWriteItem • ...
  23. $ curl -s -H 'Authorization: some-sdk-key' https://app.launchdarkly.com/sdk/latest-flags | jq .

    { "some.bool.flag": { "key": "some.bool.flag", "version": 2, "on": true, "prerequisites": [], "salt": "57c59e2cfb724c749ff438332035ccea", "sel": "cedb458484144ead94864dfff58cee51", "targets": [], "rules": [], "fallthrough": { "variation": 0 }, "offVariation": 1, "variations": [ true, false ], "trackEvents": true, "debugEventsUntilDate": null, "deleted": false }, "some.int.flag": ... }
  24. dynamodbattribute.MarshalMap/UnmarshalMap type FeatureFlag struct { Key string `json:"key" bson:"key"` Version

    int `json:"version" bson:"version"` On bool `json:"on" bson:"on"` Prerequisites []Prerequisite `json:"prerequisites" bson:"prerequisites"` Salt string `json:"salt" bson:"salt"` Sel string `json:"sel" bson:"sel"` Targets []Target `json:"targets" bson:"targets"` Rules []Rule `json:"rules" bson:"rules"` Fallthrough VariationOrRollout `json:"fallthrough" bson:"fallthrough"` OffVariation *int `json:"offVariation" bson:"offVariation"` Variations []interface{} `json:"variations" bson:"variations"` TrackEvents bool `json:"trackEvents" bson:"trackEvents"` DebugEventsUntilDate *uint64 `json:"debugEventsUntilDate" bson:"debugEventsUntilDate"` Deleted bool `json:"deleted" bson:"deleted"` }
  25. How to Store Everything in the Same DynamoDB Table? •

    Use the item's kind as the primary partition key, e.g. features or segments • Use the item's key as the primary sort key, e.g. feature.foo • Figure out how to do this without modifying LaunchDarkly's structs • ! Add a namespace attribute after using MarshalMap
  26. Making the Test Suite Happy import ldtest "gopkg.in/launchdarkly/go-client.v4/shared_test" func TestDynamoDBFeatureStore(t

    *testing.T) { table := os.Getenv("LAUNCHDARKLY_DYNAMODB_TABLE") if table == "" { t.Skip("LAUNCHDARKLY_DYNAMODB_TABLE not set in environment") } ldtest.RunFeatureStoreTests(t, func() ld.FeatureStore { store, err := NewDynamoDBFeatureStore(table, nil) if err != nil { t.Fatal(err) } return store }) }
  27. Making LaunchDarkly and AWS users happy (and this little puppy)

  28. Now Open Source: DynamoDB Store for LaunchDarkly's Go SDK github.com/mlafeldt/launchdarkly-dynamo-store

  29. Using the DynamoDB Feature Store import ld "gopkg.in/launchdarkly/go-client.v4" import "github.com/mlafeldt/launchdarkly-dynamo-store/dynamodb"

    store, err := dynamodb.NewDynamoDBFeatureStore("some-table", nil) if err != nil { ... } config := ld.DefaultConfig config.FeatureStore = store config.UseLdd = true ldClient, err := ld.MakeCustomClient("some-sdk-key", config, 5*time.Second) if err != nil { ... }
  30. Wait - we are not done yet. ! How to

    propagate flag updates?
  31. Serverless Flag Storage Pipeline

  32. ! Demo !

  33. Takeaways • Use feature flags! • Take advantage of existing

    feature flag tooling • Cache flags in environments sensitive to cold starts • Choose the right key-value store for the job • Use a (serverless) flag storage pipeline to propagate changes
  34. Thank you. @mlafeldt mathias.lafeldt@gmail.com github.com/mlafeldt/launchdarkly-dynamo-store