Central Dogma LINE’s Git-based highly-available service configuration repository LINE Corporation June 2018

Any non-trivial service needs to store its configuration somewhere, “safely”.

Central Dogma is ... ● Repository service for textual configuration – Primarily JSON – YAML, XML, INI, JavaScript, ... ● Highly available ● Version controlled ● Advanced query mechanism ● Change notification ● Fine-grained access control ● Mirroring from an external Git repository

Stores anything textual ● What’s fetched at start-time – Application parameters – Bean properties ● What’s updated at run-time – User ∙ IP blacklist – Scheduled maintenance notice – Roll-out & A/B experiment parameters ● What’s evaluated at run-time – Rule-engine scripts

Highly-available ● Multi-master ● Eventually consistent – Writing to A then reading from B → ?! – Client-side load-balancing ● Fast read / Slow write ● ZooKeeper as a replication log queue

Version-controlled ● jGit as a back-end storage – History - diffs and authors – Bigger than RAM ● Focus on simplicity – Integer revision numbers – Linear history - no branches

Advanced query mechanism ● … thanks to the first-class JSON support ● JSON path ● JSON patch – RFC6902 $.store.book[*].author $.store.book[?(@.price < 10)] $..book[?(@.author =~ /.*REES/i)] [{ “op”: “remove”, “path”: “/a/b/c” }, { “op”: “add”, “path”: “/a/b/c”, “value”: [“foo”, “bar”] }, { “op”: “replace”, “path”: “/a/b/c”, “value”: 42 }]

Change notification ● Get notified on a new commit CentralDogma client = new LegacyCentralDogmaBuilder().host("example.com").build(); Watcher watcher = client.fileWatcher("my_project", "my_repository", Query.ofJsonPath("/settings.json", "$.foo")); watcher.watch((revision, value) -> { System.err.println( "Foo has been updated to " + value + " (revision: " + revision + ')'); });

Change notification (Go) ● Get notified on a new commit c, _ = NewClientWithToken("example.com", "MyToken") q := &Query{Path: "/settings.json", Type: JSONPath, Expressions: []string{"$.foo"}} fw, _ := c.FileWatcher("my_project", "my_repository", q) listener := func(revision int, value interface{}) { fmt.Printf("Foo has been updated to %v (revision: %d)\n", value, revision) } fw.Watch(listener)

Fine-grained access control ● Apache Shiro as the authentication layer ● Four roles – Administrator, Owner, Member and Guest ● In a repository, read and write permissions can be set based on: – Roles, users and tokens ● Application token – Represents a virtual user

Mirroring from an external Git repository ● Keep your settings in a GitHub / GitLab repository ● Send a pull request to modify the configuration ● Get it reviewed and merged ● Let your services read from Central Dogma – Queryable – Watchable – Highly-available – Accessible from same network

To mirror or not to mirror ● Git-to-CD mirroring is optional – You can commit to CD directly using: ● Web dashboard ● HTTP API ● Client library: Java, Go ● CLI ● Do not commit directly to a mirrored repository ● Things that do not need mirroring: – Files updated by a non-human being

Putting it all together

Case studies Real world use cases at LINE

Scheduled maintenance ● Enter ∙ Exit scheduled maintenance – “our service is under maintenance until .” ● An administrator updates maintenance.json using a Python script ● Web application watches maintenance.json { "enabled": "false", "components": [ "stickershop", "themeshop" ], "startTimeMills": 1482234240000, "endTimeMills": 1482235801000 }

PlanOut integration ● Online field experimentations framework ● Implemented com.glassdoor.planout4j.config.Planout4jRepository which uses Central Dogma as a back-end – A/B testing – Feature roll-out namespace: unit: userIdHash segments: 100 experiment_definitions: - definition: orderA assign: !planout | order = 'recommend-generalnew-editorspick'; - definition: orderB assign: !planout | order = 'editorspick-recommend-generalnew'; - definition: orderC assign: !planout | order = 'recommend-editorspick-generalnew'; default_experiment: orderA

Using as a directory service ● Using a JSON path query to find the information about a service: $[?(@.hostname == 'TKSVR1234' && @.port == 8080)] [{ "zoneId": 0, "zone": "JP", "groups": [{ "groupSet": "ROLE", "name": "Android" }], "projectId": "talk-server", "projectPhase": "RELEASE", "hostname": "TKSVR1234", "ip": "192.168.1.234", "port": 8080, "weight": 2000, "status": "NORMAL", "keepAlive": false }]

Current status ● Doing great at production – … with continuous improvements and new features – Increasing # of internal and external customers ● We’re not “there” yet

Future work ● Multi-datacenter replication ● 1st class YAML support ● JSON Schema ● Slack, e-mail, RSS notifications ● Mirroring an HTTP query result ● SAML single sign-on ● Find more at our issue tracker: – https://github.com/line/centraldogma/issues

Let’s build Central Dogma together! ● Use it. ● Ask questions. ● Request new features. ● Tell us what rocks and sucks. ● Consider joining the effort.

Meet us at GitHub and Slack ● https://github.com/line/centraldogma ● https://line-slacknow.herokuapp.com/central-dogma/