Central Dogma: LINE's Git-based highly-available service configuration repository

Video: https://www.youtube.com/watch?v=-O19M6CC2Vw

In this session, we introduce Central Dogma, LINE's open-source multi-master replicated highly-available version-controlled service configuration repository based on Git, ZooKeeper and HTTP/2.

As well as the core features, we show the positive changes Central Dogma brought to us both in engineering and configuration management workflow perspective.

Previously presented at:

- TWJUG Meetup in Taipei on June 30, 2018
- SOSCON 2017 in Seoul on October 25, 2017

Transcript

1. Central Dogma LINE’s Git-based highly-available service configuration repository LINE Corporation June 2018
2. Any non-trivial service needs to store its configuration somewhere, “safely”.
3. Central Dogma is ... • Repository service for textual configuration – Primarily JSON – YAML, XML, INI, JavaScript, ... • Highly available • Version controlled • Advanced query mechanism • Change notification • Fine-grained access control • Mirroring from an external Git repository
4. Stores anything textual • What’s fetched at start-time – Application parameters – Bean properties • What’s updated at run-time – User ∙ IP blacklist – Scheduled maintenance notice – Roll-out & A/B experiment parameters • What’s evaluated at run-time – Rule-engine scripts
5. Highly-available • Multi-master • Eventually consistent – Writing to A then reading from B → ?! – Client-side load-balancing • Fast read / Slow write • ZooKeeper as a replication log queue
6. Version-controlled • jGit as a back-end storage – History - diffs and authors – Bigger than RAM • Focus on simplicity – Integer revision numbers – Linear history - no branches
7. Advanced query mechanism • … thanks to the first-class JSON support • JSON path • JSON patch – RFC6902 $.store.book[*].author $.store.book[?(@.price < 10)] $..book[?(@.author =~ /.*REES/i)] [{ “op”: “remove”, “path”: “/a/b/c” }, { “op”: “add”, “path”: “/a/b/c”, “value”: [“foo”, “bar”] }, { “op”: “replace”, “path”: “/a/b/c”, “value”: 42 }]
8. Change notification • Get notified on a new commit CentralDogma client = new LegacyCentralDogmaBuilder().host("example.com").build(); Watcher<JsonNode> watcher = client.fileWatcher("my_project", "my_repository", Query.ofJsonPath("/settings.json", "$.foo")); watcher.watch((revision, value) -> { System.err.println( "Foo has been updated to " + value + " (revision: " + revision + ')'); });
9. Change notification (Go) • Get notified on a new commit c, _ = NewClientWithToken("example.com", "MyToken") q := &Query{Path: "/settings.json", Type: JSONPath, Expressions: []string{"$.foo"}} fw, _ := c.FileWatcher("my_project", "my_repository", q) listener := func(revision int, value interface{}) { fmt.Printf("Foo has been updated to %v (revision: %d)\n", value, revision) } fw.Watch(listener)
10. Fine-grained access control • Apache Shiro as the authentication layer • Four roles – Administrator, Owner, Member and Guest • In a repository, read and write permissions can be set based on: – Roles, users and tokens • Application token – Represents a virtual user
11. Mirroring from an external Git repository • Keep your settings in a GitHub / GitLab repository • Send a pull request to modify the configuration • Get it reviewed and merged • Let your services read from Central Dogma – Queryable – Watchable – Highly-available – Accessible from same network
12. To mirror or not to mirror • Git-to-CD mirroring is optional – You can commit to CD directly using: • Web dashboard • HTTP API • Client library: Java, Go • CLI • Do not commit directly to a mirrored repository • Things that do not need mirroring: – Files updated by a non-human being
13. Putting it all together
14. Case studies Real world use cases at LINE
15. Scheduled maintenance • Enter ∙ Exit scheduled maintenance – “our service is under maintenance until <endTime>.” • An administrator updates maintenance.json using a Python script • Web application watches maintenance.json { "enabled": "false", "components": [ "stickershop", "themeshop" ], "startTimeMills": 1482234240000, "endTimeMills": 1482235801000 }
16. PlanOut integration • Online field experimentations framework • Implemented com.glassdoor.planout4j.config.Planout4jRepository which uses Central Dogma as a back-end – A/B testing – Feature roll-out namespace: unit: userIdHash segments: 100 experiment_definitions: - definition: orderA assign: !planout | order = 'recommend-generalnew-editorspick'; - definition: orderB assign: !planout | order = 'editorspick-recommend-generalnew'; - definition: orderC assign: !planout | order = 'recommend-editorspick-generalnew'; default_experiment: orderA
17. Using as a directory service • Using a JSON path query to find the information about a service: $[?(@.hostname == 'TKSVR1234' && @.port == 8080)] [{ "zoneId": 0, "zone": "JP", "groups": [{ "groupSet": "ROLE", "name": "Android" }], "projectId": "talk-server", "projectPhase": "RELEASE", "hostname": "TKSVR1234", "ip": "192.168.1.234", "port": 8080, "weight": 2000, "status": "NORMAL", "keepAlive": false }]
18. Current status • Doing great at production – … with continuous improvements and new features – Increasing # of internal and external customers • We’re not “there” yet
19. Future work • Multi-datacenter replication • 1st class YAML support • JSON Schema • Slack, e-mail, RSS notifications • Mirroring an HTTP query result • SAML single sign-on • Find more at our issue tracker: – https://github.com/line/centraldogma/issues
20. Let’s build Central Dogma together! • Use it. • Ask questions. • Request new features. • Tell us what rocks and sucks. • Consider joining the effort.
21. Meet us at GitHub and Slack • https://github.com/line/centraldogma • https://line-slacknow.herokuapp.com/central-dogma/