Central Dogma: LINE's Git-based highly-available service configuration repository

Video: https://www.youtube.com/watch?v=-O19M6CC2Vw

In this session, we introduce Central Dogma, LINE's open-source multi-master replicated highly-available version-controlled service configuration repository based on Git, ZooKeeper and HTTP/2.

As well as the core features, we show the positive changes Central Dogma brought to us both in engineering and configuration management workflow perspective.

Previously presented at:

- TWJUG Meetup in Taipei on June 30, 2018
- SOSCON 2017 in Seoul on October 25, 2017

Transcript

1. Central Dogma
   LINE’s Git-based highly-available service configuration repository
   LINE Corporation
   June 2018
   

   View Slide

2. Any non-trivial service needs to store its
   configuration somewhere, “safely”.
   

   View Slide

3. Central Dogma is ...
   ●
   Repository service for textual configuration
   – Primarily JSON
   – YAML, XML, INI, JavaScript, ...
   ●
   Highly available
   ●
   Version controlled
   ●
   Advanced query mechanism
   ●
   Change notification
   ●
   Fine-grained access control
   ●
   Mirroring from an external Git repository
   

   View Slide

4. Stores anything textual
   ●
   What’s fetched at start-time
   – Application parameters
   – Bean properties
   ●
   What’s updated at run-time
   – User ∙ IP blacklist
   – Scheduled maintenance notice
   – Roll-out & A/B experiment parameters
   ●
   What’s evaluated at run-time
   – Rule-engine scripts
   

   View Slide

5. Highly-available
   ●
   Multi-master
   ●
   Eventually consistent
   – Writing to A then reading from B → ?!
   – Client-side load-balancing
   ●
   Fast read / Slow write
   ●
   ZooKeeper as a replication log queue
   

   View Slide

6. Version-controlled
   ●
   jGit as a back-end storage
   – History - diffs and authors
   – Bigger than RAM
   ●
   Focus on simplicity
   – Integer revision numbers
   – Linear history - no branches
   

   View Slide

7. Advanced query mechanism
   ●
   … thanks to the first-class JSON support
   ●
   JSON path
   ●
   JSON patch – RFC6902
   $.store.book[*].author
   $.store.book[?(@.price < 10)]
   $..book[?(@.author =~ /.*REES/i)]
   [{ “op”: “remove”, “path”: “/a/b/c” },
   { “op”: “add”, “path”: “/a/b/c”, “value”: [“foo”, “bar”] },
   { “op”: “replace”, “path”: “/a/b/c”, “value”: 42 }]
   

   View Slide

8. Change notification
   ●
   Get notified on a new commit
   CentralDogma client = new LegacyCentralDogmaBuilder().host("example.com").build();
   Watcher watcher =
   client.fileWatcher("my_project", "my_repository",
   Query.ofJsonPath("/settings.json", "$.foo"));
   watcher.watch((revision, value) -> {
   System.err.println(
   "Foo has been updated to " + value + " (revision: " + revision + ')');
   });
   

   View Slide

9. Change notification (Go)
   ●
   Get notified on a new commit
   c, _ = NewClientWithToken("example.com", "MyToken")
   q := &Query{Path: "/settings.json", Type: JSONPath, Expressions: []string{"$.foo"}}
   fw, _ := c.FileWatcher("my_project", "my_repository", q)
   listener := func(revision int, value interface{}) {
   fmt.Printf("Foo has been updated to %v (revision: %d)\n", value, revision)
   }
   fw.Watch(listener)
   

   View Slide

10. Fine-grained access control
    ●
    Apache Shiro as the authentication layer
    ●
    Four roles
    – Administrator, Owner, Member and Guest
    ●
    In a repository, read and write permissions can be set based on:
    – Roles, users and tokens
    ●
    Application token
    – Represents a virtual user
    

    View Slide

11. Mirroring from an external Git repository
    ●
    Keep your settings in a GitHub / GitLab repository
    ●
    Send a pull request to modify the configuration
    ●
    Get it reviewed and merged
    ●
    Let your services read from Central Dogma
    – Queryable
    – Watchable
    – Highly-available
    – Accessible from same network
    

    View Slide

12. To mirror or not to mirror
    ●
    Git-to-CD mirroring is optional
    – You can commit to CD directly using:
    ●
    Web dashboard
    ●
    HTTP API
    ●
    Client library: Java, Go
    ●
    CLI
    ●
    Do not commit directly to a mirrored repository
    ●
    Things that do not need mirroring:
    – Files updated by a non-human being
    

    View Slide

13. Putting it all together
    

    View Slide

14. Case studies
    Real world use cases at LINE
    

    View Slide

15. Scheduled maintenance
    ●
    Enter ∙ Exit scheduled maintenance
    – “our service is under maintenance until .”
    ●
    An administrator updates maintenance.json using a Python script
    ●
    Web application watches maintenance.json
    {
    "enabled": "false",
    "components": [
    "stickershop",
    "themeshop"
    ],
    "startTimeMills": 1482234240000,
    "endTimeMills": 1482235801000
    }
    

    View Slide

16. PlanOut integration
    ●
    Online field experimentations framework
    ●
    Implemented com.glassdoor.planout4j.config.Planout4jRepository
    which uses Central Dogma as a back-end
    – A/B testing
    – Feature roll-out
    namespace:
    unit: userIdHash
    segments: 100
    experiment_definitions:
    - definition: orderA
    assign: !planout |
    order = 'recommend-generalnew-editorspick';
    - definition: orderB
    assign: !planout |
    order = 'editorspick-recommend-generalnew';
    - definition: orderC
    assign: !planout |
    order = 'recommend-editorspick-generalnew';
    default_experiment: orderA
    

    View Slide

17. Using as a directory service
    ●
    Using a JSON path query to find the information about a service:
    $[?(@.hostname == 'TKSVR1234' && @.port == 8080)]
    [{ "zoneId": 0,
    "zone": "JP",
    "groups": [{ "groupSet": "ROLE",
    "name": "Android" }],
    "projectId": "talk-server",
    "projectPhase": "RELEASE",
    "hostname": "TKSVR1234",
    "ip": "192.168.1.234",
    "port": 8080,
    "weight": 2000,
    "status": "NORMAL",
    "keepAlive": false }]
    

    View Slide

18. Current status
    ●
    Doing great at production
    – … with continuous improvements and new features
    – Increasing # of internal and external customers
    ●
    We’re not “there” yet
    

    View Slide

19. Future work
    ●
    Multi-datacenter replication
    ●
    1st class YAML support
    ●
    JSON Schema
    ●
    Slack, e-mail, RSS notifications
    ●
    Mirroring an HTTP query result
    ●
    SAML single sign-on
    ●
    Find more at our issue tracker:
    – https://github.com/line/centraldogma/issues
    

    View Slide

20. Let’s build Central Dogma together!
    ●
    Use it.
    ●
    Ask questions.
    ●
    Request new features.
    ●
    Tell us what rocks and sucks.
    ●
    Consider joining the effort.
    

    View Slide

21. Meet us at GitHub and Slack
    ●
    https://github.com/line/centraldogma
    ●
    https://line-slacknow.herokuapp.com/central-dogma/
    

    View Slide