Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Central Dogma: LINE's Git-based highly-available service configuration repository

Central Dogma: LINE's Git-based highly-available service configuration repository

Video: https://www.youtube.com/watch?v=-O19M6CC2Vw

In this session, we introduce Central Dogma, LINE's open-source multi-master replicated highly-available version-controlled service configuration repository based on Git, ZooKeeper and HTTP/2.

As well as the core features, we show the positive changes Central Dogma brought to us both in engineering and configuration management workflow perspective.

Previously presented at:

- TWJUG Meetup in Taipei on June 30, 2018
- SOSCON 2017 in Seoul on October 25, 2017

Trustin Lee

June 30, 2018

More Decks by Trustin Lee

Other Decks in Technology


  1. Central Dogma
    LINE’s Git-based highly-available service configuration repository
    LINE Corporation
    June 2018

    View full-size slide

  2. Any non-trivial service needs to store its
    configuration somewhere, “safely”.

    View full-size slide

  3. Central Dogma is ...

    Repository service for textual configuration
    – Primarily JSON
    – YAML, XML, INI, JavaScript, ...

    Highly available

    Version controlled

    Advanced query mechanism

    Change notification

    Fine-grained access control

    Mirroring from an external Git repository

    View full-size slide

  4. Stores anything textual

    What’s fetched at start-time
    – Application parameters
    – Bean properties

    What’s updated at run-time
    – User ∙ IP blacklist
    – Scheduled maintenance notice
    – Roll-out & A/B experiment parameters

    What’s evaluated at run-time
    – Rule-engine scripts

    View full-size slide

  5. Highly-available


    Eventually consistent
    – Writing to A then reading from B → ?!
    – Client-side load-balancing

    Fast read / Slow write

    ZooKeeper as a replication log queue

    View full-size slide

  6. Version-controlled

    jGit as a back-end storage
    – History - diffs and authors
    – Bigger than RAM

    Focus on simplicity
    – Integer revision numbers
    – Linear history - no branches

    View full-size slide

  7. Advanced query mechanism

    … thanks to the first-class JSON support

    JSON path

    JSON patch – RFC6902
    $.store.book[?(@.price < 10)]
    $..book[?(@.author =~ /.*REES/i)]
    [{ “op”: “remove”, “path”: “/a/b/c” },
    { “op”: “add”, “path”: “/a/b/c”, “value”: [“foo”, “bar”] },
    { “op”: “replace”, “path”: “/a/b/c”, “value”: 42 }]

    View full-size slide

  8. Change notification

    Get notified on a new commit
    CentralDogma client = new LegacyCentralDogmaBuilder().host("example.com").build();
    Watcher watcher =
    client.fileWatcher("my_project", "my_repository",
    Query.ofJsonPath("/settings.json", "$.foo"));
    watcher.watch((revision, value) -> {
    "Foo has been updated to " + value + " (revision: " + revision + ')');

    View full-size slide

  9. Change notification (Go)

    Get notified on a new commit
    c, _ = NewClientWithToken("example.com", "MyToken")
    q := &Query{Path: "/settings.json", Type: JSONPath, Expressions: []string{"$.foo"}}
    fw, _ := c.FileWatcher("my_project", "my_repository", q)
    listener := func(revision int, value interface{}) {
    fmt.Printf("Foo has been updated to %v (revision: %d)\n", value, revision)

    View full-size slide

  10. Fine-grained access control

    Apache Shiro as the authentication layer

    Four roles
    – Administrator, Owner, Member and Guest

    In a repository, read and write permissions can be set based on:
    – Roles, users and tokens

    Application token
    – Represents a virtual user

    View full-size slide

  11. Mirroring from an external Git repository

    Keep your settings in a GitHub / GitLab repository

    Send a pull request to modify the configuration

    Get it reviewed and merged

    Let your services read from Central Dogma
    – Queryable
    – Watchable
    – Highly-available
    – Accessible from same network

    View full-size slide

  12. To mirror or not to mirror

    Git-to-CD mirroring is optional
    – You can commit to CD directly using:

    Web dashboard


    Client library: Java, Go


    Do not commit directly to a mirrored repository

    Things that do not need mirroring:
    – Files updated by a non-human being

    View full-size slide

  13. Putting it all together

    View full-size slide

  14. Case studies
    Real world use cases at LINE

    View full-size slide

  15. Scheduled maintenance

    Enter ∙ Exit scheduled maintenance
    – “our service is under maintenance until .”

    An administrator updates maintenance.json using a Python script

    Web application watches maintenance.json
    "enabled": "false",
    "components": [
    "startTimeMills": 1482234240000,
    "endTimeMills": 1482235801000

    View full-size slide

  16. PlanOut integration

    Online field experimentations framework

    Implemented com.glassdoor.planout4j.config.Planout4jRepository
    which uses Central Dogma as a back-end
    – A/B testing
    – Feature roll-out
    unit: userIdHash
    segments: 100
    - definition: orderA
    assign: !planout |
    order = 'recommend-generalnew-editorspick';
    - definition: orderB
    assign: !planout |
    order = 'editorspick-recommend-generalnew';
    - definition: orderC
    assign: !planout |
    order = 'recommend-editorspick-generalnew';
    default_experiment: orderA

    View full-size slide

  17. Using as a directory service

    Using a JSON path query to find the information about a service:
    $[?(@.hostname == 'TKSVR1234' && @.port == 8080)]
    [{ "zoneId": 0,
    "zone": "JP",
    "groups": [{ "groupSet": "ROLE",
    "name": "Android" }],
    "projectId": "talk-server",
    "projectPhase": "RELEASE",
    "hostname": "TKSVR1234",
    "ip": "",
    "port": 8080,
    "weight": 2000,
    "status": "NORMAL",
    "keepAlive": false }]

    View full-size slide

  18. Current status

    Doing great at production
    – … with continuous improvements and new features
    – Increasing # of internal and external customers

    We’re not “there” yet

    View full-size slide

  19. Future work

    Multi-datacenter replication

    1st class YAML support

    JSON Schema

    Slack, e-mail, RSS notifications

    Mirroring an HTTP query result

    SAML single sign-on

    Find more at our issue tracker:
    – https://github.com/line/centraldogma/issues

    View full-size slide

  20. Let’s build Central Dogma together!

    Use it.

    Ask questions.

    Request new features.

    Tell us what rocks and sucks.

    Consider joining the effort.

    View full-size slide

  21. Meet us at GitHub and Slack



    View full-size slide