Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Central Dogma: LINE's Git-based highly-available service configuration repository

Central Dogma: LINE's Git-based highly-available service configuration repository

Video: https://www.youtube.com/watch?v=-O19M6CC2Vw

In this session, we introduce Central Dogma, LINE's open-source multi-master replicated highly-available version-controlled service configuration repository based on Git, ZooKeeper and HTTP/2.

As well as the core features, we show the positive changes Central Dogma brought to us both in engineering and configuration management workflow perspective.

Previously presented at:

- TWJUG Meetup in Taipei on June 30, 2018
- SOSCON 2017 in Seoul on October 25, 2017

Trustin Lee

June 30, 2018

More Decks by Trustin Lee

Other Decks in Technology


  1. Central Dogma
    LINE’s Git-based highly-available service configuration repository
    LINE Corporation
    June 2018

    View Slide

  2. Any non-trivial service needs to store its
    configuration somewhere, “safely”.

    View Slide

  3. Central Dogma is ...

    Repository service for textual configuration
    – Primarily JSON
    – YAML, XML, INI, JavaScript, ...

    Highly available

    Version controlled

    Advanced query mechanism

    Change notification

    Fine-grained access control

    Mirroring from an external Git repository

    View Slide

  4. Stores anything textual

    What’s fetched at start-time
    – Application parameters
    – Bean properties

    What’s updated at run-time
    – User ∙ IP blacklist
    – Scheduled maintenance notice
    – Roll-out & A/B experiment parameters

    What’s evaluated at run-time
    – Rule-engine scripts

    View Slide

  5. Highly-available


    Eventually consistent
    – Writing to A then reading from B → ?!
    – Client-side load-balancing

    Fast read / Slow write

    ZooKeeper as a replication log queue

    View Slide

  6. Version-controlled

    jGit as a back-end storage
    – History - diffs and authors
    – Bigger than RAM

    Focus on simplicity
    – Integer revision numbers
    – Linear history - no branches

    View Slide

  7. Advanced query mechanism

    … thanks to the first-class JSON support

    JSON path

    JSON patch – RFC6902
    $.store.book[?(@.price < 10)]
    $..book[?(@.author =~ /.*REES/i)]
    [{ “op”: “remove”, “path”: “/a/b/c” },
    { “op”: “add”, “path”: “/a/b/c”, “value”: [“foo”, “bar”] },
    { “op”: “replace”, “path”: “/a/b/c”, “value”: 42 }]

    View Slide

  8. Change notification

    Get notified on a new commit
    CentralDogma client = new LegacyCentralDogmaBuilder().host("example.com").build();
    Watcher watcher =
    client.fileWatcher("my_project", "my_repository",
    Query.ofJsonPath("/settings.json", "$.foo"));
    watcher.watch((revision, value) -> {
    "Foo has been updated to " + value + " (revision: " + revision + ')');

    View Slide

  9. Change notification (Go)

    Get notified on a new commit
    c, _ = NewClientWithToken("example.com", "MyToken")
    q := &Query{Path: "/settings.json", Type: JSONPath, Expressions: []string{"$.foo"}}
    fw, _ := c.FileWatcher("my_project", "my_repository", q)
    listener := func(revision int, value interface{}) {
    fmt.Printf("Foo has been updated to %v (revision: %d)\n", value, revision)

    View Slide

  10. Fine-grained access control

    Apache Shiro as the authentication layer

    Four roles
    – Administrator, Owner, Member and Guest

    In a repository, read and write permissions can be set based on:
    – Roles, users and tokens

    Application token
    – Represents a virtual user

    View Slide

  11. Mirroring from an external Git repository

    Keep your settings in a GitHub / GitLab repository

    Send a pull request to modify the configuration

    Get it reviewed and merged

    Let your services read from Central Dogma
    – Queryable
    – Watchable
    – Highly-available
    – Accessible from same network

    View Slide

  12. To mirror or not to mirror

    Git-to-CD mirroring is optional
    – You can commit to CD directly using:

    Web dashboard


    Client library: Java, Go


    Do not commit directly to a mirrored repository

    Things that do not need mirroring:
    – Files updated by a non-human being

    View Slide

  13. Putting it all together

    View Slide

  14. Case studies
    Real world use cases at LINE

    View Slide

  15. Scheduled maintenance

    Enter ∙ Exit scheduled maintenance
    – “our service is under maintenance until .”

    An administrator updates maintenance.json using a Python script

    Web application watches maintenance.json
    "enabled": "false",
    "components": [
    "startTimeMills": 1482234240000,
    "endTimeMills": 1482235801000

    View Slide

  16. PlanOut integration

    Online field experimentations framework

    Implemented com.glassdoor.planout4j.config.Planout4jRepository
    which uses Central Dogma as a back-end
    – A/B testing
    – Feature roll-out
    unit: userIdHash
    segments: 100
    - definition: orderA
    assign: !planout |
    order = 'recommend-generalnew-editorspick';
    - definition: orderB
    assign: !planout |
    order = 'editorspick-recommend-generalnew';
    - definition: orderC
    assign: !planout |
    order = 'recommend-editorspick-generalnew';
    default_experiment: orderA

    View Slide

  17. Using as a directory service

    Using a JSON path query to find the information about a service:
    $[?(@.hostname == 'TKSVR1234' && @.port == 8080)]
    [{ "zoneId": 0,
    "zone": "JP",
    "groups": [{ "groupSet": "ROLE",
    "name": "Android" }],
    "projectId": "talk-server",
    "projectPhase": "RELEASE",
    "hostname": "TKSVR1234",
    "ip": "",
    "port": 8080,
    "weight": 2000,
    "status": "NORMAL",
    "keepAlive": false }]

    View Slide

  18. Current status

    Doing great at production
    – … with continuous improvements and new features
    – Increasing # of internal and external customers

    We’re not “there” yet

    View Slide

  19. Future work

    Multi-datacenter replication

    1st class YAML support

    JSON Schema

    Slack, e-mail, RSS notifications

    Mirroring an HTTP query result

    SAML single sign-on

    Find more at our issue tracker:
    – https://github.com/line/centraldogma/issues

    View Slide

  20. Let’s build Central Dogma together!

    Use it.

    Ask questions.

    Request new features.

    Tell us what rocks and sucks.

    Consider joining the effort.

    View Slide

  21. Meet us at GitHub and Slack



    View Slide