Good for Karma: Configuration at Runtime

Applications need configuration data: for addresses for their neighboring systems and what options are active. What if changes would be possible at runtime: Without maintenance windows for all nodes in the cluster and new functions can be activated without re-deployment?
Once configuration at runtime has been implemented, it makes the lives of application owners, developers, users and operations easier.
Examples on the basis of the open source projects Togglz, Netflix Archaius and CoreOS etcd will be presented. The examples are for classical enterprise architectures, as well as for modern cloud and Continuous delivery environments.

Transcript

1. .consulting .solutions .partnership
   Good for Karma: Configuration at Runtime
   Alexander Schwartz, Principal IT Consultant
   JFall 2015 Ede (NL) – 5. November 2015
   

   View Slide

2. Configuration at Runtime
   © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 2
   Everyday configuration changes
   1
   Archaius: Accessing changing configurations
   2
   etcd: Central Configuration Store
   3
   Togglz: Activating new Functionality on demand
   4
   Better with Configuration at Runtime?
   5
   

   View Slide

3. Configuration at Runtime
   © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 5
   Everyday configuration changes
   1
   Archaius: Accessing changing configurations
   2
   etcd: Central Configuration Store
   3
   Togglz: Activating new Functionality on demand
   4
   Better with Configuration at Runtime?
   5
   

   View Slide

4. Everyday configuration changes
   If a configuration change requires a reboot, it requires a maintenance window
   © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 6
   The customer database has a new IP address,
   please change the configuration of the on-line
   registration.
   No problem, I've changed it. I just have to re-start
   the application tonight!
   Am I a hero because I get up in
   the middle of the night to restart
   the application?
   

   View Slide

5. Everyday configuration changes
   When new features are activated by a deployment, a fallback is difficult
   © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 7
   The new feature in the application,
   could you please activate it?
   Ok, we roll out the new application version
   now. Then all users can access it.
   Should you really do this on
   a Friday afternoon?
   

   View Slide

6. Everyday configuration changes
   Configurations at runtime saves time and hassle – and avoids maintenance windows
   © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 8
   • Change application‘s parameters at runtime
    target-adresses for services
    names of folders
    timeouts and business parameters
   • Central Configuration Store
    configuration parameters
    Service-Discovery
   • Activating new Features
    for named users or groups
    for a percentage of users
    timed activation
   

   View Slide

7. Everyday configuration changes
   Karma: Each of our actions has a consequence for ourselves
   © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 9
   • We are responsible for our actions
   • Configuration changes should lead to the desired result
   – without affecting the users or other stakeholders
   • Insight: We know that configurations change
   • Kindness: We treat our customers with respect and don‘t
   need maintenance windows
   • Modesty: We don‘t want to be “heroes” that get up in the
   middle of the night
   Source: unknown
   

   View Slide

8. Configuration at Runtime
   © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 10
   Everyday configuration changes
   1
   Archaius: Accessing changing configurations
   2
   etcd: Central Configuration Store
   3
   Togglz: Activating new Functionality on demand
   4
   Better with Configuration at Runtime?
   5
   

   View Slide

9. Netflix Archaius
   Netflix Archaius manages access to parameters for Java clients
   © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 11
   1. Loading configuration data
   from: property files, URLs, databases
   2. Cascading Configurations
   for: standard configurations, different environments,
   single servers
   3. Type safe access to the current value
   4. Call backs, when a value changes
   Source: http://en.wikipedia.org/wiki/Archaius_tigris
   (CC) Hans Stieglitz
   

   View Slide

10. Netflix Archaius
    Run Java with an environment parameter
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 12
    java
    -Darchaius.deployment.environment=test
    ...
    # database.properties
    db.url=jdbc:postgresql://localhost:5432/standard-db
    # database-test.properties
    db.url=jdbc:postgresql://localhost:5432/test-db
    // Java Startup
    ConfigurationManager.loadCascadedPropertiesFromResources("database");
    

    View Slide

11. Netflix Archaius
    Pass a configuration file as a parameter
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 13
    java
    -Darchaius.configurationSource.additionalUrls=file:///.../archaius.properties
    -Darchaius.fixedDelayPollingScheduler.delayMills=1000
    -Darchaius.fixedDelayPollingScheduler.initialDelayMills=1000
    –jar application.jar
    # archaius.properties
    app.timeout=1000
    app.url=http://myserver:3000/
    

    View Slide

12. Netflix Archaius
    Archaius provides type safe access to the current value
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 14
    The call back is activated every time a parameter changes its value.
    // construct type safe accessor for property
    DynamicStringProperty property =
    DynamicPropertyFactory.getInstance()
    .getStringProperty("db.url", "default value");
    // access current value
    String url = property.get();
    // get notified once the value changes
    property.addCallback(() -> {
    System.out.println("property has changed: " + property.get());
    });
    

    View Slide

13. Netflix Archaius
    Pros and Cons Archaius
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 15
    Pro:
    • Configurations can be rolled out as files as usual
    (for example using Chef, Puppet, Ansible and Salt)
    • Different storage back ends apart from files are supported
    • Hierarchical structure simplifies the setup
    • Proven library based on Apache Commons Configuration
    • Call backs enable complex reconfiguration
    Contra:
    • Changing multiple stores at the same time can be challenging
    Related projects:
    Apache DeltaSpike, Apache Commons Configuration 2.0 und Archaius 2.0, Apache Tamaya
    

    View Slide

14. Configuration at Runtime
    16
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015
    Everyday configuration changes
    1
    Archaius: Accessing changing configurations
    2
    etcd: Central Configuration Store
    3
    Togglz: Activating new Functionality on demand
    4
    Better with Configuration at Runtime?
    5
    

    View Slide

15. CoreOS etcd
    etcd is a central configuration store
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 17
    Can be used for configuration and service discovery
    • Easy to use: HTTP/REST with JSON
    • Reliable: highly available and consistent
    • Secure: optionally with SSL-Client-Certificates
    • Fast: several thousand write operations per second
    Operations: GET/PUT/DELETE/CAS/WAIT/TTL
    # PUT
    curl -L -X PUT http://127.0.0.1:4001/v2/keys/foo-service/container1 –d value="localhost:1111"
    # GET
    curl -L http://127.0.0.1:4001/v2/keys/foo-service/container1
    # WAIT
    curl -L http://127.0.0.1:4001/v2/keys/foo-service?wait=true\&recursive=true
    

    View Slide

16. CoreOS etcd + Hightower´s confd
    When a value changes in etcd, confd will update the configuration file
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 18
    confd
    

    View Slide

17. CoreOS etcd + Hightower´s confd
    When a value changes in etcd, confd will update the configuration file
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 19
    • Service description (*.toml)
    • Template for a configuration file (*.tmpl)
    # nginx.toml
    [template]
    src = "nginx.tmpl"
    dest = "/etc/nginx/sites-enabled/app.conf"
    keys = [ "/foo-service" ]
    check_cmd = "/usr/sbin/nginx -t"
    reload_cmd = "/usr/sbin/service nginx reload"
    # nginx.tmpl
    upstream app_pool {
    {{ range getvs "/foo-service/*" }}
    server {{ . }};
    {{ end }}
    }
    confd
    

    View Slide

18. CoreOS etcd + Jurmous´ Java Client
    Jurmos Java client provides a Java API for etcd
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 20
    • Java-API for the full etcd functionality
    • Implemented on top of Netty (for asynchronous, non-blocking IO)
    // Simple put
    etcd.put("/foo-service/container1","localhost:1111").send();
    // Wait for next change on foo
    EtcdResponsePromise promise = etcd.getDir("/foo-service/")
    .recursive().waitForChange().send();
    // Java 8 lambda construction
    promise.addListener(promise -> {
    // do something with change
    });
    

    View Slide

19. CoreOS etcd
    Pros und Cons etcd
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 21
    Pro:
    • Easy to access (REST), simple installation (Go-Binary), transport security (SSL)
    • Changes are published immediately to all clients
    • Generic structure for flexible usage
    Contra:
    • No access controls on a client level
    • Young open source project
    Add-ons and alternatives:
    • Vulcan (HTTP Load Balancer, stores its configuration in etcd)
    • SkyDNS (service discovery via DNS, stores its information in etcd)
    • Apache Zookeeper (central configuration store and synchronization)
    • Hazelcast (leader election, distributed locks, topics, in-memory-data-grid)
    • Netflix Eureka (service discovery)
    • Hashicorp Consul (service discovery and configuration store)
    

    View Slide

20. Configuration at Runtime
    22
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015
    Everyday configuration changes
    1
    Archaius: Accessing changing configurations
    2
    etcd: Central Configuration Store
    3
    Togglz: Activating new Functionality on demand
    4
    Better with Configuration at Runtime?
    5
    

    View Slide

21. Togglz: Feature Toggles for Java
    Togglz switches functionality on and off at runtime
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 23
    Scenarios:
    • Not yet released functions are active only in test environments
    • Activation and deactivation can be carried out via a web front end
    • Rules show new functionality only to some users or activate it at a given time
    

    View Slide

22. Togglz: Feature Toggles for Java
    Togglz has a Web UI
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 24
    Status and rollout strategy can be changed for each feature at runtime.
    

    View Slide

23. Togglz: Feature Toggles for Java
    Feature toggles are queried in the source code
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 25
    Enums declare features in Java code.
    They are queried in the code before the new code is called.
    // declaration and documentation of features
    public enum Features implements Feature {
    @Label("First Feature")
    FEATURE_ONE;
    public boolean isActive() {
    return FeatureContext.getFeatureManager().isActive(this);
    }
    } // making use of toggles within the code
    if(Features.FEATURE_ONE.isActive()) {
    // do new exciting stuff here
    }
    

    View Slide

24. Togglz: Feature Toggles for Java
    Pros and Cons Feature Toggles
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 26
    • Feature toggles are a known and well understood pattern
    • Multiple implementations for different programming languages exist
    • Feature toggles increase the complexity of your tests
    • When a feature is fully rolled out, you’ll need to remove the feature toggle
    It might be worth the additional effort,
    • if you have frequent releases (Continuous Delivery), where unfinished features would
    otherwise block finished features
    • if feature toggles help you to avoid long running feature branches that lead to a big merge at the end
    • if an early gradual roll out provides information for the completion of the feature
    

    View Slide

25. Configuration at Runtime
    27
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015
    Everyday configuration changes
    1
    Archaius: Accessing changing configurations
    2
    etcd: Central Configuration Store
    3
    Togglz: Activating new Functionality on demand
    4
    Better with Configuration at Runtime?
    5
    

    View Slide

26. Configuration at Runtime
    Better with Configuration at Runtime? Yes!
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 28
    Access to a changing configuration
    • Helps already during development of the application
    • Minimal initial effort, great flexibility for production
    Central Configuration Store
    • Cross-application configuration
    • Service discovery in cloud scenarios
    Feature Toggles
    • If it simplifies development and deployment
    • If early feedback from production helps to improve the feature
    @ahus1de
    

    View Slide

27. Configuration at Runtime
    Links
    © msg | Good for Karma: Configuration at Runtime | Alexander Schwartz | JFall Ede | 5. November 2015 29
    Netflix Archaius: https://github.com/Netflix/archaius
    Apache Commons Configuration:
    http://commons.apache.org/proper/commons-configuration/
    Apache Tamaya: http://tamaya.incubator.apache.org/
    Overview of different existing solutions:
    http://javaeeconfig.blogspot.de/2014/08/
    overview-of-existing-configuration.html
    Saltstack and Consul Examples
    https://github.com/ahus1/saltconsul-examples
    Togglz: http://www.togglz.org/
    Übersicht über Feature Toggle Implementierungen:
    http://www.beautifulbuilds.com/feature-toggle-frameworks-list/
    @ahus1de
    CoreOS etcd: https://github.com/coreos/etcd
    Jurnous etcdj: https://github.com/jurmous/etcd4j
    Vulcan: http://www.vulcanproxy.com/
    SkyDNS: https://github.com/skynetservices/skydns
    Netflix Eureka: https://github.com/Netflix/eureka
    Hazelcast: http://hazelcast.com/
    Apache Zookeeper: https://zookeeper.apache.org/
    Netflix exhibitor: https://github.com/Netflix/exhibitor
    HashCorp Consul: http://consul.io/
    

    View Slide

28. .consulting .solutions .partnership
    Alexander Schwartz
    Principal IT Consultant
    +49 171 5625767
    alexander.schwartz@msg-systems.com
    @ahus1de
    msg systems ag (Headquarters)
    Robert-Buerkle-Str. 1, 85737 Ismaning
    Germany
    www.msg-systems.com
    

    View Slide