Save 37% off PRO during our Black Friday Sale! »

Need Driven Development: Tools from Chef Server Development

Need Driven Development: Tools from Chef Server Development

Do you care about reproducible builds? Does your application manipulate JSON or interact with a SQL-based database? This talk will present three open source libraries developed to support the Erlang-powered Chef Server.

We’ll cover a rebar plugin for locking all project dependencies, a tool for accessing and validating JSON, and a library for interacting with an RDBMS with a unique "No ORM” approach.

concrete
ej
sqerl
rebar_lock_deps_plugin

49b59b4f0027999a551728da1fae3029?s=128

Seth Falcon

May 17, 2014
Tweet

Transcript

  1. NEED DRIVEN DEVELOPMENT SETH FALCON ENGINEERING LEAD AT CHEF

  2. IF YOU NEED SOME SOFTWARE, WRITE IT.

  3. WE REWROTE THE CHEF SERVER IN ERLANG

  4. CONCRETE EJ SQERL REBAR LOCK DEPS PLUGIN

  5. HELP NEW TEAM MEMBERS RAMP UP ON ERLANG

  6. HELP EXISTING TEAM MEMBERS STAY SANE

  7. DO YOU USE JSON? POSTGRESQL? SHIP ERLANG CODE?

  8. JSON POSTGRESQL SANE TEAM MEMBERS SHIP ERLANG!1! REPRODUCIBLE BUILDS

  9. concrete

  10. concrete 1. project generator with standard make targets 2. edoc

    to markdown included 3. dev-only dependencies 4. fancy dialyzer goodness 5. relx support
  11. LET'S MAKE A PROJECT!

  12. concrete init todo

  13. $ concrete init todo Creating the todo project with concrete

    Would you like an active application? (y/n): y Now try: cd todo; make
  14. $ cd todo $ ls -1 Makefile README.md concrete.mk include/

    priv/ rebar.config rebar.config.script src/ test/
  15. EDOC TO MARKDOWN INCLUDED

  16. 1. WRITE EDOCS 2. VIEW THEM ON GITHUB

  17. %% @author Seth Falcon <seth@userprimary.net> %% @copyright Copyright 2011-2012 Seth

    Falcon %% %% @doc Tools for working with Erlang terms representing JSON. %% %% The ej module is intended to make it easy to work with the Erlang %% structure used by `mochijson2' to represent JSON. You can use %% `ej:get' to walk an object and return a particular value, or %% `ej:set' to update a value. %% %% @end -module(ej).
  18. None
  19. THIS ONE WEIRD TRICK TO USE EDOWN AND AVOID ADDING

    A DEPENDENCY
  20. None
  21. DEPENDENCIES EXPENSIVE

  22. %% -- rebar.config excerpt -- {dev_only_deps, [ {proper, ".*", {git,

    "git://github.com/manopapad/proper.git", "master"}} ]}.
  23. DIALYZER GOODNESS

  24. ▸ Creates a base ~/.dializer_plt if needed ▸ deps.plt just

    for your deps ▸ make dialyzer like it says on the tin
  25. concrete HTTPS://GITHUB.COM/OPSCODE/CONCRETE

  26. ej

  27. JSON { "type" : "todo", "title" : "Tie shoes", "priority"

    : 1, "meta" : { "created_at" : "2014-05-04 02:59", "created_by" : "seth" } }
  28. EJSON {[{<<"type">>, <<"todo">>}, {<<"title">>, <<"Tie shoes">>}, {<<"priority">>, 1}, {<<"meta">>, {[{<<"created_at">>:

    <<"2014-05-04 02:59">>}, {<<"created_by">>, <<"seth">>}]}} ]}
  29. LIFE BEFORE EJ Obj = {[{<<"type">>, <<"todo">>}, {<<"title">>, <<"Ties shoes">>},

    {<<"priority">>, 1}, {<<"meta">>, {[{<<"created_at">>: <<"2014-05-04 02:59">>}, {<<"created_by">>, <<"seth">>}]}} ]}
  30. LIFE BEFORE EJ {L1} = Obj, {L2} = proplists:get_value(<<"meta">>, L1),

    CreatedBy = proplists:get_value(<<"created_by">>, L2).
  31. LIFE BEFORE EJ {L1} = Obj, {L2} = proplists:get_value(<<"meta">>, L1),

    CreatedBy = proplists:get_value(<<"created_by">>, L2).
  32. None
  33. A BRIGHTER TOMORROW WITH EJ

  34. A BRIGHTER TOMORROW WITH EJ CreatedBy = ej:get(["meta", "created_by"], Obj),

    Obj2 = ej:set(["meta", "created_by"], Obj, <<"Ezra">>)
  35. SOMEDAY SOMEONE WILL GIVE YOU SOME JSON

  36. AND YOU'LL WONDER... IS IT VALID?

  37. EJ IS HERE TO HELP Spec = {[{<<"type">>, <<"todo">>}, {<<"title">>,

    string}, {opt, {<<"description">>, string}}, {opt, {<<"priority">>, integer}}, {<<"meta">>, {[{<<"created_at">>: string}, {<<"created_by">>, string}]}} ]}, ej:valid(Spec, Obj)
  38. REQUIRED KEY, EXACT VALUE MATCH Spec = {[ {<<"type">>, <<"todo">>},

    ...
  39. REQUIRED KEY, TYPE MATCH Spec = {[ ... {<<"title">>, string},

    ...
  40. OPTIONAL KEY, TYPE MATCH Spec = {[ ... {opt, {<<"description">>,

    string}}, ...
  41. ej:valid/2 ▸ Exact match ▸ Type ▸ String match (regex)

    ▸ array map ▸ object map ▸ fun match
  42. ej HTTPS://GITHUB.COM/SETH/EJ

  43. sqerl

  44. YOU WRITE YOUR QUERIES SQERL EXECUTES YOUR QUERIES TRANSFORMS RESULTS

    TO RECORDS (IF YOU WANT)
  45. SQERL BY EXAMPLE

  46. -record(todo_item, { id :: integer() , title :: binary() ,

    description :: binary() , priority :: binary() , created_at :: binary() , created_by :: binary() }).
  47. AN ITEM Item0 = #todo_item{title = <<"write tests">>, priority =

    <<"hot">>, description = <<"use eunit">>, created_by = <<"me">>}
  48. INSERT, UPDATE, FETCH [Item1] = sqerl_rec:insert(Item0). UItem0 = Item1#todo_item{priority =

    cool}. [UItem1] = sqerl_rec:update(UItem0). [FItem] = [UItem1] = sqerl_rec:fetch(todo_item, title, <<"write tests">>).
  49. DELETE, CRAZY_TOWN {ok, 1} = sqerl_rec:delete(UItem1, id). [Items] = sqerl_rec:qfetch(todo_item,

    crazy_town, Vals).
  50. WHAT DO I HAVE TO DO TO GET THIS DELIGHTFUL

    API?!
  51. -behaviour(sqerl_rec) 7 CALLBACKS

  52. 7?!

  53. ACT NOW WE'LL INCLUDE 4 CALLBACKS FOR FREE

  54. GET 4 FREE CALLBACKS USING EXPRECS

  55. FREE CALLBACKS

  56. 1. Write your schema 2. Implement sqerl_rec behaviour module 3.

    Try it out!
  57. SCHEMA CREATE DOMAIN todo_time AS TIMESTAMP WITH TIME ZONE DEFAULT

    NOW() NOT NULL; CREATE TYPE priority_enum AS ENUM ('cool', 'warm', 'hot', 'burning');
  58. SCHEMA CREATE TABLE todo_items ( id BIGSERIAL PRIMARY KEY, title

    TEXT NOT NULL UNIQUE, priority priority_enum NOT NULL DEFAULT 'warm', description TEXT, created_at todo_time, created_by TEXT NOT NULL );
  59. USE EXPRECS AND GET THESE FOR FREE -callback '#get-'(atom(), db_rec())

    -> any(). -callback '#new-'(atom()) -> db_rec(). -callback '#fromlist-'([{atom(), _}], db_rec()) -> db_rec(). -callback '#info-'(atom()) -> [atom()].
  60. YOU'LL NEED TO WRITE THESE 3 -callback '#insert_fields'() -> [atom()].

    -callback '#update_fields'() -> [atom()]. -callback '#statements'() -> [default | {atom_list(), iolist()}].
  61. -module(todo_item). -behaviour(sqerl_rec). -export([...]). -compile({parse_transform, exprecs}). -export_records([todo_item]).

  62. -record(todo_item, { id :: integer() , title :: binary() ,

    description :: binary() , priority :: binary() , created_at :: binary() , created_by :: binary() }).
  63. '#insert_fields'() -> [title, description, priority, created_by ].

  64. '#update_fields'() -> [title, description, priority].

  65. '#statements'() -> [default, {fetch_by_title, sqerl_rec:gen_fetch(todo_item, title)}, {fetch_page, sqerl_rec:gen_fetch_page(todo_item, title)}, {fetch_all,

    sqerl_rec:gen_fetch_all(todo_item, title)} ].
  66. sqerl_rec:statements([todo_item]) [ ] todo_item_fetch_all [ ] todo_item_fetch_by_title [ ] todo_item_fetch_page

    [D] todo_item_insert [D] todo_item_update [D] todo_item_delete_by_id [D] todo_item_fetch_by_id
  67. sqerl_rec:statements([todo_item]) {todo_item_fetch_by_title, "SELECT id, title, [...] FROM todo_items WHERE title

    = $1"}, {todo_item_insert, "INSERT INTO todo_items(title, description, priority, created_by) " "VALUES ($1, $2, $3, $4) RETURNING [...]"}, {todo_item_update, "UPDATE todo_items SET title = $1, description = $2, " "priority = $3 WHERE id = $4 RETURNING [...]"},
  68. sqerl_rec:gen_fetch(todo_item, title). ["SELECT ", "id, title, description, priority, created_at, created_by",

    " FROM ","todo_items"," WHERE ","title"," = $1"]
  69. DB CONNECTION POOL PREPARED QUERIES QUERY GENERATORS MAPS TO AND

    FROM RECORDS
  70. WARTS ▸ no connection control, no transaction support ▸ timestamp

    handling doesn't include TZ ▸ hard-coded pool name in pooler ▸ some no longer needed indirection since just pg ▸ using an old-ish fork of epgsql
  71. sqerl HTTPS://GITHUB.COM/OPSCODE/SQERL

  72. None
  73. SHIP SOME ERLANG?

  74. BUILD AN OTP RELEASE

  75. make rel

  76. $ ls -1 _rel bin erts-5.10.4 lib log releases

  77. EVERY BUILD IS SPECIAL

  78. _rel/lib ej-0.0.3-19-g0332523 epgsql-1.4-13-g6ceff57 sqerl-1.0.0

  79. rebar.config {sqerl, ".*", {git, "git://github.com/opscode/sqerl.git", {branch, "master"}}}, {ej, ".*", {git,

    "git://github.com/seth/ej.git", {branch, "master"}}},
  80. JUST USE TAGS?

  81. NOPE ▸ can't trust 3rd parties. ▸ your dep might

    be tagged but what about it's deps? ▸ makes extra work at release time for code you own.
  82. CHECK IN YOUR DEPS

  83. ?!

  84. rebar_lock_deps_plugin

  85. rebar lock-deps

  86. REBAR.CONFIG.LOCK {deps,[ {rebar_lock_deps_plugin,".*", {git,"git://github.com/seth/rebar_lock_deps_plugin.git", "9711549b8a84b065eb2edc22f8eb6ff85e3c94e8"}}, {parse_trans,".*", {git,"https://github.com/uwiger/parse_trans.git", "2adfbfcc17f9cd9da19134c0ecfba30052a26e25"}}, {epgsql,".*", {git,"git://github.com/opscode/epgsql.git",

    "6ceff57e8b32b6554728a082db52584f1f65bd7e"}},
  87. make prepare_release

  88. PREPARE_RELEASE 1. remove deps/ 2. re-fetch all deps without lock

    3. lock 4. bump release version
  89. rebar commit-release

  90. New deps: lager, goldrush Removed deps: fast_log Updated deps: chef_db

    2014-02-06 a1b69c1 Merge pull request #50 from opscode/of/solr4 2014-02-06 d69d5ff Move chefp to chef_db chef_index 2014-02-21 6f44357 Merge pull request #15 from opscode/sf/solr4 2014-01-03 549054b Move chef_index_expand to chef_reindex
  91. rebar_lock_deps_plugin HTTPS://GITHUB.COM/SETH/ REBAR_LOCK_DEPS_PLUGIN SUCH A CATCHY NAME!

  92. THANK YOU

  93. GITHUB: OPSCODE GITHUB: SETH TWITTER: SFALCON

  94. None