Need Driven Development: Tools from Chef Server Development

Do you care about reproducible builds? Does your application manipulate JSON or interact with a SQL-based database? This talk will present three open source libraries developed to support the Erlang-powered Chef Server.

We’ll cover a rebar plugin for locking all project dependencies, a tool for accessing and validating JSON, and a library for interacting with an RDBMS with a unique "No ORM” approach.

concrete
ej
sqerl
rebar_lock_deps_plugin

Transcript

1. NEED DRIVEN DEVELOPMENT SETH FALCON ENGINEERING LEAD AT CHEF
2. IF YOU NEED SOME SOFTWARE, WRITE IT.
3. WE REWROTE THE CHEF SERVER IN ERLANG
4. CONCRETE EJ SQERL REBAR LOCK DEPS PLUGIN
5. HELP NEW TEAM MEMBERS RAMP UP ON ERLANG
6. HELP EXISTING TEAM MEMBERS STAY SANE
7. DO YOU USE JSON? POSTGRESQL? SHIP ERLANG CODE?
8. JSON POSTGRESQL SANE TEAM MEMBERS SHIP ERLANG!1! REPRODUCIBLE BUILDS
9. concrete
10. concrete 1. project generator with standard make targets 2. edoc to markdown included 3. dev-only dependencies 4. fancy dialyzer goodness 5. relx support
11. LET'S MAKE A PROJECT!
12. concrete init todo
13. $ concrete init todo Creating the todo project with concrete Would you like an active application? (y/n): y Now try: cd todo; make
14. $ cd todo $ ls -1 Makefile README.md concrete.mk include/ priv/ rebar.config rebar.config.script src/ test/
15. EDOC TO MARKDOWN INCLUDED
16. 1. WRITE EDOCS 2. VIEW THEM ON GITHUB
17. %% @author Seth Falcon <seth@userprimary.net> %% @copyright Copyright 2011-2012 Seth Falcon %% %% @doc Tools for working with Erlang terms representing JSON. %% %% The ej module is intended to make it easy to work with the Erlang %% structure used by `mochijson2' to represent JSON. You can use %% `ej:get' to walk an object and return a particular value, or %% `ej:set' to update a value. %% %% @end -module(ej).
19. THIS ONE WEIRD TRICK TO USE EDOWN AND AVOID ADDING A DEPENDENCY
21. DEPENDENCIES EXPENSIVE
22. %% -- rebar.config excerpt -- {dev_only_deps, [ {proper, ".*", {git, "git://github.com/manopapad/proper.git", "master"}} ]}.
23. DIALYZER GOODNESS
24. ▸ Creates a base ~/.dializer_plt if needed ▸ deps.plt just for your deps ▸ make dialyzer like it says on the tin
25. concrete HTTPS://GITHUB.COM/OPSCODE/CONCRETE
26. ej
27. JSON { "type" : "todo", "title" : "Tie shoes", "priority" : 1, "meta" : { "created_at" : "2014-05-04 02:59", "created_by" : "seth" } }
28. EJSON {[{<<"type">>, <<"todo">>}, {<<"title">>, <<"Tie shoes">>}, {<<"priority">>, 1}, {<<"meta">>, {[{<<"created_at">>: <<"2014-05-04 02:59">>}, {<<"created_by">>, <<"seth">>}]}} ]}
29. LIFE BEFORE EJ Obj = {[{<<"type">>, <<"todo">>}, {<<"title">>, <<"Ties shoes">>}, {<<"priority">>, 1}, {<<"meta">>, {[{<<"created_at">>: <<"2014-05-04 02:59">>}, {<<"created_by">>, <<"seth">>}]}} ]}
30. LIFE BEFORE EJ {L1} = Obj, {L2} = proplists:get_value(<<"meta">>, L1), CreatedBy = proplists:get_value(<<"created_by">>, L2).
31. LIFE BEFORE EJ {L1} = Obj, {L2} = proplists:get_value(<<"meta">>, L1), CreatedBy = proplists:get_value(<<"created_by">>, L2).
33. A BRIGHTER TOMORROW WITH EJ
34. A BRIGHTER TOMORROW WITH EJ CreatedBy = ej:get(["meta", "created_by"], Obj), Obj2 = ej:set(["meta", "created_by"], Obj, <<"Ezra">>)
35. SOMEDAY SOMEONE WILL GIVE YOU SOME JSON
36. AND YOU'LL WONDER... IS IT VALID?
37. EJ IS HERE TO HELP Spec = {[{<<"type">>, <<"todo">>}, {<<"title">>, string}, {opt, {<<"description">>, string}}, {opt, {<<"priority">>, integer}}, {<<"meta">>, {[{<<"created_at">>: string}, {<<"created_by">>, string}]}} ]}, ej:valid(Spec, Obj)
38. REQUIRED KEY, EXACT VALUE MATCH Spec = {[ {<<"type">>, <<"todo">>}, ...
39. REQUIRED KEY, TYPE MATCH Spec = {[ ... {<<"title">>, string}, ...
40. OPTIONAL KEY, TYPE MATCH Spec = {[ ... {opt, {<<"description">>, string}}, ...
41. ej:valid/2 ▸ Exact match ▸ Type ▸ String match (regex) ▸ array map ▸ object map ▸ fun match
42. ej HTTPS://GITHUB.COM/SETH/EJ
43. sqerl
44. YOU WRITE YOUR QUERIES SQERL EXECUTES YOUR QUERIES TRANSFORMS RESULTS TO RECORDS (IF YOU WANT)
45. SQERL BY EXAMPLE
46. -record(todo_item, { id :: integer() , title :: binary() , description :: binary() , priority :: binary() , created_at :: binary() , created_by :: binary() }).
47. AN ITEM Item0 = #todo_item{title = <<"write tests">>, priority = <<"hot">>, description = <<"use eunit">>, created_by = <<"me">>}
48. INSERT, UPDATE, FETCH [Item1] = sqerl_rec:insert(Item0). UItem0 = Item1#todo_item{priority = cool}. [UItem1] = sqerl_rec:update(UItem0). [FItem] = [UItem1] = sqerl_rec:fetch(todo_item, title, <<"write tests">>).
49. DELETE, CRAZY_TOWN {ok, 1} = sqerl_rec:delete(UItem1, id). [Items] = sqerl_rec:qfetch(todo_item, crazy_town, Vals).
50. WHAT DO I HAVE TO DO TO GET THIS DELIGHTFUL API?!
51. -behaviour(sqerl_rec) 7 CALLBACKS
52. 7?!
53. ACT NOW WE'LL INCLUDE 4 CALLBACKS FOR FREE
54. GET 4 FREE CALLBACKS USING EXPRECS
55. FREE CALLBACKS
56. 1. Write your schema 2. Implement sqerl_rec behaviour module 3. Try it out!
57. SCHEMA CREATE DOMAIN todo_time AS TIMESTAMP WITH TIME ZONE DEFAULT NOW() NOT NULL; CREATE TYPE priority_enum AS ENUM ('cool', 'warm', 'hot', 'burning');
58. SCHEMA CREATE TABLE todo_items ( id BIGSERIAL PRIMARY KEY, title TEXT NOT NULL UNIQUE, priority priority_enum NOT NULL DEFAULT 'warm', description TEXT, created_at todo_time, created_by TEXT NOT NULL );
59. USE EXPRECS AND GET THESE FOR FREE -callback '#get-'(atom(), db_rec()) -> any(). -callback '#new-'(atom()) -> db_rec(). -callback '#fromlist-'([{atom(), _}], db_rec()) -> db_rec(). -callback '#info-'(atom()) -> [atom()].
60. YOU'LL NEED TO WRITE THESE 3 -callback '#insert_fields'() -> [atom()]. -callback '#update_fields'() -> [atom()]. -callback '#statements'() -> [default | {atom_list(), iolist()}].
61. -module(todo_item). -behaviour(sqerl_rec). -export([...]). -compile({parse_transform, exprecs}). -export_records([todo_item]).
62. -record(todo_item, { id :: integer() , title :: binary() , description :: binary() , priority :: binary() , created_at :: binary() , created_by :: binary() }).
63. '#insert_fields'() -> [title, description, priority, created_by ].
64. '#update_fields'() -> [title, description, priority].
65. '#statements'() -> [default, {fetch_by_title, sqerl_rec:gen_fetch(todo_item, title)}, {fetch_page, sqerl_rec:gen_fetch_page(todo_item, title)}, {fetch_all, sqerl_rec:gen_fetch_all(todo_item, title)} ].
66. sqerl_rec:statements([todo_item]) [ ] todo_item_fetch_all [ ] todo_item_fetch_by_title [ ] todo_item_fetch_page [D] todo_item_insert [D] todo_item_update [D] todo_item_delete_by_id [D] todo_item_fetch_by_id
67. sqerl_rec:statements([todo_item]) {todo_item_fetch_by_title, "SELECT id, title, [...] FROM todo_items WHERE title = $1"}, {todo_item_insert, "INSERT INTO todo_items(title, description, priority, created_by) " "VALUES ($1, $2, $3, $4) RETURNING [...]"}, {todo_item_update, "UPDATE todo_items SET title = $1, description = $2, " "priority = $3 WHERE id = $4 RETURNING [...]"},
68. sqerl_rec:gen_fetch(todo_item, title). ["SELECT ", "id, title, description, priority, created_at, created_by", " FROM ","todo_items"," WHERE ","title"," = $1"]
69. DB CONNECTION POOL PREPARED QUERIES QUERY GENERATORS MAPS TO AND FROM RECORDS
70. WARTS ▸ no connection control, no transaction support ▸ timestamp handling doesn't include TZ ▸ hard-coded pool name in pooler ▸ some no longer needed indirection since just pg ▸ using an old-ish fork of epgsql
71. sqerl HTTPS://GITHUB.COM/OPSCODE/SQERL
73. SHIP SOME ERLANG?
74. BUILD AN OTP RELEASE
75. make rel
76. $ ls -1 _rel bin erts-5.10.4 lib log releases
77. EVERY BUILD IS SPECIAL
78. _rel/lib ej-0.0.3-19-g0332523 epgsql-1.4-13-g6ceff57 sqerl-1.0.0
79. rebar.config {sqerl, ".*", {git, "git://github.com/opscode/sqerl.git", {branch, "master"}}}, {ej, ".*", {git, "git://github.com/seth/ej.git", {branch, "master"}}},
80. JUST USE TAGS?
81. NOPE ▸ can't trust 3rd parties. ▸ your dep might be tagged but what about it's deps? ▸ makes extra work at release time for code you own.
82. CHECK IN YOUR DEPS
83. ?!
84. rebar_lock_deps_plugin
85. rebar lock-deps
86. REBAR.CONFIG.LOCK {deps,[ {rebar_lock_deps_plugin,".*", {git,"git://github.com/seth/rebar_lock_deps_plugin.git", "9711549b8a84b065eb2edc22f8eb6ff85e3c94e8"}}, {parse_trans,".*", {git,"https://github.com/uwiger/parse_trans.git", "2adfbfcc17f9cd9da19134c0ecfba30052a26e25"}}, {epgsql,".*", {git,"git://github.com/opscode/epgsql.git", "6ceff57e8b32b6554728a082db52584f1f65bd7e"}},
87. make prepare_release
88. PREPARE_RELEASE 1. remove deps/ 2. re-fetch all deps without lock 3. lock 4. bump release version
89. rebar commit-release
90. New deps: lager, goldrush Removed deps: fast_log Updated deps: chef_db 2014-02-06 a1b69c1 Merge pull request #50 from opscode/of/solr4 2014-02-06 d69d5ff Move chefp to chef_db chef_index 2014-02-21 6f44357 Merge pull request #15 from opscode/sf/solr4 2014-01-03 549054b Move chef_index_expand to chef_reindex
91. rebar_lock_deps_plugin HTTPS://GITHUB.COM/SETH/ REBAR_LOCK_DEPS_PLUGIN SUCH A CATCHY NAME!
92. THANK YOU
93. GITHUB: OPSCODE GITHUB: SETH TWITTER: SFALCON