Need Driven Development: Tools from Chef Server Development

Slide 1

Slide 1 text

NEED DRIVEN DEVELOPMENT SETH FALCON ENGINEERING LEAD AT CHEF

Slide 2

Slide 2 text

IF YOU NEED SOME SOFTWARE, WRITE IT.

Slide 3

Slide 3 text

WE REWROTE THE CHEF SERVER IN ERLANG

Slide 4

Slide 4 text

CONCRETE EJ SQERL REBAR LOCK DEPS PLUGIN

Slide 5

Slide 5 text

HELP NEW TEAM MEMBERS RAMP UP ON ERLANG

Slide 6

Slide 6 text

HELP EXISTING TEAM MEMBERS STAY SANE

Slide 7

Slide 7 text

DO YOU USE JSON? POSTGRESQL? SHIP ERLANG CODE?

Slide 8

Slide 8 text

JSON POSTGRESQL SANE TEAM MEMBERS SHIP ERLANG!1! REPRODUCIBLE BUILDS

Slide 9

Slide 9 text

concrete

Slide 10

Slide 10 text

concrete 1. project generator with standard make targets 2. edoc to markdown included 3. dev-only dependencies 4. fancy dialyzer goodness 5. relx support

Slide 11

Slide 11 text

LET'S MAKE A PROJECT!

Slide 12

Slide 12 text

concrete init todo

Slide 13

Slide 13 text

$ concrete init todo Creating the todo project with concrete Would you like an active application? (y/n): y Now try: cd todo; make

Slide 14

Slide 14 text

$ cd todo $ ls -1 Makefile README.md concrete.mk include/ priv/ rebar.config rebar.config.script src/ test/

Slide 15

Slide 15 text

EDOC TO MARKDOWN INCLUDED

Slide 16

Slide 16 text

1. WRITE EDOCS 2. VIEW THEM ON GITHUB

Slide 17

Slide 17 text

%% @author Seth Falcon %% @copyright Copyright 2011-2012 Seth Falcon %% %% @doc Tools for working with Erlang terms representing JSON. %% %% The ej module is intended to make it easy to work with the Erlang %% structure used by `mochijson2' to represent JSON. You can use %% `ej:get' to walk an object and return a particular value, or %% `ej:set' to update a value. %% %% @end -module(ej).

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

THIS ONE WEIRD TRICK TO USE EDOWN AND AVOID ADDING A DEPENDENCY

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

DEPENDENCIES EXPENSIVE

Slide 22

Slide 22 text

%% -- rebar.config excerpt -- {dev_only_deps, [ {proper, ".*", {git, "git://github.com/manopapad/proper.git", "master"}} ]}.

Slide 23

Slide 23 text

DIALYZER GOODNESS

Slide 24

Slide 24 text

▸ Creates a base ~/.dializer_plt if needed ▸ deps.plt just for your deps ▸ make dialyzer like it says on the tin

Slide 25

Slide 25 text

concrete HTTPS://GITHUB.COM/OPSCODE/CONCRETE

Slide 26

Slide 26 text

Slide 27

Slide 27 text

JSON { "type" : "todo", "title" : "Tie shoes", "priority" : 1, "meta" : { "created_at" : "2014-05-04 02:59", "created_by" : "seth" } }

Slide 28

Slide 28 text

EJSON {[{<<"type">>, <<"todo">>}, {<<"title">>, <<"Tie shoes">>}, {<<"priority">>, 1}, {<<"meta">>, {[{<<"created_at">>: <<"2014-05-04 02:59">>}, {<<"created_by">>, <<"seth">>}]}} ]}

Slide 29

Slide 29 text

LIFE BEFORE EJ Obj = {[{<<"type">>, <<"todo">>}, {<<"title">>, <<"Ties shoes">>}, {<<"priority">>, 1}, {<<"meta">>, {[{<<"created_at">>: <<"2014-05-04 02:59">>}, {<<"created_by">>, <<"seth">>}]}} ]}

Slide 30

Slide 30 text

LIFE BEFORE EJ {L1} = Obj, {L2} = proplists:get_value(<<"meta">>, L1), CreatedBy = proplists:get_value(<<"created_by">>, L2).

Slide 31

Slide 31 text

LIFE BEFORE EJ {L1} = Obj, {L2} = proplists:get_value(<<"meta">>, L1), CreatedBy = proplists:get_value(<<"created_by">>, L2).

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

A BRIGHTER TOMORROW WITH EJ

Slide 34

Slide 34 text

A BRIGHTER TOMORROW WITH EJ CreatedBy = ej:get(["meta", "created_by"], Obj), Obj2 = ej:set(["meta", "created_by"], Obj, <<"Ezra">>)

Slide 35

Slide 35 text

SOMEDAY SOMEONE WILL GIVE YOU SOME JSON

Slide 36

Slide 36 text

AND YOU'LL WONDER... IS IT VALID?

Slide 37

Slide 37 text

EJ IS HERE TO HELP Spec = {[{<<"type">>, <<"todo">>}, {<<"title">>, string}, {opt, {<<"description">>, string}}, {opt, {<<"priority">>, integer}}, {<<"meta">>, {[{<<"created_at">>: string}, {<<"created_by">>, string}]}} ]}, ej:valid(Spec, Obj)

Slide 38

Slide 38 text

REQUIRED KEY, EXACT VALUE MATCH Spec = {[ {<<"type">>, <<"todo">>}, ...

Slide 39

Slide 39 text

REQUIRED KEY, TYPE MATCH Spec = {[ ... {<<"title">>, string}, ...

Slide 40

Slide 40 text

OPTIONAL KEY, TYPE MATCH Spec = {[ ... {opt, {<<"description">>, string}}, ...

Slide 41

Slide 41 text

ej:valid/2 ▸ Exact match ▸ Type ▸ String match (regex) ▸ array map ▸ object map ▸ fun match

Slide 42

Slide 42 text

ej HTTPS://GITHUB.COM/SETH/EJ

Slide 43

Slide 43 text

sqerl

Slide 44

Slide 44 text

YOU WRITE YOUR QUERIES SQERL EXECUTES YOUR QUERIES TRANSFORMS RESULTS TO RECORDS (IF YOU WANT)

Slide 45

Slide 45 text

SQERL BY EXAMPLE

Slide 46

Slide 46 text

-record(todo_item, { id :: integer() , title :: binary() , description :: binary() , priority :: binary() , created_at :: binary() , created_by :: binary() }).

Slide 47

Slide 47 text

AN ITEM Item0 = #todo_item{title = <<"write tests">>, priority = <<"hot">>, description = <<"use eunit">>, created_by = <<"me">>}

Slide 48

Slide 48 text

INSERT, UPDATE, FETCH [Item1] = sqerl_rec:insert(Item0). UItem0 = Item1#todo_item{priority = cool}. [UItem1] = sqerl_rec:update(UItem0). [FItem] = [UItem1] = sqerl_rec:fetch(todo_item, title, <<"write tests">>).

Slide 49

Slide 49 text

DELETE, CRAZY_TOWN {ok, 1} = sqerl_rec:delete(UItem1, id). [Items] = sqerl_rec:qfetch(todo_item, crazy_town, Vals).

Slide 50

Slide 50 text

WHAT DO I HAVE TO DO TO GET THIS DELIGHTFUL API?!

Slide 51

Slide 51 text

-behaviour(sqerl_rec) 7 CALLBACKS

Slide 52

Slide 52 text

7?!

Slide 53

Slide 53 text

ACT NOW WE'LL INCLUDE 4 CALLBACKS FOR FREE

Slide 54

Slide 54 text

GET 4 FREE CALLBACKS USING EXPRECS

Slide 55

Slide 55 text

FREE CALLBACKS

Slide 56

Slide 56 text

1. Write your schema 2. Implement sqerl_rec behaviour module 3. Try it out!

Slide 57

Slide 57 text

SCHEMA CREATE DOMAIN todo_time AS TIMESTAMP WITH TIME ZONE DEFAULT NOW() NOT NULL; CREATE TYPE priority_enum AS ENUM ('cool', 'warm', 'hot', 'burning');

Slide 58

Slide 58 text

SCHEMA CREATE TABLE todo_items ( id BIGSERIAL PRIMARY KEY, title TEXT NOT NULL UNIQUE, priority priority_enum NOT NULL DEFAULT 'warm', description TEXT, created_at todo_time, created_by TEXT NOT NULL );

Slide 59

Slide 59 text

USE EXPRECS AND GET THESE FOR FREE -callback '#get-'(atom(), db_rec()) -> any(). -callback '#new-'(atom()) -> db_rec(). -callback '#fromlist-'([{atom(), _}], db_rec()) -> db_rec(). -callback '#info-'(atom()) -> [atom()].

Slide 60

Slide 60 text

YOU'LL NEED TO WRITE THESE 3 -callback '#insert_fields'() -> [atom()]. -callback '#update_fields'() -> [atom()]. -callback '#statements'() -> [default | {atom_list(), iolist()}].

Slide 61

Slide 61 text

-module(todo_item). -behaviour(sqerl_rec). -export([...]). -compile({parse_transform, exprecs}). -export_records([todo_item]).

Slide 62

Slide 62 text

-record(todo_item, { id :: integer() , title :: binary() , description :: binary() , priority :: binary() , created_at :: binary() , created_by :: binary() }).

Slide 63

Slide 63 text

'#insert_fields'() -> [title, description, priority, created_by ].

Slide 64

Slide 64 text

'#update_fields'() -> [title, description, priority].

Slide 65

Slide 65 text

'#statements'() -> [default, {fetch_by_title, sqerl_rec:gen_fetch(todo_item, title)}, {fetch_page, sqerl_rec:gen_fetch_page(todo_item, title)}, {fetch_all, sqerl_rec:gen_fetch_all(todo_item, title)} ].

Slide 66

Slide 66 text

sqerl_rec:statements([todo_item]) [ ] todo_item_fetch_all [ ] todo_item_fetch_by_title [ ] todo_item_fetch_page [D] todo_item_insert [D] todo_item_update [D] todo_item_delete_by_id [D] todo_item_fetch_by_id

Slide 67

Slide 67 text

sqerl_rec:statements([todo_item]) {todo_item_fetch_by_title, "SELECT id, title, [...] FROM todo_items WHERE title = $1"}, {todo_item_insert, "INSERT INTO todo_items(title, description, priority, created_by) " "VALUES ($1, $2, $3, $4) RETURNING [...]"}, {todo_item_update, "UPDATE todo_items SET title = $1, description = $2, " "priority = $3 WHERE id = $4 RETURNING [...]"},

Slide 68

Slide 68 text

sqerl_rec:gen_fetch(todo_item, title). ["SELECT ", "id, title, description, priority, created_at, created_by", " FROM ","todo_items"," WHERE ","title"," = $1"]

Slide 69

Slide 69 text

DB CONNECTION POOL PREPARED QUERIES QUERY GENERATORS MAPS TO AND FROM RECORDS

Slide 70

Slide 70 text

WARTS ▸ no connection control, no transaction support ▸ timestamp handling doesn't include TZ ▸ hard-coded pool name in pooler ▸ some no longer needed indirection since just pg ▸ using an old-ish fork of epgsql

Slide 71

Slide 71 text

sqerl HTTPS://GITHUB.COM/OPSCODE/SQERL

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

SHIP SOME ERLANG?

Slide 74

Slide 74 text

BUILD AN OTP RELEASE

Slide 75

Slide 75 text

make rel

Slide 76

Slide 76 text

$ ls -1 _rel bin erts-5.10.4 lib log releases

Slide 77

Slide 77 text

EVERY BUILD IS SPECIAL

Slide 78

Slide 78 text

_rel/lib ej-0.0.3-19-g0332523 epgsql-1.4-13-g6ceff57 sqerl-1.0.0

Slide 79

Slide 79 text

rebar.config {sqerl, ".*", {git, "git://github.com/opscode/sqerl.git", {branch, "master"}}}, {ej, ".*", {git, "git://github.com/seth/ej.git", {branch, "master"}}},

Slide 80

Slide 80 text

JUST USE TAGS?

Slide 81

Slide 81 text

NOPE ▸ can't trust 3rd parties. ▸ your dep might be tagged but what about it's deps? ▸ makes extra work at release time for code you own.

Slide 82

Slide 82 text

CHECK IN YOUR DEPS

Slide 83

Slide 83 text

Slide 84

Slide 84 text

rebar_lock_deps_plugin

Slide 85

Slide 85 text

rebar lock-deps

Slide 86

Slide 86 text

REBAR.CONFIG.LOCK {deps,[ {rebar_lock_deps_plugin,".*", {git,"git://github.com/seth/rebar_lock_deps_plugin.git", "9711549b8a84b065eb2edc22f8eb6ff85e3c94e8"}}, {parse_trans,".*", {git,"https://github.com/uwiger/parse_trans.git", "2adfbfcc17f9cd9da19134c0ecfba30052a26e25"}}, {epgsql,".*", {git,"git://github.com/opscode/epgsql.git", "6ceff57e8b32b6554728a082db52584f1f65bd7e"}},

Slide 87

Slide 87 text

make prepare_release

Slide 88

Slide 88 text

PREPARE_RELEASE 1. remove deps/ 2. re-fetch all deps without lock 3. lock 4. bump release version

Slide 89

Slide 89 text

rebar commit-release

Slide 90

Slide 90 text

New deps: lager, goldrush Removed deps: fast_log Updated deps: chef_db 2014-02-06 a1b69c1 Merge pull request #50 from opscode/of/solr4 2014-02-06 d69d5ff Move chefp to chef_db chef_index 2014-02-21 6f44357 Merge pull request #15 from opscode/sf/solr4 2014-01-03 549054b Move chef_index_expand to chef_reindex

Slide 91

Slide 91 text

rebar_lock_deps_plugin HTTPS://GITHUB.COM/SETH/ REBAR_LOCK_DEPS_PLUGIN SUCH A CATCHY NAME!