yapc_eu_2012

Transcript

1. Intro Deployment Testing DBTools Plack::* Bonus Yandex.Direct: our successful anti-modern

   Perl project Elena Bolshakova, Oleg Komarov 2012-08-20 1/36

2. Intro Deployment Testing DBTools Plack::* Bonus http://yandex.ru 2/36

3. Intro Deployment Testing DBTools Plack::* Bonus http://yandex.com 3/36

4. Intro Deployment Testing DBTools Plack::* Bonus There are ads on

   yandex 4/36

5. Intro Deployment Testing DBTools Plack::* Bonus http://direct.yandex.com 5/36

6. Intro Deployment Testing DBTools Plack::* Bonus Presentation available at https://speakerdeck.com/u/komarov/p/yapc-eu-2012

   http://bit.ly/N853eJ 6/36

7. Intro Deployment Testing DBTools Plack::* Bonus Team and codebase since

   2003 14 perl programmers 310 KSLOC Perl 7/36

8. Intro Deployment Testing DBTools Plack::* Bonus Production Web interfaces +

   APIs interfaces for clients, sales managers, agencies etc (> 10 user roles + complicate hierarchy) > 12 mln. transactions/day DB: app. 41 × 109 records Perl: 35 servers of 5 types DB: 50 servers Ubuntu + deb-packages app. 150 our own debs 8/36

9. Intro Deployment Testing DBTools Plack::* Bonus keeping packages ready easy

   cases: debosh, instant deb packaging Files do deploy (perl scripts, crontabs etc.) + 5 lines of metadata + debosh = .deb ready to upload and install See on github: https://github.com/komarov/debosh For complicated cases: custom script for every package And what’s your way? 9/36

10. Intro Deployment Testing DBTools Plack::* Bonus Keeping packages in order

    What version of X do we have on DB servers? When version N of X was installed on dev server? dpkg-monitor: script for gathering all that data + web-interface It seems to us to be a good practice And what’s your way? 10/36

11. Intro Deployment Testing DBTools Plack::* Bonus Keeping packages in order

    11/36

12. Intro Deployment Testing DBTools Plack::* Bonus Keeping DB in order:

    DB migrations about 5 schema or data migrations/week special files under ’deploy’ directory 12/36

13. Intro Deployment Testing DBTools Plack::* Bonus Keeping DB in order:

    DB migrations /project /deploy /2012_07_06_add_client_limit.migr # schema migration /2012_07_10_convert_favorites.pl # data migration ... /lib /t ... It seems to us to be a good practice And what’s your way? 13/36

14. Intro Deployment Testing DBTools Plack::* Bonus Keeping DB in order:

    DB migration example 2012_07_06_add_client_limit.migr: { type => ’sql’, heavy => 1, when => ’before’, time_estimate => ’3 min’, sql => ’alter table clients add column some_limit int’, } And what’s your way? 14/36

15. Intro Deployment Testing DBTools Plack::* Bonus Release testing continuous delivery

    compliance with the Sarbanes-Oxley Act JIRA as bugtracker Thanks CPAN for JIRA::Client branches in svn 15/36

16. Intro Deployment Testing DBTools Plack::* Bonus Unit tests 1 After

    every commit to mainline (+ personal notifications on fail) What to test: perl server-side js perl + DB (special DB for unit-tests, no mock) It seems to us to be a good practice http, tcp interaction Thanks CPAN for Test::HTTP::Server, AnyEvent::Socket 16/36

17. Intro Deployment Testing DBTools Plack::* Bonus Unit tests 2: non-functional

    Non-functional tests: encoding of source code i18n (documentation|test) coverage (perl|TT) syntax deprecated subs crontabs extra or missing js & TT (used but absent, present but unused) ... – whatever we can imagine It seems to us to be a good practice 17/36

18. Intro Deployment Testing DBTools Plack::* Bonus DBTools Our inhouse module

    (MySQL-specific) to handle db configs connects and reconnects boilerplate sql construction Also it is replication-aware supports transactions makes unit-tests easier 18/36

19. Intro Deployment Testing DBTools Plack::* Bonus DBTools: easy things do_insert_into_table(DB,

    "names", {id => 2, name => ’bb’} ); get_one_field_sql(DB, "SELECT count(*) FROM names"); get_one_column_sql(DB, "SELECT name FROM names"); get_all_sql(DB, "SELECT id, name FROM names"); 19/36

20. Intro Deployment Testing DBTools Plack::* Bonus DBTools: easy things my

    $id; do_in_transaction { $id = do_insert_into_table(DB, "names", {name => ’gg’} ); do_update_table(DB, "totals", {count__dont_quote => ’count + 1’}, where => {table_name => ’names’} ); }; # do smth with $id 20/36

21. Intro Deployment Testing DBTools Plack::* Bonus DBTools: sometimes you know

    better get_hashes_hash_sql(DB, [ "SELECT STRAIGHT_JOIN t1.id, t1.name, t2.status, t3.date FROM t1 JOIN t2 ON t1.id = t2.id JOIN t3 ON t2.some_id = t3.id", WHERE => { id => [1, 2, ..., 1000], ’t3.is_active’ => 1 } ]); 21/36

22. Intro Deployment Testing DBTools Plack::* Bonus DBTools no orm. why?

    full control over executed sql no FOREIGN KEYS in db tables have big rows, SELECT * is bad why not SQL::Abstract? you still need to manipulate with dbh and sth you can’t construct complex JOINs, can you? 22/36

23. Intro Deployment Testing DBTools Plack::* Bonus DBTools: one more example

    UPDATE names SET name = CASE id WHEN 1 THEN ’aa’ WHEN 2 THEN ’bb’ ... END WHERE id IN (1, 2, ...) 23/36

24. Intro Deployment Testing DBTools Plack::* Bonus DBTools: one more example

    Construct you CASE with sql_case(DB, "id", {1 => ’a’, 2 => ’b’, ..., }) Pretty simple, right? 24/36

25. Intro Deployment Testing DBTools Plack::* Bonus DBTools: one more example

    Construct you CASE with sql_case(DB, "id", {1 => ’a’, 2 => ’b’, ..., }) Pretty simple, right? When ”...” hides thousands, the performance might surprise you. 24/36

26. Intro Deployment Testing DBTools Plack::* Bonus DBTools: one more example

    Too many numeric keys? Construct binary tree of CASEs! IF(id < 50, IF(id < 25, CASE..., CASE...), IF(id < 75, CASE..., CASE...) ) 25/36

27. Intro Deployment Testing DBTools Plack::* Bonus Plack We love Plack

    26/36

28. Intro Deployment Testing DBTools Plack::* Bonus Plack::Middleware::* ACL Auth Captcha

    MemLimit Reqid Usertrack CmdLogs IPREG DevTools 27/36

29. Intro Deployment Testing DBTools Plack::* Bonus P::M::DevTools use Aspect ();

    sub call { my ($self, $env) = @_; my @devtools_sections = qw/sql http/; for my $section (@devtools_sections) { $env->{"devtools.$section"} = []; } my @hooks; ... 28/36

30. Intro Deployment Testing DBTools Plack::* Bonus P::M::DevTools ... push @hooks,

    Aspect::before { my (undef, $url) = $_->args; push @{$env->{’devtools.http’}}, $url; } Aspect::call qr/^LWP::UserAgent::(?:get|post)$/; push @hooks, Aspect::around { # DBI->trace } Aspect::call qr/::exec_sql$/; ... 29/36

31. Intro Deployment Testing DBTools Plack::* Bonus P::M::DevTools ... push @hooks,

    Aspect::before { my ($response_type, $response) = $_->args; my $devtools = { map {$_ => $env->{"devtools.$_"}} @devtools_sections }; ... # $response->{_devtools} = $devtools; } Aspect::call qr/::respond$/; return $self->app->($env); } 30/36

32. Intro Deployment Testing DBTools Plack::* Bonus Plack::ResponseHelper A very thin

    layer that abstracts Plack’s specifics. Comes with P::RH::Attachment, P::RH::JSON, P::RH::Redirect, P::RH::Text DRY, give it a name and write a helper! See on CPAN: https://metacpan.org/module/Plack::ResponseHelper 31/36

33. Intro Deployment Testing DBTools Plack::* Bonus Plack::ResponseHelper Your ”hello world”

    app.psgi use Plack::ResponseHelper text => ’Text’; sub { respond text => ’Hello world!’; } 32/36

34. Intro Deployment Testing DBTools Plack::* Bonus Plack::ResponseHelper Your real app

    use Plack::ResponseHelper csv => [’Attachment’, {content_type => ’text/csv; charset=utf-8’}], json => ’JSON’, redirect => ’Redirect’, template => [’Template’, {get_tt_object => sub {...}}], page404 => [’Redirect’, {default_location => ’/404.html’}]; sub { # routing, etc respond $controller->($env); } 33/36

35. Intro Deployment Testing DBTools Plack::* Bonus Thanks! Any questions? Drop

    us a line: [email protected] [email protected] https://speakerdeck.com/u/komarov/p/yapc-eu-2012 http://bit.ly/N853eJ 34/36

36. Intro Deployment Testing DBTools Plack::* Bonus XMLRPC::Transport::HTTP::Plack sub { my

    $env = shift; my $r = Plack::Request->new($env); my $cmd = router($r); my $multiform = $r->parameters; XMLRPC::Transport::HTTP::Plack ->dispatch_to($path, $module) ->handler($r, $cmd, $multiform); } See on CPAN: https://metacpan.org/module/XMLRPC::Transport::HTTP::Plack 35/36

37. Intro Deployment Testing DBTools Plack::* Bonus Server-side JS See on

    github: https://github.com/ezhi/javascript-v8 transparent manipulation with objects and functions memoizing of already converted objects See on github: http://bem.github.com/bem-method/html/all.en.html 36/36