To infinity and beyond

To infinity and beyond

Elastic::Model is a new framework to store your Moose objects, which uses ElasticSearch as a NoSQL document store and flexible search engine.

It is designed to make small beginnings simple, but to scale easily to Big Data requirements without needing to rearchitect your application. No job too big or small!

This talk will introduce Elastic::Model, demonstrate how to develop a simple application, introduce some more advanced techniques, and discuss how it uses ElasticSearch to scale.

https://github.com/clintongormley/Elastic-Model

D0dd23d18388ba0225bbb9bcba7ede83?s=128

Clinton Gormley

August 21, 2012
Tweet

Transcript

  1. 1.

    To infinity and beyond! A practical guide for Mooseherds (and

    other carers of livestock) @clintongormley #elasticsearch YAPC::EU 2012
  2. 21.
  3. 33.
  4. 36.
  5. 54.
  6. 55.

    package MyApp::Post; use Moose; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } );
  7. 56.

    package MyApp::Post; use Moose; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } ); package MyApp::User; use Moose; has 'name' => ( is => 'rw', isa => 'Str' ); has 'email' => ( is => 'rw', isa => 'Str', required => 1 );
  8. 57.

    package MyApp::Post; use Moose; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } ); has 'user' => ( is => 'ro', isa => 'MyApp::User', ); package MyApp::User; use Moose; has 'name' => ( is => 'rw', isa => 'Str' ); has 'email' => ( is => 'rw', isa => 'Str', required => 1 );
  9. 58.

    package MyApp::Post; use Moose; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } ); has 'user' => ( is => 'ro', isa => 'MyApp::User', ); package MyApp::User; use Moose; has 'name' => ( is => 'rw', isa => 'Str' ); has 'email' => ( is => 'rw', isa => 'Str', required => 1 );
  10. 59.

    package MyApp::Post; use Moose; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } ); has 'user' => ( is => 'ro', isa => 'MyApp::User', ); package MyApp::User; use Moose; has 'name' => ( is => 'rw', isa => 'Str' ); has 'email' => ( is => 'rw', isa => 'Str', required => 1 );
  11. 60.

    package MyApp::Post; use Elastic::Doc; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } ); has 'user' => ( is => 'ro', isa => 'MyApp::User', ); package MyApp::User; use Elastic::Doc; has 'name' => ( is => 'rw', isa => 'Str' ); has 'email' => ( is => 'rw', isa => 'Str', required => 1 );
  12. 61.

    Some definitions... * index * type * doc * alias

    Like a database Like a table Like a row in a table Like a symbolic link, points to one or more indices elasticsearch Elastic::Model * domain * namespace * model An index or an alias, used for CRUD Maps type <=> class for all associated domains Connects your app to elasticsearch.
  13. 65.
  14. 66.

    package MyApp; use Elastic::Model; has_namespace 'myapp' => { user =>

    'MyApp::User', post => 'MyApp::Post, }; # like table <=> class
  15. 70.

    use MyApp; my $model = MyApp->new; my $namespace = $model->namespace('myapp');

    # For index and alias management my $domain = $model->domain('myapp'); # For document CRUD my $view = $model->view; # For searching To do anything useful, we need:
  16. 74.

    Domain: Create a user my $domain = $model->domain('myapp'); my $user

    = $domain->new_doc( user => { name => 'Clinton', email => 'clint@foo.com', } ); $user->save;
  17. 75.

    Domain: Create a user my $domain = $model->domain('myapp'); my $user

    = $domain->create( user => { name => 'Clinton', email => 'clint@foo.com', } ); $user->save;
  18. 76.

    Domain: Create a user my $domain = $model->domain('myapp'); my $user

    = $domain->create( user => { name => 'Clinton', email => 'clint@foo.com', id => 1, } ); say $user->id; # 1 say $user->type; # user
  19. 77.

    Domain: Create a post my $domain = $model->domain('myapp'); my $post

    = $domain->create( post => { id => 2, title => 'To infinity and beyond', content => 'Elastic::Model persists Moose ' . . 'objects in elasticsearch', user => $user } );
  20. 78.

    Domain: Retrieve a doc my $domain = $model->domain('myapp'); my $post

    = $domain->get( post => 2 ); my $user = $post->user; # stub object say $user->name; # full object # Clinton say $user->id; # still stub # 1
  21. 79.

    Domain: Update a doc my $domain = $model->domain('myapp'); $post->title('Awesome blog

    post'); say $post->has_changed; # 1 say $post->has_changed('title'); # 1 say $post->old_value('title'); # To infinity and beyond $post->save;
  22. 82.

    1: $post = $domain->get(post=>2); 2: $post = $domain->get(post=>2); 1: $post->title('Awesome

    blog post'); 2: $post->title('Brilliant blog post'); 1: $post->save; 2: $post->save; *** CONFLICT ERROR ***
  23. 86.

    $post->save( on_conflict => sub { my ($old,$new) = @_; #

    do something # to resolve conflict });
  24. 87.

    $post->save( on_conflict => sub { my ($old,$new) = @_; my

    %changed = $old->old_values; $new->$_( $changed->{$_} ) for keys %changed; $new->save; $post = $new; });
  25. 89.

    Views are reusable $posts = $model->view( type => 'post' );

    $featured = $posts->filterb( featured => 1 );
  26. 94.

    my $view = $domain ->view ->type( 'post') ->filterb( created =>

    { gte => '2012-08-01' }, user => $user, ) ->queryb( title => 'awesome' ) ->sort( 'timestamp' ) ->size( 20 ) ->highlight( 'content' ) ->explain( 1 ); See "Terms of Endearment" on speakerdeck.com
  27. 98.

    Results are iterators $result = $results->next $result = $results->prev $result

    = $results->first $result = $results->last $result = $results->shift
  28. 100.

    my $results = $view->search; say "Total hits: " . $results->total;

    say "Took: " . $results->took . "ms"; while ( my $result = $results->next ) { say "Title:" . $result->object->title; say "Snippets:" . join "\n", $result->highlight('content'); say "Score:" . $result->score; say "Debug:" . $result->explain; }
  29. 106.
  30. 109.
  31. 112.

    Examples: analyzed full text has 'name' => ( is =>

    'rw', isa => 'Str', ); name: { type: "string" }
  32. 113.

    Examples: analyze and stem text has 'name' => ( is

    => 'rw', isa => 'Str', analyzer => 'english' ); name: { type: "string", analyzer: "english" }
  33. 114.

    Examples: analyze and stem text has 'name' => ( is

    => 'rw', isa => 'Str', analyzer => 'norwegian' ); name: { type: "string", analyzer: "norwegian" }
  34. 115.

    Examples: store the exact value has 'tag' => ( is

    => 'rw', isa => 'Str', index => 'not_analyzed' ); tag: { type: "string", index: "not_analyzed" }
  35. 116.

    Examples: complex data use MooseX::Types::Moose qw(Str); use MooseX::Types::Structured qw(Dict); has

    'name' => ( is => 'rw', isa => Dict[ first => Str, last => Str, middle => Optional[Str], ], ); name: { type: "object", properties: { first: { type: 'string' }, last: { type: 'string' }, middle: { type: 'string'} } }
  36. 117.

    Examples: Elastic::Doc classes has 'user' => ( is => 'rw',

    isa => 'MyApp::User', ); user: { type: "object", properties: { name: { type: 'string' }, email: { type: 'string' }, uid: { type: "object", properties: { index: {...}, type: {...}, id: {...}, routing: {...} } } } }
  37. 118.

    Examples: Elastic::Doc classes has 'user' => ( is => 'rw',

    isa => 'MyApp::User', ); user: { type: "object", properties: { name: { type: 'string' }, email: { type: 'string' }, uid: { type: "object", properties: { index: {...}, type: {...}, id: {...}, routing: {...} } } } } Denormalised data!
  38. 119.

    Examples: Elastic::Doc classes has 'user' => ( is => 'rw',

    isa => 'MyApp::User', exclude_attrs => ['email'] ); user: { type: "object", properties: { name: { type: 'string' }, email: { type: 'string' }, uid: { type: "object", properties: { index: {...}, type: {...}, id: {...}, routing: {...} } } } }
  39. 120.

    Examples: Elastic::Doc classes has 'user' => ( is => 'rw',

    isa => 'MyApp::User', include_attrs => ['email'] ); user: { type: "object", properties: { name: { type: 'string' }, email: { type: 'string' }, uid: { type: "object", properties: { index: {...}, type: {...}, id: {...}, routing: {...} } } } }
  40. 121.

    Examples: Elastic::Doc classes has 'user' => ( is => 'rw',

    isa => 'MyApp::User', include_attrs => [] ); user: { type: "object", properties: { name: { type: 'string' }, email: { type: 'string' }, uid: { type: "object", properties: { index: {...}, type: {...}, id: {...}, routing: {...} } } } }
  41. 122.

    Same data. Different purpose has 'title' => ( is =>

    'rw', isa => 'Str', } title: { type: "string" } title => 'An AMAZING talk!' title: ['amazing','talk'] What do you sort on? 'amazing' or 'talk'
  42. 125.

    Same data. Different purpose has 'title' => ( is =>

    'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' } } }
  43. 126.

    Same data. Different purpose has 'title' => ( is =>

    'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' } } } title => 'An AMAZING talk!' title: { title: ['amazing','talk'], untouched: "An AMAZING talk!" }
  44. 132.

    Analysis process "Édith Piaf" -> standard tokenizer -> ["Édith", "Piaf"]

    -> lowercase token filter -> ["édith", "piaf"] -> ascii-folding token filter -> ["edith", "piaf"] -> edge-ngrams token filter -> ["e", "ed", "edi", "edit", "edith", "p", "pi", "pia", "piaf"] Perfect for partial matching!
  45. 133.

    package MyApp; use Elastic::Model; has_namespace 'myapp' => { user =>

    'MyApp::User', type => 'MyApp::Post, }; Add a custom analyzer to our Model
  46. 134.

    package MyApp; use Elastic::Model; has_namespace 'myapp' => { user =>

    'MyApp::User', type => 'MyApp::Post, }; Add a custom analyzer to our Model
  47. 135.

    package MyApp; use Elastic::Model; has_namespace 'myapp' => { user =>

    'MyApp::User', type => 'MyApp::Post, }; has_filter 'my_edge_ngrams' => { type => 'edge_ngrams', min_gram => 1, max_gram => 15 }; Add a custom analyzer to our Model
  48. 136.

    package MyApp; use Elastic::Model; has_namespace 'myapp' => { user =>

    'MyApp::User', type => 'MyApp::Post, }; has_filter 'my_edge_ngrams' => { type => 'edge_ngrams', min_gram => 1, max_gram => 15 }; has_analyzer 'autocomplete' => { tokenizer => 'standard', filter => ['lowercase','asciifolding', 'my_edge_ngrams'] }; Add a custom analyzer to our Model
  49. 137.

    Add analyzer to our Doc class has 'title' => (

    is => 'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' } } }
  50. 138.

    Add analyzer to our Doc class has 'title' => (

    is => 'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' }, autocomplete => { analyzer => 'autocomplete' } } }
  51. 139.

    Add analyzer to our Doc class has 'title' => (

    is => 'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' }, autocomplete => { analyzer => 'autocomplete' } } } title => 'An AMAZING talk!' title: { title: ['amazing','talk'], untouched: "An AMAZING talk!" }
  52. 140.

    Add analyzer to our Doc class has 'title' => (

    is => 'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' }, autocomplete => { analyzer => 'autocomplete' } } } title => 'An AMAZING talk!' title: { title: ['amazing','talk'], untouched: "An AMAZING talk!", autocomplete: [ 'a', 'am', 'ama', 'amaz', 'amazi', 'amazin', 'amazing', 't', 'ta', 'tal', 'talk' ] }
  53. 143.
  54. 150.

    $view = $domain->view->queryb( "title.autocomplete" => { -text => { query

    => "amazing ta", operator => "or" } } ); "a OR am OR ama OR amaz OR ... OR t OR ta"
  55. 151.
  56. 152.

    $view = $domain->view->queryb( "title.autocomplete" => { -text => { query

    => "amazing ta", operator => "and" } } ); Complete words should be more relevant
  57. 153.

    $view = $domain->view->queryb( "title.autocomplete" => { -text => { query

    => "amazing ta", operator => "and" } }, "title" => "amazing ta", );
  58. 154.

    $view = $domain->view->queryb([ "title.autocomplete" => { -text => { query

    => "amazing ta", operator => "and" } }, "title" => "amazing ta", ]);
  59. 155.
  60. 156.
  61. 170.
  62. 185.
  63. 191.

    $ns->alias( 'bloggs_plumbers' )->to( myapp_v1 => { filterb => { client_id

    => 'bloggs_plumbers' }, routing => 'bloggs_plumbers' } );