Upgrade to Pro — share decks privately, control downloads, hide ads and more …

To infinity and beyond

To infinity and beyond

Elastic::Model is a new framework to store your Moose objects, which uses ElasticSearch as a NoSQL document store and flexible search engine.

It is designed to make small beginnings simple, but to scale easily to Big Data requirements without needing to rearchitect your application. No job too big or small!

This talk will introduce Elastic::Model, demonstrate how to develop a simple application, introduce some more advanced techniques, and discuss how it uses ElasticSearch to scale.

https://github.com/clintongormley/Elastic-Model

Clinton Gormley

August 21, 2012
Tweet

More Decks by Clinton Gormley

Other Decks in Programming

Transcript

  1. To infinity and beyond! A practical guide for Mooseherds (and

    other carers of livestock) @clintongormley #elasticsearch YAPC::EU 2012
  2. package MyApp::Post; use Moose; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } );
  3. package MyApp::Post; use Moose; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } ); package MyApp::User; use Moose; has 'name' => ( is => 'rw', isa => 'Str' ); has 'email' => ( is => 'rw', isa => 'Str', required => 1 );
  4. package MyApp::Post; use Moose; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } ); has 'user' => ( is => 'ro', isa => 'MyApp::User', ); package MyApp::User; use Moose; has 'name' => ( is => 'rw', isa => 'Str' ); has 'email' => ( is => 'rw', isa => 'Str', required => 1 );
  5. package MyApp::Post; use Moose; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } ); has 'user' => ( is => 'ro', isa => 'MyApp::User', ); package MyApp::User; use Moose; has 'name' => ( is => 'rw', isa => 'Str' ); has 'email' => ( is => 'rw', isa => 'Str', required => 1 );
  6. package MyApp::Post; use Moose; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } ); has 'user' => ( is => 'ro', isa => 'MyApp::User', ); package MyApp::User; use Moose; has 'name' => ( is => 'rw', isa => 'Str' ); has 'email' => ( is => 'rw', isa => 'Str', required => 1 );
  7. package MyApp::Post; use Elastic::Doc; has 'title' => ( is =>

    'rw', isa => 'Str' ); has 'content' => ( is => 'rw', isa => 'Str' ); has 'created' => ( is => 'rw', isa => 'DateTime', default => sub { DateTime->now } ); has 'user' => ( is => 'ro', isa => 'MyApp::User', ); package MyApp::User; use Elastic::Doc; has 'name' => ( is => 'rw', isa => 'Str' ); has 'email' => ( is => 'rw', isa => 'Str', required => 1 );
  8. Some definitions... * index * type * doc * alias

    Like a database Like a table Like a row in a table Like a symbolic link, points to one or more indices elasticsearch Elastic::Model * domain * namespace * model An index or an alias, used for CRUD Maps type <=> class for all associated domains Connects your app to elasticsearch.
  9. package MyApp; use Elastic::Model; has_namespace 'myapp' => { user =>

    'MyApp::User', post => 'MyApp::Post, }; # like table <=> class
  10. use MyApp; my $model = MyApp->new; my $namespace = $model->namespace('myapp');

    # For index and alias management my $domain = $model->domain('myapp'); # For document CRUD my $view = $model->view; # For searching To do anything useful, we need:
  11. Domain: Create a user my $domain = $model->domain('myapp'); my $user

    = $domain->new_doc( user => { name => 'Clinton', email => '[email protected]', } ); $user->save;
  12. Domain: Create a user my $domain = $model->domain('myapp'); my $user

    = $domain->create( user => { name => 'Clinton', email => '[email protected]', } ); $user->save;
  13. Domain: Create a user my $domain = $model->domain('myapp'); my $user

    = $domain->create( user => { name => 'Clinton', email => '[email protected]', id => 1, } ); say $user->id; # 1 say $user->type; # user
  14. Domain: Create a post my $domain = $model->domain('myapp'); my $post

    = $domain->create( post => { id => 2, title => 'To infinity and beyond', content => 'Elastic::Model persists Moose ' . . 'objects in elasticsearch', user => $user } );
  15. Domain: Retrieve a doc my $domain = $model->domain('myapp'); my $post

    = $domain->get( post => 2 ); my $user = $post->user; # stub object say $user->name; # full object # Clinton say $user->id; # still stub # 1
  16. Domain: Update a doc my $domain = $model->domain('myapp'); $post->title('Awesome blog

    post'); say $post->has_changed; # 1 say $post->has_changed('title'); # 1 say $post->old_value('title'); # To infinity and beyond $post->save;
  17. 1: $post = $domain->get(post=>2); 2: $post = $domain->get(post=>2); 1: $post->title('Awesome

    blog post'); 2: $post->title('Brilliant blog post'); 1: $post->save; 2: $post->save; *** CONFLICT ERROR ***
  18. $post->save( on_conflict => sub { my ($old,$new) = @_; #

    do something # to resolve conflict });
  19. $post->save( on_conflict => sub { my ($old,$new) = @_; my

    %changed = $old->old_values; $new->$_( $changed->{$_} ) for keys %changed; $new->save; $post = $new; });
  20. Views are reusable $posts = $model->view( type => 'post' );

    $featured = $posts->filterb( featured => 1 );
  21. my $view = $domain ->view ->type( 'post') ->filterb( created =>

    { gte => '2012-08-01' }, user => $user, ) ->queryb( title => 'awesome' ) ->sort( 'timestamp' ) ->size( 20 ) ->highlight( 'content' ) ->explain( 1 ); See "Terms of Endearment" on speakerdeck.com
  22. Results are iterators $result = $results->next $result = $results->prev $result

    = $results->first $result = $results->last $result = $results->shift
  23. my $results = $view->search; say "Total hits: " . $results->total;

    say "Took: " . $results->took . "ms"; while ( my $result = $results->next ) { say "Title:" . $result->object->title; say "Snippets:" . join "\n", $result->highlight('content'); say "Score:" . $result->score; say "Debug:" . $result->explain; }
  24. Examples: analyzed full text has 'name' => ( is =>

    'rw', isa => 'Str', ); name: { type: "string" }
  25. Examples: analyze and stem text has 'name' => ( is

    => 'rw', isa => 'Str', analyzer => 'english' ); name: { type: "string", analyzer: "english" }
  26. Examples: analyze and stem text has 'name' => ( is

    => 'rw', isa => 'Str', analyzer => 'norwegian' ); name: { type: "string", analyzer: "norwegian" }
  27. Examples: store the exact value has 'tag' => ( is

    => 'rw', isa => 'Str', index => 'not_analyzed' ); tag: { type: "string", index: "not_analyzed" }
  28. Examples: complex data use MooseX::Types::Moose qw(Str); use MooseX::Types::Structured qw(Dict); has

    'name' => ( is => 'rw', isa => Dict[ first => Str, last => Str, middle => Optional[Str], ], ); name: { type: "object", properties: { first: { type: 'string' }, last: { type: 'string' }, middle: { type: 'string'} } }
  29. Examples: Elastic::Doc classes has 'user' => ( is => 'rw',

    isa => 'MyApp::User', ); user: { type: "object", properties: { name: { type: 'string' }, email: { type: 'string' }, uid: { type: "object", properties: { index: {...}, type: {...}, id: {...}, routing: {...} } } } }
  30. Examples: Elastic::Doc classes has 'user' => ( is => 'rw',

    isa => 'MyApp::User', ); user: { type: "object", properties: { name: { type: 'string' }, email: { type: 'string' }, uid: { type: "object", properties: { index: {...}, type: {...}, id: {...}, routing: {...} } } } } Denormalised data!
  31. Examples: Elastic::Doc classes has 'user' => ( is => 'rw',

    isa => 'MyApp::User', exclude_attrs => ['email'] ); user: { type: "object", properties: { name: { type: 'string' }, email: { type: 'string' }, uid: { type: "object", properties: { index: {...}, type: {...}, id: {...}, routing: {...} } } } }
  32. Examples: Elastic::Doc classes has 'user' => ( is => 'rw',

    isa => 'MyApp::User', include_attrs => ['email'] ); user: { type: "object", properties: { name: { type: 'string' }, email: { type: 'string' }, uid: { type: "object", properties: { index: {...}, type: {...}, id: {...}, routing: {...} } } } }
  33. Examples: Elastic::Doc classes has 'user' => ( is => 'rw',

    isa => 'MyApp::User', include_attrs => [] ); user: { type: "object", properties: { name: { type: 'string' }, email: { type: 'string' }, uid: { type: "object", properties: { index: {...}, type: {...}, id: {...}, routing: {...} } } } }
  34. Same data. Different purpose has 'title' => ( is =>

    'rw', isa => 'Str', } title: { type: "string" } title => 'An AMAZING talk!' title: ['amazing','talk'] What do you sort on? 'amazing' or 'talk'
  35. Same data. Different purpose has 'title' => ( is =>

    'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' } } }
  36. Same data. Different purpose has 'title' => ( is =>

    'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' } } } title => 'An AMAZING talk!' title: { title: ['amazing','talk'], untouched: "An AMAZING talk!" }
  37. Analysis process "Édith Piaf" -> standard tokenizer -> ["Édith", "Piaf"]

    -> lowercase token filter -> ["édith", "piaf"] -> ascii-folding token filter -> ["edith", "piaf"] -> edge-ngrams token filter -> ["e", "ed", "edi", "edit", "edith", "p", "pi", "pia", "piaf"] Perfect for partial matching!
  38. package MyApp; use Elastic::Model; has_namespace 'myapp' => { user =>

    'MyApp::User', type => 'MyApp::Post, }; Add a custom analyzer to our Model
  39. package MyApp; use Elastic::Model; has_namespace 'myapp' => { user =>

    'MyApp::User', type => 'MyApp::Post, }; Add a custom analyzer to our Model
  40. package MyApp; use Elastic::Model; has_namespace 'myapp' => { user =>

    'MyApp::User', type => 'MyApp::Post, }; has_filter 'my_edge_ngrams' => { type => 'edge_ngrams', min_gram => 1, max_gram => 15 }; Add a custom analyzer to our Model
  41. package MyApp; use Elastic::Model; has_namespace 'myapp' => { user =>

    'MyApp::User', type => 'MyApp::Post, }; has_filter 'my_edge_ngrams' => { type => 'edge_ngrams', min_gram => 1, max_gram => 15 }; has_analyzer 'autocomplete' => { tokenizer => 'standard', filter => ['lowercase','asciifolding', 'my_edge_ngrams'] }; Add a custom analyzer to our Model
  42. Add analyzer to our Doc class has 'title' => (

    is => 'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' } } }
  43. Add analyzer to our Doc class has 'title' => (

    is => 'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' }, autocomplete => { analyzer => 'autocomplete' } } }
  44. Add analyzer to our Doc class has 'title' => (

    is => 'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' }, autocomplete => { analyzer => 'autocomplete' } } } title => 'An AMAZING talk!' title: { title: ['amazing','talk'], untouched: "An AMAZING talk!" }
  45. Add analyzer to our Doc class has 'title' => (

    is => 'rw', isa => 'Str', multi => { untouched => { index => 'not_analyzed' }, autocomplete => { analyzer => 'autocomplete' } } } title => 'An AMAZING talk!' title: { title: ['amazing','talk'], untouched: "An AMAZING talk!", autocomplete: [ 'a', 'am', 'ama', 'amaz', 'amazi', 'amazin', 'amazing', 't', 'ta', 'tal', 'talk' ] }
  46. $view = $domain->view->queryb( "title.autocomplete" => { -text => { query

    => "amazing ta", operator => "or" } } ); "a OR am OR ama OR amaz OR ... OR t OR ta"
  47. $view = $domain->view->queryb( "title.autocomplete" => { -text => { query

    => "amazing ta", operator => "and" } } ); Complete words should be more relevant
  48. $view = $domain->view->queryb( "title.autocomplete" => { -text => { query

    => "amazing ta", operator => "and" } }, "title" => "amazing ta", );
  49. $view = $domain->view->queryb([ "title.autocomplete" => { -text => { query

    => "amazing ta", operator => "and" } }, "title" => "amazing ta", ]);
  50. $ns->alias( 'bloggs_plumbers' )->to( myapp_v1 => { filterb => { client_id

    => 'bloggs_plumbers' }, routing => 'bloggs_plumbers' } );