Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch & .NET

Elasticsearch & .NET

A whirlwind tour of the .NET interface to Elasticsearch and how it plays to .NET's strengths

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

March 19, 2015
Tweet

More Decks by Elastic Co

Other Decks in Programming

Transcript

  1. Elasticsearch & .NET Martijn Laarman, Software Engineer

  2. { } CC-BY-ND 4.0 Who am I? • Online handle:

    mpdreamz • Jumped on Elasticsearch in 2010 because it spoke JSON, no more flattening my C# domain layers! • Joined Elasticsearch February 2014 2
  3. { } CC-BY-ND 4.0 Landscape • Started .NET client 5

    months after elasticsearch saw its first release back in 2010. • The .NET client was purely me dog-fooding elasticsearch on real projects and its feature set directly reflected what I was using. • Level of community love it has received in terms of PR’s, issues and feedback is still mind-blowing. (Even before becoming the official client). Many of the features I did not use were implemented by the community. • Even though it was a personal project the client has always been heavily unit and integration tested. 3 Prior to joining elasticsearch
  4. { } CC-BY-ND 4.0 Landscape • Existing official clients all

    implement a very similar API • .NET languages are the first statically typed in the official client family, returning dictionaries is not sufficient. • NEST 0.* already used the REST specification that other clients used to for their public API as its internal API. 4 After joining elasticsearch
  5. { } CC-BY-ND 4.0 The great .NET client schism of

    2014 or: why the heck we have two .NET clients! 5
  6. { } CC-BY-ND 4.0 Elasticsearch.NET 6 • Low-level client •

    Generated from the official elasticsearch REST API spec • No dependencies • Works like the rest of the official clients • Load balancing / cluster fail over support • Async and sync methods • Unlike NEST, does not provide typed request and response objects
  7. { } CC-BY-ND 4.0 NEST 7 • High-level client •

    Strongly typed • Json.NET • Uses and exposes elasticsearch.net under the hood • Fluent API • Load balancing / connection failover • Async variants
  8. { } CC-BY-ND 4.0 8 var client = new ElasticClient();

    client.Raw; //low level client exposed NEST var client = new ElasticsearchClient(); Elasticsearch.NET
  9. { } CC-BY-ND 4.0 9 Coordinates requests, knows when to

    ping, sniff and which result/exceptions should be retried. ITransport In charge of sending and receiving bytes, (Http, Thrift, InMemory) IConnection Manages list of known nodes and keeps track which nodes are up and down. IConnectionPool In charge of serializing and deserializing IElasticsearchSerializer
  10. { } CC-BY-ND 4.0 A whirlwind tour of the API

    or: if you spot anything thats not obvious we messed up! 10
  11. { } CC-BY-ND 4.0 API Examples var client = new

    ElasticClient(); 11 Connecting to your (localhost) cluster
  12. { } CC-BY-ND 4.0 API Examples var node = new

    Uri("http://localhost:9200"); var settings = new ConnectionSettings(node); var client = new ElasticClient(settings); 12 specifying a node and settings
  13. { } CC-BY-ND 4.0 API Examples var settings = new

    ConnectionSettings(node) .SetTimeout(1500) .MaximumRetries(5) .SetDefaultIndex("foo") .MapDefaultTypeNames(m => m.Add(typeof(Session), "session")) .MapDefaultTypeIndices(m => m.Add(typeof(Session), "some-index")) .SetJsonSerializerSettingsModifier(m => {}); 13 a closer look at ConnectionSettings
  14. { } CC-BY-ND 4.0 API Examples var nodes = new

    Uri[] { new Uri("http://localhost:9200"), new Uri("http://localhost:9201"), new Uri("http://localhost:9202") }; var pool = new StaticConnectionPool(nodes); var settings = new ConnectionSettings(pool); var client = new ElasticClient(settings); 14 using a connection pool
  15. { } CC-BY-ND 4.0 API Examples var session = new

    Session { Title = "Foo", Abstract = "Bar", Level = 100, } var response = client.Index(session); var response = client.Index(session, i => i .Index("company-index") .Type("sessie") .Id(1) .Refresh() ); 15 indexing a document
  16. { } CC-BY-ND 4.0 API Examples var response = client.Bulk(b

    => b .Index<Session>(i => i.Document(new Session { Title = "Foo" })) .Index<Session>(i => i.Document(new Session { Title = "Bar" })) ); // Shortcut var response = client.Bulk(b => b .IndexMany<Session>(sessions) ); // Shortcut to the shortcut var response = client.IndexMany<Session>(sessions); 16 indexing multiple documents
  17. { } CC-BY-ND 4.0 API Examples var response = client.Get<Session>(1);

    var response = client.Get<Session>(g => g .Index("company-index") .Type("sessie") .Id(1) ); 17 getting a document
  18. { } CC-BY-ND 4.0 API Examples var response = client.CreateIndex("company-index",

    c => c .NumberOfShards(2) .NumberOfReplicas(1) .AddMapping<Session>(map => map .Properties(props => props .String(s => s .Name(p => p.Title).Index(FieldIndexOption.NotAnalyzed) ) .Number(n => n .Name(p => p.Level) .Store(false) ) .String(s => s.Name(p => p.Abstract).Analyzer("snowball")) ) ) ); 18 creating an index
  19. { } CC-BY-ND 4.0 API Examples var response = client.Search<Session>(s

    => s .From(0) .Size(50) .Query(q => q .QueryString(qs => qs .DefaultField(o => o.Abstract) .Query("c# AND elasticsearch") ) ) ); 19 searching (query string query)
  20. { } CC-BY-ND 4.0 API Examples var response = client.Search<Session>(s

    => s .From(0) .Size(50) .Query(q => q .Bool(b => b .Should( qs=> qs.Match(m => m.OnField(o=>o.Title).Query(“elasticsearch”), qs=> qs.Match(m => m.OnField(o=>o.Abstract).Query("elasticsearch")) ) .Must(qm => qm.Match(m => m.OnField(o => o.Level).Query(100))) ) ) ); 20 searching (bool query)
  21. { } CC-BY-ND 4.0 API Examples var response = client.Search<Session>(s

    => s .From(0) .Size(50) .Query(q => (q.Match(m => m.OnField(o => o.Title).Query("elasticsearch")) || q.Match(m => m.OnField(o => o.Abstract).Query("elasticsearch")) ) && q.Match(m => m.OnField(o => o.Level).Query(100)) ) ); 21 bool query using bitwise operators
  22. { } CC-BY-ND 4.0 API Examples var response = client.Search<Session>(s

    => s .Size(0) .Aggregations(a => a .Terms("level_count", t => t .Field(o => o.LevelName) ) ) ); 22 aggregations
  23. { } CC-BY-ND 4.0 API Examples 23 No abstractions of

    our own The client API directly reflect the elasticsearch API We strive to have a strict 1 to 1 mapping with elasticsearch
  24. { } CC-BY-ND 4.0 API Examples public interface IResponse {

    bool IsValid { get; } IElasticsearchResponse ConnectionStatus { get; set; } } 24 Handling responses All responses implement IResponse, you can always use .IsValid if the response if logically valid. e.g it will be false if a bulk operation returns a HTTP 200 but with individual failures.
  25. { } CC-BY-ND 4.0 Playing to C#’s strengths or: what

    we have to offer over rolling your own with HttpClient and Json.NET 25
  26. { } CC-BY-ND 4.0 Playing to C#’s strengths 26 Plain

    Old C# Objects (POCO) There is no need to subclass your domain from a common elasticsearch type public class Person { public string FirstName { get; set; } public string LastName { get; set; } } public class Organisation { public int Id { get; set; } public string Name { get; set; } public List<Person> Employees { get; set; } } All the NEST API calls accept your POCO to map results back on to where it makes sense
  27. { } CC-BY-ND 4.0 Playing to C#’s strengths NEST eradicates

    magic strings and repeating yourself. 27 Always explicit NEST Elasticsearch.NET client.Search<Person>(s => s .Filter(f => f .Term(p =>p.Name, "martijn") ) ); client.Search("index", "person", new { filter = new { term = new { name = "martijn" } } });
  28. { } CC-BY-ND 4.0 Playing to C#’s strengths 28 .Search<Person>(search

    => search .Query(q => q.Term(p=>p.Friends.First().Name,"martijn") ) ); { "query" : { "term" : { "vrienden.naam" : "martijn" } } } Strongly typed field names There are many parts in the elasticsearch API where you have to refer to field names, search, aggregations, update API, suggest API, highlight API, etcetera. In NEST you can bind all of these to an POCO scope and refer to properties using expressions
  29. { } CC-BY-ND 4.0 Playing to C#’s strengths Controllable through

    code 29 or attributes Strongly typed field names new ConnectionSettings(uri, “company-index") .MapPropertiesFor<Person>(props => props .Rename(p => p.Name, "naam") .Rename(p => p.Friends, "vrienden") ) ); public class Person { [ElasticProperty(Name = "vrienden")] public IList<Person> Friends { get; set; } [ElasticProperty(Name = "naam")] public string Name { get; set; } }
  30. { } CC-BY-ND 4.0 Playing to C#’s strengths 30 client.Search<Person>();

    client.Search<Person>(s=>s.Type("x")); client.Search<Person>(s=>s.AllIndices()); .Search<P>(s=>s.AllIndices().AllTypes()); /default_index/person/_search /default_index/x/_search /_all/person/_search /_search Index and Type name inference
  31. { } CC-BY-ND 4.0 Playing to C#’s strengths Controllable through

    code 31 or attributes Index and Type name inference new ConnectionSettings(uri, “company-index") .MapDefaultTypeNames(p => p .Add(typeof(Car), "automobile") .Add(typeof(Person), "humans") .Add(typeof(Organisation), "organisation") .Add(typeof(Developer), "coder") ) .MapDefaultTypeIndices(i => i .Add(typeof(Car), "car-catalog") ); [ElasticType(Name = "human")] public class Person { }
  32. { } CC-BY-ND 4.0 Playing to C#’s strengths 32 var

    human = new Person { Id = 12 }; client.Index<Person>(human); client.Index(human); .Index(human, i => i.Type("x")); We look at properties called Id by default /default_index/human/12 /default_index/human/12 /default_index/x/12 Id inference
  33. { } CC-BY-ND 4.0 Playing to C#’s strengths Controllable through

    code 33 or attributes Id inference new ConnectionSettings(uri, “company-index") .MapIdPropertyFor<Project>(p => p.Name); [ElasticType(IdProperty = "Name")] public class Project { public string Name { get; set; } }
  34. { } CC-BY-ND 4.0 Playing to C#’s strengths Can be

    used in unforeseen places 34 Inference is fully exposed var marker = Property.Path<Person>(p=>p.Name); var path = client.Infer.PropertyPath(marker); var typeName = client.Infer.TypeName<Person>(); var inferrer = new ElasticInferrer(settings); returns an instance of PropertyPathMarker() returns string representation. returns "human" based on the clients settings client not necessary Great for writing unit tests against, or write your own extension methods to support 3rd party plugins or missing elasticsearch functionality (although that should not happen!).
  35. { } CC-BY-ND 4.0 Playing to C#’s strengths Great for

    exploration and prototyping, terser then forcing you to new generic builders with the same type information everywhere. 35 Lambda’s everywhere! client.Search<Person>(search => search .Query(q => { //do complex if statements here }) ); client.Search<Person>(search => search .Query(BuildQuery) ); public QueryContainer BuildQuery(QueryDescriptor<Person> d) { //do complex if statements here } Can still be abstracted quite neatly into named methods, just extract method on the lambda!
  36. { } CC-BY-ND 4.0 Playing to C#’s strengths Not for

    everyone, therefor NEST 1.0 now also accepts object initialiser syntax for all methods as well as fluent lambda syntax 36 Lambda’s everywhere! IQueryContainer query = new MatchQuery { Field = "abstract", Query = "elasticsearch" }; var request = new SearchRequest { From = 0, Size = 50, Query = query, Filter = ..., Aggregations = ... }; client.Search<Session>(request);
  37. { } CC-BY-ND 4.0 Playing to C#’s strengths 37 Covariant

    search results var r = client.Search<ISearchable>(s => s .Types( typeof(Car), typeof(Organization) ) ); var cars = r.Documents.OfType<Car>(); var orgs = r.Documents.OfType<Organisation>(); foreach(var result in r.Documents) { if (result is Car) {} else if (result is Organization) {} NEST can deal with searches returning multiple types out of the box and instantiate real subclass instances int the returned results. /default-index/car,organization/_search r.Documents is IEnumarable<ISearchable> ISearchable is just a marker interface both types implement, can also be a common base class.
  38. { } CC-BY-ND 4.0 Playing to C#’s strengths Elasticsearch boolean

    query implements unary boolean logic not binary using must/ must_not/should clauses. A common misconception is to use boolean queries as followed: 38 Real binary boolean support bool must X must_not Y should Z In order to represent: (X && !Y) || Z what it actually means is X MUST be present and Y MUST NOT in the document and SHOULD Z be present it will score higher then those documents without Z.
  39. { } CC-BY-ND 4.0 Playing to C#’s strengths When using

    NEST boolean operators are expanded into the correct unary representation. 39 Real binary boolean support .Query(q => (q.Term(X) && !q.Term(Y)) || q.Term(Z)) bool should bool must X must_not Y bool must Z This does in fact translate to: (X && Y) || Z You can use parenthesis to force order and write binary bool queries of any complexity
  40. { } CC-BY-ND 4.0 Playing to C#’s strengths Some methods

    exists of multiple elasticsearch calls 40 Exposing IObservables for coordinated request var observable = this.Client.Reindex<object>(r => r .FromIndex(ElasticsearchConfiguration.DefaultIndex) .ToIndex(toIndex) ); var observer = new ReindexObserver<object>( onNext: (reindex) => Logger.Info(“reindexed portion") onError: (e) => Logger.Error(e.Message), completed: () => { }); observable.Subscribe(observer); Available as observable: Reindex, Snapshot & Restore
  41. { } CC-BY-ND 4.0 Elasticsearch.NET & NEST 2.0 41

  42. { } CC-BY-ND 4.0 Elasticsearch.NET & NEST 2.0 55% Test

    coverage NEST 70% Test coverage Elasticsearch.NET Great! but code is basically 5 years worth of writing tests differently and scattered over 5 test projects New test framework will combine literate programming through roslyn to combine writing tests with documentation. 42 Rethinking testing, documentation
  43. { } CC-BY-ND 4.0 Elasticsearch.NET & NEST 2.0 Concise effort

    to go over all the Elasticsearch API and (hopefully) come up with an official contract of requests and responses. 43 Official API specification
  44. { } CC-BY-ND 4.0 Elasticsearch.NET & NEST 2.0 Currently we

    support .NET 4.0 and up on both .NET and Mono. With 2.0 we are going to have proper PCL builds for various platforms and full support for the new .NET core framework. 44 PCL/CoreFX support
  45. { } CC-BY-ND 4.0 Quick Recaps • Official high level

    .NET client • 99% API coverage WITH request and response types • In almost all cases the client you want to start out with 45 • Official low level .NET client • 100% API endpoint coverage but no provided types for request and responses • No dependencies • Use when you have a small isolated use case or integration point with elasticsearch. NEST Elasticsearch.NET
  46. { } GitHub: https://github.com/elasticsearch/elasticsearch-net Documentation: http://nest.azurewebsites.net Elasticsearch Documentation: http://www.elasticsearch.org/guide/ Twitter:

    @mpdreamz @gregmarzouka GitHub: mpdreamz, gmarz Email: dotnet@elasticsearch.com
  47. { } This work is licensed under the Creative Commons

    Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0