Elasticsearch & .NET

A whirlwind tour of the .NET interface to Elasticsearch and how it plays to .NET's strengths

1. Elasticsearch & .NET Martijn Laarman, Software Engineer
2. { } CC-BY-ND 4.0 Who am I? • Online handle: mpdreamz • Jumped on Elasticsearch in 2010 because it spoke JSON, no more flattening my C# domain layers! • Joined Elasticsearch February 2014 2
3. { } CC-BY-ND 4.0 Landscape • Started .NET client 5 months after elasticsearch saw its first release back in 2010. • The .NET client was purely me dog-fooding elasticsearch on real projects and its feature set directly reflected what I was using. • Level of community love it has received in terms of PR’s, issues and feedback is still mind-blowing. (Even before becoming the official client). Many of the features I did not use were implemented by the community. • Even though it was a personal project the client has always been heavily unit and integration tested. 3 Prior to joining elasticsearch
4. { } CC-BY-ND 4.0 Landscape • Existing official clients all implement a very similar API • .NET languages are the first statically typed in the official client family, returning dictionaries is not sufficient. • NEST 0.* already used the REST specification that other clients used to for their public API as its internal API. 4 After joining elasticsearch
5. { } CC-BY-ND 4.0 The great .NET client schism of 2014 or: why the heck we have two .NET clients! 5
6. { } CC-BY-ND 4.0 Elasticsearch.NET 6 • Low-level client • Generated from the official elasticsearch REST API spec • No dependencies • Works like the rest of the official clients • Load balancing / cluster fail over support • Async and sync methods • Unlike NEST, does not provide typed request and response objects
7. { } CC-BY-ND 4.0 NEST 7 • High-level client • Strongly typed • Json.NET • Uses and exposes elasticsearch.net under the hood • Fluent API • Load balancing / connection failover • Async variants
8. { } CC-BY-ND 4.0 8 var client = new ElasticClient(); client.Raw; //low level client exposed NEST var client = new ElasticsearchClient(); Elasticsearch.NET
9. { } CC-BY-ND 4.0 9 Coordinates requests, knows when to ping, sniff and which result/exceptions should be retried. ITransport In charge of sending and receiving bytes, (Http, Thrift, InMemory) IConnection Manages list of known nodes and keeps track which nodes are up and down. IConnectionPool In charge of serializing and deserializing IElasticsearchSerializer
10. { } CC-BY-ND 4.0 A whirlwind tour of the API or: if you spot anything thats not obvious we messed up! 10
11. { } CC-BY-ND 4.0 API Examples var client = new ElasticClient(); 11 Connecting to your (localhost) cluster
12. { } CC-BY-ND 4.0 API Examples var node = new Uri("http://localhost:9200"); var settings = new ConnectionSettings(node); var client = new ElasticClient(settings); 12 specifying a node and settings
13. { } CC-BY-ND 4.0 API Examples var settings = new ConnectionSettings(node) .SetTimeout(1500) .MaximumRetries(5) .SetDefaultIndex("foo") .MapDefaultTypeNames(m => m.Add(typeof(Session), "session")) .MapDefaultTypeIndices(m => m.Add(typeof(Session), "some-index")) .SetJsonSerializerSettingsModifier(m => {}); 13 a closer look at ConnectionSettings
14. { } CC-BY-ND 4.0 API Examples var nodes = new Uri[] { new Uri("http://localhost:9200"), new Uri("http://localhost:9201"), new Uri("http://localhost:9202") }; var pool = new StaticConnectionPool(nodes); var settings = new ConnectionSettings(pool); var client = new ElasticClient(settings); 14 using a connection pool
15. { } CC-BY-ND 4.0 API Examples var session = new Session { Title = "Foo", Abstract = "Bar", Level = 100, } var response = client.Index(session); var response = client.Index(session, i => i .Index("company-index") .Type("sessie") .Id(1) .Refresh() ); 15 indexing a document
16. { } CC-BY-ND 4.0 API Examples var response = client.Bulk(b => b .Index<Session>(i => i.Document(new Session { Title = "Foo" })) .Index<Session>(i => i.Document(new Session { Title = "Bar" })) ); // Shortcut var response = client.Bulk(b => b .IndexMany<Session>(sessions) ); // Shortcut to the shortcut var response = client.IndexMany<Session>(sessions); 16 indexing multiple documents
17. { } CC-BY-ND 4.0 API Examples var response = client.Get<Session>(1); var response = client.Get<Session>(g => g .Index("company-index") .Type("sessie") .Id(1) ); 17 getting a document
18. { } CC-BY-ND 4.0 API Examples var response = client.CreateIndex("company-index", c => c .NumberOfShards(2) .NumberOfReplicas(1) .AddMapping<Session>(map => map .Properties(props => props .String(s => s .Name(p => p.Title).Index(FieldIndexOption.NotAnalyzed) ) .Number(n => n .Name(p => p.Level) .Store(false) ) .String(s => s.Name(p => p.Abstract).Analyzer("snowball")) ) ) ); 18 creating an index
19. { } CC-BY-ND 4.0 API Examples var response = client.Search<Session>(s => s .From(0) .Size(50) .Query(q => q .QueryString(qs => qs .DefaultField(o => o.Abstract) .Query("c# AND elasticsearch") ) ) ); 19 searching (query string query)
20. { } CC-BY-ND 4.0 API Examples var response = client.Search<Session>(s => s .From(0) .Size(50) .Query(q => q .Bool(b => b .Should( qs=> qs.Match(m => m.OnField(o=>o.Title).Query(“elasticsearch”), qs=> qs.Match(m => m.OnField(o=>o.Abstract).Query("elasticsearch")) ) .Must(qm => qm.Match(m => m.OnField(o => o.Level).Query(100))) ) ) ); 20 searching (bool query)
21. { } CC-BY-ND 4.0 API Examples var response = client.Search<Session>(s => s .From(0) .Size(50) .Query(q => (q.Match(m => m.OnField(o => o.Title).Query("elasticsearch")) || q.Match(m => m.OnField(o => o.Abstract).Query("elasticsearch")) ) && q.Match(m => m.OnField(o => o.Level).Query(100)) ) ); 21 bool query using bitwise operators
22. { } CC-BY-ND 4.0 API Examples var response = client.Search<Session>(s => s .Size(0) .Aggregations(a => a .Terms("level_count", t => t .Field(o => o.LevelName) ) ) ); 22 aggregations
23. { } CC-BY-ND 4.0 API Examples 23 No abstractions of our own The client API directly reflect the elasticsearch API We strive to have a strict 1 to 1 mapping with elasticsearch
24. { } CC-BY-ND 4.0 API Examples public interface IResponse { bool IsValid { get; } IElasticsearchResponse ConnectionStatus { get; set; } } 24 Handling responses All responses implement IResponse, you can always use .IsValid if the response if logically valid. e.g it will be false if a bulk operation returns a HTTP 200 but with individual failures.
25. { } CC-BY-ND 4.0 Playing to C#’s strengths or: what we have to offer over rolling your own with HttpClient and Json.NET 25
26. { } CC-BY-ND 4.0 Playing to C#’s strengths 26 Plain Old C# Objects (POCO) There is no need to subclass your domain from a common elasticsearch type public class Person { public string FirstName { get; set; } public string LastName { get; set; } } public class Organisation { public int Id { get; set; } public string Name { get; set; } public List<Person> Employees { get; set; } } All the NEST API calls accept your POCO to map results back on to where it makes sense
27. { } CC-BY-ND 4.0 Playing to C#’s strengths NEST eradicates magic strings and repeating yourself. 27 Always explicit NEST Elasticsearch.NET client.Search<Person>(s => s .Filter(f => f .Term(p =>p.Name, "martijn") ) ); client.Search("index", "person", new { filter = new { term = new { name = "martijn" } } });
28. { } CC-BY-ND 4.0 Playing to C#’s strengths 28 .Search<Person>(search => search .Query(q => q.Term(p=>p.Friends.First().Name,"martijn") ) ); { "query" : { "term" : { "vrienden.naam" : "martijn" } } } Strongly typed field names There are many parts in the elasticsearch API where you have to refer to field names, search, aggregations, update API, suggest API, highlight API, etcetera. In NEST you can bind all of these to an POCO scope and refer to properties using expressions
29. { } CC-BY-ND 4.0 Playing to C#’s strengths Controllable through code 29 or attributes Strongly typed field names new ConnectionSettings(uri, “company-index") .MapPropertiesFor<Person>(props => props .Rename(p => p.Name, "naam") .Rename(p => p.Friends, "vrienden") ) ); public class Person { [ElasticProperty(Name = "vrienden")] public IList<Person> Friends { get; set; } [ElasticProperty(Name = "naam")] public string Name { get; set; } }
30. { } CC-BY-ND 4.0 Playing to C#’s strengths 30 client.Search<Person>(); client.Search<Person>(s=>s.Type("x")); client.Search<Person>(s=>s.AllIndices()); .Search<P>(s=>s.AllIndices().AllTypes()); /default_index/person/_search /default_index/x/_search /_all/person/_search /_search Index and Type name inference
31. { } CC-BY-ND 4.0 Playing to C#’s strengths Controllable through code 31 or attributes Index and Type name inference new ConnectionSettings(uri, “company-index") .MapDefaultTypeNames(p => p .Add(typeof(Car), "automobile") .Add(typeof(Person), "humans") .Add(typeof(Organisation), "organisation") .Add(typeof(Developer), "coder") ) .MapDefaultTypeIndices(i => i .Add(typeof(Car), "car-catalog") ); [ElasticType(Name = "human")] public class Person { }
32. { } CC-BY-ND 4.0 Playing to C#’s strengths 32 var human = new Person { Id = 12 }; client.Index<Person>(human); client.Index(human); .Index(human, i => i.Type("x")); We look at properties called Id by default /default_index/human/12 /default_index/human/12 /default_index/x/12 Id inference
33. { } CC-BY-ND 4.0 Playing to C#’s strengths Controllable through code 33 or attributes Id inference new ConnectionSettings(uri, “company-index") .MapIdPropertyFor<Project>(p => p.Name); [ElasticType(IdProperty = "Name")] public class Project { public string Name { get; set; } }
34. { } CC-BY-ND 4.0 Playing to C#’s strengths Can be used in unforeseen places 34 Inference is fully exposed var marker = Property.Path<Person>(p=>p.Name); var path = client.Infer.PropertyPath(marker); var typeName = client.Infer.TypeName<Person>(); var inferrer = new ElasticInferrer(settings); returns an instance of PropertyPathMarker() returns string representation. returns "human" based on the clients settings client not necessary Great for writing unit tests against, or write your own extension methods to support 3rd party plugins or missing elasticsearch functionality (although that should not happen!).
35. { } CC-BY-ND 4.0 Playing to C#’s strengths Great for exploration and prototyping, terser then forcing you to new generic builders with the same type information everywhere. 35 Lambda’s everywhere! client.Search<Person>(search => search .Query(q => { //do complex if statements here }) ); client.Search<Person>(search => search .Query(BuildQuery) ); public QueryContainer BuildQuery(QueryDescriptor<Person> d) { //do complex if statements here } Can still be abstracted quite neatly into named methods, just extract method on the lambda!
36. { } CC-BY-ND 4.0 Playing to C#’s strengths Not for everyone, therefor NEST 1.0 now also accepts object initialiser syntax for all methods as well as fluent lambda syntax 36 Lambda’s everywhere! IQueryContainer query = new MatchQuery { Field = "abstract", Query = "elasticsearch" }; var request = new SearchRequest { From = 0, Size = 50, Query = query, Filter = ..., Aggregations = ... }; client.Search<Session>(request);
37. { } CC-BY-ND 4.0 Playing to C#’s strengths 37 Covariant search results var r = client.Search<ISearchable>(s => s .Types( typeof(Car), typeof(Organization) ) ); var cars = r.Documents.OfType<Car>(); var orgs = r.Documents.OfType<Organisation>(); foreach(var result in r.Documents) { if (result is Car) {} else if (result is Organization) {} NEST can deal with searches returning multiple types out of the box and instantiate real subclass instances int the returned results. /default-index/car,organization/_search r.Documents is IEnumarable<ISearchable> ISearchable is just a marker interface both types implement, can also be a common base class.
38. { } CC-BY-ND 4.0 Playing to C#’s strengths Elasticsearch boolean query implements unary boolean logic not binary using must/ must_not/should clauses. A common misconception is to use boolean queries as followed: 38 Real binary boolean support bool must X must_not Y should Z In order to represent: (X && !Y) || Z what it actually means is X MUST be present and Y MUST NOT in the document and SHOULD Z be present it will score higher then those documents without Z.
39. { } CC-BY-ND 4.0 Playing to C#’s strengths When using NEST boolean operators are expanded into the correct unary representation. 39 Real binary boolean support .Query(q => (q.Term(X) && !q.Term(Y)) || q.Term(Z)) bool should bool must X must_not Y bool must Z This does in fact translate to: (X && Y) || Z You can use parenthesis to force order and write binary bool queries of any complexity
40. { } CC-BY-ND 4.0 Playing to C#’s strengths Some methods exists of multiple elasticsearch calls 40 Exposing IObservables for coordinated request var observable = this.Client.Reindex<object>(r => r .FromIndex(ElasticsearchConfiguration.DefaultIndex) .ToIndex(toIndex) ); var observer = new ReindexObserver<object>( onNext: (reindex) => Logger.Info(“reindexed portion") onError: (e) => Logger.Error(e.Message), completed: () => { }); observable.Subscribe(observer); Available as observable: Reindex, Snapshot & Restore
41. { } CC-BY-ND 4.0 Elasticsearch.NET & NEST 2.0 41
42. { } CC-BY-ND 4.0 Elasticsearch.NET & NEST 2.0 55% Test coverage NEST 70% Test coverage Elasticsearch.NET Great! but code is basically 5 years worth of writing tests differently and scattered over 5 test projects New test framework will combine literate programming through roslyn to combine writing tests with documentation. 42 Rethinking testing, documentation
43. { } CC-BY-ND 4.0 Elasticsearch.NET & NEST 2.0 Concise effort to go over all the Elasticsearch API and (hopefully) come up with an official contract of requests and responses. 43 Official API specification
44. { } CC-BY-ND 4.0 Elasticsearch.NET & NEST 2.0 Currently we support .NET 4.0 and up on both .NET and Mono. With 2.0 we are going to have proper PCL builds for various platforms and full support for the new .NET core framework. 44 PCL/CoreFX support
45. { } CC-BY-ND 4.0 Quick Recaps • Official high level .NET client • 99% API coverage WITH request and response types • In almost all cases the client you want to start out with 45 • Official low level .NET client • 100% API endpoint coverage but no provided types for request and responses • No dependencies • Use when you have a small isolated use case or integration point with elasticsearch. NEST Elasticsearch.NET
46. { } GitHub: https://github.com/elasticsearch/elasticsearch-net Documentation: http://nest.azurewebsites.net Elasticsearch Documentation: http://www.elasticsearch.org/guide/ Twitter: @mpdreamz @gregmarzouka GitHub: mpdreamz, gmarz Email: dotnet@elasticsearch.com
47. { } This work is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0