Elastic: An Elasticsearch Client for Go

I presented these slides at the Munich Search Meetup on 5th February 2015.

Transcript

1. Elastic
 An Elasticsearch Client for Go Oliver Eilhard
 Meplato GmbH
 2015-02-05
2. Search Meetup Munich _ 5th Feb 2015 Who is this guy? • Programming addict, consultant, trainer • Go, Ruby, C#/.NET • E-Procurement • Past: wallmedien AG, CTO • Currently: Meplato GmbH, CTO
3. Why is this guy standing between me and Shay^H beer?
4. Search Meetup Munich _ 5th Feb 2015 Agenda • Show off Elastic:
 Our Elasticsearch client for Go • Using Elasticsearch, Elastic and Go for a fun project
5. Search Meetup Munich _ 5th Feb 2015 What is Go? Go is an open source programming language that makes it easy to build simple, reliable, and efficient software. • https://www.golang.org/ • Robert Griesemer, Rob Pike, Ken Thompson, Brad Fitzpatrick etc. • Open sourced on Nov 10, 2009 • Statically typed, compiled language,
 garbage collected
6. Search Meetup Munich _ 5th Feb 2015 Pros/Cons of Go • Compiles crazy fast • Simple: 25 keywords • Good at concurrency: Channels, light-weight processes (goroutines) • Novel ideas: Interfaces are implicit, type embedding • Creates a static binary: No dependency headaches • Very good at cross-compiling • Solid standard library • Integrated package management system (without versioning) • Opinionated: No generics, no type inheritance, no method overloading, no operators, no exceptions, no pointer arithmetic.
7. package  main   //  Import  the  fmt  package  from  the  standard  library   import  "fmt"   //  Program  starts  here   func  main()  {          fmt.Println("Hello")   }   Hello World in Go
8. func  worker(id  int,  jobs  <-­‐chan  int)  {          for  job  :=  range  jobs  {                  fmt.Printf("Worker  %d  processing  job  %d\n",  id,  job)                  time.Sleep(time.Duration(rand.Int63n(1000))  *  time.Millisecond)          }   }   func  main()  {          //  Create  buffered  channel  with  up  to  100  entries          jobs  :=  make(chan  int,  100)          //  Start  3  workers          for  w  :=  1;  w  <=  3;  w++  {                  go  worker(w,  jobs)          }          //  Send  data  down  the  channel  to  the  workers          for  j  :=  1;  j  <=  1000;  j++  {                  jobs  <-­‐  j          }          close(jobs)   } Concurrency in Go
9. Search Meetup Munich _ 5th Feb 2015 What I like about Go • Very intuitive and clean • Productive within days • Rob Pike and Ken Thompson obviously know a bit or two about programming • They have a mountain of experience, good taste, and solid opinions • They favor simplicity, minimalism, and clean code Recently on Hacker News: “I'm more surprised that the Go 1.4 compiler is written in C. Writing a compiler in C sounds very unpleasant compared to a language like OCaml. Does anyone know why they made this choice for the initial design?“ — “Ken Thompson wrote it. He's pretty good at C.“ Give Go a try e.g. by visiting the Go Tour at https://tour.golang.org/
10. Search Meetup Munich _ 5th Feb 2015 Elasticsearch at Meplato • Product catalogs • Analytics
11. Search Meetup Munich _ 5th Feb 2015 Elastic No solid Elasticsearch client for Go at the time of writing our code. So we wrote our own. We released it as open source last year. https://olivere.github.com/elastic (Home)
 https://github.com/olivere/elastic (Source)
12. package  main   import  (          "fmt"          "net/http"          "github.com/olivere/elastic"   )   func  main()  {          //  Get  a  client          client,  _  :=  elastic.NewClient(http.DefaultClient)          //  Get  the  version  for  some  ES  server          version,  _  :=  client.ElasticsearchVersion("http://127.0.0.1:9200")          fmt.Printf("You're  running  Elasticsearch  version  %s",  version)   } Example: Determine Elasticsearch version
13. Search Meetup Munich _ 5th Feb 2015 Elastic: Central concepts • The client exposes services, e.g. CreateIndex or Search • Services follow the "builder pattern" • Services have a Do method to execute • Follow the Java client API
14. Search Meetup Munich _ 5th Feb 2015 Elastic: Features • Focus on search/data management, not on cluster/administration (for now) • APIs: 14 of 23 implemented • Query DSL: 26 of 39 • Filters: 20 of 27 • Facets: 9 of 9 • Aggregations: 28 of 29
15. //  If  "test"  index  exists,  delete  it   exists,  err  :=  client.IndexExists("test").Do()   if  err  !=  nil  {          panic(err)   }   if  exists  {          if  _,  err  :=  client.DeleteIndex("test").Do();  err  !=  nil  {                  panic(err)          }          fmt.Println("Index  deleted")   }   //  Create  a  new  index  named  "test"   mapping  :=  `{"settings":{"number_of_shards":1,"number_of_replicas":0}}`   _,  err  =  client.CreateIndex("test").Body(mapping).Do()   if  err  !=  nil  {        panic(err)   }   fmt.Println("Index  created") Example: Creating and deleting indices
16. type  Tweet  struct  {          User        string  `json:"user"`          Message  string  `json:"message"`   }   …   //  Create  a  new  tweet   tweet  :=  Tweet{          User:        "telexico",          Message:  "At  the  Elasticsearch  meetup,  having  fun  with  the  good  people  here.",   }   //  Add  document  to  Elasticsearch   _,  err  =  client.Index().
                      Index("test").
                      Type("tweet").
                      Id("1").
                      BodyJson(tweet).
                      Do()   if  err  !=  nil  {        panic(err)   }   fmt.Println("Document  added") Example: Adding a document
17. //  Get  document  of  type  "tweet"  with  id="1"  from  index  "test"   doc,  err  :=  client.Get().
                            Index("test").
                            Type("tweet").
                            Id("1").
                            Do()   if  err  !=  nil  {        panic(err)   }   //  Unmarshal  JSON   var  tweet  Tweet   if  err  :=  json.Unmarshal(*doc.Source,  &tweet);  err  !=  nil  {        panic(err)   }   fmt.Printf("Tweet  %s  by  %s:\n%q\n",  
                        doc.Id,  tweet.User,  tweet.Message) Example: Retrieving a document by ID
18. //  Search  for  documents  of  user  "telexico"   termQuery  :=  elastic.NewTermQuery("user",  "telexico")   res,  err  :=  client.Search().
                            Index("test").
                            Query(termQuery).
                            Do()   if  err  !=  nil  {          panic(err)   }   //  Print  results   fmt.Printf("%d  hits:\n\n",  res.Hits.TotalHits)   for  _,  doc  :=  range  res.Hits.Hits  {          var  tweet  Tweet          json.Unmarshal(*doc.Source,  &tweet)          fmt.Printf("@%s:  %s\n",  tweet.User,  tweet.Message)   } Example: Searching by Term Query
19. Search Meetup Munich _ 5th Feb 2015 Let’s do something different How do you learn stuff best?
 By applying it to a real-world problem. So we went here…
21. Search Meetup Munich _ 5th Feb 2015 Meplato Racing Racing is our passion. We took part in the complete 2014 VLN season at the Nordschleife. Maybe we can apply technology here, too.
 
 
 VLN: http://www.vln.de
23. Search Meetup Munich _ 5th Feb 2015 Problem identified We were too slow: ~20 seconds per lap.
 So where exactly did we lose time? What do good engineers do? 1. Gather data
 2. Analyze data
24. Search Meetup Munich _ 5th Feb 2015 GPS Tracking • Cars have GPS on board • Cars also offer tons of other data from gearbox, clutch, brakes etc. • We only need: • Latitude/Longitude • Speed • Steering wheel position
25. Search Meetup Munich _ 5th Feb 2015 GPS Tracking Ideas • Downloading data at pitstops? • Too slow • We can track only our own car
26. Search Meetup Munich _ 5th Feb 2015 GPS Tracking Ideas • „Racing App“ by GPSoverIP GmbH • Provides live car tracking over the Internet • Let’s write a proxy and record data into Elasticsearch
 Racing App: https://itunes.apple.com/de/app/racing-app/id444591012?mt=8
27. {      "_index"  :  "racetrack-­‐2014-­‐08-­‐02",      "_type"  :  "record",      "_id"  :  "bXyMvt-­‐-­‐RlGG43n3zJP8ug",      "_score"  :  1.0,      "_source":{              "id":"nbr24@216654@2014-­‐08-­‐02T11:09:27.040151138Z",              "track":"nbr24",              "device":"216654",              "name":"Opel  Astra  OPC  CUP",              "nr":"357",              "class":"CUP1",              "location":"50.348095,6.971377",              "direction":237,              "speed":234.3,              "ts":"2014-­‐08-­‐02T11:09:23Z",              "created":"2014-­‐08-­‐02T11:09:27.040157548Z"      }   } GPS Tracking
28. Search Meetup Munich _ 5th Feb 2015 How do we track sector times? • GeoJSON for map and sectors • http://geojson.io/
30. Search Meetup Munich _ 5th Feb 2015 Analyze data with Elasticsearch • Run a program to track sector entry and exit times for the cars • Elasticsearch has strong Geo query capabilities that make this super simple • Make a nice PDF for the non-geeks
32. Search Meetup Munich _ 5th Feb 2015 Analyze the race afterwards • We are great fans of learning by visualization • So we used D3.js to run the race again, fast-forward
34. Search Meetup Munich _ 5th Feb 2015 What we learned • Elasticsearch is a realtime document store • Elasticsearch is crazy fast • Elasticsearch has exhaustive query capabilities • GeoJSON for geo data • D3.js for visualization • How to run the Nordschleife 
 in sub-9:30m
35. Thank you
 
 Oliver Eilhard
 Meplato GmbH
 oliver.eilhard@meplato.com
 https://www.meplato.com/
 @telexico