Share
Embed
Elastic: An Elasticsearch Client for Go
I presented these slides at the Munich Search Meetup on 5th February 2015.
Oliver Eilhard
February 05, 2015
Transcript
35 slides
-
Search Meetup Munich _ 5th Feb 2015 Who is this
guy? • Programming addict, consultant, trainer • Go, Ruby, C#/.NET • E-Procurement • Past: wallmedien AG, CTO • Currently: Meplato GmbH, CTO -
Search Meetup Munich _ 5th Feb 2015 Agenda • Show
off Elastic: Our Elasticsearch client for Go • Using Elasticsearch, Elastic and Go for a fun project -
Search Meetup Munich _ 5th Feb 2015 What is Go?
Go is an open source programming language that makes it easy to build simple, reliable, and efficient software. • https://www.golang.org/ • Robert Griesemer, Rob Pike, Ken Thompson, Brad Fitzpatrick etc. • Open sourced on Nov 10, 2009 • Statically typed, compiled language, garbage collected -
Search Meetup Munich _ 5th Feb 2015 Pros/Cons of Go
• Compiles crazy fast • Simple: 25 keywords • Good at concurrency: Channels, light-weight processes (goroutines) • Novel ideas: Interfaces are implicit, type embedding • Creates a static binary: No dependency headaches • Very good at cross-compiling • Solid standard library • Integrated package management system (without versioning) • Opinionated: No generics, no type inheritance, no method overloading, no operators, no exceptions, no pointer arithmetic. -
package main // Import the fmt package from the
standard library import "fmt" // Program starts here func main() { fmt.Println("Hello") } Hello World in Go -
func worker(id int, jobs <-‐chan int) {
for job := range jobs { fmt.Printf("Worker %d processing job %d\n", id, job) time.Sleep(time.Duration(rand.Int63n(1000)) * time.Millisecond) } } func main() { // Create buffered channel with up to 100 entries jobs := make(chan int, 100) // Start 3 workers for w := 1; w <= 3; w++ { go worker(w, jobs) } // Send data down the channel to the workers for j := 1; j <= 1000; j++ { jobs <-‐ j } close(jobs) } Concurrency in Go -
Search Meetup Munich _ 5th Feb 2015 What I like
about Go • Very intuitive and clean • Productive within days • Rob Pike and Ken Thompson obviously know a bit or two about programming • They have a mountain of experience, good taste, and solid opinions • They favor simplicity, minimalism, and clean code Recently on Hacker News: “I'm more surprised that the Go 1.4 compiler is written in C. Writing a compiler in C sounds very unpleasant compared to a language like OCaml. Does anyone know why they made this choice for the initial design?“ — “Ken Thompson wrote it. He's pretty good at C.“ Give Go a try e.g. by visiting the Go Tour at https://tour.golang.org/ -
Search Meetup Munich _ 5th Feb 2015 Elasticsearch at Meplato
• Product catalogs • Analytics -
Search Meetup Munich _ 5th Feb 2015 Elastic No solid
Elasticsearch client for Go at the time of writing our code. So we wrote our own. We released it as open source last year. https://olivere.github.com/elastic (Home) https://github.com/olivere/elastic (Source) -
package main import ( "fmt"
"net/http" "github.com/olivere/elastic" ) func main() { // Get a client client, _ := elastic.NewClient(http.DefaultClient) // Get the version for some ES server version, _ := client.ElasticsearchVersion("http://127.0.0.1:9200") fmt.Printf("You're running Elasticsearch version %s", version) } Example: Determine Elasticsearch version -
Search Meetup Munich _ 5th Feb 2015 Elastic: Central concepts
• The client exposes services, e.g. CreateIndex or Search • Services follow the "builder pattern" • Services have a Do method to execute • Follow the Java client API -
Search Meetup Munich _ 5th Feb 2015 Elastic: Features •
Focus on search/data management, not on cluster/administration (for now) • APIs: 14 of 23 implemented • Query DSL: 26 of 39 • Filters: 20 of 27 • Facets: 9 of 9 • Aggregations: 28 of 29 -
// If "test" index exists, delete it exists, err
:= client.IndexExists("test").Do() if err != nil { panic(err) } if exists { if _, err := client.DeleteIndex("test").Do(); err != nil { panic(err) } fmt.Println("Index deleted") } // Create a new index named "test" mapping := `{"settings":{"number_of_shards":1,"number_of_replicas":0}}` _, err = client.CreateIndex("test").Body(mapping).Do() if err != nil { panic(err) } fmt.Println("Index created") Example: Creating and deleting indices -
type Tweet struct { User
string `json:"user"` Message string `json:"message"` } … // Create a new tweet tweet := Tweet{ User: "telexico", Message: "At the Elasticsearch meetup, having fun with the good people here.", } // Add document to Elasticsearch _, err = client.Index(). Index("test"). Type("tweet"). Id("1"). BodyJson(tweet). Do() if err != nil { panic(err) } fmt.Println("Document added") Example: Adding a document -
// Get document of type "tweet" with id="1" from index
"test" doc, err := client.Get(). Index("test"). Type("tweet"). Id("1"). Do() if err != nil { panic(err) } // Unmarshal JSON var tweet Tweet if err := json.Unmarshal(*doc.Source, &tweet); err != nil { panic(err) } fmt.Printf("Tweet %s by %s:\n%q\n", doc.Id, tweet.User, tweet.Message) Example: Retrieving a document by ID -
// Search for documents of user "telexico" termQuery :=
elastic.NewTermQuery("user", "telexico") res, err := client.Search(). Index("test"). Query(termQuery). Do() if err != nil { panic(err) } // Print results fmt.Printf("%d hits:\n\n", res.Hits.TotalHits) for _, doc := range res.Hits.Hits { var tweet Tweet json.Unmarshal(*doc.Source, &tweet) fmt.Printf("@%s: %s\n", tweet.User, tweet.Message) } Example: Searching by Term Query -
Search Meetup Munich _ 5th Feb 2015 Let’s do something
different How do you learn stuff best? By applying it to a real-world problem. So we went here… -
Search Meetup Munich _ 5th Feb 2015 Meplato Racing Racing
is our passion. We took part in the complete 2014 VLN season at the Nordschleife. Maybe we can apply technology here, too. VLN: http://www.vln.de -
Search Meetup Munich _ 5th Feb 2015 Problem identified We
were too slow: ~20 seconds per lap. So where exactly did we lose time? What do good engineers do? 1. Gather data 2. Analyze data -
Search Meetup Munich _ 5th Feb 2015 GPS Tracking •
Cars have GPS on board • Cars also offer tons of other data from gearbox, clutch, brakes etc. • We only need: • Latitude/Longitude • Speed • Steering wheel position -
Search Meetup Munich _ 5th Feb 2015 GPS Tracking Ideas
• Downloading data at pitstops? • Too slow • We can track only our own car -
Search Meetup Munich _ 5th Feb 2015 GPS Tracking Ideas
• „Racing App“ by GPSoverIP GmbH • Provides live car tracking over the Internet • Let’s write a proxy and record data into Elasticsearch Racing App: https://itunes.apple.com/de/app/racing-app/id444591012?mt=8 -
{ "_index" : "racetrack-‐2014-‐08-‐02", "_type" :
"record", "_id" : "bXyMvt-‐-‐RlGG43n3zJP8ug", "_score" : 1.0, "_source":{ "id":"nbr24@216654@2014-‐08-‐02T11:09:27.040151138Z", "track":"nbr24", "device":"216654", "name":"Opel Astra OPC CUP", "nr":"357", "class":"CUP1", "location":"50.348095,6.971377", "direction":237, "speed":234.3, "ts":"2014-‐08-‐02T11:09:23Z", "created":"2014-‐08-‐02T11:09:27.040157548Z" } } GPS Tracking -
Search Meetup Munich _ 5th Feb 2015 How do we
track sector times? • GeoJSON for map and sectors • http://geojson.io/ -
Search Meetup Munich _ 5th Feb 2015 Analyze data with
Elasticsearch • Run a program to track sector entry and exit times for the cars • Elasticsearch has strong Geo query capabilities that make this super simple • Make a nice PDF for the non-geeks -
Search Meetup Munich _ 5th Feb 2015 Analyze the race
afterwards • We are great fans of learning by visualization • So we used D3.js to run the race again, fast-forward -
Search Meetup Munich _ 5th Feb 2015 What we learned
• Elasticsearch is a realtime document store • Elasticsearch is crazy fast • Elasticsearch has exhaustive query capabilities • GeoJSON for geo data • D3.js for visualization • How to run the Nordschleife in sub-9:30m