Bleve - modern text indexing for Go

Slide 1

Slide 1 text

Bleve modern text indexing for Go GopherCon India - Feb 2015

Slide 2

Slide 2 text

2 Say What? blev-ee bih-leev

Slide 3

Slide 3 text

3 Marty Schoch • NoSQL Document Database • Ofﬁcial Go SDK • Projects Using Go • N1QL Query Language • Secondary Indexing • Cross Data-Center Replication

Slide 4

Slide 4 text

4 Why? • Lucene/Solr/Elasticsearch are awesome • Could we build 50% of Lucene's text analysis, combine it with off-the-shelf KV stores and get something interesting? • Pluggable text analysis • Pluggable KV storage

Slide 5

Slide 5 text

What is Search?

Slide 6

Slide 6 text

6 Simple Search

Slide 7

Slide 7 text

7 Advanced Search

Slide 8

Slide 8 text

8 Search Results

Slide 9

Slide 9 text

9 Faceted Search

Slide 10

Slide 10 text

Getting Started

Slide 11

Slide 11 text

11 Install Bleve go get github.com/blevesearch/bleve/...

Slide 12

Slide 12 text

12 Import import "github.com/blevesearch/bleve" type Person struct { Name string } func main() { mapping := bleve.NewIndexMapping() index, err := bleve.New("people.bleve", mapping) if err != nil { log.Fatal(err) } person := Person{"Marty Schoch"} err = index.Index("m1", person) if err != nil { log.Fatal(err) } fmt.Println("Indexed Document") } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Slide 13

Slide 13 text

13 Data Model import "github.com/blevesearch/bleve" type Person struct { Name string } func main() { mapping := bleve.NewIndexMapping() index, err := bleve.New("people.bleve", mapping) if err != nil { log.Fatal(err) } person := Person{"Marty Schoch"} err = index.Index("m1", person) if err != nil { log.Fatal(err) } fmt.Println("Indexed Document") } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Slide 14

Slide 14 text

14 Index Mapping import "github.com/blevesearch/bleve" type Person struct { Name string } func main() { mapping := bleve.NewIndexMapping() index, err := bleve.New("people.bleve", mapping) if err != nil { log.Fatal(err) } person := Person{"Marty Schoch"} err = index.Index("m1", person) if err != nil { log.Fatal(err) } fmt.Println("Indexed Document") } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Slide 15

Slide 15 text

15 Create New Index import "github.com/blevesearch/bleve" type Person struct { Name string } func main() { mapping := bleve.NewIndexMapping() index, err := bleve.New("people.bleve", mapping) if err != nil { log.Fatal(err) } person := Person{"Marty Schoch"} err = index.Index("m1", person) if err != nil { log.Fatal(err) } fmt.Println("Indexed Document") } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Slide 16

Slide 16 text

16 Index Data import "github.com/blevesearch/bleve" type Person struct { Name string } func main() { mapping := bleve.NewIndexMapping() index, err := bleve.New("people.bleve", mapping) if err != nil { log.Fatal(err) } person := Person{"Marty Schoch"} err = index.Index("m1", person) if err != nil { log.Fatal(err) } fmt.Println("Indexed Document") } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Slide 17

Slide 17 text

17 Index Data import "github.com/blevesearch/bleve" type Person struct { Name string } func main() { mapping := bleve.NewIndexMapping() index, err := bleve.New("people.bleve", mapping) if err != nil { log.Fatal(err) } person := Person{"Marty Schoch"} err = index.Index("m1", person) if err != nil { log.Fatal(err) } fmt.Println("Indexed Document") } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Indexed Document Program exited.

Slide 18

Slide 18 text

18 Open Index func main() { index, err := bleve.Open("people.bleve") if err != nil { log.Fatal(err) } query := bleve.NewTermQuery("marty") request := bleve.NewSearchRequest(query) result, err := index.Search(request) if err != nil { log.Fatal(err) } fmt.Println(result) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Slide 19

Slide 19 text

19 Build Query func main() { index, err := bleve.Open("people.bleve") if err != nil { log.Fatal(err) } query := bleve.NewTermQuery("marty") request := bleve.NewSearchRequest(query) result, err := index.Search(request) if err != nil { log.Fatal(err) } fmt.Println(result) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Slide 20

Slide 20 text

20 Build Request func main() { index, err := bleve.Open("people.bleve") if err != nil { log.Fatal(err) } query := bleve.NewTermQuery("marty") request := bleve.NewSearchRequest(query) result, err := index.Search(request) if err != nil { log.Fatal(err) } fmt.Println(result) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Slide 21

Slide 21 text

21 Search func main() { index, err := bleve.Open("people.bleve") if err != nil { log.Fatal(err) } query := bleve.NewTermQuery("marty") request := bleve.NewSearchRequest(query) result, err := index.Search(request) if err != nil { log.Fatal(err) } fmt.Println(result) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Slide 22

Slide 22 text

22 Search func main() { index, err := bleve.Open("people.bleve") if err != nil { log.Fatal(err) } query := bleve.NewTermQuery("marty") request := bleve.NewSearchRequest(query) result, err := index.Search(request) if err != nil { log.Fatal(err) } fmt.Println(result) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 matches, showing 1 through 1, took 24.713µs 1. m1 (0.216978) Program exited.

Slide 23

Slide 23 text

More Realistic Data

Slide 24

Slide 24 text

24 Conference Schedule type Event struct { UID string `json:"uid"` Summary string `json:"summary"` Description string `json:"description"` Speaker string `json:"speaker"` Start time.Time `json:"start"` Duration float64 `json:"duration"` } 1 2 3 4 5 6 7 8

Slide 25

Slide 25 text

25 Phrase Search func main() { index, err := bleve.Open("gopherconin.bleve") if err != nil { log.Fatal(err) } phrase := []string{"quality", "search", "results"} q := bleve.NewPhraseQuery(phrase, "description") req := bleve.NewSearchRequest(q) req.Highlight = bleve.NewHighlightWithStyle("html") req.Fields = []string{"summary", "speaker"} res, err := index.Search(req) if err != nil { log.Fatal(err) } fmt.Println(res) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Slide 26

Slide 26 text

26 Phrase Search func main() { index, err := bleve.Open("gopherconin.bleve") if err != nil { log.Fatal(err) } phrase := []string{"quality", "search", "results"} q := bleve.NewPhraseQuery(phrase, "description") req := bleve.NewSearchRequest(q) req.Highlight = bleve.NewHighlightWithStyle("html") req.Fields = []string{"summary", "speaker"} res, err := index.Search(req) if err != nil { log.Fatal(err) } fmt.Println(res) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1 matches, showing 1 through 1, took 46.134µs 1. bleve_-‐_modern_text_indexing_for_go (1.033644) description …earch component. But delivering high quality search results requires a long list of text analysis and indexing techniques. With the bleve library, we bring advanced text indexing and search to your Go… summary bleve -‐ modern text indexing for Go speaker Martin Schoch Program exited.

Slide 27

Slide 27 text

27 Query String func main() { index, err := bleve.Open("gopherconin.bleve") if err != nil { log.Fatal(err) } qString := `+description:text ` qString += `summary:"text indexing" ` qString += `summary:believe~2 ` qString += `-‐description:lucene ` qString += `duration:<30` q := bleve.NewQueryStringQuery(qString) req := bleve.NewSearchRequest(q) req.Highlight = bleve.NewHighlightWithStyle("html") req.Fields = []string{"summary", "speaker", "description", "duration"} res, err := index.Search(req) if err != nil { log.Fatal(err) } fmt.Println(res) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Slide 28

Slide 28 text

28 Query String func main() { index, err := bleve.Open("gopherconin.bleve") if err != nil { log.Fatal(err) } qString := `+description:text ` qString += `summary:"text indexing" ` qString += `summary:believe~2 ` qString += `-‐description:lucene ` qString += `duration:<30` q := bleve.NewQueryStringQuery(qString) req := bleve.NewSearchRequest(q) req.Highlight = bleve.NewHighlightWithStyle("html") req.Fields = []string{"summary", "speaker", "description", "duration"} res, err := index.Search(req) if err != nil { log.Fatal(err) } fmt.Println(res) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 1 matches, showing 1 through 1, took 220.08µs 1. bleve_-‐_modern_text_indexing_for_go (0.338882) summary bleve -‐ modern text indexing for Go description …ist of text analysis and indexing techniques. With the bleve library, we bring advanced text indexing and search to your Go applications. This talk will start with a brief introduction to text search … speaker Martin Schoch duration 25 Program exited.

Slide 29

Slide 29 text

29 HTTP Handlers (optional) import "github.com/blevesearch/bleve/http" • All major bleve operations mapped • Assume JSON document bodies • See bleve-explorer sample app https://github.com/blevesearch/bleve-‐explorer

Slide 30

Slide 30 text

Putting It All Together

Slide 31

Slide 31 text

31 Schedule Search http://gopherconin.blevesearch.com/

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Join the Community

Slide 36

Slide 36 text

36 Community • #bleve is small/quiet room, talk to us real time • Discuss your use-case • Plan a feature implementation • Apache License v2.0 • Report Issues, Submit Pull Requests

Slide 37

Slide 37 text

37 Contributors

Slide 38

Slide 38 text

38 Roadmap • Result Sorting (other than score) • Better Spell Suggest/Fuzzy Search • Performance • Prepare for 1.0 Release

Slide 39

Slide 39 text

One More Thing…

Slide 40

Slide 40 text

40 Hindi Analyzer

Slide 41

Slide 41 text

41 Thanks • Marty Schoch • marty@couchbase.com • http://github.com/blevesearch/bleve • @mschoch • @blevesearch