Elasticsearch with Node.js

Transcript

1. ELASTICSEARCH WITH NODE.JS Shaunak Kashyap • @shaunak
2. WHY USE ELASTICSEARCH?
3. Full text search
4. Structured search
5. Faceted navigation
6. Analytics
7. FEATURES Easy to get started — , extract, run! Scalable — just add or remove nodes as needed HTTP API Supported clients for Javascript, Python, ..., even Perl ☻ download Well-documented
8. CONCEPTS Document The unit of data fed into Elasticsearch, in JSON format Index A collection of documents stored in Elasticsearch Type The category of the document within an index
9. CONCEPTS Shard A part of an index, consisting of a subset of documents in that index Node A running Elasticsearch process Cluster A collection of nodes with the same c l u s t e r . n a m e
11. INDEX AN EMAIL HTTP c u r l - X P U T h t t p : / / l o c a l h o s t : 9 2 0 0 / e m a i l s / e m a i l / 1 - d ' { " f r o m " : " t h o r @ a v e n g e r s . o r g " , " t o " : [ " t o n y @ s t a r k . c o m " , " h u l k @ a v e n g e r s . o r g " ] , " s u b j e c t " : " G r e e t i n g s , E a r t h l i n g s ! " , " b o d y " : " H e y g u y s , h e a r d f r o m J a n e l a t e l y ? " , " s e n t " : " 2 0 1 5 - 0 6 - 2 9 T 0 8 : 3 0 : 2 3 - 0 7 0 0 " } ' JS v a r e s = r e q u i r e ( " e l a s t i c s e a r c h " ) ; v a r c l i e n t = n e w e s . C l i e n t ( { h o s t : " l o c a l h o s t : 9 2 0 0 " } ) ; c l i e n t . i n d e x ( { i n d e x : " e m a i l s " , t y p e : " e m a i l " , i d : 1 , b o d y : { f r o m : " t h o r @ a v e n g e r s . o r g " , t o : [ " t o n y @ s t a r k . c o m " , " h u l k @ a v e n g e r s . o r g " ] , s u b j e c t : " G r e e t i n g s , E a r t h l i n g s ! " , b o d y : " H e y g u y s , h e a r d f r o m J a n e l a t e l y ? " , s e n t : " 2 0 1 5 - 0 6 - 2 9 T 0 8 : 3 0 : 2 3 - 0 7 0 0 " } } , f u n c t i o n ( e r r o r , r e s p o n s e ) { / / r e s p o n s e = { _ i n d e x : " e m a i l s " , _ t y p e : " e m a i l " , _ i d : " 1 " , _ v e r s i o n : 1 , c r e a t e d : t r u e } } ) ;
12. RETRIEVE EMAIL HTTP c u r l - X G E T h t t p : / / l o c a l h o s t : 9 2 0 0 / e m a i l s / e m a i l / 1 JS c l i e n t . g e t ( { i n d e x : " e m a i l s " , t y p e : " e m a i l " , i d : 1 } , f u n c t i o n ( e r r o r , r e s p o n s e ) { / / r e s p o n s e = { _ i n d e x : " e m a i l s " , _ t y p e : " e m a i l " , _ i d : " 1 " , _ v e r s i o n : 1 , f o u n d : t r u e , _ s o u r c e : { . . . } } } ) ;
13. INDEX MULTIPLE EMAILS HTTP c u r l - X P O S T h t t p : / / l o c a l h o s t : 9 2 0 0 / e m a i l s / e m a i l / _ b u l k - d ' { " i n d e x " : { " _ i d " : 2 } } { " f r o m " : " t o n y @ s t a r k . c o m " , " t o " : [ " t h o r @ a v e n g e r s . o r g " , " h u l k @ a v e n g e r s . o r g " ] , " s u b j e c t " : " R e : G r e e t i n g s , E a r t h l i n g s ! " { " i n d e x " : { " _ i d " : 3 } } { " f r o m " : " t h o r @ a v e n g e r s . o r g " , " t o " : [ " t o n y @ s t a r k . c o m " , " h u l k @ a v e n g e r s . o r g " ] , " s u b j e c t " : " R e : G r e e t i n g s , E a r t h l i n g s ! " { " i n d e x " : { " _ i d " : 4 } } { " f r o m " : " h u l k @ a v e n g e r s . o r g " , " t o " : [ " t o n y @ s t a r k . c o m " , " t h o r @ a v e n g e r s . o r g " ] , " s u b j e c t " : " R e : G r e e t i n g s , E a r t h l i n g s ! " { " i n d e x " : { " _ i d " : 5 } } { " f r o m " : " t o n y @ s t a r k . c o m " , " t o " : " n a t a s h a @ a v e n g e r s . o r g " , " s u b j e c t " : " F w d : G r e e t i n g s , E a r t h l i n g s ! " , " b o d y " : " B a n n e r i s f r e a k i n g { " i n d e x " : { " _ i d " : 6 } } { " f r o m " : " n a t a s h a @ a v e n g e r s . o r g " , " t o " : " h u l k @ a v e n g e r s . o r g " , " s u b j e c t " : " C o f f e e ? " , " b o d y " : " H e y B r u c e , l o n g t i m e n o s e e . F r e e f ' JS c l i e n t . b u l k ( { i n d e x : " e m a i l s " , t y p e : " e m a i l " , b o d y : [ { i n d e x : { _ i d : 2 } } , { f r o m : " t o n y @ s t a r k . c o m " , t o : [ " t h o r @ a v e n g e r s . o r g " , " h u l k @ a v e n g e r s . o r g " ] , s u b j e c t : " R e : G r e e t i n g s , E a r t h l i n g s ! " { i n d e x : { _ i d : 3 } } , { f r o m : " t h o r @ a v e n g e r s . o r g " , t o : [ " t o n y @ s t a r k . c o m " , " h u l k @ a v e n g e r s . o r g " ] , s u b j e c t : " R e : G r e e t i n g s , E a r t h l i n g s ! " { i n d e x : { _ i d : 4 } } , { f r o m : " h u l k @ a v e n g e r s . o r g " , t o : [ " t o n y @ s t a r k . c o m " , " t h o r @ a v e n g e r s . o r g " ] , s u b j e c t : " R e : G r e e t i n g s , E a r t h l i n g s ! " { i n d e x : { _ i d : 5 } } , { f r o m : " t o n y @ s t a r k . c o m " , t o : " n a t a s h a @ a v e n g e r s . o r g " , s u b j e c t : " F w d : G r e e t i n g s , E a r t h l i n g s ! " , b o d y : " B a n n e r i s f r e a k i n g o u t { i n d e x : { _ i d : 6 } } , { f r o m : " n a t a s h a @ a v e n g e r s . o r g " , t o : " h u l k @ a v e n g e r s . o r g " , s u b j e c t : " C o f f e e ? " , b o d y : " H e y B r u c e , l o n g t i m e n o s e e . F r e e f o r c ] } , f u n c t i o n ( e r r o r , r e s p o n s e ) { / / r e s p o n s e = { t o o k : 3 , e r r o r s : f a l s e , i t e m s : [ { i n d e x : . . . } , { i n d e x : . . . } , . . . ] } } ) ;
14. SEARCH FOR EMAILS TO THE HULK AFTER A CERTAIN TIME HTTP c u r l - X P O S T h t t p : / / l o c a l h o s t : 9 2 0 0 / e m a i l s / e m a i l / _ s e a r c h - d ' { " q u e r y " : { " b o o l " : { " m u s t " : [ { " m a t c h " : { " t o " : " h u l k @ a v e n g e r s . o r g " } } , { " r a n g e " : { " s e n t " : { " g t e " : " 2 0 1 5 - 0 6 - 3 0 T 0 0 : 0 0 : 0 0 Z " } } } ] } } } ' JS c l i e n t . s e a r c h ( { i n d e x : " e m a i l s " , t y p e : " e m a i l " , b o d y : { q u e r y : { b o o l : { m u s t : [ { m a t c h : { t o : " h u l k @ a v e n g e r s . o r g " } } , { r a n g e : { s e n t : { g t e : " 2 0 1 5 - 0 6 - 3 0 T 0 0 : 0 0 : 0 0 Z " } } } ] } } } } , f u n c t i o n ( e r r o r , r e s p o n s e ) { / / r e s p o n s e = { t o o k : 1 3 , h i t s : { t o t a l : 4 , m a x _ s c o r e : 1 . 1 0 3 5 8 9 , h i t s : [ . . . ] } } } ) ;
15. COUNT EMAILS GROUPED BY SENDER HTTP c u r l - X P O S T h t t p : / / l o c a l h o s t : 9 2 0 0 / e m a i l s / e m a i l / _ s e a r c h - d ' { " a g g s " : { " c o u n t _ b y _ s e n d e r " : { " t e r m s " : { " f i e l d " : " f r o m " } } } } ' JS c l i e n t . s e a r c h ( { i n d e x : " e m a i l s " , t y p e : " e m a i l " , b o d y : { a g g s : { c o u n t _ b y _ s e n d e r : { t e r m s : { f i e l d : " f r o m " } } } } } , f u n c t i o n ( e r r o r , r e s p o n s e ) { / / r e s p o n s e = { t o o k : 5 , h i t s : { . . . } , a g g r e g a t i o n s : { c o u n t _ b y _ s e n d e r : { b u c k e t s : [ . . . ] } } } } ) ;
16. WHY USE THE JAVASCRIPT CLIENT? Uses Javascript language idioms Low-level and unopinionated: maps 1-1 with HTTP API Cluster sniffing Load balances amongst nodes Detects and gracefully handles node failures Officially supported by Elastic Well-documented API
17. THANK YOU https://discuss.elastic.co/