RAIDING THE MONGODB TOOLBOX Jeremy Mikola jmikola

Agenda Full-text Indexing Geospatial Queries Data Aggregation Creating a Job Queue Tailable Cursors

Full-text Indexing

You have an awesome PHP blog { " _ i d " : O b j e c t I d ( " 5 4 4 f d 6 3 8 6 0 d a b 3 b 1 2 5 2 1 3 7 9 b " ) , " t i t l e " : " T e n S e c r e t s A b o u t P S R - 7 Y o u W o n ' t B e l i e v e ! " , " c o n t e n t " : " P h i l S t u r g e o n c a u s e d q u i t e a s t i r o n t h e P H P - F I G m a i l i n g l i s t t h i s m o r n i n g w h e n h e u n a n i m o u s l y p a s s e d M a t t h e w W e i e r O ' P h i n n e y ' s c o n t r o v e r s i a l P S R - 7 s p e c i f i c a t i o n . P H P - F I G m e m b e r s w e r e o u t r a g e d a s t h e s e l f - p r o c l a i m e d G o r d o n R a m s a y o f P H P … " , " p u b l i s h e d " : t r u e , " c r e a t e d _ a t " : I S O D a t e ( " 2 0 1 4 - 1 0 - 2 8 T 1 7 : 4 6 : 3 6 . 0 6 5 Z " ) }

We’d like to search the content Store arrays of keyword strings Query with regular expressions Sync data to Solr, Elasticsearch, etc. Create a full-text index

Creating a full-text index $ c o l l e c t i o n - > c r e a t e I n d e x ( [ ' c o n t e n t ' = > ' t e x t ' ] ) ; Compound indexing with other fields $ c o l l e c t i o n - > c r e a t e I n d e x ( [ ' c o n t e n t ' = > ' t e x t ' , ' c r e a t e d _ a t ' = > 1 ] ) ; Indexing multiple string fields $ c o l l e c t i o n - > c r e a t e I n d e x ( [ ' c o n t e n t ' = > ' t e x t ' , ' t i t l e ' = > ' t e x t ' ] ) ;

Step 1: Tokenization [ P h i l , S t u r g e o n , c a u s e d , q u i t e , a , s t i r , o n , t h e , P H P - F I G , m a i l i n g , l i s t , t h i s , m o r n i n g , w h e n , h e , u n a n i m o u s l y , p a s s e d , … ]

Step 2: Trim stop-words [ P h i l , S t u r g e o n , c a u s e d , q u i t e , s t i r , P H P - F I G , m a i l i n g , l i s t , m o r n i n g , u n a n i m o u s l y , p a s s e d , … ]

Step 3: Stemming [ P h i l , S t u r g e o n , c a u s e , q u i t e , s t i r , P H P - F I G , m a i l , l i s t , m o r n i n g , u n a n i m o u s , p a s s , … ]

Step 4: Profit?

Querying a text index $ c u r s o r = $ c o l l e c t i o n - > f i n d ( [ ' $ t e x t ' = > [ ' $ s e a r c h ' = > ' P h i l S t u r g e o n ' ] ] ) ; f o r e a c h ( $ c u r s o r a s $ d o c u m e n t ) { e c h o $ d o c u m e n t [ ' c o n t e n t ' ] . " \ n \ n " ; } ↓ P h i l S t u r g e o n c a u s e d q u i t e a s t i r o n t h e P H P - F I G … P h i l J e r k s o n , b e t t e r k n o w n a s @ p h p j e r k o n T w i t t e r …

and Phrases negations $ c u r s o r = $ c o l l e c t i o n - > f i n d ( [ ' $ t e x t ' = > [ ' $ s e a r c h ' = > ' P H P - " P h i l S t u r g e o n " ' ] ] ) ; f o r e a c h ( $ c u r s o r a s $ d o c u m e n t ) { e c h o $ d o c u m e n t [ ' c o n t e n t ' ] . " \ n \ n " ; } ↓ B e p r e p a r e d f o r t h e l a t e s t a n d g r e a t e s t v e r s i o n o f P H P w i t h …

Sorting by the match score $ c u r s o r = $ c o l l e c t i o n - > f i n d ( [ ' $ t e x t ' = > [ ' $ s e a r c h ' = > ' P h i l S t u r g e o n ' ] ] , [ ' s c o r e ' = > [ ' $ m e t a ' = > ' t e x t S c o r e ' ] ] ) ; $ c u r s o r - > s o r t ( [ ' s c o r e ' = > [ ' $ m e t a ' = > ' t e x t S c o r e ' ] ] ) ; f o r e a c h ( $ c u r s o r a s $ d o c u m e n t ) { p r i n t f ( " % . 6 f : % s \ n \ n " , $ d o c u m e n t [ ' s c o r e ' ] , $ d o c u m e n t [ ' c o n t e n t ' ] ) ; } ↓ 1 . 0 3 5 7 1 4 : P h i l S t u r g e o n c a u s e d q u i t e a s t i r o n t h e P H P - F I G … 0 . 5 5 5 5 5 6 : P h i l J e r k s o n , b e t t e r k n o w n a s @ p h p j e r k o n T w i t t e r …

Supporting multiple languages $ c o l l e c t i o n - > c r e a t e I n d e x ( [ ' c o n t e n t ' = > ' t e x t ' ] , [ ' d e f a u l t _ l a n g u a g e ' = > ' e n ' ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' c o n t e n t ' = > ' W e a r e p l a n n i n g a h o t d o g c o n f e r e n c e ' , ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' c o n t e n t ' = > ' D i e K o n f e r e n z w i r d W u r s t C o n b e n a n n t w e r d e n ' , ' l a n g u a g e ' = > ' d e ' , ] ) ; $ c o l l e c t i o n - > f i n d ( [ ' $ t e x t ' = > [ ' $ s e a r c h ' = > ' s a u c i s s e ' , ' $ l a n g u a g e ' = > ' f r ' ] ] , ) ;

Geospatial Queries

Because some of us ❤ maps

Geospatial indexes , , and 2 d s p h e r e for earth-like geometry Supports objects 2 d for flat geometry Legacy point format: [ x , y ] Inclusion proximity intersection GeoJSON

in a nutshell GeoJSON { " t y p e " : " P o i n t " , " c o o r d i n a t e s " : [ 1 0 0 . 0 , 0 . 0 ] } { " t y p e " : " L i n e S t r i n g " , " c o o r d i n a t e s " : [ [ 1 0 0 . 0 , 0 . 0 ] , [ 1 0 1 . 0 , 1 . 0 ] ] } { " t y p e " : " P o l y g o n " , " c o o r d i n a t e s " : [ [ [ 1 0 0 . 0 , 0 . 0 ] , [ 1 0 1 . 0 , 0 . 0 ] , [ 1 0 1 . 0 , 1 . 0 ] , [ 1 0 0 . 0 , 1 . 0 ] , [ 1 0 0 . 0 , 0 . 0 ] ] , [ [ 1 0 0 . 2 , 0 . 2 ] , [ 1 0 0 . 8 , 0 . 2 ] , [ 1 0 0 . 8 , 0 . 8 ] , [ 1 0 0 . 2 , 0 . 8 ] , [ 1 0 0 . 2 , 0 . 2 ] ] ] } { " t y p e " : " M u l t i P o l y g o n " , " c o o r d i n a t e s " : [ [ [ [ 1 0 2 , 2 ] , [ 1 0 3 , 2 ] , [ 1 0 3 , 3 ] , [ 1 0 2 , 3 ] , [ 1 0 2 , 2 ] ] ] , [ [ [ 1 0 0 , 0 ] , [ 1 0 1 , 0 ] , [ 1 0 1 , 1 ] , [ 1 0 0 , 1 ] , [ 1 0 0 , 0 . 0 ] ] ] ] } { " t y p e " : " G e o m e t r y C o l l e c t i o n " , " g e o m e t r i e s " : [ { … } , { … } ] }

ARRAYS ARRAYS EVERYWHERE

No content

Indexing some places of interest $ c o l l e c t i o n - > i n s e r t ( [ ' n a m e ' = > ' H y a t t R e g e n c y S a n t a C l a r a ' , ' t y p e ' = > ' h o t e l ' , ' l o c ' = > [ ' t y p e ' = > ' P o i n t ' , ' c o o r d i n a t e s ' = > [ - 1 2 1 . 9 7 6 5 5 7 , 3 7 . 4 0 4 9 7 7 ] , ] , ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' n a m e ' = > ' I n - N - O u t B u r g e r ' , ' t y p e ' = > ' r e s t a u r a n t ' , ' l o c ' = > [ ' t y p e ' = > ' P o i n t ' , ' c o o r d i n a t e s ' = > [ - 1 2 1 . 9 8 2 1 0 2 , 3 7 . 3 8 7 9 9 3 ] , ] ] ) ; $ c o l l e c t i o n - > e n s u r e I n d e x ( [ ' l o c ' = > ' 2 d s p h e r e ' ] ) ;

Inclusion queries / / D e f i n e a G e o J S O N p o l g y o n $ p o l y g o n = [ ' t y p e ' = > ' P o l y g o n ' , ' c o o r d i n a t e s ' = > [ [ [ - 1 2 1 . 9 7 6 5 5 7 , 3 7 . 4 0 4 9 7 7 ] , / / H y a t t R e g e n c y [ - 1 2 1 . 9 8 2 1 0 2 , 3 7 . 3 8 7 9 9 3 ] , / / I n - N - O u t B u r g e r [ - 1 2 1 . 9 9 2 3 1 1 , 3 7 . 4 0 4 3 8 5 ] , / / R a b b i t ' s F o o t M e a d e r y [ - 1 2 1 . 9 7 6 5 5 7 , 3 7 . 4 0 4 9 7 7 ] , ] , ] , ] ; / / F i n d d o c u m e n t s w i t h i n t h e p o l y g o n ' s b o u n d s $ c o l l e c t i o n - > f i n d ( [ ' l o c ' = > [ ' $ g e o W i t h i n ' = > $ p o l y g o n ] ] ) ; / / F i n d d o c u m e n t s w i t h i n c i r c u l a r b o u n d s $ c o l l e c t i o n - > f i n d ( [ ' l o c ' = > [ ' $ g e o W i t h i n ' = > [ ' $ c e n t e r S p h e r e ' = > [ [ - 1 2 1 . 9 7 6 5 5 7 , 3 7 . 4 0 4 9 7 7 ] , / / C e n t e r c o o r d i n a t e 5 / 3 9 5 9 , / / C o n v e r t m i l e s t o r a d i a n s ] ] ] ] ) ;

Sorted proximity queries $ p o i n t = [ ' t y p e ' = > ' P o i n t ' , ' c o o r d i n a t e s ' = > [ - 1 2 1 . 9 7 6 5 5 7 , 3 7 . 4 0 4 9 7 7 ] ] ; / / F i n d l o c a t i o n s n e a r e s t a p o i n t $ c o l l e c t i o n - > f i n d ( [ ' l o c ' = > [ ' $ n e a r ' = > $ p o i n t ] ] ) ; / / F i n d t h e n e a r e s t 5 0 r e s t a u r a n t s w i t h i n 5 k m $ c o l l e c t i o n - > f i n d ( [ ' l o c ' = > [ ' $ n e a r ' = > $ p o i n t , ' $ m a x D i s t a n c e ' = > 5 0 0 0 ] , ' t y p e ' = > ' r e s t u a r a n t ' , ] ) - > l i m i t ( 5 0 ) ;

No content

Data Aggregation

We have some commands for this aggregate count distinct group mapReduce

Count $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' A 1 2 3 ' , ' n u m ' = > 5 0 0 ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' A 1 2 3 ' , ' n u m ' = > 2 5 0 ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' B 2 1 2 ' , ' n u m ' = > 2 0 0 ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' A 1 2 3 ' , ' n u m ' = > 3 0 0 ] ) ; $ c o l l e c t i o n - > c o u n t ( ) ; / / R e t u r n s 4 $ c o l l e c t i o n - > c o u n t ( [ ' n u m ' = > [ ' $ g t e ' = > 2 5 0 ] ] ) ; / / R e t u r n s 3

Distinct $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' A 1 2 3 ' , ' n u m ' = > 5 0 0 ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' A 1 2 3 ' , ' n u m ' = > 2 5 0 ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' B 2 1 2 ' , ' n u m ' = > 2 0 0 ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' A 1 2 3 ' , ' n u m ' = > 3 0 0 ] ) ; $ c o l l e c t i o n - > d i s t i n c t ( ' c o d e ' ) ; / / R e t u r n s [ " A 1 2 3 " , " B 2 1 2 " ]

Group $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' A 1 2 3 ' , ' n u m ' = > 5 0 0 ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' A 1 2 3 ' , ' n u m ' = > 2 5 0 ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' B 2 1 2 ' , ' n u m ' = > 2 0 0 ] ) ; $ c o l l e c t i o n - > i n s e r t ( [ ' c o d e ' = > ' A 1 2 3 ' , ' n u m ' = > 3 0 0 ] ) ; $ r e s u l t = $ c o l l e c t i o n - > g r o u p ( [ ' c o d e ' = > 1 ] , / / f i e l d ( s ) o n w h i c h t o g r o u p [ ' s u m ' = > 0 ] , / / i n i t i a l a g g r e g a t e v a l u e n e w M o n g o C o d e ( ' f u n c t i o n ( c u r , a g g ) { a g g . s u m + = c u r . n u m } ' ) ) ; f o r e a c h ( $ r e s u l t [ ' r e t v a l ' ] a s $ g r o u p e d ) { p r i n t f ( " % s : % d \ n " , $ g r o u p e d [ ' c o d e ' ] , $ g r o u p e d [ ' s u m ' ] ) ; } ↓ A 1 2 3 : 1 0 5 0 B 2 1 2 : 2 0 0

MapReduce Extremely versatile, powerful Intended for complex data analysis Overkill for simple aggregation tasks e.g. averages, summation, grouping Incremental data processing

Aggregating query profiler output { " o p " : " q u e r y " , " n s " : " d b . c o l l e c t i o n " , " q u e r y " : { " c o d e " : " A 1 2 3 " , " n u m " : { " $ g t " : 2 2 5 } } , " n t o r e t u r n " : 0 , " n t o s k i p " : 0 , " n s c a n n e d " : 1 1 4 2 6 , " l o c k S t a t s " : { … } , " n r e t u r n e d " : 0 , " r e s p o n s e L e n g t h " : 2 0 , " m i l l i s " : 1 2 , " t s " : I S O D a t e ( " 2 0 1 3 - 0 5 - 2 3 T 2 1 : 2 4 : 3 9 . 3 2 7 Z " ) , }

Constructing a query skeleton { " c o d e " : " A 1 2 3 " , " n u m " : { " $ g t " : 2 2 5 } } ↓ { " c o d e " : < s t r i n g > , " n u m " : { " $ g t " : < n u m b e r > } } Aggregate stats for similar queries (e.g. execution time, index performance)

No content

Aggregation framework Process a stream of documents Original input is a collection Outputs one or more result documents Series of operators Filter or transform data Input/output chain p s a x | g r e p m o n g o d | h e a d - n 1

Executing an aggregation pipeline $ c o l l e c t i o n - > a g g r e g a t e C u r s o r ( [ [ ' $ m a t c h ' = > [ ' s t a t u s ' = > ' A ' ] ] , [ ' $ g r o u p ' = > [ ' _ i d ' = > ' $ c u s t _ i d ' , ' t o t a l ' = > [ ' $ s u m ' = > ' $ a m o u n t ' ] ] ] ] ) ;

Pipeline operators $ m a t c h $ g e o N e a r $ p r o j e c t $ g r o u p $ u n w i n d $ s o r t $ l i m i t $ s k i p $ r e d a c t $ o u t

Aggregating fractals

Solving symbolic equations and calculus

Creating a Job Queue

Things not to do in your controllers Send email messages Upload files to S3 Blocking API calls Heavy data processing Mining cryptocurrency

Creating a job $ c o l l e c t i o n - > i n s e r t ( [ ' d a t a ' = > [ … ] , ' p r o c e s s e d ' = > f a l s e , ' c r e a t e d A t ' = > n e w M o n g o D a t e , ] ) ; $ c o l l e c t i o n - > c r e a t e I n d e x ( [ ' p r o c e s s e d ' = > 1 , ' c r e a t e d A t ' = > 1 ] ) ;

Selecting a job $ j o b = $ c o l l e c t i o n - > f i n d A n d M o d i f y ( [ ' p r o c e s s e d ' = > f a l s e ] , [ ' $ s e t ' = > [ ' p r o c e s s e d ' = > t r u e , ' r e c e i v e d A t ' = > n e w M o n g o D a t e ] ] , n u l l , / / f i e l d p r o j e c t i o n ( i f a n y ) [ ' s o r t ' = > [ ' c r e a t e d A t ' = > 1 ] , ' n e w ' = > t r u e , ] ) ; ↓ { " _ i d " : O b j e c t I d ( " 5 4 5 1 5 e 1 6 b a 5 a 4 d a 1 b 1 5 a 1 7 6 6 " ) , " d a t a " : { … } , " p r o c e s s e d " : t r u e , " c r e a t e d A t " : I S O D a t e ( " 2 0 1 4 - 1 0 - 2 9 T 2 1 : 3 7 : 2 6 . 4 0 5 Z " ) , " r e c e i v e d A t " : I S O D a t e ( " 2 0 1 4 - 1 0 - 2 9 T 2 1 : 3 7 : 3 3 . 1 1 8 Z " ) }

Schedule jobs in the future $ c o l l e c t i o n - > i n s e r t ( [ ' d a t a ' = > [ … ] , ' p r o c e s s e d ' = > f a l s e , ' c r e a t e d A t ' = > n e w M o n g o D a t e , ' s c h e d u l e d A t ' = > n e w M o n g o D a t e ( s t r t o t i m e ( ' 1 h o u r ' ) ) , ] ) ; ↓ $ n o w = n e w M o n g o D a t e ; $ j o b = $ c o l l e c t i o n - > f i n d A n d M o d i f y ( [ ' p r o c e s s e d ' = > f a l s e , ' s c h e d u l e d A t ' = > [ ' $ l t ' = > $ n o w ] ] , [ ' $ s e t ' = > [ ' p r o c e s s e d ' = > t r u e , ' r e c e i v e d A t ' = > $ n o w ] ] , n u l l , [ ' s o r t ' = > [ ' c r e a t e d A t ' = > 1 ] , ' n e w ' = > t r u e , ] ) ;

Prioritize job selection $ c o l l e c t i o n - > i n s e r t ( [ ' d a t a ' = > [ … ] , ' p r o c e s s e d ' = > f a l s e , ' c r e a t e d A t ' = > n e w M o n g o D a t e , ' p r i o r i t y ' = > 0 , ] ) ; / / I n d e x : { " p r o c e s s e d " : 1 , " p r i o r i t y " : - 1 , " c r e a t e d A t " : 1 } ↓ $ n o w = n e w M o n g o D a t e ; $ j o b = $ c o l l e c t i o n - > f i n d A n d M o d i f y ( [ ' p r o c e s s e d ' = > f a l s e ] , [ ' $ s e t ' = > [ ' p r o c e s s e d ' = > t r u e , ' r e c e i v e d A t ' = > $ n o w ] ] , n u l l , [ ' s o r t ' = > [ ' p r i o r i t y ' = > - 1 , ' c r e a t e d A t ' = > 1 ] , ' n e w ' = > t r u e , ] ) ;

Gracefully handle failed jobs $ c o l l e c t i o n - > i n s e r t ( [ ' d a t a ' = > [ … ] , ' p r o c e s s e d ' = > f a l s e , ' c r e a t e d A t ' = > n e w M o n g o D a t e , ' a t t e m p t s ' = > 0 , ] ) ; ↓ $ n o w = n e w M o n g o D a t e ; $ j o b = $ c o l l e c t i o n - > f i n d A n d M o d i f y ( [ ' p r o c e s s e d ' = > f a l s e ] , [ ' $ s e t ' = > [ ' p r o c e s s e d ' = > t r u e , ' r e c e i v e d A t ' = > $ n o w ] , ' $ i n c ' = > [ ' a t t e m p t s ' = > 1 ] , ] , n u l l , [ ' s o r t ' = > [ ' c r e a t e d A t ' = > 1 ] , ' n e w ' = > t r u e , ] ) ;

Tailable Cursors

Capped collections $ d a t a b a s e - > c r e a t e C o l l e c t i o n ( ' t a i l m e ' , [ ' c a p p e d ' = > t r u e , ' s i z e ' = > 1 6 7 7 7 2 1 6 , / / 1 6 M i B ' m a x ' = > 1 0 0 0 , ] ) ;

Producer f o r ( $ i = 0 ; + + $ i ; ) { $ c o l l e c t i o n - > i n s e r t ( [ ' x ' = > $ i ] ) ; p r i n t f ( " I n s e r t e d : % d \ n " , $ i ) ; s l e e p ( 1 ) ; } ↓ I n s e r t e d : 1 I n s e r t e d : 2 I n s e r t e d : 3 I n s e r t e d : 4 I n s e r t e d : 5 …

Consumer $ c u r s o r = $ c o l l e c t i o n - > f i n d ( ) ; $ c u r s o r - > t a i l a b l e ( t r u e ) ; $ c u r s o r - > a w a i t D a t a ( t r u e ) ; w h i l e ( t r u e ) { i f ( $ c u r s o r - > d e a d ( ) ) { b r e a k ; } i f ( ! $ c u r s o r - > h a s N e x t ( ) ) { c o n t i n u e ; } p r i n t f ( " C o n s u m e d : % d \ n " , $ c u r s o r - > g e t N e x t ( ) [ ' x ' ] ) ; } ↓ C o n s u m e d : 1 C o n s u m e d : 2 …

Replica set oplog $ c o l l e c t i o n - > i n s e r t ( [ ' x ' = > 1 , ] ) ; ↓ { " t s " : T i m e s t a m p ( 1 4 1 4 6 2 4 9 2 9 , 1 ) , " h " : N u m b e r L o n g ( " 2 6 3 1 3 8 2 8 9 4 3 8 7 4 3 4 4 8 4 " ) , " v " : 2 , " o p " : " i " , " n s " : " t e s t . f o o " , " o " : { " _ i d " : O b j e c t I d ( " 5 4 5 1 7 6 a 1 4 a b 5 c 0 c 9 9 9 d a 7 0 f 0 " ) , " x " : 1 } }

Replica set oplog $ c o l l e c t i o n - > u p d a t e ( [ ' x ' = > 1 ] , [ ' $ i n c ' = > [ ' x ' = > 1 ] ] ) ; ↓ { " t s " : T i m e s t a m p ( 1 4 1 4 6 2 4 9 6 2 , 1 ) , " h " : N u m b e r L o n g ( " 5 0 7 9 4 2 5 1 0 6 8 5 0 5 5 0 7 0 1 " ) , " v " : 2 , " o p " : " u " , " n s " : " t e s t . f o o " , " o 2 " : { " _ i d " : O b j e c t I d ( " 5 4 5 1 7 6 a 1 4 a b 5 c 0 c 9 9 9 d a 7 0 f 0 " ) } , " o " : { " $ s e t " : { " x " : 2 } } }

Fun with MongoDB’s oplog Syncing MongoDB to Solr with PHP MongoDB river plugin for Elasticsearch Building real-time systems @ Stripe Scalable oplog tailing @ Meteor

THANKS! Questions?

Image Credits Books designed by from the Aggregator designed by from the Register designed by from the Ouroboros designed by from the Catherine Please Noun Project stuart mcmorris Noun Project Wilson Joseph Noun Project Silas Reeves Noun Project http://mariompittore.com/wp-content/uploads/2013/08/Social-Gnomes1.png