Raiding the MongoDB Toolbox

RAIDING THE
MONGODB
TOOLBOX
Jeremy Mikola
jmikola

View Slide

Agenda
Full-text Indexing
Geospatial Queries
Data Aggregation
Creating a Job Queue
Tailable Cursors

View Slide

Full-text
Indexing

View Slide

You have an awesome PHP blog
{
"
_
i
d
"
: O
b
j
e
c
t
I
d
(
"
5
4
4
f
d
6
3
8
6
0
d
a
b
3
b
1
2
5
2
1
3
7
9
b
"
)
,
"
t
i
t
l
e
"
: "
T
e
n S
e
c
r
e
t
s A
b
o
u
t P
S
R
-
7 Y
o
u W
o
n
'
t B
e
l
i
e
v
e
!
"
,
"
c
o
n
t
e
n
t
"
: "
P
h
i
l S
t
u
r
g
e
o
n c
a
u
s
e
d q
u
i
t
e a s
t
i
r o
n t
h
e P
H
P
-
F
I
G
m
a
i
l
i
n
g l
i
s
t t
h
i
s m
o
r
n
i
n
g w
h
e
n h
e u
n
a
n
i
m
o
u
s
l
y
p
a
s
s
e
d M
a
t
t
h
e
w W
e
i
e
r O
'
P
h
i
n
n
e
y
'
s c
o
n
t
r
o
v
e
r
s
i
a
l
P
S
R
-
7 s
p
e
c
i
f
i
c
a
t
i
o
n
. P
H
P
-
F
I
G m
e
m
b
e
r
s w
e
r
e o
u
t
r
a
g
e
d
a
s t
h
e s
e
l
f
-
p
r
o
c
l
a
i
m
e
d G
o
r
d
o
n R
a
m
s
a
y o
f P
H
P
…
"
,
"
p
u
b
l
i
s
h
e
d
"
: t
r
u
e
,
"
c
r
e
a
t
e
d
_
a
t
"
: I
S
O
D
a
t
e
(
"
2
0
1
4
-
1
0
-
2
8
T
1
7
:
4
6
:
3
6
.
0
6
5
Z
"
)
}

View Slide

We’d like to search the content
Store arrays of keyword strings
Query with regular expressions
Sync data to Solr, Elasticsearch, etc.
Create a full-text index

View Slide

Creating a full-text index
$
c
o
l
l
e
c
t
i
o
n
-
>
c
r
e
a
t
e
I
n
d
e
x
(
[
'
c
o
n
t
e
n
t
' =
> '
t
e
x
t
'
]
)
;
Compound indexing with other fields
$
c
o
l
l
e
c
t
i
o
n
-
>
c
r
e
a
t
e
I
n
d
e
x
(
[
'
c
o
n
t
e
n
t
' =
> '
t
e
x
t
'
, '
c
r
e
a
t
e
d
_
a
t
' =
> 1
]
)
;
Indexing multiple string fields
$
c
o
l
l
e
c
t
i
o
n
-
>
c
r
e
a
t
e
I
n
d
e
x
(
[
'
c
o
n
t
e
n
t
' =
> '
t
e
x
t
'
, '
t
i
t
l
e
' =
> '
t
e
x
t
'
]
)
;

View Slide

Step 1: Tokenization
[
P
h
i
l
, S
t
u
r
g
e
o
n
, c
a
u
s
e
d
, q
u
i
t
e
, a
, s
t
i
r
,
o
n
, t
h
e
, P
H
P
-
F
I
G
, m
a
i
l
i
n
g
, l
i
s
t
, t
h
i
s
,
m
o
r
n
i
n
g
, w
h
e
n
, h
e
, u
n
a
n
i
m
o
u
s
l
y
, p
a
s
s
e
d
,
…
]

View Slide

Step 2: Trim stop-words
[
P
h
i
l
, S
t
u
r
g
e
o
n
, c
a
u
s
e
d
, q
u
i
t
e
, s
t
i
r
,
P
H
P
-
F
I
G
, m
a
i
l
i
n
g
, l
i
s
t
,
m
o
r
n
i
n
g
, u
n
a
n
i
m
o
u
s
l
y
, p
a
s
s
e
d
,
…
]

View Slide

Step 3: Stemming
[
P
h
i
l
, S
t
u
r
g
e
o
n
, c
a
u
s
e
, q
u
i
t
e
, s
t
i
r
,
P
H
P
-
F
I
G
, m
a
i
l
, l
i
s
t
,
m
o
r
n
i
n
g
, u
n
a
n
i
m
o
u
s
, p
a
s
s
,
…
]

View Slide

Step 4: Profit?

View Slide

Querying a text index
$
c
u
r
s
o
r = $
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
(
[
'
$
t
e
x
t
' =
> [
'
$
s
e
a
r
c
h
' =
> '
P
h
i
l S
t
u
r
g
e
o
n
'
]
]
)
;
f
o
r
e
a
c
h (
$
c
u
r
s
o
r a
s $
d
o
c
u
m
e
n
t
) {
e
c
h
o $
d
o
c
u
m
e
n
t
[
'
c
o
n
t
e
n
t
'
] . "
\
n
\
n
"
;
}
↓
P
h
i
l S
t
u
r
g
e
o
n c
a
u
s
e
d q
u
i
t
e a s
t
i
r o
n t
h
e P
H
P
-
F
I
G
…
P
h
i
l J
e
r
k
s
o
n
, b
e
t
t
e
r k
n
o
w
n a
s @
p
h
p
j
e
r
k o
n T
w
i
t
t
e
r
…

View Slide

and
Phrases negations
$
c
u
r
s
o
r = $
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
(
[
'
$
t
e
x
t
' =
> [
'
$
s
e
a
r
c
h
' =
> '
P
H
P -
"
P
h
i
l S
t
u
r
g
e
o
n
"
'
]
]
)
;
f
o
r
e
a
c
h (
$
c
u
r
s
o
r a
s $
d
o
c
u
m
e
n
t
) {
e
c
h
o $
d
o
c
u
m
e
n
t
[
'
c
o
n
t
e
n
t
'
] . "
\
n
\
n
"
;
}
↓
B
e p
r
e
p
a
r
e
d f
o
r t
h
e l
a
t
e
s
t a
n
d g
r
e
a
t
e
s
t v
e
r
s
i
o
n o
f P
H
P w
i
t
h
…

View Slide

Sorting by the match score
$
c
u
r
s
o
r = $
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
(
[
'
$
t
e
x
t
' =
> [
'
$
s
e
a
r
c
h
' =
> '
P
h
i
l S
t
u
r
g
e
o
n
'
]
]
,
[
'
s
c
o
r
e
' =
> [
'
$
m
e
t
a
' =
> '
t
e
x
t
S
c
o
r
e
'
]
]
)
;
$
c
u
r
s
o
r
-
>
s
o
r
t
(
[
'
s
c
o
r
e
' =
> [
'
$
m
e
t
a
' =
> '
t
e
x
t
S
c
o
r
e
'
]
]
)
;
f
o
r
e
a
c
h (
$
c
u
r
s
o
r a
s $
d
o
c
u
m
e
n
t
) {
p
r
i
n
t
f
(
"
%
.
6
f
: %
s
\
n
\
n
"
, $
d
o
c
u
m
e
n
t
[
'
s
c
o
r
e
'
]
, $
d
o
c
u
m
e
n
t
[
'
c
o
n
t
e
n
t
'
]
)
;
}
↓
1
.
0
3
5
7
1
4
: P
h
i
l S
t
u
r
g
e
o
n c
a
u
s
e
d q
u
i
t
e a s
t
i
r o
n t
h
e P
H
P
-
F
I
G
…
0
.
5
5
5
5
5
6
: P
h
i
l J
e
r
k
s
o
n
, b
e
t
t
e
r k
n
o
w
n a
s @
p
h
p
j
e
r
k o
n T
w
i
t
t
e
r
…

View Slide

Supporting multiple languages
$
c
o
l
l
e
c
t
i
o
n
-
>
c
r
e
a
t
e
I
n
d
e
x
(
[
'
c
o
n
t
e
n
t
' =
> '
t
e
x
t
'
]
,
[
'
d
e
f
a
u
l
t
_
l
a
n
g
u
a
g
e
' =
> '
e
n
'
]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
n
t
e
n
t
' =
> '
W
e a
r
e p
l
a
n
n
i
n
g a h
o
t d
o
g c
o
n
f
e
r
e
n
c
e
'
,
]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
n
t
e
n
t
' =
> '
D
i
e K
o
n
f
e
r
e
n
z w
i
r
d W
u
r
s
t
C
o
n b
e
n
a
n
n
t w
e
r
d
e
n
'
,
'
l
a
n
g
u
a
g
e
' =
> '
d
e
'
,
]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
(
[
'
$
t
e
x
t
' =
> [
'
$
s
e
a
r
c
h
' =
> '
s
a
u
c
i
s
s
e
'
, '
$
l
a
n
g
u
a
g
e
' =
> '
f
r
'
]
]
,
)
;

View Slide

Geospatial Queries

View Slide

Because some of us ❤ maps

View Slide

Geospatial indexes
, , and
2
d
s
p
h
e
r
e for earth-like geometry
Supports objects
2
d for flat geometry
Legacy point format: [
x
,
y
]
Inclusion proximity intersection
GeoJSON

View Slide

in a nutshell
GeoJSON
{ "
t
y
p
e
"
: "
P
o
i
n
t
"
, "
c
o
o
r
d
i
n
a
t
e
s
"
: [
1
0
0
.
0
, 0
.
0
] }
{ "
t
y
p
e
"
: "
L
i
n
e
S
t
r
i
n
g
"
, "
c
o
o
r
d
i
n
a
t
e
s
"
: [
[
1
0
0
.
0
, 0
.
0
]
, [
1
0
1
.
0
, 1
.
0
]
] }
{ "
t
y
p
e
"
: "
P
o
l
y
g
o
n
"
,
"
c
o
o
r
d
i
n
a
t
e
s
"
: [
[
[
1
0
0
.
0
, 0
.
0
]
, [
1
0
1
.
0
, 0
.
0
]
, [
1
0
1
.
0
, 1
.
0
]
, [
1
0
0
.
0
, 1
.
0
]
, [
1
0
0
.
0
, 0
.
0
]
]
,
[
[
1
0
0
.
2
, 0
.
2
]
, [
1
0
0
.
8
, 0
.
2
]
, [
1
0
0
.
8
, 0
.
8
]
, [
1
0
0
.
2
, 0
.
8
]
, [
1
0
0
.
2
, 0
.
2
]
]
]
}
{ "
t
y
p
e
"
: "
M
u
l
t
i
P
o
l
y
g
o
n
"
,
"
c
o
o
r
d
i
n
a
t
e
s
"
: [
[
[
[
1
0
2
, 2
]
, [
1
0
3
, 2
]
, [
1
0
3
, 3
]
, [
1
0
2
, 3
]
, [
1
0
2
, 2
]
]
]
,
[
[
[
1
0
0
, 0
]
, [
1
0
1
, 0
]
, [
1
0
1
, 1
]
, [
1
0
0
, 1
]
, [
1
0
0
, 0
.
0
]
]
]
]
}
{ "
t
y
p
e
"
: "
G
e
o
m
e
t
r
y
C
o
l
l
e
c
t
i
o
n
"
,
"
g
e
o
m
e
t
r
i
e
s
"
: [ { … }
, { … } ]
}

View Slide

ARRAYS
ARRAYS EVERYWHERE

View Slide

View Slide

Indexing some places of interest
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
n
a
m
e
' =
> '
H
y
a
t
t R
e
g
e
n
c
y S
a
n
t
a C
l
a
r
a
'
,
'
t
y
p
e
' =
> '
h
o
t
e
l
'
,
'
l
o
c
' =
> [
'
t
y
p
e
' =
> '
P
o
i
n
t
'
,
'
c
o
o
r
d
i
n
a
t
e
s
' =
> [
-
1
2
1
.
9
7
6
5
5
7
, 3
7
.
4
0
4
9
7
7
]
,
]
,
]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
n
a
m
e
' =
> '
I
n
-
N
-
O
u
t B
u
r
g
e
r
'
,
'
t
y
p
e
' =
> '
r
e
s
t
a
u
r
a
n
t
'
,
'
l
o
c
' =
> [
'
t
y
p
e
' =
> '
P
o
i
n
t
'
,
'
c
o
o
r
d
i
n
a
t
e
s
' =
> [
-
1
2
1
.
9
8
2
1
0
2
, 3
7
.
3
8
7
9
9
3
]
,
]
]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
e
n
s
u
r
e
I
n
d
e
x
(
[
'
l
o
c
' =
> '
2
d
s
p
h
e
r
e
'
]
)
;

View Slide

Inclusion queries
/
/ D
e
f
i
n
e a G
e
o
J
S
O
N p
o
l
g
y
o
n
$
p
o
l
y
g
o
n = [
'
t
y
p
e
' =
> '
P
o
l
y
g
o
n
'
,
'
c
o
o
r
d
i
n
a
t
e
s
' =
> [
[
[
-
1
2
1
.
9
7
6
5
5
7
, 3
7
.
4
0
4
9
7
7
]
, /
/ H
y
a
t
t R
e
g
e
n
c
y
[
-
1
2
1
.
9
8
2
1
0
2
, 3
7
.
3
8
7
9
9
3
]
, /
/ I
n
-
N
-
O
u
t B
u
r
g
e
r
[
-
1
2
1
.
9
9
2
3
1
1
, 3
7
.
4
0
4
3
8
5
]
, /
/ R
a
b
b
i
t
'
s F
o
o
t M
e
a
d
e
r
y
[
-
1
2
1
.
9
7
6
5
5
7
, 3
7
.
4
0
4
9
7
7
]
,
]
,
]
,
]
;
/
/ F
i
n
d d
o
c
u
m
e
n
t
s w
i
t
h
i
n t
h
e p
o
l
y
g
o
n
'
s b
o
u
n
d
s
$
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
(
[
'
l
o
c
' =
> [
'
$
g
e
o
W
i
t
h
i
n
' =
> $
p
o
l
y
g
o
n
]
]
)
;
/
/ F
i
n
d d
o
c
u
m
e
n
t
s w
i
t
h
i
n c
i
r
c
u
l
a
r b
o
u
n
d
s
$
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
(
[
'
l
o
c
' =
> [
'
$
g
e
o
W
i
t
h
i
n
' =
> [
'
$
c
e
n
t
e
r
S
p
h
e
r
e
' =
> [
[
-
1
2
1
.
9
7
6
5
5
7
, 3
7
.
4
0
4
9
7
7
]
, /
/ C
e
n
t
e
r c
o
o
r
d
i
n
a
t
e
5 / 3
9
5
9
, /
/ C
o
n
v
e
r
t m
i
l
e
s t
o r
a
d
i
a
n
s
]
]
]
]
)
;

View Slide

Sorted proximity queries
$
p
o
i
n
t = [
'
t
y
p
e
' =
> '
P
o
i
n
t
'
,
'
c
o
o
r
d
i
n
a
t
e
s
' =
> [
-
1
2
1
.
9
7
6
5
5
7
, 3
7
.
4
0
4
9
7
7
]
]
;
/
/ F
i
n
d l
o
c
a
t
i
o
n
s n
e
a
r
e
s
t a p
o
i
n
t
$
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
(
[
'
l
o
c
' =
> [
'
$
n
e
a
r
' =
> $
p
o
i
n
t
]
]
)
;
/
/ F
i
n
d t
h
e n
e
a
r
e
s
t 5
0 r
e
s
t
a
u
r
a
n
t
s w
i
t
h
i
n 5
k
m
$
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
(
[
'
l
o
c
' =
> [
'
$
n
e
a
r
' =
> $
p
o
i
n
t
, '
$
m
a
x
D
i
s
t
a
n
c
e
' =
> 5
0
0
0
]
,
'
t
y
p
e
' =
> '
r
e
s
t
u
a
r
a
n
t
'
,
]
)
-
>
l
i
m
i
t
(
5
0
)
;

View Slide

View Slide

Data Aggregation

View Slide

We have some commands for this
aggregate
count
distinct
group
mapReduce

View Slide

Count
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
A
1
2
3
'
, '
n
u
m
' =
> 5
0
0 ]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
A
1
2
3
'
, '
n
u
m
' =
> 2
5
0 ]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
B
2
1
2
'
, '
n
u
m
' =
> 2
0
0 ]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
A
1
2
3
'
, '
n
u
m
' =
> 3
0
0 ]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
c
o
u
n
t
(
)
; /
/ R
e
t
u
r
n
s 4
$
c
o
l
l
e
c
t
i
o
n
-
>
c
o
u
n
t
(
[
'
n
u
m
' =
> [
'
$
g
t
e
' =
> 2
5
0
]
]
)
; /
/ R
e
t
u
r
n
s 3

View Slide

Distinct
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
A
1
2
3
'
, '
n
u
m
' =
> 5
0
0 ]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
A
1
2
3
'
, '
n
u
m
' =
> 2
5
0 ]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
B
2
1
2
'
, '
n
u
m
' =
> 2
0
0 ]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
A
1
2
3
'
, '
n
u
m
' =
> 3
0
0 ]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
d
i
s
t
i
n
c
t
(
'
c
o
d
e
'
)
; /
/ R
e
t
u
r
n
s [
"
A
1
2
3
"
, "
B
2
1
2
"
]

View Slide

Group
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
A
1
2
3
'
, '
n
u
m
' =
> 5
0
0 ]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
A
1
2
3
'
, '
n
u
m
' =
> 2
5
0 ]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
B
2
1
2
'
, '
n
u
m
' =
> 2
0
0 ]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
c
o
d
e
' =
> '
A
1
2
3
'
, '
n
u
m
' =
> 3
0
0 ]
)
;
$
r
e
s
u
l
t = $
c
o
l
l
e
c
t
i
o
n
-
>
g
r
o
u
p
(
[
'
c
o
d
e
' =
> 1
]
, /
/ f
i
e
l
d
(
s
) o
n w
h
i
c
h t
o g
r
o
u
p
[
'
s
u
m
' =
> 0
]
, /
/ i
n
i
t
i
a
l a
g
g
r
e
g
a
t
e v
a
l
u
e
n
e
w M
o
n
g
o
C
o
d
e
(
'
f
u
n
c
t
i
o
n
(
c
u
r
, a
g
g
) { a
g
g
.
s
u
m +
= c
u
r
.
n
u
m }
'
)
)
;
f
o
r
e
a
c
h (
$
r
e
s
u
l
t
[
'
r
e
t
v
a
l
'
] a
s $
g
r
o
u
p
e
d
) {
p
r
i
n
t
f
(
"
%
s
: %
d
\
n
"
, $
g
r
o
u
p
e
d
[
'
c
o
d
e
'
]
, $
g
r
o
u
p
e
d
[
'
s
u
m
'
]
)
;
}
↓
A
1
2
3
: 1
0
5
0
B
2
1
2
: 2
0
0

View Slide

MapReduce
Extremely versatile, powerful
Intended for complex data analysis
Overkill for simple aggregation tasks
e.g. averages, summation, grouping
Incremental data processing

View Slide

Aggregating query profiler output
{
"
o
p
" : "
q
u
e
r
y
"
,
"
n
s
" : "
d
b
.
c
o
l
l
e
c
t
i
o
n
"
,
"
q
u
e
r
y
" : {
"
c
o
d
e
" : "
A
1
2
3
"
,
"
n
u
m
" : {
"
$
g
t
" : 2
2
5
}
}
,
"
n
t
o
r
e
t
u
r
n
" : 0
,
"
n
t
o
s
k
i
p
" : 0
,
"
n
s
c
a
n
n
e
d
" : 1
1
4
2
6
,
"
l
o
c
k
S
t
a
t
s
" : { … }
,
"
n
r
e
t
u
r
n
e
d
" : 0
,
"
r
e
s
p
o
n
s
e
L
e
n
g
t
h
" : 2
0
,
"
m
i
l
l
i
s
" : 1
2
,
"
t
s
" : I
S
O
D
a
t
e
(
"
2
0
1
3
-
0
5
-
2
3
T
2
1
:
2
4
:
3
9
.
3
2
7
Z
"
)
,
}

View Slide

Constructing a query skeleton
{
"
c
o
d
e
" : "
A
1
2
3
"
,
"
n
u
m
" : {
"
$
g
t
" : 2
2
5
}
}
↓
{
"
c
o
d
e
" : <
s
t
r
i
n
g
>
,
"
n
u
m
" : {
"
$
g
t
" : <
n
u
m
b
e
r
>
}
}
Aggregate stats for similar queries
(e.g. execution time, index performance)

View Slide

View Slide

Aggregation framework
Process a stream of documents
Original input is a collection
Outputs one or more result documents
Series of operators
Filter or transform data
Input/output chain
p
s a
x | g
r
e
p m
o
n
g
o
d | h
e
a
d -
n 1

View Slide

Executing an aggregation pipeline
$
c
o
l
l
e
c
t
i
o
n
-
>
a
g
g
r
e
g
a
t
e
C
u
r
s
o
r
(
[
[
'
$
m
a
t
c
h
' =
> [
'
s
t
a
t
u
s
' =
> '
A
'
]
]
,
[
'
$
g
r
o
u
p
' =
> [
'
_
i
d
' =
> '
$
c
u
s
t
_
i
d
'
, '
t
o
t
a
l
' =
> [
'
$
s
u
m
' =
> '
$
a
m
o
u
n
t
'
]
]
]
]
)
;

View Slide

Pipeline operators
$
m
a
t
c
h
$
g
e
o
N
e
a
r
$
p
r
o
j
e
c
t
$
g
r
o
u
p
$
u
n
w
i
n
d
$
s
o
r
t
$
l
i
m
i
t
$
s
k
i
p
$
r
e
d
a
c
t
$
o
u
t

View Slide

Aggregating fractals

View Slide

Solving symbolic equations and calculus

View Slide

Creating a Job Queue

View Slide

Things not to do in your controllers
Send email messages
Upload files to S3
Blocking API calls
Heavy data processing
Mining cryptocurrency

View Slide

Creating a job
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
d
a
t
a
' =
> [ … ]
,
'
p
r
o
c
e
s
s
e
d
' =
> f
a
l
s
e
,
'
c
r
e
a
t
e
d
A
t
' =
> n
e
w M
o
n
g
o
D
a
t
e
,
]
)
;
$
c
o
l
l
e
c
t
i
o
n
-
>
c
r
e
a
t
e
I
n
d
e
x
(
[
'
p
r
o
c
e
s
s
e
d
' =
> 1
, '
c
r
e
a
t
e
d
A
t
' =
> 1
]
)
;

View Slide

Selecting a job
$
j
o
b = $
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
A
n
d
M
o
d
i
f
y
(
[
'
p
r
o
c
e
s
s
e
d
' =
> f
a
l
s
e
]
,
[
'
$
s
e
t
' =
> [
'
p
r
o
c
e
s
s
e
d
' =
> t
r
u
e
, '
r
e
c
e
i
v
e
d
A
t
' =
> n
e
w M
o
n
g
o
D
a
t
e
]
]
,
n
u
l
l
, /
/ f
i
e
l
d p
r
o
j
e
c
t
i
o
n (
i
f a
n
y
)
[
'
s
o
r
t
' =
> [
'
c
r
e
a
t
e
d
A
t
' =
> 1
]
,
'
n
e
w
' =
> t
r
u
e
,
]
)
;
↓
{
"
_
i
d
" : O
b
j
e
c
t
I
d
(
"
5
4
5
1
5
e
1
6
b
a
5
a
4
d
a
1
b
1
5
a
1
7
6
6
"
)
,
"
d
a
t
a
" : { … }
,
"
p
r
o
c
e
s
s
e
d
" : t
r
u
e
,
"
c
r
e
a
t
e
d
A
t
" : I
S
O
D
a
t
e
(
"
2
0
1
4
-
1
0
-
2
9
T
2
1
:
3
7
:
2
6
.
4
0
5
Z
"
)
,
"
r
e
c
e
i
v
e
d
A
t
" : I
S
O
D
a
t
e
(
"
2
0
1
4
-
1
0
-
2
9
T
2
1
:
3
7
:
3
3
.
1
1
8
Z
"
)
}

View Slide

Schedule jobs in the future
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
d
a
t
a
' =
> [ … ]
,
'
p
r
o
c
e
s
s
e
d
' =
> f
a
l
s
e
,
'
c
r
e
a
t
e
d
A
t
' =
> n
e
w M
o
n
g
o
D
a
t
e
,
'
s
c
h
e
d
u
l
e
d
A
t
' =
> n
e
w M
o
n
g
o
D
a
t
e
(
s
t
r
t
o
t
i
m
e
(
'
1 h
o
u
r
'
)
)
,
]
)
;
↓
$
n
o
w = n
e
w M
o
n
g
o
D
a
t
e
;
$
j
o
b = $
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
A
n
d
M
o
d
i
f
y
(
[
'
p
r
o
c
e
s
s
e
d
' =
> f
a
l
s
e
, '
s
c
h
e
d
u
l
e
d
A
t
' =
> [
'
$
l
t
' =
> $
n
o
w
]
]
,
[
'
$
s
e
t
' =
> [
'
p
r
o
c
e
s
s
e
d
' =
> t
r
u
e
, '
r
e
c
e
i
v
e
d
A
t
' =
> $
n
o
w
]
]
,
n
u
l
l
,
[
'
s
o
r
t
' =
> [
'
c
r
e
a
t
e
d
A
t
' =
> 1
]
,
'
n
e
w
' =
> t
r
u
e
,
]
)
;

View Slide

Prioritize job selection
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
d
a
t
a
' =
> [ … ]
,
'
p
r
o
c
e
s
s
e
d
' =
> f
a
l
s
e
,
'
c
r
e
a
t
e
d
A
t
' =
> n
e
w M
o
n
g
o
D
a
t
e
,
'
p
r
i
o
r
i
t
y
' =
> 0
,
]
)
;
/
/ I
n
d
e
x
: { "
p
r
o
c
e
s
s
e
d
"
: 1
, "
p
r
i
o
r
i
t
y
"
: -
1
, "
c
r
e
a
t
e
d
A
t
"
: 1 }
↓
$
n
o
w = n
e
w M
o
n
g
o
D
a
t
e
;
$
j
o
b = $
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
A
n
d
M
o
d
i
f
y
(
[
'
p
r
o
c
e
s
s
e
d
' =
> f
a
l
s
e
]
,
[
'
$
s
e
t
' =
> [
'
p
r
o
c
e
s
s
e
d
' =
> t
r
u
e
, '
r
e
c
e
i
v
e
d
A
t
' =
> $
n
o
w
]
]
,
n
u
l
l
,
[
'
s
o
r
t
' =
> [
'
p
r
i
o
r
i
t
y
' =
> -
1
, '
c
r
e
a
t
e
d
A
t
' =
> 1
]
,
'
n
e
w
' =
> t
r
u
e
,
]
)
;

View Slide

Gracefully handle failed jobs
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
d
a
t
a
' =
> [ … ]
,
'
p
r
o
c
e
s
s
e
d
' =
> f
a
l
s
e
,
'
c
r
e
a
t
e
d
A
t
' =
> n
e
w M
o
n
g
o
D
a
t
e
,
'
a
t
t
e
m
p
t
s
' =
> 0
,
]
)
;
↓
$
n
o
w = n
e
w M
o
n
g
o
D
a
t
e
;
$
j
o
b = $
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
A
n
d
M
o
d
i
f
y
(
[
'
p
r
o
c
e
s
s
e
d
' =
> f
a
l
s
e
]
,
[
'
$
s
e
t
' =
> [
'
p
r
o
c
e
s
s
e
d
' =
> t
r
u
e
, '
r
e
c
e
i
v
e
d
A
t
' =
> $
n
o
w
]
,
'
$
i
n
c
' =
> [
'
a
t
t
e
m
p
t
s
' =
> 1
]
,
]
,
n
u
l
l
,
[
'
s
o
r
t
' =
> [
'
c
r
e
a
t
e
d
A
t
' =
> 1
]
,
'
n
e
w
' =
> t
r
u
e
,
]
)
;

View Slide

Tailable Cursors

View Slide

Capped collections
$
d
a
t
a
b
a
s
e
-
>
c
r
e
a
t
e
C
o
l
l
e
c
t
i
o
n
(
'
t
a
i
l
m
e
'
,
[
'
c
a
p
p
e
d
' =
> t
r
u
e
,
'
s
i
z
e
' =
> 1
6
7
7
7
2
1
6
, /
/ 1
6 M
i
B
'
m
a
x
' =
> 1
0
0
0
,
]
)
;

View Slide

Producer
f
o
r (
$
i = 0
; +
+
$
i
; ) {
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
x
' =
> $
i
]
)
;
p
r
i
n
t
f
(
"
I
n
s
e
r
t
e
d
: %
d
\
n
"
, $
i
)
;
s
l
e
e
p
(
1
)
;
}
↓
I
n
s
e
r
t
e
d
: 1
I
n
s
e
r
t
e
d
: 2
I
n
s
e
r
t
e
d
: 3
I
n
s
e
r
t
e
d
: 4
I
n
s
e
r
t
e
d
: 5
…

View Slide

Consumer
$
c
u
r
s
o
r = $
c
o
l
l
e
c
t
i
o
n
-
>
f
i
n
d
(
)
;
$
c
u
r
s
o
r
-
>
t
a
i
l
a
b
l
e
(
t
r
u
e
)
;
$
c
u
r
s
o
r
-
>
a
w
a
i
t
D
a
t
a
(
t
r
u
e
)
;
w
h
i
l
e (
t
r
u
e
) {
i
f (
$
c
u
r
s
o
r
-
>
d
e
a
d
(
)
) {
b
r
e
a
k
;
}
i
f ( ! $
c
u
r
s
o
r
-
>
h
a
s
N
e
x
t
(
)
) {
c
o
n
t
i
n
u
e
;
}
p
r
i
n
t
f
(
"
C
o
n
s
u
m
e
d
: %
d
\
n
"
, $
c
u
r
s
o
r
-
>
g
e
t
N
e
x
t
(
)
[
'
x
'
]
)
;
}
↓
C
o
n
s
u
m
e
d
: 1
C
o
n
s
u
m
e
d
: 2
…

View Slide

Replica set oplog
$
c
o
l
l
e
c
t
i
o
n
-
>
i
n
s
e
r
t
(
[
'
x
' =
> 1
,
]
)
;
↓
{
"
t
s
" : T
i
m
e
s
t
a
m
p
(
1
4
1
4
6
2
4
9
2
9
, 1
)
,
"
h
" : N
u
m
b
e
r
L
o
n
g
(
"
2
6
3
1
3
8
2
8
9
4
3
8
7
4
3
4
4
8
4
"
)
,
"
v
" : 2
,
"
o
p
" : "
i
"
,
"
n
s
" : "
t
e
s
t
.
f
o
o
"
,
"
o
" : {
"
_
i
d
" : O
b
j
e
c
t
I
d
(
"
5
4
5
1
7
6
a
1
4
a
b
5
c
0
c
9
9
9
d
a
7
0
f
0
"
)
,
"
x
" : 1
}
}

View Slide

Replica set oplog
$
c
o
l
l
e
c
t
i
o
n
-
>
u
p
d
a
t
e
(
[
'
x
' =
> 1
]
,
[
'
$
i
n
c
' =
> [
'
x
' =
> 1
]
]
)
;
↓
{
"
t
s
" : T
i
m
e
s
t
a
m
p
(
1
4
1
4
6
2
4
9
6
2
, 1
)
,
"
h
" : N
u
m
b
e
r
L
o
n
g
(
"
5
0
7
9
4
2
5
1
0
6
8
5
0
5
5
0
7
0
1
"
)
,
"
v
" : 2
,
"
o
p
" : "
u
"
,
"
n
s
" : "
t
e
s
t
.
f
o
o
"
,
"
o
2
" : {
"
_
i
d
" : O
b
j
e
c
t
I
d
(
"
5
4
5
1
7
6
a
1
4
a
b
5
c
0
c
9
9
9
d
a
7
0
f
0
"
)
}
,
"
o
" : {
"
$
s
e
t
" : {
"
x
" : 2
}
}
}

View Slide

Fun with MongoDB’s oplog
Syncing MongoDB to Solr with PHP
MongoDB river plugin for Elasticsearch
Building real-time systems @ Stripe
Scalable oplog tailing @ Meteor

View Slide

THANKS!
Questions?

View Slide

Image Credits
Books designed by from the
Aggregator designed by from the
Register designed by from the
Ouroboros designed by from the
Catherine Please Noun Project
stuart mcmorris Noun Project
Wilson Joseph Noun Project
Silas Reeves Noun Project
http://mariompittore.com/wp-content/uploads/2013/08/Social-Gnomes1.png

View Slide

Raiding the MongoDB Toolbox

Raiding the MongoDB Toolbox

Jeremy Mikola

More Decks by Jeremy Mikola

Other Decks in Programming

Featured

Transcript