Slide 1
Slide 1 text
Save The World with
MongoDB Data Lake
Joe Karlsson | Developer Advocate | @JoeKarlsson1
joekarlsson.dev/MDB-Data-Lake
Slide 2
Slide 2 text
The Goal
Save Money
Save the World
Slide 3
Slide 3 text
The Problem
Slide 4
Slide 4 text
The Problem
Slide 5
Slide 5 text
The Problem
Slide 6
Slide 6 text
The Problem
Slide 7
Slide 7 text
The Problem
Slide 8
Slide 8 text
The Problem
Slide 9
Slide 9 text
The Problem
Slide 10
Slide 10 text
{
name: “Joe Karlsson”,
company: “MongoDB”,
title: [
“Developer Advocate”,
“Software Engineer”
],
}
twitter: “@JoeKarlsson1”,
twitch: “joe_karlsson”,
tiktok: “joekarlsson”,
website: “joekarlsson.com”,
opinions: “my own”,
links: “joekarlsson.dev/MDB-Data-Lake”
joekarlsson.dev/iot-kitty-bf04b
joekarlsson.dev/MDB-Data-Lake
Slide 11
Slide 11 text
Only One Planet
More Data Centers
More Electricity
Consumption
Less Money
Slide 12
Slide 12 text
Cold Data
Hot Data
Slide 13
Slide 13 text
MongoDB Atlas Data Lake
Slide 14
Slide 14 text
@JoeKarlsson1
Agenda
Online Archive
Demo: Archive with $out
Demo: Archive with a Realm function
MongoDB Atlas Data Lake
joekarlsson.dev/MDB-Data-Lake
Slide 15
Slide 15 text
@JoeKarlsson1
Atlas Data Lake - Features and Benefits
Query your S3 and MongoDB Atlas data in-place and in its native format using the MongoDB Query Language (MQL).
Work with rich data easily &
intuitively
Leverage a serverless &
scalable query service
Easy to use with your favorite
tools
Integrated with the MongoDB
Cloud Platform
Eliminate cost & complexity of
data movement
Slide 16
Slide 16 text
MongoDB Atlas Data Lake - Architecture
Slide 17
Slide 17 text
Demo Time
Slide 18
Slide 18 text
First, let’s insert some data
Slide 19
Slide 19 text
@JoeKarlsson1
Insert Data – show script and data in the cluster
Slide 20
Slide 20 text
First, let’s insert some data
Slide 21
Slide 21 text
Configuring Our Data Lake
Slide 22
Slide 22 text
Configuring Our Data Lake
Slide 23
Slide 23 text
pipeline_s3 = [
{'$match': {'date': {'$gte': date_start, '$lt': date_stop}}},
{
'$out': {
's3': {
'bucket': 'cold-data-mongodb',
'region': 'eu-west-1',
'filename': date_start.isoformat('T', 'milliseconds') + 'Z-' + date_stop.isoformat('T', 'milliseconds') + 'Z',
'format': {'name': 'json', 'maxFileSize': '200MiB'}
}
}
}
]
iot_data_lake.aggregate(pipeline_s3)
Archiving with pymongo & $out
joekarlsson.dev/
MDB-Data-Lake
Slide 24
Slide 24 text
Running the Archive Script
Slide 25
Slide 25 text
Confirm data is in S3
Slide 26
Slide 26 text
@JoeKarlsson1
Slide 27
Slide 27 text
Online Archive
Slide 28
Slide 28 text
To Sum Up
We can archive in S3
Saving the World
Saving lot of money
We have access to ALL
the data
Slide 29
Slide 29 text
@JoeKarlsson1
Questions
Slide 30
Slide 30 text
@JoeKarlsson1
What’s next?
Slide 31
Slide 31 text
No content
Slide 32
Slide 32 text
@JoeKarlsson1
MongoDB Community
joekarlsson.dev/MongoDBUniversity
Recommended Course:
M320: Data Modeling
joekarlsson.dev/MongoDBDeveloper
joekarlsson.dev/MongoDBCommunity
joekarlsson.dev/
MDB-Data-Lake
Slide 33
Slide 33 text
Want $100 in FREE MongoDB Atlas
credits?
Use code JoeK100
joekarlsson.dev/free-atlas-credits
joekarlsson.dev/free-atlas-credits
Slide 34
Slide 34 text
@JoeKarlsson1
Additional Resources
joekarlsson.dev/
MDB-Data-Lake
[Docs] MongoDB Atlas Data Lake Documentation:
https://docs.mongodb.com/datalake/
[Docs] Archiving a MongoDB Cluster:
https://docs.atlas.mongodb.com/online-archive/manage-online-
archive/
[DevHub Post] MongoDB Data Lake Setup Tutorial:
https://developer.mongodb.com/how-to/atlas-data-lake-setup
[GitHub] Save The World And Money With MongoDB Data Lake:
https://github.com/JoeKarlsson/mongodb-datalake-save-the-world
[GitHub] MongoDB IoT Sample Data Generator:
https://github.com/joekarlsson/IoT-generator-mongodb
Slide 35
Slide 35 text
{
name: “Joe Karlsson”,
company: “MongoDB”,
title: [
“Developer Advocate”,
“Software Engineer”
],
}
twitter: “@JoeKarlsson1”,
twitch: “joe_karlsson”,
tiktok: “joekarlsson”,
website: “joekarlsson.com”,
links: “joekarlsson.dev/MDB-Data-Lake”
joekarlsson.dev/
MDB-Data-Lake
joekarlsson.dev/MDB-Data-Lake
Slide 36
Slide 36 text
@JoeKarlsson1
Thank you!
Slide 37
Slide 37 text
{
name: “Joe Karlsson”,
company: “MongoDB”,
title: [
“Developer Advocate”,
“Software Engineer”
],
}
twitter: “@JoeKarlsson1”,
twitch: “joe_karlsson”,
tiktok: “joekarlsson”,
website: “joekarlsson.com”,
links: “joekarlsson.dev/MDB-Data-Lake”
joekarlsson.dev/
MDB-Data-Lake
joekarlsson.dev/MDB-Data-Lake
Slide 38
Slide 38 text
@JoeKarlsson1