re:Invent 2021 re:Cap – Part II

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Danilo Poccia
AWS Chief Evangelist (EMEA)
@danilop
PART II

View Slide

Agenda
Analytics
Databases
AI/ML
IoT

View Slide

Analytics

View Slide

GA
Data Exchange for APIs
RESTful or GraphQL API calls directly to AWS Data
Exchange
Use the AWS SDK in the programming language of
your choice.
Manage API access centrally with AWS Identity and
Access Management (IAM)

View Slide

GA
Granular access control
Define row- and cell-level access policies on
both governed and traditional Amazon S3
tables. Manage and consistently enforce
policies in a single place.
Governed tables
A new type of Amazon S3 table that allows
you to ingest and manage data at scale, easily
and reliably.
Storage optimization
Automatically optimize how data is stored to
improve query performance as data volume
increases.
A new storage API to simplify
how data is ingested, stored,
and managed, together with
row-level security to protect
your data
AWS Lake Formation Governed Tables

View Slide

GA
New storage API that solves for those key challenges
7
Fine-grained
access controls
Simple to ingest
and manage data
Integration APIs
Define column-, row-, and cell-
level permissions in a single
place and enforce them
consistently across AWS native
and third-party tools
Ingest streaming and batch data
with ACID transactions
Automatic small file compaction
and data management
Unified data access APIs make it
easy to securely integrate your
applications and tools with data
in the data lake

View Slide

Preview
Amazon Athena ACID transactions
Write, delete, update, and time travel operations
Enables concurrent users to make row-level
modifications to Amazon S3 data
Use Athena's console, API, and ODBC and JDBC
drivers
Use ACID transactions either on Lake Formation
Governed tables, or using the Apache Iceberg table
format

View Slide

GA
Amazon Athena Engine Upgrade
Queries up to 3x faster
Up to 70% cost savings

View Slide

Preview
Faster time to market
Don’t worry about data warehouse management
Pay for what you use
Compute provisioning
Automated patching
Routine maintenance
Backup and recovery
Automatic scaling
Automatic failover
Security and industry compliance
Advanced monitoring
YOU
focus on insights
takes care of the rest
Automatically provisions and scales data warehouse capacity

View Slide

Preview
Amazon Redshift Serverless – Easy Start Experience
Activate Amazon
Redshift Serverless for
your AWS account
Connect from your
favorite BI tool or Amazon
Redshift Query Editor
BI tool
Developer and
business analyst Load data seamlessly and
Amazon Redshift executes
queries by automatically
provisioning capacity
Amazon Redshift
Pay for compute and
storage used during analysis
Pay
Easy to get started
SQL features Performance at scale Security
Scalability Analyze all your data
No compromises

View Slide

Preview
Redshift Materialized views auto refresh & query rewrite
S P E E D U P Q U E R Y P E R F O R M A N C E B Y O R D E R S O F M A G N I T U D E W I T H P R E C O M P U T E D R E S U L T S
item store cust price
i1 s1 c1 12.00
i2 s2 c1 3.00
i3 s2 c2 7.00
store owner loc
s1 Joe SF
s2 Ann NY
s3 Lisa SF
loc total_sales
SF 12.00
NY 10.00
sales store_info
loc_sales
Simplify and accelerate iterative and predictable
workloads, such as ETL, BI/dashboarding queries
MVs can be based on one or more Amazon Redshift
tables or external tables (Spectrum, Federated)
Efficient incremental maintenance
Scheduled, automatic, or manually timed refresh
Amazon Redshift auto query rewrite optimizes queries
by replacing native tables with materialized views
“The Amazon Redshift materialized view auto query rewrite feature reduced dashboard load times from 8 minutes to just
500 ms. The best part is that this is completely transparent for Tableau and the business user.”
—Arman Nasrollahi, Home24

View Slide

Preview
Redshift Query Editor v2 – Notebook Support
Data users engaging in advanced analytics work on
multiple queries at a time to perform various tasks
for their data analysis.
organize related queries by saving them together
in a folder, or combining them into a single saved
query with multiple statements.
The Notebooks support provides an alternative
way to embed all queries required for a complete
data analysis in a single document using SQL cells.
You can share your Notebooks with team
members, similar to how you share your saved
queries in Query Editor V2. Documenting your
work precisely enables further collaboration with
other users.

View Slide

Preview
Amazon
EMR
Serverless
Easily run petabyte-scale data analytics in the cloud without
managing, tuning, optimizing, securing, and operating clusters
Quickly run large-scale applications that use open-source
frameworks of your choice, including Spark, Hive, and Presto
Automatically scale resources up and down as needed based on
the changing requirements of your application
Pay only for the resources you use for data analytics at scale

View Slide

Preview
Easily run Apache Kafka clusters without rightsizing
cluster capacity
Instantly scale I/O without worrying about scaling
capacity up and down or reassigning partitions
Pay for the data volume you stream and retain
Amazon MSK Serverless

View Slide

GA
Simple to use
Simplify streaming data processing by eliminating capacity management
Flexible scaling
Automatically scale capacity in response to changing data volumes
Automated high availability
Provide built-in availability and fault tolerance by default
Lower your costs
Pay per gigabyte of data written, read, and stored
Amazon Kinesis Data Streams on demand

View Slide

AWS Glue Autoscaling
Cost =
f(compute)
t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11
Without Autoscaling With Autoscaling
Job execution timeline
List operation
Wide transform
Uneven distribution of
data partitions
AWS Glue job Tear-down
Setup
Potential
savings
Preview

View Slide

Streaming Autoscaling (preview)
Cost =
f(compute)
t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11
Without Autoscaling With Autoscaling
Job execution timeline
Low stream
activity Peak streaming
record arrival rate
Scale down as stream
activity decreases
Glue job Tear-down
Setup
Potential
Savings
Initial
scale up
Scale down to match
stream activity

View Slide

AWS Glue Studio Notebook (Preview)
Interactive AWS Glue
jobs development
Submit AWS Glue jobs from the
AWS Glue Studio notebook
Use notebook magic to define
transforms in SQL and control cost
Built-in monitoring support
Preview

View Slide

AWS Glue Interactive Sessions
Development
tool of your
choice
Rapid
development
Built-in cost
control
Preview
A dedicated cluster created in seconds for developers to
interactively build data pipelines with choice of IDE…

View Slide

… and advanced transforms like PII detection for
advanced cleansing
Scan Data Detect Entities Remeditate
1 2 3
Full Scan
Sample Scan
Built-in Entities
(e.g. SSN, passport)
Custom Entities
Store results
Redact/mask
results

View Slide

How does PII Detection work Preview
Entities to detect
Remediation
Full Scan Sample Scan

View Slide

Databases

View Slide

GA
Amazon RDS Custom
Amazon RDS Custom is a
managed database service for
legacy, custom, and packaged
applications that require
access to the underlying
operating system and
database environment
Amazon RDS Custom is
available for RDS for Oracle
and for SQL Server

View Slide

GA
ML-powered database advisor for Amazon RDS
First released for Amazon Aurora
Delivers insights and recommendations to help resolve issues in minutes
Uses ML models informed by years of operating thousands of databases for
Amazon.com
Automatically detects and diagnoses hard-to-find performance bottlenecks and
operational issues
Continuously monitors database workloads at scale
No ML experience required
Amazon DevOps Guru for RDS

View Slide

GA
Amazon DynamoDB Standard-
Infrequent Access (Standard-IA)
table class
The Standard-IA table class
offers 60% lower storage
costs than DynamoDB
Standard tables.
Standard-IA tables offer the
same performance, durability,
data availability, and massive
scalability as existing
DynamoDB Standard tables.
Switch between table classes
with a single click in the
DynamoDB console, or using
the AWS CLI or AWS SDK.
Also, use the same DynamoDB
APIs and service endpoints.
Lower storage costs No performance trade-offs No developer overhead

View Slide

GA
What’s new in Amazon Timestream
R E A L - T I M E A N A L Y T I C S A R E O R D E R S O F M A G N I T U D E F A S T E R A N D C H E A P E R
Multi-measure records
Scheduled queries
Magnetic store writes
• A new data modeling capability for cost-effective data storage, performant data access, and ease of
use
• Store multiple measurements and events in a single table row, instead of storing one measure per
table row
• Improve query performance and reduce costs by an order of magnitude
• Fully managed, serverless, and scalable solution for calculating and storing aggregates, rollups, and
other real-time analytics typically used to power dashboards, business reports, and other applications
• Simply define the computation/query to precompute and its schedule. Amazon Timestream will
automatically and periodically run the queries and reliably write the results into a separate table.
• Flexible and cost-effective solution for late-arriving data
• Ability to ingest data directly into the magnetic store
• As you ingest data, Amazon Timestream will automatically determine whether the data gets written to
the memory store or the magnetic store, based on the timestamp of the data and the configured data
retention window for the memory and magnetic stores

View Slide

Preview
AWS DMS Fleet Advisor
A C C E L E R A T E D A T A B A S E A N D A N A L Y T I C S M I G R A T I O N S W I T H A U T O M A T E D
I N V E N T O R Y A N D M I G R A T I O N R E C O M M E N D A T I O N S
Generates a list of
possible migration
targets
Automatically
gathers
database fleet
information
Results in hours
instead of weeks
and months

View Slide

GA
Database Migration Service – Time Travel
Store and encrypt AWS DMS logs using Amazon S3,
and view, download and obfuscate the logs within a
certain time frame
Log and debug replication tasks
Time Travel can be used in all AWS Regions with
DMS-supported PostgreSQL source endpoints and
DMS-supported PostgreSQL and MySQL target
endpoints

View Slide

AI/ML

View Slide

The AWS ML Stack

View Slide

Preview
Amazon EC2 Trn1 Instances for ML Training
• Training larger models requires an increase in
compute power, which also increases costs
• Increasing costs become a barrier to
innovation and growth
INCREASING ML COMPLEXITY
• Businesses need higher precision in their model
predictions
• Larger and more complex models result in
more frequent retraining
INCREASING TRAINING COSTS

View Slide

Preview
Amazon EC2 Trn1 Instances for ML Training
16 AWS Trainium chips,
purpose-built for
deep learning training
Most cost efficient for large-
scale distributed training
AWS Neuron SDK integrated
with TensorFlow
and PyTorch
Deployed in Amazon EC2 UltraClusters
with tens of thousands of Trainium
chips and petabit-scale networking
800 Gbps networking and ultra-
highspeed interconnect between
accelerators
Fastest ML training in the cloud

View Slide

The AWS ML Stack

View Slide

GA
Amazon SageMaker Notebooks
P E R F O R M D A T A E N G I N E E R I N G , A N A L Y T I C S , A N D M L W O R K F L O W S I N O N E N O T E B O O K
Discover, manage, create, terminate, and
connect to Amazon EMR clusters
Interactively access, transform, and analyze a
wide range of data
Build, train, and deploy models using
your preferred framework
Amazon SageMaker

View Slide

GA
Amazon SageMaker Training Compiler
A C C E L E R A T E T H E T R A I N I N G O F D E E P L E A R N I N G M O D E L S
Training compiler
Accelerate training times by up to 50% through more
efficient use of GPUs
Reduce training costs
Save time and money associated with training
Integrated with existing frameworks
Use familiar framework APIs with minimal code
changes
Amazon SageMaker

View Slide

GA
Amazon SageMaker Inference Recommender
A C C E L E R A T E T H E D E P L O Y M E N T O F M O D E L E N D P O I N T S
Automatic deployment recommendations
Optimal instance type/count and container
parameters
Reduce inference costs
Receive endpoint configuration recommends for best
price performance
Fully managed load testing
Run and review fully managed load tests on a set of
instance types to evaluate performance and trade-offs
Amazon SageMaker

View Slide

Preview
Amazon SageMaker Serverless Inference
D E P L O Y M L M O D E L S F O R I N F E R E N C E W I T H O U T C O N F I G U R I N G O R M A N A G I N G U N D E R L Y I N G I N F R A S T R U C T U R E
Optimize for intermittent or unpredictable workloads
Automatic and fast scaling when usage is required
Pay per use
Pay only for the resources consumed, not idle time
Fully managed
Underlying infrastructure, security, patching,
monitoring and logging is all managed
Amazon SageMaker

View Slide

GA
Amazon SageMaker MLOps
T R A C K , A S S E S S , A N D M O N I T O R Y O U R M L M O D E L S
Improved lineage management
Retrieve all ML lineage across accounts and in a single
query, from data to endpoint
Model quality and bias detection in workflows
SageMaker Model Monitor and SageMaker Clarify
integrated into SageMaker Pipeline workflows
Monitor endpoints from SageMaker Model Registry
Track endpoints, custom metadata, and model metrics
Amazon SageMaker

View Slide

GA
Amazon SageMaker Ground Truth Plus
D E L I V E R H I G H - Q U A L I T Y T R A I N I N G D A T A S E T S F A S T A N D R E D U C E D A T A L A B E L L I N G C O S T S
Increase data quality through
ML-powered data labeling
Access expert data labelers
Reduce data labeling costs
with assistive labeling features
Improve operational efficiency by
reviewing project metrics
Make data labeling accessible to data
operations and program managers

View Slide

GA
Amazon SageMaker Canvas
G E N E R A T E M L P R E D I C T I O N S – N O C O D E R E Q U I R E D
Quickly access and prepare data sources for
ML
AutoML built in to generate accurate
predictions
Share ML models with data science teams
Amazon SageMaker

View Slide

Preview
Amazon SageMaker Studio Lab
L E A R N A N D E X P E R I M E N T W I T H M A C H I N E L E A R N I N G
Amazon SageMaker
Studio Lab
Free, no-configuration
development environment
All you need is an email address to get started
As many CPU and GPU sessions as needed
Automatically saves session data
Models ready for production

View Slide

The AWS ML Stack

View Slide

GA
Amazon Textract
S P E C I A L I Z E D S U P P O R T F O R I D E N T I T Y D O C U M E N T S
Amazon Textract
OUTPUT
First Name: JORGE
Last Name: SOUZA
Middle Name:
Address Line1: 100 MAIN STREET
Address Line 2:
City: ANYTOWN
State: MA
Document Number: 820BAC729CBAC
Expiration Date: 01/20/2020
Date of Birth: 03/18/1978
ID Type: Driver License
Date of Issue: 03/18/1978
Issued By: MASSACHUSETTS
Class: D
Restrictions: NONE
Endorsements: NONE
DOCUMENT

View Slide

GA
Amazon Personalize
I N T E L L I G E N T U S E R S E G M E N T A T I O N
Amazon Personalize
• Identify users interested in a genre,
category, or any other item attribute
• Identify users interested in a given item
such as a movie, product, and so on
• More effective campaigns through
marketing channels
• Acquire users for new product categories,
genres, channels, and more
• Improve return on investment for your
marketing spend
Action movie
fans

View Slide

GA
Amazon Personalize
U S E - C A S E O P T I M I Z E D R E C O M M E N D E R S F O R R E T A I L , A N D M E D I A A N D E N T E R T A I N M E N T
Retail
• Recommended for you
• Customers who viewed this
also viewed
• Frequently bought
together
• Most viewed
• Best sellers
Media and
entertainment
• Top picks for you
• Because you watched X
• More like Y
• Most popular
Amazon Personalize

View Slide

Preview
Amazon Lex
A U T O M A T E D C H A T B O T D E S I G N E R
Amazon Lex
Analyze thousands
of lines of transcripts
in minutes
Discover the most common
intents and the information
needed to fulfill them
Accelerate chatbot
deployment
Allows you to design
chatbots in hours rather
than weeks
Improve customer
experience
Helps ensure that intents
are well defined and
separated

View Slide

AI & ML Scholarship Program
Learn ML faster with AWS DeepRacer Student
An all-new free service for students over the age of 16
Fast-track your career with scholarships
2,000 students per year to be awarded scholarships as
part of the AWS AI & ML Scholarship program
Receive mentorship from AWS and Intel ML experts
Students in the AWS AI & ML Scholarship program
receive mentoring and career guidance from industry
experts

View Slide

IoT

View Slide

New services:
• AWS IoT ExpressLink
• AWS IoT TwinMaker
• AWS IoT FleetWise
• AWS IoT RoboRunner
• FreeRTOS out-of-the-box
AWS IoT Announcements
Other:
• AWS IoT Greengrass & Systems Manager
• AWS IoT Greengrass Software Catalog
• AWS IoT SiteWise Hot/Cold Storage Tiers
• AWS IoT Fleet Indexing enhancements
• FreeRTOS extended maintenance

View Slide

Preview
AWS IoT ExpressLink
AWS IoT ExpressLink packages all the
complex and security-critical code into
a single hardware component.
A 3-wire serial interface is all you need
to connect any embedded device to
the cloud.
TX
RX
GND WAKE
INT
RST

View Slide

Preview
AWS IoT ExpressLink
IoT
library
Hardware
security
Radio
library
Your application
Host processor
Hardware
security
Radio
library
Your
application
Wireless module
IoT
library
AWS IoT ExpressLink
Host processor
Hardware
security library
Without AWS IoT ExpressLink With AWS IoT ExpressLink

View Slide

Preview
AWS IoT ExpressLink
int main()
{
print("AT+CONNECT\n");
while(1){
print("AT+SEND data {\"A\"=%d}", getSensorA());
delays(1);
}
}
In less than 10 lines of code

View Slide

Preview
AWS IoT ExpressLink – Four evaluation kits
Vendor Evaluation Kit Technology Buy
Espressif ESP32-C3-DevKitM1 Wi-Fi
Infineon CCM Eval Kit Wi-Fi
u-blox Nora-W2 Mini Evaluation kit Wi-Fi
u-blox Sara – R5 AWS Evaluation kit Cellular

View Slide

Preview
AWS IoT TwinMaker
E A S I L Y C R E A T E D I G I T A L T W I N S O F R E A L - W O R L D S Y S T E M S T O O P T I M I Z E O P E R A T I O N S
Easily access data no matter where it lives
Accurately model your built environment
Create immersive 3D views

View Slide

Preview
AWS IoT TwinMaker – How it works
AWS IoT TwinMaker
Digital twin
Operations Planning
Operational and
maintenance history
Real-time
process data
Real-time IoT
sensor data
CAD/BIM
models
Point cloud
scans
Video feeds Geo-spatial
metadata
Statistical and
physics-based
models
Integrated asset model
3D visualization
Insights
AWS Cloud
End-user web application

View Slide

Preview
AWS IoT TwinMaker – Features
Data connectors
Access data from diverse
sources using built-in
connectors, or easily
create your own
connectors to AWS and
third-party data sources.
Model builder
Create entities to virtually
represent physical systems,
specify relationships
between them, and connect
them to different data
sources to form a digital
twin graph
Scene composer
Combine existing 3D
visual models with real-
world data from the
digital twin graph to
compose an interactive
3D view of your physical
environment
App toolkit
Integrate digital twins
into 3D-enabled
applications using
Amazon Managed
Grafana or AWS Partners
for end users to monitor
and improve operations

View Slide

Preview

View Slide

Preview
Build virtual representations of
vehicles and apply a common data
format to structure and label vehicle
attributes, sensors, and signals
Easily access
standardized, fleet-wide
vehicle data
AWS IoT FleetWise
Easily collect, transform, and transfer vehicle data to the cloud at scale
Reduce costs with
intelligent data
filtering
Detect and mitigate problems
more quickly by surfacing
vehicle data in near real time
Select which data to transfer, define
rules and events for when to transfer it,
and automatically reduce redundant
data through dynamic data selection
Take quick, corrective action by notifying
operations or manufacturing groups in
near real time when problems occur

View Slide

Preview
AWS IoT RoboRunner
B U I L D A P P L I C A T I O N S T H A T H E L P R O B O T S W O R K T O G E T H E R S E A M L E S S L Y
Connect robots and
work management
systems to a common
infrastructure
Enable robots
from different
vendors to work
together
Simplify building
applications for
optimizing robot
fleets

View Slide

GA
IoT devices Hub/gateway
AWS IoT Greengrass
AWS Systems Manager integrates AWS IoT Greengrass
Systems Manager
OS upgrades, schedule maintenance tasks, and remotely access their edge device fleet

View Slide

Greengrass Software Catalog
A W S I O T G R E E N G R A S S S O F T W A R E C O M P O N E N T S D E V E L O P E D B Y T H E G R E E N G R A S S C O M M U N I T Y

View Slide

GA
AWS IoT SiteWise cold tier storage support
1 0 1 0 1 1 0
0 1 0 1 0 1 0
0 1 0 1 0 0 1
0 1 0 1 0 1 0
1 0 1 0 1 0 1
101011010
01010 10101
0101001010
01010 10101
1010101010
1010101010
1010101010
1 0 1 0 1 1 0
0 1 0 1 0 1 0
0 1 0 1 0 0 1
0 1 0 1 0 1 0
1 0 1 0 1 0 1
1 0 1 0 1 0 1 0 1 0 1 0 1 0
1 0 1 0 1 0 1 0 1 0 1 0 1 0
1 0 1 0 1 0 1 0 1 0 1 0 1 0
1 0 1 0 1 0 1 0 1 0 1 0 1 0
1 0 1 0 1 0 1 0
01010101
ASSET MODEL
Property: Attribute
tool_id :
Property: Time-series
pressforce-ch1 :
Property: Formula
Avg Pressforce : f
(pressforce-ch1)
Ingest equipment data
into AWS in minutes
Structure data and specify
performance metrics for your
equipment and processes
Store data in a time-series-
optimized data store
Create and share dashboards to
visualize live and historical
equipment data

View Slide

GA
FreeRTOS out-of-the-box connectivity

View Slide

GA
FreeRTOS Extended Maintenance Plan

View Slide

Agenda
Analytics
Databases
AI/ML
IoT

View Slide

Thank you!
Danilo Poccia
AWS Chief Evangelist (EMEA)
@danilop

View Slide

re:Invent 2021 re:Cap – Part II

re:Invent 2021 re:Cap – Part II

Danilo Poccia

More Decks by Danilo Poccia

Other Decks in Programming

Featured

Transcript