Introduction to AWS Glue DataBrew

For this session, I introduced AWS Glue DataBrew - a zero-code visual data preparation tool.

What personas it caters to and how it can be used to simplify and drive efficiency in terms of data transformation as well as data quality.

AWS Glue DataBrew: https://aws.amazon.com/glue/features/databrew/

Transcript

1. AWS AI/ML Users Meetup
2. Diving Deep on AWS Data Lakes featuring AWS Glue DataBrew Kyle Escosia kescosia@info-alchemy.net
3. Kyle Escosia Jr. Data Science Specialist @ Info Alchemy AWS Community Builder Program 2+ years experience in AWS Big Data and Analytics 2x AWS Certifications linkedin.com/in/kyle-escosia/ dev.to/klescosia
4. bit.ly/databrew-may2021 Slides are downloadable at
5. What to expect Overview Introduction AWS Glue DataBrew Key Features Demo Use Cases
6. “Over the last 20 years, there’s been a surge in the variety and volume of data that companies can collect”
7. Preparing data involves a lot of complex tasks ELT ETL Orchestration Needs a lot of heavy-lifting to work at scale
8. 80% of time is spent preparing data Data Engineer ETL Developers Data Analysts Data Scientists
10. “Needs the right tool for the right persona”
11. Challenges with traditional data preparation Manual Needs a lot of code-based heavy-lifting for it to work at scale Siloed Often requires moving large amounts of data into silos, at times out of VPCs Time consuming Needs the right tools for the right persona that are integrated
12. Introducing AWS Glue DataBrew
14. Clean and normalize data up to 80% faster
15. Built for Data Analysts and Data Scientists Understand data quality Understand patterns and detect anomalies using profiles Clean and normalize data Over 250 built-in transformations Visually map data lineage Understand steps that the data has been through Automate at scale Save transformations and apply to new data as it comes in Data preparation made easy
16. Demo
17. What we’ll see in the demo ELO < 1000 RATED = TRUE GROUP BY OPENING Create and Start job Chess Dataset AWS Glue DataBrew
18. Build a recipe Profile the data Run a job Operationalize at scale Schedule jobs Use APIs/SDK Reuse recipes What we saw
19. Popular usecases
20. One-time data analysis for business reporting AWS Glue DataBrew Amazon QuickSight Amazon S3 output bucket Data catalog data sources Amazon S3 Amazon Redshift Amazon RDS Amazon Simple Storage Service (Amazon S3) Local file
21. Enrich data with unions and joins
22. Update theschema
23. Amazon Simple Notification Service Amazon EventBridge Email notification AWS Lambda AWS Glue DataBrew Recurring raw data feed Amazon S3 Set up data quality rules with AWS Lambda
24. Generate dataprofiles
25. Data preprocessing for machine learning Amazon S3 AWS Glue DataBrew JupyterLab environment Inference Amazon S3 output bucket Model training
26. Use withJupyter Notebooks
27. Remove outliers
28. Feature engineering
29. Orchestrating data preparation in workflows AWS Step Functionsworkflow AWS Glue DataBrew AWS Glue Data catalog Amazon Redshift Crawler
30. AWS GlueDataBrew Cleanand normalize data up to 80% faster https://aws.amazon.com/glue/features/databrew/ https://github.com/aws/aws-glue-databrew-jupyter-extension
31. Thank you! Kyle Escosia kescosia@info-alchemy.net