Building Serverless Data Infrastructure in the AWS Cloud

Presentation given at the Utah Big Mountain Data & Developer Conference in November 2017. Describes the new data economy, a reference architecture for Big Data infrastructure, and its application to Amazon Web Services serverless services.

Transcript

1. Building Serverless Data Infrastructure in the AWS Cloud Ryan Plant @ryan_plant November 10, 2017
2. Thanks to our Sponsors! Partners Premier Marquee: Prize:
3. Get the app! Give feedback!
4. WHAT WE’LL COVER The New Data Economy Reference Architecture Using the AWS Cloud
5. The world’s most valuable resource is no longer oil, but data… May 6th, 2017
6. Data => Revenue (but extraction, refinement, packaging, and distribution needed)
7. DW Traditional Data Warehousing Volume, variety, and velocity… Advanced analytics… Artificial intelligence… ”What got us here won’t (entirely) get us there…” Mostly proprietary… Costly and complex to scale…
8. Next Generation Data Infrastructure (i.e. the “data lake”)
9. James “Data Lake” Dixon
10. If you think of a datamart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state…
11. From Data Warehouses to Lakes A data pond, lake, ocean is not a product it’s an architecture… (and architecture is a principled and pattern-oriented approach to building systems) Any and all data… Any source and format… Any time…
12. WHAT WE’LL COVER The New Data Economy Reference Architecture Using the AWS Cloud
13. APPS & SOURCES STORAGE AND PROCESSING LAYER SERVING LAYER Storage Catalog Processing Analytics & Artificial Intelligence Ingestion Models & Marts DATA OPS API Search Security Config Telemetry Cost Mgmt
14. DATA OPS Security Config Telemetry Cost Mgmt SERVING LAYER Models & Marts API Search APPS & SOURCES STORAGE AND PROCESSING LAYER Storage Ingestion Catalog Processing Analytics & Artificial Intelligence
15. Data Ingestion Pipelines SERVICE SERVICE SERVICE MONOLITH MONOLITH MONOLITH Change Data Capture (CDC) STREAMS MESSAGING FILE EXTRACTS STORAGE source data aggregated, stored indefinitely many supported formats append append PUT Security segregation & encryption
16. Storage and Catalog STORAGE RAW REFINED Catalog • Register source and schema • Data attribute inventory • Relationships and dependencies • Etc… data Ingestion
17. Catalog Raw to Refined Processing Pipelines STORAGE RAW REFINED Processing Pipelines data Ingestion C1 C2 C3 C..n • Preserve RAW data; enrich only • Apply transforms to create new, REFINED datasets (e.g. customer partitioned views) • Catalog new datasets • Enable new use cases: • Reporting/Analytical views • Machine/Deep Learning X Y Z ALL DATA
18. Processing Pipelines Catalog Analytics and AI STORAGE RAW REFINED data Ingestion Analytics and Artificial Intelligence C1 C2 C3 C..n ALL DATA X Y Z … … …
19. DATA OPS Security Config Telemetry Cost Mgmt APPS & SOURCES STORAGE AND PROCESSING LAYER Storage Ingestion Catalog Processing Analytics & Artificial Intelligence SERVING LAYER Models & Marts API Search
20. Processing Pipelines Catalog Curation and Serving STORAGE RAW REFINED data Ingestion Analytics and Artificial Intelligence C1 C2 C3 C..n ALL DATA X Y Z Models and Marts … … … Search … … …
21. Processing Pipelines Catalog STORAGE RAW REFINED data Ingestion Analytics and Artificial Intelligence C1 C2 C3 C..n ALL DATA X Y Z Models and Marts … … … Search … … … API
22. APPS & SOURCES STORAGE AND PROCESSING LAYER SERVING LAYER Storage Catalog Processing Analytics & Artificial Intelligence Ingestion Models & Marts DATA OPS API Search Security Config Telemetry Cost Mgmt
23. WHAT WE’LL COVER The New Data Economy Reference Architecture Using the AWS Cloud
24. Lots of software, hardware, etc.
25. TRADITIONAL INVESTMENT IN NEXT GENERATION DATA
26. CAPITAL AND RISK BARRIERS acquire/write and maintain software procure, install, and maintain hardware get commercial real estate license
28. PUBLIC CLOUD ECONOMIES OF SCALE
29. CLOUD OPTIMIZATION Infrastructure as a Service Someone else’s hardware and real estate Your software, your (virtual) servers Platform as a Service Someone else’s software, servers, hardware and real estate Your custom application software Software as a Service Someone else’s application software, you provide the data (everything else doesn’t matter) Cycle Time Capital Optimization Differentiation Focus High Higher Highest
30. Go Serverless! (as much as possible)
31. everything is an event: messages, log entries, file I/Os, clock alarms, etc. listen for events: trigger a handler with an event stateless event handling: avoid state, persist as event source, handoff as soon as possible automation through orchestration and coordination Principles for event-driven, reactive data infrastructure primed for serverless architectures
32. Storage Ingestion SQS SNS Kinesis DynamoDB/RDS event triggers y = f (x) y = f (x, y) y = f ([x, y]) event handlers AWS Glacier (archival) /{source}-raw/{key}/YYYY-MM-DD /{source}-refined/{key}/YYYY-MM-DD AWS Lambda AWS S3 (ready) KMS (encryption) lifecycle policies IAM + Directory (access control) CloudWatch/Trail to S3 direct AWS Step Functions (coordinated state)
33. Catalog Storage Sources Ingestion AWS Glue (serverless ETL/ELT) source crawlers metadata classifier classifier doSomething(…) {…} trigger Processing Pipelines jobs and job runner To Targets
34. Catalog Storage Sources & Targets Ingestion Processing Pipelines AWS Glue (serverless ETL/ELT) AWS EMR (Managed Hadoop) Streaming Kinesis Batch AWS Batch Targets & Sources Ingestion Serving Layer
35. Catalog Storage Processing Pipelines AWS Glue (serverless ETL/ELT) Serving Layer AWS ElasticSearch (managed ES) AWS RedShift Spectrum (Parallel DW) Sources Ingestion AWS Athena (Ad-hoc Query)
36. Catalog Storage Processing Pipelines Serving Layer Sources Ingestion AWS API Gateway (serverless APIs) AWS QuickSight (visualization) AWS Cognito (Web/Mobile Identity and SSO)
37. WHAT WE’LL COVER The New Data Economy Reference Architecture Using the AWS Cloud
38. CLOUD OPTIMIZATION Infrastructure as a Service Someone else’s hardware and real estate Your software, your (virtual) servers Platform as a Service Someone else’s software, servers, hardware and real estate Your custom application software Software as a Service Someone else’s application software, you provide the data (everything else doesn’t matter) Cycle Time Capital Optimization Differentiation Focus High Higher Highest
39. CLOUD OPTIMIZATION Infrastructure as a Service Someone else’s hardware and real estate Your software, your (virtual) servers Platform as a Service Someone else’s software, servers, hardware and real estate Your custom application software Software as a Service Someone else’s application software, you provide the data (everything else doesn’t matter) You are likely here… Aim here… TBD Opportunity! Public Cloud R&D Investment
40. SERVERLESS: USE CAUTION The floor is wet (and is constantly getting mopped!) The edges are sharp: • Development, Test, Debug tools and experience • Configuration and Deployment challenges • Variable, non-deterministic performance Extremely new (but inevitable) paradigm…