Lessons learned running data pipeline on AWS

Here are the slides from the talk at ConFoo Montreal 2019.

Transcript

1. DATA PIPELINE ON AWS LESSONS LEARNED RUNNING
2. Majid Fatemian R E D V E N T U R E S mfatemian@redventures.com @majidfn aws CERTIFIED - SAA
3. TECH & DATA-DRIVEN CUSTOMER ACQUISITION SALES & MARKETING R E D V E N T U R E S
4. MACHINE
 LEARNING PRODUCT
 RECOMMENDATION DATA
 SCIENCE REALTIME
 ANALYSIS USER
 EXPERIENCE DATA
 WAREHOUSE DATA
5. ▸ Collect ▸ Process ▸ Protect ▸ Access DATA PIPELINE RESPONSIBILITIES
6. DATA PIPELINE AVAILABLE D U R A B L E DESIRED PROPERTIES
7. AVAILABILITY Every request gets a response Availability Down time / Year 99% 3.65 days 99.9% 8.77 hours 99.99% 52.60 minutes 99.999% 5.26 minutes
8. DURABILITY Guaranteed delivery when acknowledged APP Msg Ack
9. DATA IMMUTABLE DESIRED PROPERTIES S C H E M A
10. IMMUTABLE DATA ▸ No UPDATE (Immutable data) ▸ Changes are stored as sequence of events ▸ Current State = Applying the history of changes
11. IMMUTABLE DATA UserId Date City 10001 2015-07-01 Montréal 10001 2012-09-12 Toronto 10001 2009-07-26 New York 10001 2007-01-01 San Francisco
12. EVENT-SOURCING ▸ Event-Sourcing ▸ User / Application events 㲗 Data events ▸ Reliable audit logs ▸ Data consistency in multiple destinations ▸ Complex queries ▸ One way of designing data pipeline
13. SCHEMA ▸ Enforced schema ▸ Data is structured ▸ Avoids data corruption
14. ▸ Apache Thrift ▸ Protocol Buffers ▸ Avro ▸ … SERIALIZATION FRAMEWORKS
15. PROTOCOL BUFFERS + GRPC ▸ Serialization framework ▸ Schema ▸ gRPC is a binary RPC protocol grpc.io
16. PROTOCOL BUFFERS Protobuf v2 | developers.google.com // Defining a Message message Person { required string name = 1; required int32 id = 2; optional string email = 3; } // Defining a service service SearchService { rpc Search (SearchRequest) returns (SearchResponse); }
17. PROTOCOL BUFFERS COMPILER Stub Stub Stub Scala NodeJS Go
18. PROTOCOL BUFFERS COMPILER Scala Go Stub Stub NodeJS Stub
19. PROTOCOL BUFFERS + GRPC Scala Go Stub Stub Stub gRPC
 server
20. GRPC Handling gRPC traffic
21. GRPC APP Handling gRPC traffic
22. GRPC APP Persisting gRPC traffic
23. GRPC APP Persisting gRPC traffic
24. GRPC ENDPOINT - TECHNOLOGIES ▸ Go application ▸ AWS Elastic Container Service (ECS) ▸ Network Load Balancer (NLB) ▸ AWS PrivateLink ▸ Kinesis stream ▸ Kinesis Firehose to S3 (disaster recovery)
25. GRPC APP Persisting gRPC traffic
26. GRPC APP WEB Handling web/http traffic
27. GRPC APP HTTP WEB Handling web/http traffic
28. GRPC APP WEB HTTP Persisting web/http traffic
29. HTTP ENDPOINT - TECHNOLOGIES ▸ Go application ▸ AWS Elastic Container Service (ECS) ▸ Application Load Balancer (ALB) ▸ Kinesis stream
30. GRPC APP WEB HTTP Persisting web/http traffic
31. GRPC APP WEB HTTP TRANSCODER Converting JSON to Protobuf
32. HTTP GRPC TRANSCODER WEB APP Pushing converted data to gRPC
33. TRANSCODER - TECHNOLOGIES ▸ Scala application ▸ AWS Elastic Container Service (ECS) ▸ Kinesis Enhanced Fanout
34. HTTP GRPC TRANSCODER WEB APP Pushing converted data to gRPC
35. HTTP GRPC TRANSCODER WEB APP Handling Sensitive data - Redaction
36. HTTP GRPC TRANSCODER WEB APP Consuming data, ETL - Apache Spark
37. APACHE SPARK ▸ Apache Spark through DataBricks ▸ Extensive tools ▸ Runs on your AWS infrastructure (VPC) ▸ Ability to extend with custom tools ▸ Easy to use
38. HTTP GRPC TRANSCODER WEB APP
39. HTTP GRPC TRANSCODER WEB APP
40. DATA LAKE HTTP GRPC TRANSCODER WEB APP
41. DATA LAKE HTTP GRPC TRANSCODER WEB APP
42. TECHNOLOGIES ▸ Redis ▸ RedShift Spectrum ▸ Parquet ▸ S3 - multi-tenant storage ▸ AWS Relational Database Service (RDS)
43. DATA LAKE HTTP GRPC TRANSCODER WEB APP
44. LEARNED 6 LESSONS
45. OBSERVABILITY 1/6
46. @honest_update
47. ▸ Golden metrics ▸ Latency ▸ Traffic ▸ Errors ▸ Saturation MONITORING Google | Site Reliability Engineering
48. ▸ Analyzing long-term trends ▸ Predict future capacity ▸ Detect anomalies in trends ▸ Symptoms vs. Causes ▸ What’s broken & why MONITORING Google | Site Reliability Engineering
49. ▸ Black box monitoring ▸ Periodic test message ▸ Measure end-to-end ▸ White box monitoring ▸ Monitor individual components MONITORING Google | Site Reliability Engineering
50. DISTRIBUTED TRACING
51. ▸ Visibility to system over time ▸ Locally aggregated logs ▸ Centralized logs DISTRIBUTED TRACING
52. ▸ Define thresholds ▸ Define appropriate resolution ▸ Signal vs. Noise ▸ Define escalation paths ALERTING
53. DURABILITY 2/6
54. APP 1 2 3 4
55. APP 1 2 3 APP 1 4 2 4 3
56. APP 1 2 3 APP 1 4 2 4 3 X
57. APP 1 2 3 APP 1 4 2 4 3 Fire & Forget 
Guaranteed Delivery
58. ▸ Fire & Forget ▸ Lower latency ▸ Lower durability = Higher data loss ▸ Guaranteed delivery ▸ Higher durability = Lower data loss ▸ Higher latency DURABILITY
59. ▸ Delivery semantics ▸ At least once ▸ Exactly once ▸ At most once DURABILITY
60. ▸ Human error / Machine error ▸ Simple collectors ▸ Immutable data ▸ Disaster recovery / Replay of historical data ▸ Dead Letter Queue (DLQ) for invalid data FAULT TOLERANCE
61. APP DLQ - INVALID DATA
62. ARCHITECTURE & Technologies 3/6
63. ARCHITECTURE & TECHNOLOGIES AS SIMPLE AS POSSIBLE, BUT NOT SIMPLER Albert Einstein
64. ▸ Understand the technologies ▸ Understand the trade-offs ▸ Build vs. Buy TECHNOLOGIES
65. ▸ Kinesis vs. Kafka ▸ gRPC vs. HTTP ▸ Apache Spark: AWS EMR vs. DataBricks ▸ … TECHNOLOGIES
66. ▸ External facing techs ▸ Consistent interface ▸ Familiar techs ▸ Internal techs ▸ Ability to experiment ARCHITECTURE & TECHNOLOGIES
67. SDK Software Development Kit 4/6
68. 
HTTP gRPC WEB APP SDK
69. ▸ Provide in early stages ▸ Easier adoption of service ▸ Take control ▸ Build resiliency ▸ Retry mechanism ▸ Circuit breaker SDK
70. SECURITY Everyone’s Responsibility 5/6
71. ▸ Do not collect the data you do not need SECURITY
72. ▸ Encryption: Rest vs. In-transit ▸ Anonymizing vs. Pseudonymizing ▸ Life cycle policies ▸ Enrichment SECURITY
73. ▸ Multiple copies of the data? ▸ Right to be forgotten (GDPR) SECURITY
74. AUTOMATION 6/6
75. ▸ Builds and Deployments (CI / CD) ▸ Automate infrastructure provisioning AUTOMATE
76. ▸ Automatic provisioning ▸ Parameterized infrastructure code ▸ Reproducible environments ▸ Short-lived environments (tests) INFRASTRUCTURE AS CODE
77. ▸ Unit & integration tests ▸ LocalStack (AWS services in container) ▸ End-to-End tests ▸ Actual infrastructure (smaller capacity) TESTING
78. TRADE-OFFS
79. ▸ Maintainability ▸ Operations ▸ Security ▸ Simplicity ▸ Adoption ▸ Performance ▸ … TRADE-OFFS
80. INFORMED DECISIONS
81. QUESTIONS?
82. ▸ Google | Site Reliability Engineering ▸ Designing Data Intensive Applications RESOURCES