apidays Paris 2024 - Building APIs for Big Data Services - Alexander Günsche, Amazon Web Services

Building APIs for Big Data Services
Alexander Günsche, Senior Solutions Architect at Amazon Web Services

apidays Paris 2024 - The Future API Stack for Mass Innovation
December 3 - 5, 2024

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data as a Service: APIs for Big Data Applications Alexander Günsche Senior Solutions Architect lxg@amazon.com APIDAYS PARIS · 3/4/5 DECEMBER 2024
2. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is special about Big Data APIs?
3. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Unprecedented Growth of Data 010101010101010101101010101010101010101010 110101010101010101010101010101010101101010 101010101010100101010010101010101010101101 010101010101010101010110101010101010101010 101010101010101010101010101010101101010101 010101010101010110101010101010101010101010 101010101010101010101010101101010101010101 010101010110101010101010101010101010101010 101010101010101010101010101010101101010101 010101010101010110101010101010101010101010 101010101010100101010101000101010100101010 101001010101010010101010101001010101010100 101010010101010100101010101010010101000010 101010100101010101010100101010101010010100 100101010100101010101001010101010100101010 000101010101001010101010101001010101010100 101001001010101010100110001001100101001001 There is more data and more diversity of data than people think Data growth To live for To scale 15+ 1,000x years >10x every 5 years Data platforms needs IDC, “Data Age 2025”
4. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Recency Real time Seconds Minutes Hours Days Months Time critical decisions Business Intelligence Predictive Actionable Reactive Historical
5. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Lake fundamentals OLTP IoT BI Data sources Storage Data access Consumers AI/ML Ingestion Infrastructure Metadata/Governance
6. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cost drivers Compute Storage Data Transfer With Terrabytes of total volumes and Gigabytes per transaction, be aware of typical cost drivers …
7. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. API Paradigms
8. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Everything HTTP! Verbs Ressource locators Extensible (e.g. WebDav) Proxy support Multipart responses Content Negotiation Streaming (Websockets) Multiplexing Caching Status codes Compression Broad adoption HTTP+UDP (QUIC) Encryption Custom Headers
9. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. The challenge “Get orders from Berlin in March 2023 or later” { "id": "123", "user": { "id": "987", "name": "Jack Johnson", "email": "jj@example.com", "city": "Aachen" }, "items": [ { "p_id": "456", "qty": 1, "price": 37.81 }, { "p_id": "567", "qty": 2, "price": 42.35 } ], "datetime": "2024-05-23T18:23:18Z" }
10. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. The usual suspects GraphQL Prompt REST gRPC/Protobuf OData expensive NLP non-deterministic error-prone Low adoption/maturity expensive object mapping over-/underfetching Unflexible queries Complex translation logic
11. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. SELECT c.*, o.*, oi.* FROM customers c JOIN orders o ON c.id = o.id JOIN items oi ON o.id = oi.id WHERE c.city = 'Berlin' AND o.date >= DATE '2023-03-01' SQL as a data access layer Extremely flexible query format incl. filters, ranges, pagination Single fetch even for complex data structures PROs Very permissive by default Caching hardly possible CONs
12. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Response formats
13. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. The usual suspects JSON XML Graph Protobuf CSV expensive response parsing only scalar values expensive object mapping expensive resolution/ transformation
14. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. JSONL { "id": "123", "user": { "id": "987", "name": "Jack Johnson", "email": " jj@example.com", "city": "Aachen" }, "items": [ {"p_id": "456", "qty": 1, "price": 37.81}, {"p_id": "567", "qty": 2, "price": 42.35} ], "datetime": "2024-05-23T18:23:18Z" } { "id": "124", "user": { "id": "988", "name": "John Jackson", "email": "jj@example.net", "city": "Berlin" }, "items": [ { "p_id": "678", "qty": 3, "price": 43.19 }, { "p_id": "789", "qty": 25, "price": 2.88 } ], "time": "2024-05-24T03:48:10Z" } …
15. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Flat database (binary) row-based column-based
16. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Implementation
17. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Query engine SELECT c.*, o.*, oi.* FROM customers c JOIN orders o ON c.id = o.id JOIN items oi ON o.id = oi.id WHERE c.city = 'Berlin' AND o.date >= DATE '2023-03-01' Amazon Athena Query Engine
18. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data API Security
19. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Column-level Access
20. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Row-level Access
21. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example: Native Integration Data Catalog Data Lake Query Engine Business Intelligence IAM
22. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example: Sync API API Gateway IAM FaaS Data Catalog Data Lake Query Engine
23. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example: Async API API Gateway IAM FaaS Data Catalog Data Lake Query Engine Queue Temp Storage Notification
24. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Takeaways Everything is HTTP, but there’s more than REST. Understand the nature of your data and the needs of your customers. Be aware of cost drivers and performance killers.
25. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you! Alexander Günsche Senior Solutions Architect lxg@amazon.com