Building Cloud Data Infrastructure with Ruby @ SF Ruby 2025

Transcript

1. Evgeny Li, co-founder of Bemi AI Building Cloud Data Infrastructure with Ruby
2. I ❤ Ruby
3. I Also Love Pickleball Swimming Muay Thai
4. I Also Love Pickleball Swimming Muay Thai
5. I ❤ Ruby Type of Software Programming Language Web / API Mobile Desktop Game Database AR / VR Blockchain / Web3 Infrastructure / DevOps ML / Data Science Embedded / IoT Ruby Ruby Ruby Ruby Ruby Ruby Ruby Ruby Ruby Ruby
6. Choosing the Right Tool Type of Software Programming Language Web / API Ruby Mobile Kotlin / Swift Desktop TypeScript Game C++ Database Rust AR / VR C# Blockchain / Web3 Solidity Infrastructure / DevOps Go ML / Data Science Python Embedded / IoT C
7. Jose Valim at SF Ruby? Type of Software Programming Language Web / API Ruby Mobile Kotlin / Swift Desktop TypeScript Game C++ Database Rust AR / VR C# Blockchain / Web3 Solidity Infrastructure / DevOps Elixir ML / Data Science Elixir Embedded / IoT Elixir Type of Software Programming Language Web / API Ruby Mobile Kotlin / Swift Desktop TypeScript Game C++ Database Rust AR / VR C# Blockchain / Web3 Solidity Infrastructure / DevOps ML / Data Science Embedded / IoT
8. 🔥 Hot Take #1 Ruby is the best programming language for scripting and automation
9. Ruby History
10. ChatGPT 👌
11. Gemini 👌
12. Claude Hallucination 🤔
13. I’m a Prompt Engineer 👌
14. Bemi AI: Product
15. Bemi AI: Product
16. Bemi AI: Multi-Tenant Infrastructure
17. Data Engineering 101 Postgres: logical replication Notion: API Dialpad: webhooks Immutable CSV on steroids JSON metadata SQLite for analytics
18. Data Engineering 101 Postgres: logical replication Notion: API Dialpad: webhooks Immutable CSV on steroids JSON metadata SQLite for analytics Kafka JVM Table compaction Snapshotting Iceberg v3, Delta Lake, Hudi, etc. Data catalog Data transformation, ETL, ELT, reverse ETL, WTF 🤯 Data streaming Orchestration Vectors Data quality
19. SPA stacks are trivial in comparison Postgres: logical replication Notion: API Dialpad: webhooks Immutable CSV on steroids JSON metadata SQLite for analytics Kafka JVM Table compaction Snapshotting Iceberg v3, Delta Lake, Hudi, etc. Data catalog Data transformation, ETL, ELT, reverse ETL, WTF 🤯 Data streaming Orchestration Vectors Data quality
20. 🔥 Hot Take #2 Data engineering stacks are 10x more complex and costly than software engineering
21. Infrastructure Automation • Ansible (YAML), Chef (Ruby) • Terraform (HCL), Pulimi (JavaScript) • Docker (Docker fi le), Kubernetes (YAML)
22. How to Automate the Automation?
23. Rich Standard Library Data Parsing: JSON, YAML, CSV
24. Rich Standard Library Template Rendering
25. Rich Standard Library HTTP Requests
26. Rich Standard Library File Manipulation
27. Rich Standard Library Logging
28. Rich Standard Library Running shell commands
29. Iteration Speed as Advantange • No boilerplate • No compilation • Easy to run • Easy to debug
30. 🔥 Hot Take #3 Move fast by being more e ff i cient
31. 🔥 Hot Take #3 Uses only legs
32. Dry run pattern
33. Sidecar pattern
34. Sidecar pattern
35. 🔥 Hot Take #4 When a monolith isn’t an option, go o ff -Rails and use plain Ruby
36. GitOps
37. Gradual typing
38. Gradual typing Can’t type-annotate structs Sorry 😿
39. 🔥 Hot Take #5 Gradual static typing is a useful tool, but its current state in Ruby is a mess
40. 🔥 Hot Take #5 Gradual static typing is a useful tool, but its current state in Ruby is a mess RBS (aka C++ *.h fi les for Ruby) Steep Sorbet RBI Static type checking Runtime type checking Tapioca TypeProf
41. Best Practices • Limit dependencies • Reduce mutable global state • Make scripts idempotent • Mask sensitive data before logging • Validate and sanitize user inputs • DRY without needless abstractions
42. Advanced Techniques • Timeout handling • Feature fl ags • Rate limits • Retry with exponential backo ff • Circuit breaker • Chaos engineering
43. Ruby: More Than Web Ruby was designed for scripting and has applications beyond just web development
44. evgeny@bemi.ai | @exaspark 👋 • Ruby, AI, data engineering • Startups, founders, pivots • Swimming, pickleball, martial arts