Did It Sync, or Did It Just Not Error? Building Observable Data Pipelines

Transcript

1. Did It Sync? or, Did It Just Not Error ? Building Observable Data Pipelines
2. The Ecommerce Ecosystem • Product Information Management • Customer Relationship Management • Warehouse Management System • Order Management System • Enterprise Resource Planning • Search and Personalisation Suite • Shopping Feeds • Conversational AI • …
3. “Everything fails, all the time ” - Werner Vogels CTO, Amazon
4. Loud Failures • Visible • Unambiguous • Fail fast Ooops!
5. It ran • Cron / Message Consumer ran • No exceptions thrown • APIs return successful responses • Logs are clean
6. It ran, vs… • Cron / Message Consumer ran • No exceptions thrown • APIs return successful responses • Logs are clean It worked • Record actually gets updated… • …with the right information • Downstream systems get notified • Logs report success
7. It ran, vs… • Cron / Message Consumer ran A lack of errors does not guarantee correctness • No exceptions thrown • APIs return successful responses • Logs are clean It worked • Record actually gets updated… • …with the right information • Downstream systems get notified • Logs report success
8. It ran, vs… • Cron / Message Consumer ran Success states do not guarantee correctness • No exceptions thrown • APIs return successful responses • Logs are clean It worked lied • Record actually says it was updated… • …with the right information • Downstream might be notified • Logs still report success
9. Common Silent Failure Causes * Hidden Errors Unmonitored Paths The process ran through a path you weren’t expecting or watching Race Conditions The failure is there, but the system doesn’t surface it properly Async messages processed “out of order” reverting stale data Stranded Work Missing Data Processes start, fail (or time out), and disappear from view The pipeline runs without issue, but data either wasn’t present or disappeared * not a comprehensive list!
10. What Does Observable Mean?
11. “The most effective debugging tool is still careful thought , coupled with judiciously placed print statements ” - Brian Kernighan
12. Improving Observability • Log decision points - not noise • “Fingers Crossed” logging • Make exclusions visible • Separate transport success from business success • Carry session IDs through the flow • Toggleable trace logs through key processes • Nothing happened !== Nothing to do Intentional Logging
13. Improving Observability • Answers available without special access • Run summaries > Raw traces • Telemetry data must be durable • Make diagnostics searchable and queryable Intentional Logging Accessible Diagnostics
14. Improving Observability Intentional Logging • Payload previews • Dry-run decision input -> output • Support tracing entities in isolation Accessible Diagnostics • Available in live environments Proactive Visibility
15. But… What about AI?
16. Harnessing AI Yes, But • Mining data • Hallucinations • Pattern detection • Root cause analysis • Analysis is only as good as the telemetry provided • Finding blind spots - especially in legacy systems • Cannot reconstruct data that was never recorded • Agent-led system monitors • Additional costs
17. Takeaways • A pipeline that ran isn’t the same as one that worked • A false success is worse than an error • Knowing what’s missing is as important as knowing what’s wrong • Legacy systems are not lost causes • AI can’t make up for missing or incorrect data • Test and review your observability strategies , P V M is s ic t s o n g ia D ” e v a h o t e ic n “ a t no
18. Thank you! Any Questions? /in/abiverde @abiverderci Abi Green