running queries ▧ Understand postgresql WALs ▧ Data pipelines break. Exception handling, notifications, logging is utmost important ▧ We wired luigi exceptions to slack for notifications ▧ Pandas transformations are slow for large datasets ▧ PySpark to the rescue! ▧ Use monitoring tools like Munin for profiling