What Data Analysts Wish Application Developers Knew

August 2016 version

Transcript

1. What Data Analysts Wish Applica/on Developers Knew Alison Stanton @alison985 me@alisonstanton.com Pronouns: She/Her/Hers #DataAnalystsWish
2. #DataAnalystsWish
3. Foundational Concepts #DataAnalystsWish
4. Reconcilia/on #DataAnalystsWish
5. Developers Are Upstream of Data Folk Business •  Priori/zes what gets built Developers •  Build Data •  Reports to Business on what Business wants to know about the build and the resul/ng built thing
6. Fictional scenarios based on real life events. Names and details have been changed, merged, and muddled to protect the guilty.
7. Run the World, Inc. #DataAnalystsWish
8. First Request Jackie and Beyonce (Sales Team) come to Kerry with ques/ons from the client about why the demo they gave a client with the clients data doesn’t match what the client is seeing in their trial instance. #DataAnalystsWish
9. The Inves/ga/on Reveals •  It’s visible. •  It’s not reproducible. #DataAnalystsWish
10. Kerry's Core Problem Missing data. Not enough data was saved. #DataAnalystsWish
11. Kerry's Detailed Problems •  Can’t answer people’s ques/ons at all. •  Can’t make decisions based on facts. •  There’s no baseline or historical informa/on available for past informa/on forever. #DataAnalystsWish
12. Dev. Ques/on How do I know what to save? #DataAnalystsWish
13. Answer •  Save the dis/nc/ons for reconcilia/on. •  Think of them as data migra/ons. •  Keep them in one place. #DataAnalystsWish
14. Second Request bell hooks (Support Director) and Marian Wright-Edelman (Lawyer) are reques/ng all the /mes we responded to a customer in IL with ques/ons about the guarantee on feature Y. #DataAnalystsWish
15. Can't get to the data. #DataAnalystsWish Kerry's Core Problem
16. Kerry's Detailed Problems •  Ge_ng access permissions •  ETL headaches #DataAnalystsWish
17. Dev. Ques/on Where do I save it? #DataAnalystsWish
18. Answer •  A place that everyone can get to. •  Data is in one place. Documenta/on is in one place. •  One. Singular. One. #DataAnalystsWish
19. Third Request Audre Lorde (CEO) doesn’t get her Monday morning dashboard. She got an error message instead. #DataAnalystsWish
20. The Inves/ga/on Reveals •  Kerry's code hasn't changed but the results did. #DataAnalystsWish
21. Kerry's Core Problem Produc/on iterated and dev team didn’t tell anybody. #DataAnalystsWish
22. Kerry's Detail Problems •  Automated reports failed because schema changed. •  Meanings changed without warning. #DataAnalystsWish
23. Fourth Issue Oprah Winfrey (CMO) starts making ad buys because her sales report says we have 50 new customers this month. In actuality we have already surpassed the goal of 500.
24. Dev. Ques/on What are the relevant pieces of informa/on about a deploy to communicate to the data team? #DataAnalystsWish
25. Answer •  Something is changing •  Why is it changing? •  What is it now and what will it change to be when released? •  When is it changing? •  ASK ME #DataAnalystsWish
26. Fiah Request Rosa Parks (Finance Director) is preparing for an investor audit. She’s reques/ng help calcula/ng FIFO on all products. Run the World, Inc. Run the World, Inc. #DataAnalystsWish
27. The Inves/ga/on Reveals •  Aren’t saving history or edi_ng diffs. #DataAnalystsWish
28. Kerry's Core Problem Not enough data to fulfill the request. #DataAnalystsWish
29. Kerry's Detailed Problem •  Impossible task #DataAnalystsWish
30. Dev. Ques/on What can we do to prevent this in the future? #DataAnalystsWish
31. Answer •  created by, created at, updated by, updated at, status •  Separate audit table •  What's the risk? #DataAnalystsWish
32. Sixth Request Oprah (CMO) and Angelica (CSO) are figh/ng with Harriet (COO) about who brings in the most business. Every order needs to be afributed to either the sales or marke/ng department. #DataAnalystsWish
33. Kerry's Core Problem Can’t integrate data across systems. #DataAnalystsWish
34. Kerry's Detail Problems •  JOINing problems because of a lack of foreign system keys. #DataAnalystsWish
35. Dev. Ques/on How do I prepare without over- engineering? #DataAnalystsWish
36. Answer •  Store other systems keys whenever possible. •  Understand, maintain, and enforce uniqueness. •  Tools around mapping values. #DataAnalystsWish
37. Seventh Request Toni Morrison (Accounts Director) comes over complaining that invoices to clients are being generated incorrectly. Instead of charging your biggest customer $500,000 the invoice was for $5,000. #DataAnalystsWish
38. Kerry's Core Problem There’s a code bug that is wri/ng inaccurate data. #DataAnalystsWish
39. Kerry's Detailed Problems •  Lots of /me spent playing detec/ve. •  Time spent communica/ng with developers. •  Time spent fixing. •  Time spent communica/ng with stakeholders. #DataAnalystsWish
40. Dev. Ques/on How do I minimize organiza/onal pain? #DataAnalystsWish
41. Answer •  Fix: Stop the Bleeding. •  Communicate: Time started, /me ended, what was effected and how. •  Prevent: –  Version control with release tags –  documenta/on of soaware (including the bug and how it was fixed) –  great tests –  Code review #DataAnalystsWish
42. Dear Applica/on Developers, Please remember: 1) Save as much as possible. Anything you will need to know to re-create a result is relevant. 2) We adore standardized, accessible storage that is properly documented. 3) Please communicate about changes you're making. 4) Save an audit trail when needed and/or in the context of money. 5) Assume everything has to be able to talk to each other eventually. 6) When bugs arise: Fix. Communicate. Prevent. Sincerely, The Data Folk #DataAnalystsWish
43. WARNING You will be cri2cized for working to solve the root of the issue. You will probably be cri2cized for talking to people instead of only wri2ng code. #DataAnalystsWish
44. #DataAnalystsWish
45. #DataAnalystsWish
46. Thank You @alison985 me@alisonstanton.com What Data Analysts Wish Applica/on Developers Knew #DataAnalystsWish