Architecture Decisions for Tiny & Big Data

Enterprises want to be data driven from the very beginning or want to join the race for data supremacy. Being data driven requires the system to store and process every single transaction and interaction the customer makes with the product, thus enabling the business to make better decisions.

But storing, processing, and analyzing data comes with a cost. This cost is distributed across the choice of technology, infrastructure, and go-to-market strategy.

Nischal HP and Raghotham Sripadraj share their experience building data science platforms for various enterprises, with an emphasis on making the right architecture choices for things such as databases, queues, caching mechanisms, distribution of the workload, underlying technology for machine learning and predicitive models, visualization, and prototyping. Nischal and Raghotham stress the importance of using distributed and fault-tolerant tools, which themselves come with the cost of managing the infrastructure (including, by implication, a dedicated team to monitor the infra). However, with small data, simple tools take you a long way.

Many things can go unnoticed in building an end-to-end data science system, like the importance of logging, building a data pipeline that sends notifications to the required medium of communication, exposing data science as a service via APIs, or A/B testing for data science-backed feature releases when required. Only when the data science solution is in production does it power the organization the right way.

When building data science products you should live by the motto “fail fast.” Nischal and Raghotham themselves have failed fast when making these choices, but in time they came to understand that adopting the latest and the coolest technology on the planet just for the sake of it is not the right thing to do.

Transcript

1. Architecture Decisions for Big & Tiny Data.
2. www.unnati.xyz UnnatiData Labs We are @raghothams @nischalhp
3. What do we do? Build Data Science Platforms
4. What do we do? Build Data Science Platforms Educate + =
5. What do we do? Build Data Science Platforms Educate Frame the problem + + =
6. What do we do? Build Data Science Platforms Educate Frame the problem = Technology + +
7. 3 Stories #Fintech #CampaignManagement #UnstructuredHealthcare
8. 3 Criteria V.V.V Talent Pool Cost
9. #Fintech Tiny data | Early Startup | Data Driven
10. What are we solving?
11. Creditworthiness of Students What are we solving?
12. Challenges?
13. No data Challenges?
14. No data Challenges? Credit history of Students is scarce
15. No data Challenges? Credit history of Students is scarce Using Alternate Data Sources
16. Architecture
17. Criteria V.V.V 100K records in the first 6 months 60K Students in the first year
18. Criteria Talent Pool 3 member tech team MEAN stack shop
19. Criteria Cost Early Stage Startup
20. Learnings
21. Tiny data problems are tricky Learnings
22. Tiny data problems are tricky Get hold of the low hanging fruits Learnings
23. Tiny data problems are tricky Get hold of the low hanging fruits Beware of Data Sanity with NoSQL Learnings
24. Tiny data problems are tricky Get hold of the low hanging fruits Beware of Data Sanity with NoSQL Embrace Data Science Early Learnings
25. #Campaign Management Medium data | Established Startup
26. What are we solving?
27. What are we solving? Predict behavior of Users
28. What are we solving? Predict behavior of Users Educate the team on Data Science
29. What are we solving? Predict behavior of Users Educate the team on Data Science Optimize campaign costs
30. Challenges?
31. Challenges? No analytics DW in place
32. Challenges? No analytics DW in place Time to market
33. Architecture
34. Criteria V.V.V 50+ million rows Exponential growth
35. Criteria Talent Pool 5 member tech team Python+Django Shop
36. Criteria Cost Bootstrapped Established Startup
37. Learnings
38. Know your databases well Learnings
39. Know your databases well Learnings Data pipelines break
40. Know your databases well Learnings Data pipelines break Exception handling - Logging - Notifications
41. Know your databases well Learnings Data pipelines break Exception handling - Logging - Notifications Monitor performance of your libraries
42. #Unstructured Healthcare Big data | Generic Platform
43. What are we solving?
44. What are we solving? Analytics on healthcare spend
45. What are we solving? Analytics on healthcare spend Medical claims - Many Data Providers - no standard
46. Challenges?
47. Challenges? Ever changing schema
48. Challenges? Ever changing schema Large number of ETL jobs
49. Architecture
50. Adding more servers is easy, writing good code is hard.
51. Criteria V.V.V 150+ million rows Every source has their own schema
52. Criteria Talent Pool SQL + Tableau team
53. Criteria Cost Limited budget
54. Learnings
55. Data security is very important Learnings
56. Data security is very important Learnings Parallelize ingestion
57. Data security is very important Learnings Parallelize ingestion Data redundancy is not an issue
58. Data security is very important Learnings Parallelize ingestion Data redundancy is not an issue Polyglot of services
59. Tools
60. THANKS! Any questions? www.unnati.xyz