How to Update Your Data Management Program in the Age of AI

Transcript

1. How to Update Your Data Governance and Management Strategies for the AI Era Karen Lopez InfoAdvisors 1
2. Karen Lopez Microsoft MVP, Data Platform Microsoft Certified Trainer, vExpert Data management expert, space enthusiast, and #TeamData evangelist www.datamodel.com @datachick.bksy.social
3. ANY TIME Ask questions Raise issues Leave kudos Don’t hold them until the end.
4. Nature of Data Management Data management principles haven’t changed much in the last 30 years Advance in technology and business innovation have increased the need for stronger data management requirements AI is no different, but does bring new emphasis on certain aspect.
5. Two-phased AI Requirements • Preparing for AI uses • Protecting against AI abuses
6. Key Takeaways AI as a Copilot, Not an Autopilot AI-Powered Query Optimization Managing Data Complexity Mastering Multi- Platform Complexity Governance at Scale The Importance of Synthetic Data Generation Data Architect Productivity Managing Risk Importance of Laziness
7. We can’t manage data we don’t know we have
8. Biggest Mistake in Data Complexity? Not knowing what data we have, where it is, how it is protected, when it is updated, is it trusted, how old it is, what does is mean, is it named correctly, who governs it, why it’s being collected…
9. Data Inventory and Discovery Finding and identifying relevant data within an organization Identifying data that is usable for a specific purpose Helping organizations find new insights, make better decisions, and meet compliance targets
10. Scaling Data Governance Roles and responsibilities Usable standards Robust tools & automation Collaboration incentives Classifications and grouping Monitoring & alerting
11. How AI Will Help Us With Complexity We aren’t all SQL Developers It’s not just about SQL syntax A good data pro is a lazy data pro We need help understanding data in other practices
12. Karen’s Directives on AI & ML Never autonomous Paired “programming” Responsible AI must be foremost Understand bias Understand security and privacy
13. Where AI Can be Used in Data Governance • Data Quality • Data Classification • Data Preparation • Data Security • Data Observation and Monitoring • Data Support and Literacy • Auditing and Compliance • Anomaly Detection • Data Profiling • Data Growth and Capacity Planning • Generative use cases – test data • ..and more
14. Data Capacity Planning • Predictive Analytics, but more. • Cost savings • Outage protection • Auto scaling
15. Test Data Generation Synthetic Data NO MORE PRODUCTION TEST DATA More realistic test data Faster, cheaper, better
17. Give it a try Ask a bot to generate data for your use case. Refine, validate, refine
18. Data Observation and Monitoring
19. Data Literacy Understanding data sources and types Data exploration Meta data collaboration Meta data “translations” Data visualization and communication Interpreting data analysis
20. Simple Queries with challenging computations
21. Simple Queries with challenging computations, with a fix ☺
22. Give it a try Ask a bot to generate a simple table, then to generate a not so simple query about it.
23. Data Preparation • Identifying data sources • Validating data relevance • Data cleansing • Data transformation • Data integration • Data loading
24. Data Quality • Data Types • Error Detection • Data Deduplication • Data Validation • Data Standardization • Anomaly Dection
25. Data Quality • Data Cleansing • Data Matching • Monitoring • Automation • Alerting
30. Analyze this data for data quality and data privacy issues 1.Missing values – Certain fields (e.g., in the Year column and possibly in Date or Duration fields) have missing or invalid entries (such as a non-date value or 'nat’). 2.Inconsistent formatting – The Duration column is in text format while a derived numerical column (Duration_minutes) exists. It is important to ensure that time-related values are consistently converted and validated. 3.Data integrity – There could be duplicate entries or inconsistencies between related fields (for instance, discrepancies between the Duration and Duration_minutes columns). 4.Data parsing – Special care is needed in handling non-standard or erroneous entries (for example, a 'nat' entry in place of a valid date).
32. Data Classification Data profiling vs. data meta data inspection Natural Language Processing Pattern recognition Anti-patterns in profiling Data Exceptions 80/20 for now
33. Give it a try Ask a bot to generate a simple table with sensitive data as columns, then ask it to classify the data.
34. Where Else Can AI be Used in Data Governance & Data Management?
35. Data Management AI Readiness Invest in Data Governance Integrate Data Governance with AI Governance Improve Data Quality processes Strengthen Data Integration Governance Integrate AI with Data Analytics
36. Coffee Talk | Expert View: How To Update Your Data Management Strategy for the AI Era Data Management AI Readiness Refocus on metadata programs Strengthen security measures Do more automation Build a data-driven culture Build ethical AI skills, knowledge, and methods
37. More Data- Related AI Threats Cybersecurity Bias and Discrimination Jobs? Compliance breaches Flirting and bullying Ethical Abuses IP issues
38. Data-related AI Protection • Defensive AI • Employee training • Data Governance • More auditing • AI-driven data protection • Data risk assessments
39. Questions to Answer Before AI What are our real goals for AI? Do we have the right data infrastructure? Do we understand the legal and ethical uses of our data? How will we protect the data, models, and systems? How will we monitor and assess the outcomes? How does AI integrate with our existing systems and tools? Do we have resources to do this responsibly?
40. 10 Tips 1. Do personal learning and education on AI topics 2. Do personal learning and education AI ethics 3. Build up your Data Governance programs 4. Bring meta data methods back to the forefront 5. Communicate, often, the importance of monitoring and auditing
41. 10 Tips 6. Leverage AI, smartly and ethically to build these programs 7. Enhance all data management segments with AI 8. Ensure data professionals are part of the strategic planning 9. Evangelize the importance of cross-group collaboration 10.Build Data Literacy programs
42. Resources More for you…
43. Frameworks for Ethical and Responsible AI 1.Asilomar AI Principles: Developed during the 2017 Asilomar conference, these principles focus on research issues, ethics and values, and longer-term issues 2.IEEE’s Ethically Aligned Design: A set of guidelines that prioritize human rights and well-being in the development of autonomous and intelligent systems 3.EU’s Ethics Guidelines for Trustworthy AI: Created by the High-Level Expert Group on AI, these guidelines emphasize lawful, ethical, and robust AI 4.Montreal Declaration for Responsible AI: A framework that outlines principles for responsible AI development, including well-being, respect for autonomy, and democratic participation 5.AI4People’s Ethical Framework for a Good AI Society: Offers recommendations and outlines principles for the ethical implementation of AI in society
44. Frameworks for Ethical and Responsible AI 6. Google’s AI Principles: Google’s own set of ethical principles for AI development, which includes being socially beneficial, avoiding creating or reinforcing unfair bias, and being accountable 7.Microsoft’s AI Principles: Microsoft’s framework focuses on fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability 8.The Toronto Declaration: A declaration focusing on protecting the right to equality and non-discrimination in machine learning system 9.The Beijing AI Principles: Developed by the Beijing Academy of Artificial Intelligence, these principles focus on harmony and friendliness, fairness and justice, and inclusivity and sharing 10.The Public Voice’s Universal Guidelines for AI: A set of principles that include the right to transparency, the right to human determination, and the right to redress2.
47. Karen Lopez Datachick.bksy.social karen@infoadvisors.com