Saheed LASISI , JPMorgan Chase & Co API Days Paris | December 2024 Bridging Knowledge gaps and Elevating Developer Experience

Agenda 1. Introduction 2. Challenges in API Documentation 3. Generative AI: A Game Changer 4. Capabilities of AI in API Documentation 5. Real world AI challenges 6. Addressing Challenges and Governance 7. Future Vision and Key Takeaways 8. Q &A

Introduction ❑ Over 50% of buy decisions are influenced by developers ❑ Developers are decision makers in companies of all sizes - DevRelx1 Technical Documentation is key in getting in front of Decision makers https://www.devrelx.com/post/how-education-helps-developers-reach-purchasing-decisions-and-product-adoption of-b2b-sales-interactions-between-su

The opportunity ❑ Nearly 40% of developers report educational resources as top expectation from vendors1 ❑ “80% of B2B sales interactions will occur in Digital channels by 2025” – Gartner2 https://www.devrelx.com/post/how-education-helps-developers-reach-purchasing-decisions-and-product-adoption 2 – Source: https://www.gartner.com/en/newsroom/press-releases/2020-09-15-gartner-says-80--of-b2b-sales-interactions- between-su ❑ When it comes to choosing a public API, good API documentation outranked performance or security3

API Documentation Challenges ❑ Inconsistencies ❑ Inaccuracies ❑ Complexity ❑ Outdated Information 4 – Source https://www.postman.com/state-of-api/2023/executing-on-apis/#executing-on-apis/ ❑ 80% of developers find poor documentation to be their top pain point consuming APIs.4

API Documentation Expertise Navigating documentation challenges

Trends in API Documentation Interactive Documentatio n Real-time Collaboration Focus on User Experience Localization and Globalization Doc As Code Gen AI and Automation

Generative AI: the Game Changer Intelligence Machine Learning Deep Learning Generative AI (GenAI) e.g., LLM chatbot to generate creative content, create images, or summarize text Artificial Intelligence (AI) Broad term for the field that enables machines to mimic human intelligence, solving problems and performing tasks. Machine Learning (ML) Subset of AI, this involves computers learning from data to make decisions or predictions without being explicitly programmed. Deep Learning Branch of machine learning that uses multi- layered (“deep”) artificial neural networks to handle large data sets and perform complex tasks. Generative AI (GenAI) Type of deep learning that focuses on creating new, realistic outputs from unstructured inputs like text, images, or audio. Large Language Models (LLM) Model trained on vast amounts of text data to understand, generate, and predict text with human-like proficiency across various contexts and tasks. Retrieval Augmented Generation (RAG) A technique that enhances the accuracy and relevance of the model's responses by integrating external knowledge through information retrieval systems .

Localization and Globalization: GenAI can automatically translate API documentation into multiple languages, making APIs accessible to a wider audience of developers. Unlocking new possibilities: ❑ API spec, test and code generation ❑ Interactive API Mocks ❑ Enhanced AI search for API discoverability Enhanced Adoption: ❑ Faster onboarding ❑ Simplified explanations and adoption for bridging stakeholders ❑ Multilingual support Enhancing Consistency and Accuracy: ❑ GenAI leverages templates, style- guides, blueprints, data dictionaries, glossaries and playbooks to enforce governance and generate quality content for cross-platform adaptation. Improving Accessibility and Engagement: ❑ Enabling Natural language search ❑ Intelligent examples and contextual code snippets Automating the grunt work: dynamic content generation from API artifacts: specs, code snippets, requirements, collections, tests etc Enhancing Developer experience: ❑ Personalized documentation experience based on profile and usage context. ❑ Interactive tutorials and examples Interactive guidance and chat assistance ❑ Troubleshooting and debugging Extending Scale: GenAI can help organize thoughts and provide basic drafts or comprehensive outlines . It can automate content creation, editing, rewriting and content organization necessary of new projects. GenAI is revolutionizing API Documentation 1 5 2 4 8 6 7 3 IMPORTANT! Content produced by GenAI must be reviewed, validated and fact checked against credible sources by a human in the loop before publication or dissemination. Capabilities of GenAI in API Documentation

Real world Use-cases ❏ Plaid embeds AI Robot into its API documentation: “Ask Bill”1. ❏ Stripe, known for its excellent developer experience: ❏ Brings AI enabled Stripe docs to VSCode IDE through Github copilot extension2 ❏ Stripe leverages GPT4 and Stripe docs for enhanced developer support 3. Result: Reduction in support effort. 1: https://plaid.com/docs/support/?showChat=true 2: https://insiders.stripe.dev/t/join-the-vs-code-extension-github-copilot-preview/107 3: https://openai.com/index/stripe/

Challenges in AI enabled API documentation ● Maintaining Accuracy, Consistency, and Style. ● Risks of Hallucinations and bias in AI-Generated Content. ● Risk of Prompt injections and brute-force attacks ● Compliance risks “These challenges are not unique to API Documentation.”

Real world GenAI Challenges Generative AI is no longer a futuristic fantasy. Everyone is racing to integrate, but are they overlooking critical roadblocks ? (Source: Writer.com, State of AI Report 2024) ➔ Security, Data Protection and compliance are top pain points in AI integrations. - (Postman, State of API Report 2024 and Writer.com State of AI Report 2024)

AI RISKS Compliance Bias Data Privacy Accuracy Copyright Ethics Building Trust in the age of GenAI ❏ To truly unlock the potential of AI, we need to build systems that are trustworthy, reliable, and ethical. ❏ "88% of companies estimate it will take at least 6 months to have a reliable in-house AI solution." (Source: State of AI Report 2004, Writer) This highlights the need for careful planning, investment, and a long- term perspective on AI adoption.

○ "Data controls and access leads security controls needed for generative AI " ○ "61% of companies have already experienced accuracy issues with generative AI." - (Source: Writer.com, State of AI Report 2024) Start with Data Risks and governance This underscores the critical need for robust data strategies to manage data access, control, and governance. (Source: Writer.com, State of AI Report 2024)

❏ Unreliable outputs ❏ AI hallucinations and confabulations ❏ Weak Operational integrity of AI driven processes Impact: ❏ Misinformed or flawed decisions ❏ Misleading, false and harmful information ❏ Erosion of Trust and acceptance Mitigation Strategies: ❏ Regularly update training data to improve accuracy. ❏ Implement feedback mechanisms for ongoing model improvement. ❏ Implement validation mechanisms (e.g., fact-checking) to detect and prevent AI hallucinations and factual grounding. ❏ Incorporate human oversight and intervention in critical decision-making processes. ❏ Continuously train AI models on real-world data to reduce hallucination tendencies (aka self-evolving models). ❏ Use techniques like retrieval-augmented generation (RAG) (contextual and knowledge graphs) to improve generated content accuracy and relevance. ❏ Continuously score trusted, untrusted and new retrieval sources to establish varying degree of trust. ❏ Continuously certify training data sets and knowledge sources for augmented retrieval. ❏ Implement explainable AI (XAI) methods such as LIME(Local Interpretable Model-Agnostic Explanations), SHAP(SHapley Additive exPlanations), or counterfactual explanations to provide insights into model reasoning. ❏ Use a combination of explainability strategies taking into account context, domain specific knowledge and invariants. Tackle Accuracy and Explainability Risks

Transform your lifecycle with DDD

Address Adoption Pitfalls - 1 Stay up-to-date with RAG Retrieval-Augmented Generation (RAG)1 is an advanced technique in natural language processing that enhances the capabilities of LLMs by integrating external information retrieval mechanisms to augment the generation process. ❏ This approach enables LLMs to access up-to-date and domain-specific data. ❏ Contextual RAGs, Semantic chunking, high fidelity agentic retrieval and Knowledge graphs aims to solve semantics and context problems even further. Research shows higher relevance, accuracy and lower hallucinations - (RobustQA Benchmark, ACL 2023, https://aclanthology.org/2023.findings-acl.263/) 1: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401

Confidential Copyright © (Source: Weaviate - https://weaviate.io/blog/what-is-agentic-rag) ❏ RAG is a pattern that can improve the efficacy of LLM applications by using your own custom data. ❏ It can facilitate the development of high-quality articles, documentation, reports, chatbots, search engines and summaries using relevant context. Contd ..

01 02 Address Adoption Pitfalls - 2 Mitigate Data Privacy Risk Potential Risks:Unauthorized access Impact: violation of privacy regulations Mitigation Strategies: Explicit user consent for data usage, access controls, encryption, anonymization, audits and compliance with privacy regulations, input sanitization, zero-trust architecture Mitigate Bias and Fairness Potential Risks: Biased training datasets Impact: discriminatory outcomes Mitigation Strategies: Use diverse dataset for training, data pre-processing, conduct fairness assessments, data annotations and labelling, establish a feedback loop, update models regularly, incorporate automatic fairness verification and fairness awareness learning in model training.

Confidential Copyright © 20 Address Adoption Pitfalls - 3 03 04 Mitigate Prompt Injection and Security attacks Potential Risks: Unauthorized access Impact: misinformation, privacy breaches and system compromises. Mitigation Strategies: input sanitization, rate limiting, RBAC, code review and static analysis, encryption, secure storage and zero trust architecture. Maintanability Potential Risks: Difficulties in maintaining, updating, and refactoring designs Impact: Technical debt and misinformation arising from poor and outdated designs Mitigation Strategies: ❏ Adopt standardized diagramming frameworks ("diagram as code") such as Mermaid, PlantUML, and Architecture as Code tools. ❏ Enforce version control for Documentation diagrams just like code.

Confidential Copyright © 21 Address Adoption Pitfalls - 4 05 Compliance Potential Risks: Non-compliance with legal and industry standards Impact: Legal repercussions and penalties Mitigation Strategies: ❏ Shift-left design approach to address compliance risks with a compliance styleguide. ❏ Implement automated compliance guardrails based on a compliance styleguide as part of every genAI app stack. ❏ Establish clear compliance monitoring and reporting procedures. ❏ Stay informed about the latest laws, regulations, and industry standards. ❏ Establish patterns and mechanisms to mitigate compliance risks. ❏ Collaborate early with compliance, legal and regulatory experts to ensure compliance is incorporated early in the development lifecycle.

Confidential Copyright © Envisioning the future ❏ Reduced Developer Friction: fewer support tickets . ❏ Faster Onboarding: Developers onboard nX faster. ❏ Improved Satisfaction: Higher adoption rates and enhanced feedback. ❏ APIs that are Self-Documenting and Developer-Centric. ❏ Context-aware Dynamic Content Adaptation Based on Developer Interactions.

Wrapping up ❏ The Age of AI Enablement: powering API documentation to deliver dynamic and intelligent experiences that adapt to developers' needs. ❏ Elevate the developer experience: Combine AI with strong governance, patterns and standards to drive API adoption and client satisfaction. ❏ Shift-left Governance: Shift-left to design-time, risk, compliance, and conformance to establish a robust governance framework to unlock GenAI's full potential. ❏ Knowledge is power: Continuously or retrospectively build certified knowledge bases to fuel AI accuracy and explainability. ❏ Don't wait, innovate: Start small, iterate, learn, refine and evolve.

Questions Connect: https://www.linkedin.com/in/saheed-abiola-lasisi-4b62891b/

Confidential Copyright © APPENDIX

Confidential Copyright © GenAI Risk Mitigation Strategies -1 Risk Category Potential Risks IMPACT Mitigation Strategies Data Privacy Unauthorized data access Violation of privacy regulations Implement robust access controls, data encryption, and anonymization techniques. Regularly audit and monitor data access. Acquire explicit user consent for data usage. Adhere to relevant data privacy regulations (e.g., GDPR, CCPA). Educate employees on data privacy best practices. Bias and Fairness Biased training datasets Discriminatory outcomes Utilize diverse and representative data for AI model training. Regularly audit, annotate dataset and mitigate potential biases in AI models Conduct fairness assessments to identify and address biases. Include diverse stakeholders in the AI model development lifecycle and a feedback loop. Maintain transparency in the AI decision-making process. Continuously monitor and update AI models to address emerging biases.

Confidential Copyright © GenAI Risk Mitigation Strategies - 2 Compliance Non-compliance with legal and industry standards Legal repercussions and penalties Stay informed about the latest laws, regulations, and industry standards. Establish clear compliance monitoring and reporting procedures. Conduct regular compliance audits to identify gaps. Provide comprehensive training to employees on compliance requirements. Collaborate with legal and regulatory experts to ensure compliance. Shift-left design approach for compliance mitigation. Develop mechanisms for addressing and reporting non- compliance. Risk Category Potential Risks IMPACT Mitigation Strategies

Confidential Copyright © GenAI Risk Mitigation Strategies -3 Ethical Considerations Unforeseen harmful outcomes Negative societal effects Develop and implement clear ethical guidelines for AI development and deployment. Conduct ethical impact assessments to evaluate potential societal implications. Actively engage with relevant stakeholders to discuss ethical concerns. Promote transparency and openness in AI development and usage. Regularly review and update ethical guidelines to adapt to evolving AI capabilities. Cultivate a culture of ethical AI use within the organization. Establish channels for reporting ethical concerns related to AI. Regularly educate employees on ethical considerations in AI development and use. Encourage interdisciplinary collaboration to address ethical challenges. Risk Category Potential Risks IMPACT Mitigation Strategies

Confidential Copyright © GenAI Risk Mitigation Strategies - 4 Risk Category Potential Risks IMPACT Mitigation Strategies Accuracy Inaccurate or unreliable outputs Misinformed or flawed decisions Regularly update training data to improve accuracy. Conduct rigorous testing and validation of AI models. Implement feedback mechanisms for ongoing model improvement. Monitor and address biases in training and evaluation data. Provide clear explanations and maintain transparency in AI predictions. AI hallucinations or confabulations Misleading, false or harmful information Implement validation mechanisms (e.g., fact- checking) to detect and prevent AI hallucinations. Regularly review and update language models to minimize hallucination risks. Incorporate human oversight and intervention in critical decision-making processes. Continuously train AI models on real-world data to reduce hallucination tendencies. Use techniques like retrieval-augmented generation (RAG) to improve generated content accuracy and relevance. Continuously certify training data sets and knowledge sources for augmented retrieval.

Confidential Copyright © GenAI Risk Mitigation Strategies -5 Risk Category Potential Risks IMPACT Mitigation Strategies Copyright Infringement of copyrighted materials Legal repercussions and penalties Obtain necessary licenses and permissions for using copyrighted materials. Implement content filtering and copyright protection measures. Maintainability and Scalability Difficulties in maintaining, updating, and refactoring designs (workflows, flowcharts, UML, sequence diagrams, architecture diagrams, etc.) Technical debt arising from poor and outdated designs Adopt standardized diagramming frameworks ("diagram as code") such as Mermaid, PlantUML, and Architecture as Code tools. Enforce version control for diagrams, just like code.

Confidential Copyright © GenAI Risk Mitigation Strategies - 6 Risk Category Potential Risks IMPACT Mitigation Strategies Explainability Lack of understanding of how AI models arrive at decisions Erosion of Trust and acceptance: If users and stakeholders don't understand the reasoning behind AI-driven decisions, they may be hesitant to adopt and utilize these systems, especially in high-stakes applications Implement explainable AI (XAI) methods such as LIME(Local Interpretable Model-Agnostic Explanations), SHAP(SHapley Additive exPlanations), or counterfactual explanations to provide insights into model reasoning. Develop techniques to visualize and interpret model behavior, such as attention maps or decision trees. Document model architecture, training data, and decision-making processes to enhance transparency. Use a combination of explainability strategies taking into account context, domain specific knowledge and constraints. Verification Difficulty in verifying the correctness and reliability of AI models Misinformation and safety risks Establish robust testing and validation procedures for AI models, including unit tests, integration tests, and A/B testing. Erosion of Trust Use diverse datasets and scenarios for comprehensive testing, including edge cases and adversarial examples. Reputational damage Incorporate human review and feedback in the verification process to identify potential biases or errors.

Confidential Copyright © GenAI Risk Mitigation Strategies -7 Risk Category Potential Risks IMPACT Mitigation Strategies Jurisdiction and Regulatory Risks Varying legal and regulatory requirements across different jurisdictions Operational Complexities in managing compliance Conduct thorough research and analysis of the legal and regulatory landscape in each relevant jurisdiction. Uncertainty regarding the applicability of existing laws and regulations to AI Market entry barriers Engage with legal experts and regulatory bodies to ensure compliance with current and evolving requirements. Potential conflicts between different legal frameworks Increased legal and operational cost Develop a comprehensive regulatory strategy that addresses the specific requirements of each jurisdiction.