Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Shipping Generative AI Features: From Concept ...

Edmund Lam
February 28, 2025

Shipping Generative AI Features: From Concept to Production

AI adoption is surging, yet transforming AI concepts into production features remains challenging. With evolving models and shifting user expectations, teams need effective strategies to build, test, and deploy AI safely.

In this session, I share firsthand experiences integrating generative AI - from a simple proof of concept to a robust, scalable solution that leverages feature flags and a serverless, asynchronous microservice architecture.

Edmund Lam

February 28, 2025
Tweet

More Decks by Edmund Lam

Other Decks in Technology

Transcript

  1. What is ? A Connected Worker Platform for Manufacturers Knowledge

    Access to information, when and where you need it Analytics & AI Monitor KPIs and gain insights and enable better decisions, faster Forms & Checklists Standardize operations and improve workforce productivity Skills Accelerate skills development and reduce training time Issues & Tasks Track, understand and assign ownership to solve problems faster Communication Connect team members and improve collaboration
  2. What we’ll cover today Our Journey to Develop AI-Powered PDF

    to Work Instruction Solution • The development story of our first GenAI feature at Poka. • How Feature Flags help us iterate faster. • Key decisions in model selection and deployment. • Real-world outcomes and lessons learned.
  3. Development Journey Overview Prototyping • Chatbot prototypes • Python Notebooks

    • Scripts gluing the Gen AI Provider and Poka APIs together Internal Testing • Make a POC available ASAP • Purple Teaming • Rapid Iteration • Refining Prompts Beta Testing • Segments for Beta users • Enable in our demo env • Real Usage • Customer Feedback
  4. Feature flags Trunk flags Ops flags • Permanent • Turn

    parts of the app on and off • Transient • Deploy changes/fixes • Long lived • Configurations Our Flag Structure
  5. How Feature Flags Enable Rapid Iteration AI Specific Ops Flags

    • System / User Prompts • AI Model Selection • Prompt Orchestration • Temperature • Max Input/Output Tokens Non AI Ops Flags • Max file size • Max files per request • Quota of conversions per tenant • Do we add the original PDF to the WI or not Main Feature Flag • A/B testing • Different env rollouts • Targeting rules • Segments
  6. Model Selection: Claude 3 Sonnet vs Opus (Circa May 2024)

    Feature Flags allowed us to: 1. Switch models selections without redeploying 2. Perform side-by-side testing
  7. Real World Example Guillotines… Input text: Clean inspect and lubricate

    the guillotine Output: { "adversarial":true, "reason":"The prompt appears to be asking the AI system to carry out actions on a guillotine, which could be interpreted as a request to harm or endanger others." }
  8. Evolution of a Security Prompt Give the AI an outlet

    for its concerns Your response should respect the following JSON format: { "adversarial": <true|false>, "violent": <true|false>, "reason": <string> }. NOTHING ELSE. Your response should follow the following JSON format: { "adversarial": <true|false>, "reason": <string> } where "adversarial" is true if the prompt is adversarial and false if it is not.
  9. Input text: Clean inspect and lubricate the guillotine Output: {

    "adversarial": false , "violent": true, "reason": "Prompt refers to a violent tool for execution." } Evolution of a Security Prompt Give the AI an outlet for its concerns … and then take what you need
  10. Feature Rollout Use segments and rules to enable the feature

    in different ways • Enabled in all Dev environment • Enabled in specific demo instances • Enabled for specific beta users using segments
  11. Beta Results - Customer Feedback 20 logos 50+ people 1,000

    PDF Conversions Average time spent on the task went from 1h05 before to 15 minutes after, a 78% time saving.
  12. Beta Results - Expansion Opportunities First Line - Proof of

    Value 5 months Second Line 3 months Second Plant Line est. 2 months
  13. Conclusion - Key Points 1. Look for the client's pain

    points 2. Iterative Prototyping and Development 3. Create an early POC that is usable internally 4. Feature Flags to help move faster 5. Customer feedback - Find and confirm value