TCFPro24 Building Real-Time Generative AI Pipelines

https://princetonacm.acm.org/tcfpro/
18th Annual IEEE IT Professional Conference (ITPC)
Armstrong Hall at The College of New Jersey
Friday, March 15th, 2024 | 10:00 AM to 5:00 PM

IT Professional Conference at Trenton Computer Festival
IEEE Information Technology Professional Conference on Friday, March 15th, 2024

TCFPro24 Building Real-Time Generative AI Pipelines

Building Real-Time Generative AI Pipelines
In this talk, Tim will delve into the exciting realm of building real-time generative AI pipelines with streaming capabilities. The discussion will revolve around the integration of cutting-edge technologies to create dynamic and responsive systems that harness the power of generative algorithms.

From leveraging streaming data sources to implementing advanced machine learning models, the presentation will explore the key components necessary for constructing a robust real-time generative AI pipeline. Practical insights, use cases, and best practices will be shared, offering a comprehensive guide for developers and data scientists aspiring to design and implement dynamic AI systems in a streaming environment.

Tim will show a live demo showing we can use Apache NiFi to provide a live chat between a person in Slack and several LLM models all orchestrated with Apache NiFi, Apache Kafka and Python. We will use RAG against Chroma and Pinecone vector data stores, Hugging Face and WatsonX.AI LLM, and add additional context with NiFi lookups of stocks, weather and other data streams in real-time.

Timothy Spann
Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming.

Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark.

Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.

Transcript

1. © 2024 Cloudera, Inc. All rights reserved. Building Real-Time Generative AI Pipelines Tim Spann Principal Developer Advocate March 15, 2024
2. © 2024 Cloudera, Inc. All rights reserved.
3. © 2024 Cloudera, Inc. All rights reserved. 3 This week in Apache NiFi, Apache Flink, Apache Kafka, ML, AI, Apache Spark, Apache Iceberg, Python, Java and Open Source friends. https://bit.ly/32dAJft https://www.meetup.com/futureofdata- princeton/ FLaNK Stack Weekly by Tim Spann
4. © 2024 Cloudera, Inc. All rights reserved. 4 Confidential—Restricted @PaasDev https://www.meetup.com/futureofdata-princeton/ From Big Data to AI to Streaming to Containers to Cloud to Analytics to Cloud Storage to Fast Data to Machine Learning to Microservices to ... Future of Data - NYC + NJ + Philly + Virtual https://linktr.ee/tspannhw
5. © 2024 Cloudera, Inc. All rights reserved. 5 Tim Spann Twitter: @PaasDev Blog: datainmotion.dev Principal Developer Advocate Princeton Future of Data Meetup ex-Pivotal, ex-Hortonworks, ex-StreamNative, ex-HPE, ex-PwC, ex-EY. https://medium.com/@tspann https://github.com/tspannhw
6. © 2024 Cloudera, Inc. All rights reserved. 6 Some common Vector DBs Open Community & Open Models RAPID INNOVATION IN THE LLM SPACE Too much to cover today.. but you should know the common LLMs, Frameworks, Tools Notable LLMs Closed Models Open Models GPT3.5 GPT4 Llama2 Mistral7B Mixtral8x7B Claude2 ++ 100s more… check out the HuggingFace LLM Leaderboard (pretrained, domain fine-tuned, chat models, …) Code Llama Popular LLM Frameworks When to use one over the other? Use Langchain if you need a general-purpose framework with flexibility and extensibility. Consider LlamaIndex if you’re building a RAG only app (retrieval/search) Langchain is a framework for developing apps powered by LLMs • Python and JavaScript Libraries • Provides modules for LLM Interface, Retrieval, & Agents LLamaIndex is a framework designed specifically for RAG apps • Python and JavaScript Libraries • Provides built in optimizations / techniques for advanced RAG HuggingFace is an ML community for hosting & collaborating on models, datasets, and ML applications • Latest open source LLMs are in HuggingFace • + great learning resources / demos https://huggingface.co/ Open Source vs Self Hosted vs SaaS option
7. © 2024 Cloudera, Inc. All rights reserved. 7 Enterprise Knowledge Base / Chatbot / Q&A - Customer Support & Troubleshooting - Enable open ended conversations with user provided prompts Code assistant: - Provide relevant snippets of code as a response to a request written in natural language. - Assist with creating test cases and synthetic test data. - Reference other relevant data such as a company’s documentation to help provide more accurate responses. Social and emotional sensing - Gauge emotions and opinions based on a piece of text. - Understand and deliver a more nuanced message back based on sentiment. ENTERPRISE WIDE USE CASES FOR AN LLM Classification and Clustering - Categorize and sort large volumes of data into common themes and trends to support more informed decision making. Language Translation - Globalize your content by feeding web pages through LLMs for translation. - Combine with chatbots to provide multilingual support to your customer base. Document Summarization - Distill large amounts of text down to the most relevant points. Content Generation - Provide detailed and contextually relevant prompts to develop outlines, brainstorm ideas and approaches for content. L Adoption dependent upon an Enterprise’s risk tolerance, restrictions, decision rights and disclosure obligations.
8. © 2024 Cloudera, Inc. All rights reserved. NLP / AI / LLM Generative AI
9. 9 Which Model and When? Use the right model for right job: closed or open-source Closed Source Usage can easily scale but so can your costs Rapidly improving AI models Most advanced AI models Excel at more specialized tasks Great for a wide range of tasks Open Source Better cost planning Compliance, privacy, and security risks More control over where & how models are deployed
10. 10 Adoption of Generative AI is a Journey Identifying AI challenges in the enterprise Data integration barriers • Streamlined access to enterprise data Rigid model infrastructure • Modularity • Flexibility • AI Ops Lack of security and transparency • Model control • Built-in security • Visibility & governance What’s missing Challenges
11. 11 Data = Organization Context Your data enables contextually accurate responses from LLMs Large Language Model User Query Contextually Inaccurate Response Data Organization Context User Query Large Language Model Contextually Accurate Response
12. © 2024 Cloudera, Inc. All rights reserved. 12 CLOSED-SOURCE FOUNDATION MODELS MODEL HUBS OPEN SOURCE FOUNDATION MODELS FINE-TUNED MODELS PRIVATE VECTOR STORE MANAGED VECTOR STORE CLOUD INFRASTRUCTURE Milvus, Solr* Meta (Llama 2) Applied Machine Learning Prototypes (AMPs) Hugging Face Pinecone SPECIALIZED HARDWARE APIs: OpenAI (GPT-4 Turbo) Amazon Bedrock: Anthropic (Claude 2), Cohere… DATA WRANGLING REAL-TIME DATA INGEST & ROUTING AI MODEL TRAINING & INFERENCE DATA STORE & VISUALIZATION Open Data Lakehouse DATA WRANGLING REAL-TIME DATA INGEST & ROUTING AI MODEL TRAINING & SERVING DATA STORE & VISUALIZATION AI APPLICATIONS
13. Live Q&A Travel Advisories Weather Reports Documents Social Media Databases Transactions Public Data Feeds S3 / Files Logs ATM Data Live Chat … ARCHITECTURE INTERACT COLLECT STORE ENRICH, REPORT Distribute Collect Report REPORT Visualize Report, Automate AI BASED ENHANCEMENTS Predict, Automate VECTOR DATABASE LLM Machine Learning Data Visualization Data Flow Data Warehouse SQL Stream Builder Data Visualization Input Sentences Generated Text Timestamp Input Sentence Timestamps Enrichments Messaging Broker Real-time alerting Real-time alerting Aggregations
14. © 2024 Cloudera, Inc. All rights reserved. DATAFLOW / STREAMING
15. LLM USE CASE Vector DB AI Model Unstructured file types Data in Motion on Cloudera Data Platform (CDP) Capture, process & distribute any data, anywhere Other enterprise data Open Data Lakehouse Materialized Views Structured Sources Applications/API’s Streams
16. https://medium.com/cloudera-inc/getting-ready-for-apache-nifi-2-0-5a5e6a67f450 NiFi 2.0.0 Features • Python Integration • Parameters • JDK 21+ • JSON Flow Serialization • Rules Engine for Development Assistance • Run Process Group as Stateless • flow.json.gz https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Release+Goals
17. 17 DataFlow Pipelines Can Help External Context Ingest Ingesting, routing, clean, enrich, transforming, parsing, chunking and vectorizing structured, unstructured, semistructured, binary data and documents Prompt engineering Crafting and structuring queries to optimize LLM responses Context Retrieval Enhancing LLM with external context such as Retrieval Augmented Generation (RAG) Roundtrip Interface Act as a Discord, REST, Kafka, SQL, Slack bot to roundtrip discussions
18. 18 UNSTRUCTURED DATA WITH NIFI • Archives - tar, gzipped, zipped, … • Images - PNG, JPG, GIF, BMP, … • Documents - HTML, Markdown, RSS, PDF, Doc, RTF, Plain Text, … • Videos - MP4, Clips, Mov, Youtube URL… • Sound - MP3, … • Social / Chat - Slack, Discord, Twitter, REST, Email, … • Identify Mime Types, Chunk Documents, Store to Vector Database • Parse Documents - HTML, Markdown, PDF, Word, Excel, Powerpoint
19. 19 CLOUD ML/DL/AI/Vector Database Services • Cloudera ML • Amazon Polly, Translate, Textract, Transcribe, Bedrock, … • Hugging Face • IBM Watson X.AI • Vector Stores Anywhere: Weaviate, Pinecone, Milvus, Chroma DB, SOLR, …
20. 20 https://medium.com/cloudera-inc/google-gemma-for-real-time-lightweight-open-llm-infe rence-88efe98e580f
21. © 2024 Cloudera, Inc. All rights reserved.
22. © 2024 Cloudera, Inc. All rights reserved. Python Processors
23. © 2024 Cloudera, Inc. All rights reserved. Extract Company Names • Python 3.10+ • Hugging Face, NLP, SpaCY, PyTorch https://github.com/tspannhw/FLaNK-python-ExtractCompanyName-processor
24. © 2024 Cloudera, Inc. All rights reserved. WatsonX SDK To Foundation • Python 3.10+ • LLM • WatsonX.AI Foundation Models • Inference • Secure • Official SDK from IBM https://github.com/tspannhw/FLaNK-python-watsonx-processor
25. © 2024 Cloudera, Inc. All rights reserved. CaptionImage • Python 3.10+ • Hugging Face • Salesforce/blip-image-captioning-large • Generate Captions for Images • Adds captions to FlowFile Attributes • Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
26. © 2024 Cloudera, Inc. All rights reserved. RESNetImageClassification • Python 3.10+ • Hugging Face • Transformers • Pytorch • Datasets • microsoft/resnet-50 • Adds classification label to FlowFile Attributes • Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
27. © 2024 Cloudera, Inc. All rights reserved. NSFWImageDetection • Python 3.10+ • Hugging Face • Transformers • Falconsai/nsfw_image_detection • Adds normal and nsfw to FlowFile Attributes • Gives score on safety of image • Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
28. © 2024 Cloudera, Inc. All rights reserved. FacialEmotionsImageDetection • Python 3.10+ • Hugging Face • Transformers • facial_emotions_image_detection • Image Classification • Adds labels/scores to FlowFile Attributes • Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
29. © 2024 Cloudera, Inc. All rights reserved. Other Python Processors • Chunk Document, Parse Document • Prompt Chat GPT • Put Chroma, Query Chroma • Put Pinecone, Query Pinecone
30. 30 DEMO
31. Startup Grind AI Max Summit - April 12 NJ
32. 32 TH N Y U
33. CLOUDERA STREAM PROCESSING Two Major Capabilities: Enterprise Messaging and Powerful Stream Processing Enterprise grade messaging products for Apache Kafka. Streams Messaging Manager to monitor/operate clusters, Streams Replication Manager for HA/DR deployments, Schema Registry for centralized schema management, and support for Kafka Connect and Cruise Control Cloudera Streaming Analytics (CSA) Powered By Apache Flink Cloudera Streams Messaging (CSM) Powered by Apache Kafka Powered by Apache Flink with SQL StreamBuilder, it provides low-latency stream processing capabilities with advanced windowing & state management made simple with SQL
34. ENTERPRISE MANAGEMENT CAPABILITIES FOR APACHE KAFKA Extend streams messaging services for Schema Mgmt, Replication & Monitoring Schema Registry Kafka Schema Governance Streams Replication Manager Kafka Replication Service for Disaster Recovery Streams Messaging Manager Management & Monitoring Service for all of your Kafka clusters
35. Kafka Data Movement, Operations and Security Made Easier ENTERPRISE MANAGEMENT CAPABILITIES FOR APACHE KAFKA Kafka Connect Support Simple Data Movement Change Data Capture Connectors Build Custom Connectors with NiFi Ranger Security Improved ACL and Audit for Kafka, KConnect and Schema Registry Cruise Control Support Intelligent Rebalancing & Self-Healing of your Kafka Clusters
36. CLOUDERA STREAM PROCESSING Two Major Capabilities: Enterprise Messaging and Powerful Stream Processing Enterprise grade messaging products for Apache Kafka. Streams Messaging Manager to monitor/operate clusters, Streams Replication Manager for HA/DR deployments, Schema Registry for centralized schema management, and support for Kafka Connect and Cruise Control Cloudera Streaming Analytics (CSA) Powered By Apache Flink Cloudera Streams Messaging (CSM) Powered by Apache Kafka Powered by Apache Flink with SQL StreamBuilder, it provides low-latency stream processing capabilities with advanced windowing & state management made simple with SQL
37. NEXT GENERATION STREAMING ANALYTICS WITH APACHE FLINK Low latency stateful stream processing • Flink is a distributed data processing systems ideally suited for real-time, event driven applications. • Unifies stream and batch processing • Advanced features - late arriving data, checkpointing, event time processing, Exactly Once Processing Real-Time Insights Event Processing Low Latency
38. SQL STREAM BUILDER (SSB) SQL STREAM BUILDER allows developers, analysts, and data scientists to write streaming applications with industry standard SQL. No Java or Scala code development required. Simplifies access to data in Kafka & Flink. Connectors to batch data in HDFS, Kudu, Hive, S3, JDBC, CDC and more Enrich streaming data with batch data in a single tool Democratize access to real-time data with just SQL
39. © 2024 Cloudera, Inc. All rights reserved. 39 LLMs ARE FOUNDATION MODELS Base models that can be adapted for a wide range of use cases Terabytes of Data (Multiple Formats) Foundation Models (Billions of Parameters) Train Adapt Question/Answering Sentiment Analysis Doc summarization … ++ more ➔ Historically, data scientists trained specialized models against narrow datasets to solve specific tasks. ➔ LLMs are Foundation models that can be adapted to perform a variety of tasks. ◆ It is faster to “adapt” a foundation model than it is to train a specialized model from scratch ◆ Decouples “knowledge” from “intelligence” ◆ Opens up AI use cases to software developers (instead of just specialised data scientists)