Case Study for Repurposing Video Content With Generative AI Kazuki Miura Hokkaido Television Broadcasting Co., Ltd. Sonu Kim Serverless Operations, Inc.

Speakers Kazuki Miura Hokkaido Television Broadcasting Co., Ltd. Sonu Kim Serverless Operations, Inc. Media-JAWS Organizer AWS User Group - Japan focusing on Media industry

Do You Know This TV Program?

Key Concepts to know to start with today's talk

Amazon Bedrock

APIs provided by AWS Access to various generative AI models Pricing model On-demand by token consumption Provisioned mode Integration convenience with other AWS services Testing convenience with various models Why we use Amazon Bedrock?

Various models available in Amazon Bedrock As of September 16, 2024, us-west-2 region

A simple Q&A design using Amazon Bedrock AWS SDK (boto3) Amazon Bedrock Claude3 AWS Credentials (SigV4) Simple design that calls API provided by Amazon Bedrock AWS SDK is available in the same way as other AWS services

What is Retrieval-Augmented Generation (RAG)? Questioner Ask Answer 1.Similarity search 2.Extended (Augmented) context Vector store (Knowledge contents) 3.Question+Search +Augmented Context 4.Generated Answer LLM (Text generative AI Model)

Generative AI’s use cases

Questions about use cases of generative AI Can text generation AI help us in our daily work? Can we use it in revenue-generating services?

Exploring use cases for text generative AI Search engine/solutions Internal document land history in real estate and construction industries Providing Q&A functionality for customers Offering product-specific Q&A based on user manuals Inquiries from seminar or lecture content Guidance on documentation processes in HR New hire onboarding Year-end tax adjustments Various administrative procedures

Case Study: “SODANE” HTB’s Blog Platform

What is “SODANE”?

What is “SODANE”? Owned media site with 1,000 blog posts updated annually Converting video content from one-time broadcasts and cooking shows into blog articles manually Content creation workload 30% of articles written by TV program staff 70% created by the media operations team Managing this could not be easy alongside their primary tasks

What Inspired Us to Use Gen AI for Blog Writing After 5 years of blogging and just as ad revenue began to increase, the key team member was reassigned Content creation does not start from a zero-bases, as articles are based on previously aired programs Problem-solving criteria Maintain the pace of blog posting even with new members joining Reduce dependency on specific individuals Establish a systematic approach to media operations

If the article has to be written based on the video, It should be possible to transcribe the audio to text (though it is NOT 100% accurate) Could a text-generating AI be used to proofread the transcription, and create a blog post with the content summarized? Our Breakthrough Idea Amazon Transcribe Amazon Bedrock Audio

1.Upload video file 2.Transcribe 3.Capture images 4. Place images for each paragraphs Technical Challenge How to capture screenshots from a video and specify them to be relevant to the article content Video capture images must be placed to match each paragraph of the article as a MUST requirement. ? ? Amazon S3 Amazon Transcribe

Solving Technical Challenge Through Advanced RAG design 1. Take multiple captures at intervals of a few seconds in advance 2. Use LLM capable of image input to convert the images into text description and embed them into vector store 3. When creating the article, retrieve image descriptions from vector store, and let the LLM select the specific images to use. Take source video captures at intervals of few seconds Load images Convert to text Save (Embedding) Retrieve text description Generate article

A portion of the actual prompt used (for text conversion of captured images) You are an AI assistant designed to provide detailed descriptions of captured images from videos. After describing the image, remember which number this image is and which article ID it is associated with. Also, use numbering notation to clearly indicate which number description this image is. The response must be in the following format: --- Article ID: *Article ID received from the question* Image number: *Image number received from the question* Description: *Description of the image* --- Now, please describe the attached image.

A portion of the actual prompt used (for article generation) I would like to request the creation of a blog post with images in Japanese. Please structure the content of the article as a blog post composed of multiple paragraphs, based on the text of the question including transcription content. Each paragraph should be as detailed as possible while remaining easy to read. Please specify the image numbers that correspond to the article ID, matching them to the appropriate content. {context} Please specify the image numbers in ascending order for each paragraph, ensuring there are no duplicates across all paragraphs. Furthermore, you must only use image numbers from the following list. You are not allowed to use any image numbers that are not listed below. {capturedImages} For each paragraph, please indicate the image number using the following URL format. The actual image number will be inserted where it says 【imageNumber】. https://sample.cloudfront.net/{article_id}/【imageNumber】.jpg Lastly, please provide a title for the article. Now, please begin the task. Please write a blog article with the following contents:

Technical Challenge problem-solved design 1.Upload video file 2.Transcribe 3.Capture images 4. Convert & Embed image as text 5. Retrieve and generate article Amazon S3 Amazon Transcribe Vector store Claude 3

Overall System Architecture Momento: External Solution for real-time messaging & Vector store

Step 1 : Upload video file

Step 2 : Transcription

Step 3 : Create Capture Image

Step 4: Create Captured Image description

Step 5: Generate article

Step 5: Generate article

How to use & Outcomes Create 7-8 AI-generated articles in advance for each program Review & proofread by writers before updating the blog. Average article creation time reduced by over 50%

Showcased at AWS Summit Tokyo 2024 as a case study Case studies of customers using generative AI in production environments

Thank you! Kazuki Miura Hokkaido Television Broadcasting Co., Ltd. Sonu Kim Serverless Operations, Inc.