02-ad-breaks-and-contextual-ad-targeting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c0b82119-b336-4f19-af09-56827ed617af",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Ad break detection and contextual Ad targeting\n",
    "\n",
    "Contextual advertising is a form of targeted advertising where the advertisement is matched to the context of the webpage or media being consumed by the user. This process involves three key players: the publisher (website or content owner), the advertiser, and the consumer. Publishers provide the platform and content, while advertisers create ads tailored to the context. Consumers engage with the content, and relevant ads are displayed based on the context, creating a more personalized and relevant advertising experience.\n",
    "\n",
    "\n",
    "\n",
    "A challenging area of contextual advertising is inserting ads in media content for streaming on video on demand (VOD) platforms. This process traditionally relied on manual tagging, where human experts analyzed the content, identified breaks in the narrative and assigned relevant keywords or categories. However, this approach is time-consuming, subjective, and may not capture the full context or nuances of the content. Traditional AI/ML solutions can automate this process, but they often require extensive training data and can be expensive and limited in their capabilities.\n",
    "\n",
    "![Ad decisions](./static/images/02-ad-breaks.jpg)\n",
    "\n",
    "Generative AI, powered by large language models, offers a promising solution to this challenge. By leveraging the vast knowledge and contextual understanding of these models, publishers can automatically generate contextual insights and taxonomies for their media assets. This approach streamlines the process and provides accurate and comprehensive contextual understanding, enabling effective ad targeting and monetization of media archives.\n",
    "\n",
    "When you are done with this part of the workshop, you'll have created the following metadata for a video:\n",
    "* a list of high quality ad placement opportunities  or _breaks_ available in the video\n",
    "* contextual information for the video before and after each break, including classification using the IAB Content Taxonomy that is used by advertisers to classify content for automated placement using Ad Decision Servers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ca97816-35b5-40c4-b194-ba88c68b0222",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88dadf3d-559a-4a5a-9e9d-509f9139a0b5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "To run this notebook, you need to have run all the previous foundation notebooks, where you set up the notebook environment and segmented the video using audio, visual and semantic information:\n",
    "1. [00-prerequisites.ipynb](00-prerequisites.ipyn)\n",
    "2. [01A-visual-segments-frames-shots-scenes.ipynb](01A-visual-segments-frames-shots-scenes.ipynb) \n",
    "3. [01B-audio-segments.ipynb](01B-audio-segments.ipynb) nts-frames-shots-scenes.ipynb) \n",
    "3. [01B-audio-segments.ipynb](01B-audio-segments.ipynb) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "432f3368-b3ea-406d-bc46-328e3e322d8f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Import python packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e9d4c51-6580-4386-a398-df098834657b",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import os\n",
    "import json\n",
    "import json\n",
    "import boto3\n",
    "from botocore.exceptions import ClientError\n",
    "import json_repair\n",
    "import copy\n",
    "from termcolor import colored\n",
    "from IPython.display import JSON\n",
    "from IPython.display import Video\n",
    "from IPython.display import Pretty\n",
    "from IPython.display import Image as DisplayImage\n",
    "from lib.frames import VideoFrames\n",
    "from lib.shots import Shots\n",
    "from lib.scenes import Scenes\n",
    "from lib.transcript import Transcript\n",
    "#from lib.chapters import Chapters\n",
    "from lib import frame_utils\n",
    "from lib import util\n",
    "from PIL import Image, ImageDraw, ImageFont\n",
    "from io import BytesIO\n",
    "import copy\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2636a0da-4572-4c3b-8b80-4ee2ed465181",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Retrieve saved values from previous notebooks\n",
    "To run this notebook, you need to have run the previous notebook: 00_prerequisites.ipynb, where you installed package dependencies and gathered some information from the SageMaker environment.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d96940dd-7733-4a7d-8782-8c7779f15432",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "store -r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04e3677d-8b4c-422b-afa0-f633d2b86d51",
   "metadata": {},
   "source": [
    "# Architecture\n",
    "\n",
    "This hands-on workflow uses AWS services from SageMaker.  It takes scenes, conversation topics and advertising content taxonomies as inputs and produces contextual ad breaks and chapter segments as outputs.\n",
    "\n",
    "\n",
    "![Contextual Ads workflow with outputs](./static/images/02-contextual-ads-workflow-w-outputs-drawio.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25bb30cd-62be-40e5-8370-db452a029a06",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "\n",
    "# Find ad placement opportunities by aligning scenes and topics to identify chapters in the narrative\n",
    "\n",
    "In the [Video segmentation notebook](video-understanding-with-generative-ai-on-aws-main/01-video-time-segmentation.ipynb), we have separately processed the visual and audio cues from the video. Now, we will do one more step to bring them together and ensure that the transcription topics align with the scenes. The last thing you want is to insert an ad during an ongoing conversation or scene. To create alignment, we will iterate over each conversational topic, represented by its start and end timestamps, and a text description summarizing the topic. For each topic, the code identifies the relevant video scenes that overlap or fall within the topic's timestamp range. The output of this process is a list of chapters, where each chapter contains a list of scene IDs representing the video scenes that align with the corresponding audio conversation. After the alignment process, we have combined visual and audio cues into the final chapters. The breaks between chapters are ideal places for ad insertion because they occur between contextual changes in the content of the video. \n",
    "\n",
    "In real-world applications, we recommend surfacing these breaks as suggestions to the operator and having a human-in-the-loop step to confirm the final ad placements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9437b9ed-1bb4-4522-8e47-249388ac7719",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Chapters:\n",
    "    def __init__(self, topics, scenes, frames):\n",
    "        self.video_asset_dir = frames.video_asset_dir()\n",
    "        self.chapters = self.align_scenes_in_chapters(topics, scenes, frames)\n",
    "        \n",
    "    def align_scenes_in_chapters(self, topics, scenes, frames):\n",
    "        \"\"\"\n",
    "        Aligns video scenes with conversation topics to create chronological chapters.\n",
    "    \n",
    "        Args:\n",
    "            topics: List of conversation topics with start_ms, end_ms, and reason\n",
    "            scenes: List of scene metadata with start_ms and end_ms\n",
    "            frames: List of video frame metadata\n",
    "    \n",
    "        Returns:\n",
    "            List of chapters, each containing aligned scenes and associated text\n",
    "    \n",
    "        Note:\n",
    "            - Handles scenes without conversations\n",
    "            - Merges overlapping topics\n",
    "            - Preserves chronological order\n",
    "            - Creates Chapter objects for each segment\n",
    "    \"\"\"\n",
    "        scenes = copy.deepcopy(scenes)\n",
    "    \n",
    "        chapters = []\n",
    "        for topic in topics:\n",
    "            \n",
    "            topic_start_ms = topic['start_ms']\n",
    "            topic_end_ms = topic['end_ms']\n",
    "            text = topic['reason']\n",
    "\n",
    "            # find all the frames that align with the conversation topic\n",
    "            stack = []\n",
    "            while len(scenes) > 0:\n",
    "                scene = scenes[0]\n",
    "                frame_start = scene['start_ms']\n",
    "                frame_end = scene['end_ms']\n",
    "\n",
    "                \n",
    "                if frame_start > topic_end_ms:\n",
    "                    # topic overlaps scenes that belong to previous topic - merge the text\n",
    "                    if not stack:\n",
    "                        num_chapters = len(chapters)\n",
    "                        if num_chapters > 0:\n",
    "                            chapters[num_chapters-1]['text'] = chapters[num_chapters-1]['text'] + ' ' + text\n",
    "                        \n",
    "                    break\n",
    "    \n",
    "                # scenes before any conversation starts\n",
    "                if frame_end < topic_start_ms:\n",
    "                    chapter = Chapter(len(chapters), [scene], frames).__dict__\n",
    "                    chapters.append(chapter)\n",
    "                    scenes.pop(0)\n",
    "                    continue\n",
    "    \n",
    "                stack.append(scene)\n",
    "                scenes.pop(0)\n",
    "    \n",
    "            if stack:\n",
    "                chapter = Chapter(len(chapters), stack, frames, text).__dict__\n",
    "                chapters.append(chapter)\n",
    "    \n",
    "        ## There could be more scenes without converations, append them\n",
    "        for scene in scenes:\n",
    "            chapter = Chapter(len(chapters), [scene], frames).__dict__\n",
    "            chapters.append(chapter)\n",
    "    \n",
    "        return chapters\n",
    "\n",
    "class Chapter:\n",
    "    def __init__(self, chapter_id, scenes, frames, text = ''):\n",
    "        self.scene_ids = [scene['id'] for scene in scenes]\n",
    "        self.start_frame_id = scenes[0]['start_frame_id']\n",
    "        self.end_frame_id = scenes[-1]['end_frame_id']\n",
    "        self.start_ms = scenes[0]['start_ms']\n",
    "        self.end_ms = scenes[-1]['end_ms']\n",
    "        self.id = chapter_id\n",
    "        self.text = text\n",
    "        #folder = os.path.join(frames.video_asset_dir(), 'chapters')\n",
    "        #os.makedirs(folder, exist_ok=True) \n",
    "        self.composite_images = frames.create_composite_images(frames.frames[self.start_frame_id:self.end_frame_id+1], 'chapters', prefix=\"chapter_\")\n",
    "        \n",
    "        return \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1339c1bf-6ade-4a05-868d-069424bccdf0",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "video['chapters'] = Chapters(video['topics'], video['scenes'].scenes, video['frames'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03f341c0-1e68-487a-b1fd-ad6323bc7214",
   "metadata": {},
   "source": [
    "Examine the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f30eb92-f37f-46f5-98b7-1d238056785a",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "display(JSON(video['chapters'].chapters))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be3b4ffc-bc67-4a1f-8749-362815e1154a",
   "metadata": {},
   "source": [
    "#### Visualize the chapters\n",
    "\n",
    "Now let's visualize the frames and the text for each of the chapters.  These will be the input to our prompt to generate contextual information for the ad breaks. Note that some chapters will not have any text associated with them.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "💡 Use the scroll bar in the output box to view the chapters.  Some chapters contain more frames than can fit on a single composite image, so the may be multiple composite images displayed for each chapter.\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6036728d-ea60-47a3-9167-d2f5949f9b96",
   "metadata": {
    "editable": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# visualize the chapters\n",
    "\n",
    "STOP=10\n",
    "for counter, b in enumerate(video[\"chapters\"].chapters):\n",
    "    print(f'\\nChapter {counter}: frames {b[\"start_frame_id\"] } to {b[\"end_frame_id\"] }, scenes { b[\"scene_ids\"][0] } to { b[\"scene_ids\"][-1] }, time { b[\"start_ms\"]} to { b[\"end_ms\"] } =======\\n')\n",
    "    if len(b[\"text\"]) > 0: \n",
    "        print(f'\\nChapter Text: { b[\"text\"] }')\n",
    "    else:\n",
    "        print(f'\\nChapter Text (conversation topic): None')\n",
    "\n",
    "    video['frames'].display_frames(start=b['start_frame_id'], end=b['end_frame_id']+1)\n",
    "\n",
    "    # ALTERNATIVE: view the composite images that will be used in prompts\n",
    "    #for image_file in b['composite_images']:\n",
    "    #    display(DisplayImage(filename=image_file['file'], height=100))\n",
    "    #if counter == STOP:\n",
    "    #    break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81e484ed-cf3c-41a5-9bab-771c56fa47f6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Generate chapter level contextual information \n",
    "\n",
    "The last step is to send both the visually and audio-aligned data to Claude 3 Sonnet to generate contextual information for each topic. This approach that takes advantage of the multimodal capabilities of the Claude 3 family of models. From our testing, these models have demonstrated the ability to capture minute details from large images and follow image sequences when provided with appropriate instructions.\n",
    "\n",
    "To prepare the input for Claude3 Sonnet, we first assemble video frames associated with each topic and create a composite image grid. Through our experimentation, we have found that the optimum image grid ratio is 7 rows by 4 columns, which will assemble a 1568 x 1540 pixel image that fits under Claude's 5 MB image file size limit while still preserving enough detail in each individual frame tile. Furthermore, you can also assemble multiple images if needed.\n",
    "\n",
    "Subsequently, the composite images, the transcription, the IAB Content taxonomy definitions, and GARM taxonomy definitions are fed into the prompt to generate descriptions, sentiment, IAB taxonomy, GARM taxonomy, and other relevant information in a single query to the Claude3 Haiku model. Not only that, but we can adapt this approach to any taxonomy or custom labeling use cases without the need to train a model each time. This is where the true power of this approach lies. The final output can be presented to a human reviewer for final confirmation if needed. Here is an example of a composite image grid and the corresponding contextual output for a specific topic.\n",
    "\n",
    "![Contextualized chapters](./static/images/02-chapter-contextualization.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64e01a83-55cf-4ab7-944c-a6c9d3448414",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Download the IAB Content Taxonomy definition\n",
    "\n",
    "The IAB (Interactive Advertising Bureau) Taxonomy is a standardized classification system for digital advertising content and audiences. It provides a hierarchical structure to categorize digital content, making it easier for advertisers and publishers to organize, target, and measure digital advertising.\n",
    "\n",
    "You will instruct Anthropic Claude to use this taxonomy to classify the chapters to help identify the kinds of advertisements that might fit between different chapters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77fdff3d-466b-4d0e-8f3e-6df0188f29e5",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "iab_file = 'iab_content_taxonomy_v3.json'\n",
    "url = f\"https://dx2y1cac29mt3.cloudfront.net/iab/{iab_file}\"\n",
    "\n",
    "!curl {url} -o {iab_file}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d84fd39-1e6a-4c2f-861f-ae0a3f87d43e",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def load_iab_taxonomy(file):\n",
    "    \"\"\"\n",
    "    Loads IAB taxonomy definitions from a JSON file.\n",
    "    Args:\n",
    "        file: Path to the IAB taxonomy JSON file\n",
    "    Returns:\n",
    "        Dictionary containing IAB taxonomy definitions\n",
    "    \"\"\"\n",
    "    with open(file) as f:\n",
    "        iab_taxonomies = json.load(f)\n",
    "    return iab_taxonomies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "514bd101-affd-4630-a192-bee997980abb",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "iab_taxonomy = load_iab_taxonomy(iab_file)\n",
    "display(JSON(iab_taxonomy))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9efdb888-fb4b-41ea-b85f-59be51f89e93",
   "metadata": {},
   "source": [
    "## Construct a prompt to generate contextual metadata for each chapter segment\n",
    "\n",
    "This example uses the [Anthropic Claude Messages API (aka Conversations API)](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages.html) with Amazon Bedrock for this prompt that simulates a multi-turn conversation with the Foundation model. \n",
    "\n",
    "First, let's create some helper functions to create text for parts of the prompts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8b9cdc1-1e9b-486b-b284-eb8deefd405a",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Constructors for parts of prompt messages\n",
    "\n",
    "def make_iab_taxonomoies(iab_list):\n",
    "    iab = [item['name'] for item in iab_list]\n",
    "    iab.append('None')\n",
    "\n",
    "    return iab\n",
    "\n",
    "def make_image_message(composite_images):\n",
    "    \"\"\"\n",
    "    Converts a list of image files into a formatted message with base64-encoded images.\n",
    "    Args:\n",
    "        composite_images: List of dicts containing image file paths\n",
    "    Returns:\n",
    "        Dict with 'role' and 'content' containing:\n",
    "        - Text description of number of images\n",
    "        - List of base64-encoded images with metadata\n",
    "    \"\"\"\n",
    "    # adding the composite image sequences\n",
    "    image_contents = [{\n",
    "        'type': 'text',\n",
    "        'text': 'Here are {0} images containing frame sequence that describes a scene.'.format(len(composite_images))\n",
    "    }]\n",
    "\n",
    "    open_images = []\n",
    "    for image in composite_images:\n",
    "        with open(image['file'], \"rb\") as image_file:\n",
    "            image_data = image_file.read()\n",
    "            open_images.append(image_file)\n",
    "        image_pil = Image.open(BytesIO(image_data))\n",
    "        bas64_image = frame_utils.image_to_base64(image_pil)\n",
    "        image_contents.append({\n",
    "            'type': 'image',\n",
    "            'source': {\n",
    "                'type': 'base64',\n",
    "                'media_type': 'image/jpeg',\n",
    "                'data': bas64_image\n",
    "            }\n",
    "        })\n",
    "\n",
    "    # close the images\n",
    "    for image in open_images:\n",
    "        image.close()\n",
    "\n",
    "    return {\n",
    "        'role': 'user',\n",
    "        'content': image_contents\n",
    "    }\n",
    "\n",
    "def make_output_example():\n",
    "    \"\"\"\n",
    "    Creates a template message for AI model output formatting.\n",
    "    Returns:\n",
    "        Dict with 'role' and 'content' keys containing example JSON structure for:\n",
    "        - Scene description\n",
    "        - Sentiment analysis\n",
    "        - IAB and GARM taxonomies\n",
    "        - Brand/logo detection\n",
    "        - Relevant tags\n",
    "    Note:\n",
    "        Used as part of the prompt to ensure consistent response formatting\n",
    "    \"\"\"\n",
    "    example = {\n",
    "        'description': {\n",
    "            'text': 'The scene describes...',\n",
    "            'score': 98\n",
    "        },\n",
    "        'sentiment': {\n",
    "            'text': 'Positive',\n",
    "            'score': 90\n",
    "        },\n",
    "        'iab_taxonomy': {\n",
    "            'text': 'Station Wagon',\n",
    "            'score': 80\n",
    "        },\n",
    "        'garm_taxonomy': {\n",
    "            'text': 'Online piracy',\n",
    "            'score': 90\n",
    "        },\n",
    "        'brands_and_logos': [\n",
    "            {\n",
    "                'text': 'Amazon',\n",
    "                'score': 95\n",
    "            },\n",
    "            {\n",
    "                'text': 'Nike',\n",
    "                'score': 85\n",
    "            }\n",
    "        ],\n",
    "        'relevant_tags': [\n",
    "            {\n",
    "                'text': 'auto racing',\n",
    "                'score': 95\n",
    "            }\n",
    "        ]            \n",
    "    }\n",
    "    \n",
    "    return {\n",
    "        'role': 'user',\n",
    "        'content': 'Return JSON format. An example of the output:\\n{0}\\n'.format(json.dumps(example))\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56f8187c-bc2e-458b-8abe-94822775f64a",
   "metadata": {},
   "source": [
    "This next block of code constructs the prompt using the helper functions and calls Amazon Bedrock to make the inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf3d5343-bb92-4def-a323-8c0b8db3e2c0",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "MODEL_ID = 'anthropic.claude-3-sonnet-20240229-v1:0'\n",
    "MODEL_PRICING = (0.003, 0.015)\n",
    "MODEL_VER = 'bedrock-2023-05-31'\n",
    "\n",
    "def get_chapter_description(images, text, iab_definitions):\n",
    "    \"\"\"\n",
    "    Generates chapter descriptions using image analysis, text, and IAB classifications.\n",
    "    Args:\n",
    "        images: List of image analysis results (max 19 images)\n",
    "        text: Transcribed conversation/text (optional)\n",
    "        iab_definitions: IAB taxonomy definitions with tier1 classifications\n",
    "    Returns:\n",
    "        Dict containing chapter description, IAB classifications, and sentiment analysis\n",
    "    Note:\n",
    "        - Uses Claude model for analysis\n",
    "        - Requires make_iab_taxonomies(), make_output_example(), make_image_message()\n",
    "        - Implements retry logic for failed inference calls\n",
    "    \"\"\"\n",
    "\n",
    "      \n",
    "    system = '''You are a media operation engineer. Your job is to review a clip from a video \n",
    "    content presented in a sequence of consecutive images. Each image\n",
    "    contains a sequence of frames presented in a 4x7 grid reading from left to\n",
    "    right and then from top to bottom. Interpret the frames as the time \n",
    "    progression of a video clip.  Don't refer to specific frames, instead, think\n",
    "    about what is happening over time in the scene.  You may also optionally be given the\n",
    "    conversation of the scene you can use to understand the context of\n",
    "    the scene. \n",
    "\n",
    "    You are asked to provide the following information: a detailed \n",
    "    description to describe the scene using the visual and audio, identify the most relevant IAB taxonomy, \n",
    "    GARM, sentiment, and brands and logos that \n",
    "    may appear in the scene, and five most relevant tags from the scene.\n",
    "    \n",
    "    It is important to return the results in JSON format and also includes a\n",
    "    confidence score from 0 to 100. Skip any explanation.\n",
    "    '''\n",
    "\n",
    "    other_information = []\n",
    "    other_information.append(\n",
    "        {\n",
    "            \"type\": \"text\",\n",
    "            \"text\": f'''\n",
    "                    Here is a list of IAB Taxonomies in <iab> tag. Only answer \n",
    "                    the IAB taxonomy from this list:\n",
    "                    <iab>\n",
    "                    { json.dumps(make_iab_taxonomoies(iab_definitions['tier1'])) }\n",
    "                    </iab>\n",
    "                    '''\n",
    "        })\n",
    "    \n",
    "    other_information.append(\n",
    "        {\n",
    "            \"type\": \"text\",\n",
    "            \"text\": f'''\n",
    "                    Here is a list of GARM Taxonomies in <garm> tag. Only answer\n",
    "                    the GARM taxonomy from this list:\n",
    "                    <garm>\n",
    "                    [\n",
    "                        'Adult & Explicit Sexual Content',\n",
    "                        'Arms & Ammunition',\n",
    "                        'Crime & Harmful acts to individuals and Society, Human Right Violations',\n",
    "                        'Death, Injury or Military Conflict',\n",
    "                        'Online piracy',\n",
    "                        'Hate speech & acts of aggression',\n",
    "                        'Obscenity and Profanity, including language, gestures, and explicitly gory, graphic or repulsive content intended to shock and disgust',\n",
    "                        'Illegal Drugs, Tobacco, ecigarettes, Vaping, or Alcohol',\n",
    "                        'Spam or Harmful Content',\n",
    "                        'Terrorism',\n",
    "                        'Debated Sensitive Social Issue',\n",
    "                        'None',\n",
    "                    ]\n",
    "                    </garm>\n",
    "                    '''\n",
    "        })\n",
    "\n",
    "    other_information.append(\n",
    "        {\n",
    "            \"type\": \"text\",\n",
    "            \"text\": f'''\n",
    "                Here is a list of Sentiments in <sentiment> tag. Only answer the\n",
    "                sentiment from this list:\n",
    "\n",
    "                <sentiment>\n",
    "                ['Positive', 'Neutral', 'Negative', 'None']\n",
    "                </sentiment>\n",
    "                '''\n",
    "        })\n",
    "\n",
    "    output_format_message = make_output_example()\n",
    "\n",
    "    messages = []\n",
    " \n",
    "    # adding sequences of composite images to the prompt.  Limit is 20.\n",
    "    message_images = make_image_message(images[:19])\n",
    "    messages.append(message_images)\n",
    "\n",
    "    # adding the conversation to the prompt\n",
    "    messages.append({\n",
    "        'role': 'assistant',\n",
    "        'content': 'Got the images. Do you have the conversation of the scene?'\n",
    "    })\n",
    "\n",
    "    message_conversation = {\n",
    "        'role': 'user',\n",
    "        'content': 'No conversation.'\n",
    "    }\n",
    "    if text:\n",
    "        message_conversation['content'] = f'''\n",
    "            Here is the conversation of the scene in <conversation> tag.\n",
    "            <conversation>\n",
    "            { text }\n",
    "            </conversation>\n",
    "            '''\n",
    "\n",
    "    messages.append(message_conversation)\n",
    "\n",
    "    # other information\n",
    "    messages.append({\n",
    "        'role': 'assistant',\n",
    "        'content': 'OK. Do you have other information to provdie?'\n",
    "    })\n",
    "\n",
    "    messages.append({\n",
    "        'role': 'user',\n",
    "        'content': other_information\n",
    "    })\n",
    "\n",
    "    # output format\n",
    "    messages.append({\n",
    "        'role': 'assistant',\n",
    "        'content': 'OK. What output format?'\n",
    "    })\n",
    "    messages.append(output_format_message)\n",
    "\n",
    "    # prefill '{'\n",
    "    messages.append({\n",
    "        'role': 'assistant',\n",
    "        'content': '{'\n",
    "    })\n",
    "    \n",
    "    model_params = {\n",
    "        'anthropic_version': MODEL_VER,\n",
    "        'max_tokens': 4096,\n",
    "        'temperature': 0.1,\n",
    "        'top_p': 0.7,\n",
    "        'top_k': 20,\n",
    "        'stop_sequences': ['\\n\\nHuman:'],\n",
    "        'system': system,\n",
    "        'messages': messages\n",
    "    }\n",
    "    \n",
    "    try:\n",
    "        response = inference(model_params)\n",
    "    \n",
    "    except Exception as e:\n",
    "        print(colored(f\"ERR: inference: {str(e)}\\n RETRY...\", 'red'))\n",
    "        response = inference(model_params)\n",
    "\n",
    "    return response\n",
    "\n",
    "def display_prompt(model_params):\n",
    "    \"\"\"\n",
    "    Displays the model parameters including system prompt and messages for debugging or logging purposes.\n",
    "    Args:\n",
    "        model_params (dict): Dictionary containing model parameters with:\n",
    "            - system (str): The system prompt text\n",
    "            - messages (list): List of message objects to be displayed       \n",
    "    Returns:\n",
    "        None: This function prints to console and doesn't return any value\n",
    "    \"\"\"\n",
    "    print (f'MODEL_ID: {MODEL_ID}\\n')\n",
    "    print (f'System Prompt:\\n\\n{model_params[\"system\"]}')\n",
    "    print (f'Messages:\\n\\n')\n",
    "    for message in model_params['messages']:\n",
    "        print (json.dumps(message))\n",
    "\n",
    "    print('\\n')\n",
    "\n",
    "    return\n",
    "\n",
    "def inference(model_params):\n",
    "    \"\"\"\n",
    "    Invokes an Amazon Bedrock model for inference using the specified parameters.\n",
    "\n",
    "    Args:\n",
    "        model_params (dict): Parameters for the model inference, including:\n",
    "            - Any model-specific parameters required for the inference call\n",
    "            - Must be JSON-serializable\n",
    "\n",
    "    Returns:\n",
    "        dict: The processed response containing:\n",
    "            - content (list): List of response contents where each item contains:\n",
    "                - text (str): Raw text response from the model\n",
    "                - json (dict): Parsed JSON response (if successful)\n",
    "            - model_params (dict): Original input parameters\n",
    "            - Additional response metadata from the model\n",
    "    \"\"\"\n",
    "    model_id = MODEL_ID\n",
    "    accept = 'application/json'\n",
    "    content_type = 'application/json'\n",
    "\n",
    "    bedrock_runtime_client = boto3.client(service_name='bedrock-runtime')\n",
    "\n",
    "    response = bedrock_runtime_client.invoke_model(\n",
    "        body=json.dumps(model_params),\n",
    "        modelId=model_id,\n",
    "        accept=accept,\n",
    "        contentType=content_type\n",
    "    )\n",
    "\n",
    "    response_body = json.loads(response.get('body').read())\n",
    "\n",
    "    # patch the json string output with '{' and parse it\n",
    "    response_content = response_body['content'][0]['text']\n",
    "    if response_content[0] != '{':\n",
    "        response_content = '{' + response_content\n",
    "\n",
    "    try:\n",
    "        response_content = json.loads(response_content)\n",
    "    except Exception as e:\n",
    "        print(colored(\"Malformed JSON response. Try to repair it...\", 'red'))\n",
    "        try:\n",
    "            response_content = json_repair.loads(response_content, strict=False)\n",
    "        except Exception as e:\n",
    "            print(colored(\"Failed to repair the JSON response...\", 'red'))\n",
    "            print(colored(response_content, 'red'))\n",
    "            raise e\n",
    "\n",
    "    response_body['content'][0]['json'] = response_content\n",
    "    response_body['model_params'] = model_params\n",
    "\n",
    "    return response_body\n",
    "\n",
    "\n",
    "\n",
    "def display_contextual_cost(usage):\n",
    "    \"\"\"\n",
    "    Calculate and display the estimated cost of using the model based on input and output tokens.\n",
    "    Args:\n",
    "        usage (dict): A dictionary containing token usage information with keys:\n",
    "            - input_tokens (int): Number of input tokens used\n",
    "            - output_tokens (int): Number of output tokens generated\n",
    "    Returns:\n",
    "        dict: A dictionary containing cost calculation details:\n",
    "            - input_per_1k (float): Cost per 1000 input tokens\n",
    "            - output_per_1k (float): Cost per 1000 output tokens\n",
    "            - input_tokens (int): Number of input tokens used\n",
    "            - output_tokens (int): Number of output tokens generated\n",
    "            - estimated_cost (float): Total estimated cost in USD\n",
    "    \"\"\"\n",
    "    # us-east-1 pricing\n",
    "    input_per_1k, output_per_1k = MODEL_PRICING\n",
    "\n",
    "    input_tokens = usage['input_tokens']\n",
    "    output_tokens = usage['output_tokens']\n",
    "\n",
    "    contextual_cost = (\n",
    "        input_per_1k * input_tokens +\n",
    "        output_per_1k * output_tokens\n",
    "    ) / 1000\n",
    "\n",
    "    print('\\n')\n",
    "    print('========================================================================')\n",
    "    print('Estimated cost:', colored(f\"${round(contextual_cost, 4)}\", 'green'), f\"in us-east-1 region with {colored(input_tokens, 'green')} input tokens and {colored(output_tokens, 'green')} output tokens.\")\n",
    "    print('========================================================================')\n",
    "\n",
    "    return {\n",
    "        'input_per_1k': input_per_1k,\n",
    "        'output_per_1k': output_per_1k,\n",
    "        'input_tokens': input_tokens,\n",
    "        'output_tokens': output_tokens,\n",
    "        'estimated_cost': contextual_cost,\n",
    "    }\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6ca1d65-48b7-4738-b600-352d67191756",
   "metadata": {},
   "source": [
    "## Execute the prompt against all the chapter segments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99effde5-caba-4d24-9199-7346042c34c9",
   "metadata": {
    "editable": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "total_usage = {\n",
    "    'input_tokens': 0,\n",
    "    'output_tokens': 0,\n",
    "}\n",
    "\n",
    "iab_definitions = load_iab_taxonomy(iab_file)\n",
    "\n",
    "for chapter in video['chapters'].chapters:\n",
    "\n",
    "    composite_images = chapter['composite_images']\n",
    "    num_images = len(composite_images)\n",
    "\n",
    "    chapter_id = chapter['id']\n",
    "    text = chapter['text'] \n",
    "\n",
    "    contextual_response = get_chapter_description(composite_images, chapter['text'], iab_definitions)\n",
    "    time.sleep(5)\n",
    "    usage = contextual_response['usage']\n",
    "    contextual = contextual_response['content'][0]['json']\n",
    "\n",
    "    # save the contextual to the chapter\n",
    "    chapter['contextual'] = {\n",
    "        'usage': usage,\n",
    "        **contextual\n",
    "    }\n",
    "\n",
    "    total_usage['input_tokens'] += usage['input_tokens']\n",
    "    total_usage['output_tokens'] += usage['output_tokens']\n",
    "\n",
    "    print(f\"==== Chapter #{chapter['id']:02d}: Contextual information ======\")\n",
    "    video['frames'].display_frames(start=chapter['start_frame_id'], end=chapter['end_frame_id']+1)\n",
    "    for key in ['description', 'sentiment', 'iab_taxonomy', 'garm_taxonomy']:\n",
    "        print(f\"{key.capitalize()}: {colored(contextual[key]['text'], 'green')} ({contextual[key]['score']}%)\")\n",
    "\n",
    "    for key in ['brands_and_logos', 'relevant_tags']:\n",
    "        items = ', '.join([item['text'] for item in contextual[key]])\n",
    "        if len(items) == 0:\n",
    "            items = 'None'\n",
    "        print(f\"{key.capitalize()}: {colored(items, 'green')}\")\n",
    "    print(f\"================================================\\n\\n\")\n",
    "\n",
    "output_file = os.path.join(video[\"output_dir\"], 'scenes_in_chapters.json')\n",
    "util.save_to_file(output_file, video['chapters'].chapters)\n",
    "\n",
    "contextual_cost = display_contextual_cost(total_usage)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03269195-fcfd-4a9b-9d5d-a7de1d7cfe7c",
   "metadata": {},
   "source": [
    "## Ad breaks\n",
    "\n",
    "At this point, we have generated video segments that have clear visual breaks between scenes, and we have grouped the scenes into chapters that have clear breaks between topics from the speech in the audio. The breaks between chapters are all candidate ad placement opportunities.  We can use the IAB taxonomy of the chapter segments adjacent to the breaks to make better decisions about what ad to place in that break.  \n",
    "\n",
    "🤔 As you view the chapter segments, imagine yourself as a company who wants to advertise your brand, are there breaks you prefer over others in terms of brand safety?\n",
    "\n",
    "🤔 Now imagine you are a viewer.  What products would be interesting if you chose this title?\n",
    "\n",
    "In practice, ad breaks would be ranked by a value function that takes into account the needs of the consumer, the publisher, and the advertisers.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c40ec109-e2a7-44a4-b1b9-2636fd518046",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Visualize the ad breaks \n",
    "\n",
    "In this section, we'll insert a test Ad into one of the breaks to visualize the ad experience.  You can change the value of BREAK_CHAPTER_ID to try different chapter breaks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04ddc979-bf0f-4aca-a0a8-72147b95aeb4",
   "metadata": {
    "editable": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import moviepy\n",
    "from moviepy.editor import VideoFileClip, concatenate_videoclips\n",
    "\n",
    "BREAK_CHAPTER_ID = 8\n",
    "\n",
    "ad_demo_file= f\"ad_break_{ BREAK_CHAPTER_ID }_demo.mp4\"\n",
    "adbreak_start = video['chapters'].chapters[BREAK_CHAPTER_ID]['start_ms']/1000\n",
    "\n",
    "clip1 = VideoFileClip(video[\"path\"], target_resolution=(360, 640)).subclip(adbreak_start-10, adbreak_start)\n",
    "clip2 = VideoFileClip(\"static/images/CountdownClock_0.mp4\", target_resolution=(360, 640))\n",
    "clip3 = VideoFileClip(video[\"path\"], target_resolution=(360, 640)).subclip(adbreak_start, adbreak_start+10)\n",
    "final_clip = concatenate_videoclips([clip1,clip2,clip3], method=\"compose\")\n",
    "final_clip.write_videofile(ad_demo_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8181640e-de93-4f58-8d87-d82bc49ce33a",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "Video(url=ad_demo_file, width=640, height=360)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79b4a880-fca8-4fef-93ab-3f39c8db23c8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# What's Next?\n",
    "\n",
    "You can try another use case or, if you are done, continue to the [Additional Resources](09-resources.ipynb) lab.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa21bf8a-83f4-4120-94c4-17faa02ef16c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}