2_2-langchain-rag.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a2a28162-8f3e-42f8-83f6-1b9ef38cd394",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/vanderbilt-data-science/ai_summer/blob/main/2_2-langchain-rag.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "\n",
    "# AI Solutions with Langchain and RAG\n",
    "> For Vanderbilt University AI Summer 2024<br>Prepared by Dr. Charreau Bell\n",
    "\n",
    "_Code versions applicable: May 14, 2024_\n",
    "\n",
    "## Learning Outcomes:\n",
    "* Participants will be able to articulate the essential steps and components of a retrieval-augmented generation (RAG) system and implement a standard RAG system using langchain.\n",
    "* Participants will gain familiarity in inspecting the execution pathways of LLM-based systems.\n",
    "* Participants will gain familiarity in approaches for the evaluation of LLM-based systems."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c1cebcf-1ac1-48aa-b998-9f0b2a77567f",
   "metadata": {},
   "source": [
    "### Computing Environment Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15202e4f-5a92-4231-a557-43daa913beb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install langchain==0.1.20 langchain_openai grandalf sentence-transformers\n",
    "! pip install pypdf chromadb faiss-cpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd8f69f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Best practice is to do all imports at the beginning of the notebook, but we have separated them here for learning purposes.\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0e06178",
   "metadata": {},
   "outputs": [],
   "source": [
    "# auth replicated here for reference just in case you choose to do something similar\n",
    "from google.colab import userdata\n",
    "os.environ['OPENAI_API_KEY'] = userdata.get('OPENAI_API_KEY')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88934c0a-c340-4678-be23-563a4fe331f1",
   "metadata": {},
   "source": [
    "## Langchain Introduction\n",
    "\n",
    "### Overview of System\n",
    "\n",
    "[Overview of Langchain](https://python.langchain.com/v0.1/docs/get_started/introduction/)\n",
    "\n",
    "<figure>\n",
    "<img src='https://python.langchain.com/v0.1/svg/langchain_stack.svg' height=600/>\n",
    "    <figcaption>\n",
    "        Langchain Overview, from <a href=https://python.langchain.com/v0.1/docs/get_started/introduction>Langchain Introduction</a>\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "### Quick Start\n",
    "To, as it says - start quickly - get started using the [Quick Start](https://python.langchain.com/v0.1/docs/get_started/quickstart/) page.\n",
    "\n",
    "### Details of Individual Composition Components\n",
    "To learn more about any of the individual components used below, use the [Components Page](https://python.langchain.com/v0.1/docs/modules/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "477bff1b",
   "metadata": {},
   "source": [
    "## Review of python formatted strings\n",
    "To prepare ourselves for langchain, we'll first review formatted strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cff2bcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# basic functionality of print\n",
    "print('Tell me a story about cats')\n",
    "\n",
    "# with variables\n",
    "prompt_string = 'Tell me a story about cats'\n",
    "print('As string ', prompt_string)\n",
    "\n",
    "# as formatted string\n",
    "prompt_string = 'Tell me a story about cats'\n",
    "print(f\"With formatted string: {prompt_string}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d708c91c",
   "metadata": {},
   "source": [
    "Motivating example: you are building a GPT that tells stories. The user just needs to provide the topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afd22772",
   "metadata": {},
   "outputs": [],
   "source": [
    "# as a template string\n",
    "string_prompt_template = f\"Tell me a story about {{topic}}\"\n",
    "string_prompt_template"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70e55c40",
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can fill in the template at a later time\n",
    "string_prompt_template.format(topic='cats')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a7ffaf4",
   "metadata": {},
   "source": [
    "## Langchain Prompt Templates\n",
    "> Formatting and arranging prompt strings\n",
    "\n",
    "Langchain prompt templates work just like this, but with additional functionality targeted towards LLM interaction. There are lots of different prompt templates, but here, we'll focus on two: `PromptTemplate`, and `ChatPromptTemplate`.\n",
    "\n",
    "**Additional resources**: [Guide on Prompt Templates](https://python.langchain.com/v0.1/docs/modules/model_io/prompts/quick_start/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71f3192e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import PromptTemplate, ChatPromptTemplate, MessagesPlaceholder\n",
    "from langchain_core.output_parsers import StrOutputParser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef3a0cbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create system messsage for shorter responses\n",
    "brief_system_message = 'You are a helpful assistant. Be brief, succinct, and clear in your responses. Only answer what is asked.'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f581ae79",
   "metadata": {},
   "source": [
    "### PromptTemplate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dda49b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a langchain prompt template\n",
    "lc_prompt\n",
    "\n",
    "# has standard behavior of f-strings\n",
    "lc_prompt\n",
    "\n",
    "# but also has additional functionality\n",
    "lc_prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38033eec",
   "metadata": {},
   "source": [
    "### ChatPromptTemplate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f253a330",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create prompt template\n",
    "lc_chat_prompt_template = ChatPromptTemplate.from_template(\"tell me a story about {topic}\")\n",
    "\n",
    "# has invocation functionality resulting to chat-style messages\n",
    "lc_chat_prompt_template.invoke({'topic':'cats'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6896b8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create message-based chat prompt template\n",
    "lc_chat_prompt_template = ChatPromptTemplate.from_messages(\n",
    "    # create messages similar to OpenAI API\n",
    ")\n",
    "\n",
    "# invoke the chat prompt template\n",
    "lc_chat_prompt_template.invoke({'topic':'cats'})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20703088",
   "metadata": {},
   "source": [
    "## Langchain Expression Language (LCEL)\n",
    "**Resource:** [LCEL Overview](https://python.langchain.com/v0.1/docs/expression_language/)\n",
    "Main Points:\n",
    "* Runnable Protocol\n",
    "* Known inputs and outputs on invoke\n",
    "* Flexibility in chain assembly\n",
    "* [Standard Interface](https://python.langchain.com/v0.1/docs/expression_language/interface/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2D0GyPPpedXw",
   "metadata": {},
   "source": [
    "# Basic Model Chains/ Model I/O\n",
    "\n",
    "**Resource**: [Detailed Guide](https://python.langchain.com/v0.1/docs/modules/)\n",
    "\n",
    "## Basic Prompt/Model Chain\n",
    "See [Prompt+LLM](https://python.langchain.com/docs/expression_language/cookbook/prompt_llm_parser) for more information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5d1b35f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import ChatOpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f068a62-9a56-420a-be6f-2bf3bb805d04",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = ChatPromptTemplate.from_template(\"tell me a joke about {foo}\")\n",
    "model \n",
    "chain "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1GVTTau0fsdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Observe what the prompt looks like when we substitute words into it\n",
    "prompt.invoke({'foo':\"cats\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "VP7khR3zd2OJ",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now, actually call the entire chain on it\n",
    "\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d753e10d",
   "metadata": {},
   "source": [
    "A little helper visualization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53dc608a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create visualization\n",
    "chain.get_graph().print_ascii()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "LUCuzx2HfeVx",
   "metadata": {},
   "source": [
    "## Even more simplified prompt chains"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eM3wDIpnfg3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create total user prompt chain\n",
    "prompt = ChatPromptTemplate.from_template(\"{text}\")\n",
    "\n",
    "# Add output parser\n",
    "chain = prompt | model | StrOutputParser()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8hStGS1kfonX",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now, the user can submit literally whatever\n",
    "res = chain.invoke({'text':\"Briefly and succintly summarize Episodes 4-6 of Star Wars.\"})\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "958be68e",
   "metadata": {},
   "source": [
    "## What just happened? Inspecting model behavior\n",
    "Several ways to do this:\n",
    "* `langchain` verbosity/debugging\n",
    "* `langsmith`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db9684a3",
   "metadata": {},
   "source": [
    "### Langchain\n",
    "Resource: [Guides -> Langchain Debugging](https://python.langchain.com/v0.1/docs/guides/development/debugging/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e76bf06e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.globals import set_debug, set_verbose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88edac26",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_debug(True)\n",
    "set_verbose(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1984a1a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Basic prompt -> model -> parser chain\n",
    "chain = prompt | model | StrOutputParser()\n",
    "chain.invoke('What is a python f-string?')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e60883c",
   "metadata": {},
   "source": [
    "### Langsmith\n",
    "Resource: [Tracing Langchain with Langsmith](https://docs.smith.langchain.com/how_to_guides/tracing/trace_with_langchain)\n",
    "\n",
    "Don't have a langsmith API Key yet? You'll need a user account on [Langsmith](https://smith.langchain.com/). Then, follow these [instructions provided by langsmith](https://docs.smith.langchain.com/#2-create-an-api-key)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6948026e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# reset this\n",
    "set_debug(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e427ad5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# enable tracing and set project name\n",
    "os.environ['LANGCHAIN_TRACING_V2'] = \"false\"\n",
    "\n",
    "# uncomment the following two lines before running the cell if you have a Langchain/Langsmith API Key\n",
    "#os.environ['LANGCHAIN_API_KEY'] = userdata.get('LANGCHAIN_API_KEY')\n",
    "#os.environ['LANGCHAIN_TRACING_V2'] = \"true\"\n",
    "\n",
    "# set langchain project\n",
    "os.environ['LANGCHAIN_PROJECT'] = 'May15'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c9e76df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# use a the basic chain from above\n",
    "chain = (prompt | model | StrOutputParser()) #add new component for tracing\n",
    "response = chain.invoke(\"What is a python f-string?\")\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11d4399e",
   "metadata": {},
   "source": [
    "#### View langsmith traces\n",
    "We can take a look at this trace on [Langsmith](https://smith.langchain.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "545b8841",
   "metadata": {},
   "source": [
    "## Adding Memory\n",
    "Adapted from: [LCEL Adding Message History](https://python.langchain.com/v0.1/docs/expression_language/how_to/message_history/)\n",
    "Also see:\n",
    "- [Langchain -> Use Cases -> Chatbots -> Memory Management](https://python.langchain.com/v0.1/docs/use_cases/chatbots/memory_management/)\n",
    "- [Components -> More -> Memory](https://python.langchain.com/v0.1/docs/modules/memory/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0873b34a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.chat_message_histories import ChatMessageHistory\n",
    "from langchain_core.chat_history import BaseChatMessageHistory\n",
    "from langchain_core.runnables.history import RunnableWithMessageHistory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9843f11f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create chat template with standard elements\n",
    "model = ChatOpenAI(name='gpt-3.5-turbo')\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", brief_system_message),\n",
    "        MessagesPlaceholder(variable_name=\"chat_history\"),\n",
    "        (\"human\", \"{most_recent_user_message}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "turns_chain = prompt | model | StrOutputParser()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37b39d13",
   "metadata": {},
   "outputs": [],
   "source": [
    "# quickly try out chain, pretending we've already said something to the system\n",
    "first_chat_turn_messages = [(\"human\", \"Tell me a joke about cats\"),\n",
    "                            (\"ai\", \"Cats jump on beds\")]\n",
    "\n",
    "next_user_message = \"What was funny about that joke?\"\n",
    "turns_chain.invoke({'most_recent_user_message': next_user_message,\n",
    "                    'chat_history': first_chat_turn_messages})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f37fa72",
   "metadata": {},
   "outputs": [],
   "source": [
    "# all saved conversations\n",
    "chat_conversation_threads = {}\n",
    "\n",
    "# define function to create new conversation or load old one based on session_id\n",
    "def get_session_history(session_id: str) -> BaseChatMessageHistory:\n",
    "    if session_id not in chat_conversation_threads:\n",
    "        chat_conversation_threads[session_id] = ChatMessageHistory()\n",
    "    return chat_conversation_threads[session_id]\n",
    "\n",
    "# create chat history enabled chain\n",
    "chat_with_message_history = RunnableWithMessageHistory(\n",
    "    turns_chain,\n",
    "    get_session_history,\n",
    "    input_messages_key=\"most_recent_user_message\",\n",
    "    history_messages_key=\"chat_history\", \n",
    ").with_config(run_name = 'Chat with Message History')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ac18605",
   "metadata": {},
   "source": [
    "Let's try it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "076cfa99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# send first message\n",
    "user_message_1 = \"Tell me a joke about cats\"\n",
    "session_id_1 = \"convo_1\"\n",
    "chat_with_message_history.invoke({'most_recent_user_message': # add first message},\n",
    "                                config={\"configurable\": {\"session_id\": # add session_id}})        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "441d9331",
   "metadata": {},
   "outputs": [],
   "source": [
    "# send second message\n",
    "chat_with_message_history.invoke({'most_recent_user_message': # add another message to the chat},\n",
    "                                    config={\"configurable\": {\"session_id\": session_id_1}})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47da336f",
   "metadata": {},
   "source": [
    "#### View langsmith traces\n",
    "We can take a look at this trace on [Langsmith](https://smith.langchain.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "j-T1I1nDiM-A",
   "metadata": {},
   "source": [
    "# Retrieval Augmented Generation (RAG)\n",
    "## Review\n",
    "* Conceptual and step-by-step guide about [RAG](https://python.langchain.com/v0.1/docs/use_cases/question_answering/)\n",
    "* Learn more about implementing [RAG](https://python.langchain.com/docs/expression_language/cookbook/retrieval)\n",
    "\n",
    "**Data Ingestion (Creating a Vector Store of Documents)**\n",
    "<figure>\n",
    "<img src='https://python.langchain.com/v0.1/assets/images/rag_indexing-8160f90a90a33253d0154659cf7d453f.png' height=300/>\n",
    "    <figcaption>\n",
    "        Source: Data Ingestion (Preparing Embeddings), from <a href=https://python.langchain.com/v0.1/docs/use_cases/question_answering/>Langchain Use Case: Q&A with RAG</a>\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "**Retrieval and Generation**\n",
    "<figure>\n",
    "<img src='https://python.langchain.com/v0.1/assets/images/rag_retrieval_generation-1046a4668d6bb08786ef73c56d4f228a.png' height=300/>\n",
    "    <figcaption>\n",
    "        Source: Retrieval and Generation, from <a href=https://python.langchain.com/v0.1/docs/use_cases/question_answering/>Langchain Use Case: Q&A with RAG</a>\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "HOs-D7XQlmzt",
   "metadata": {},
   "source": [
    "## Document Loaders and Splitters\n",
    "[Data Ingestion/Vector Store Preparation Guide ](https://python.langchain.com/docs/modules/data_connection/)\n",
    "<figure>\n",
    "<img src='https://python.langchain.com/v0.1/assets/images/data_connection-95ff2033a8faa5f3ba41376c0f6dd32a.jpg' height=300/>\n",
    "    <figcaption>\n",
    "        Langchain Retrieval Component, from <a href=https://python.langchain.com/docs/modules/data_connection/>Langchain Components</a>\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "**Other extremely useful resources**:\n",
    "* **[Components -> Retrieval -> Document Loaders](https://python.langchain.com/v0.1/docs/modules/data_connection/document_loaders/)**: Use the sidebar to navigate through different types of document loaders. For all available integrations available through langchain, see [Components -> Integrations -> Components](https://python.langchain.com/v0.1/docs/integrations/document_loaders/)\n",
    "* **[Components -> Retrieval -> Text Splitters](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/)**: Use the sidebar to navigate through different types of text splitters. For all available integrations available through langchain, see [Components -> Integrations -> Components](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88cf0508",
   "metadata": {},
   "outputs": [],
   "source": [
    "# example pdf links\n",
    "doc_1 = 'https://registrar.vanderbilt.edu/documents/Undergraduate_School_Catalog_2023-24_UPDATED2.pdf'\n",
    "doc_2 = 'https://www.tnmd.uscourts.gov/sites/tnmd/files/Pro%20Se%20Nonprisoner%20Handbook.pdf'\n",
    "doc_3 = 'https://www.uscis.gov/sites/default/files/document/guides/M-654.pdf'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83aff93e",
   "metadata": {},
   "source": [
    "### Example: pdfloader and recursive character splitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "Sk4B1Ihklod8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b31d1836",
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = #choose loader and document\n",
    "\n",
    "# Add the kind of text splitter\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=800,\n",
    "    chunk_overlap=250,\n",
    ")\n",
    "\n",
    "# use the text splitter to split the document\n",
    "chunks = # load and split chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f9490d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# see how many chunks were made\n",
    "print(len(chunks))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7e3261f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect a single chunk\n",
    "chunks[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08ab1136",
   "metadata": {},
   "outputs": [],
   "source": [
    "# view first 3 chunks\n",
    "for chunk_index, chunk in enumerate(chunks[:3]):\n",
    "    print(f'****** Chunk {chunk_index} ******\\n{chunk.page_content}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1fd5bbb",
   "metadata": {},
   "source": [
    "### Example: Loading website data and splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f15b971",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import SoupStrainer\n",
    "from langchain_community.document_loaders import WebBaseLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fa1ea14",
   "metadata": {},
   "outputs": [],
   "source": [
    "constitution_website = \"https://constitutioncenter.org/the-constitution/full-text\"\n",
    "\n",
    "# load using WebBaseLoader\n",
    "web_loader = WebBaseLoader(constitution_website,\n",
    "                       bs_kwargs = {'parse_only':SoupStrainer(['article'])})\n",
    "\n",
    "# read the document from the website (without splitting)\n",
    "web_document = #load document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c754f1f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# only the first few characters\n",
    "print(web_document[0].page_content[:330])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "330f96da",
   "metadata": {},
   "source": [
    "Now, we'll split in a slightly different way. Since we've already scraped the website, we will just directly use the splitter. Note that after we load the website, we have a data type of (list of) `Document`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e15cfb1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "website_splitter = RecursiveCharacterTextSplitter(chunk_size=330, chunk_overlap=100, # add ability to use start_index\n",
    "website_chunks = website_splitter.split_documents(web_document)\n",
    "len(website_chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f581f865",
   "metadata": {},
   "outputs": [],
   "source": [
    "website_chunks[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "841ab566",
   "metadata": {},
   "source": [
    "If you know less about the constitution and more about Star wars (or another topic available on Wikipedia), feel free to run the cells below to use that text moving forward. It will replace the `website_chunks` variable. You may need to adjust the `chunk_size` and `chunk_overlap` options. Uncomment and run these cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08bb3ff8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# alternate data\n",
    "webloader = WebBaseLoader('https://simple.wikipedia.org/wiki/Star_Wars_Episode_IV:_A_New_Hope',\n",
    "                       bs_kwargs = {'parse_only':SoupStrainer('div', id='bodyContent')})\n",
    "web_chunks = webloader.load_and_split(RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=100, add_start_index=True))\n",
    "print('Number of chunks generated: ', len(web_chunks))\n",
    "print('\\n\\nSample: ')\n",
    "web_chunks[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mKdzgo6ci0MD",
   "metadata": {},
   "source": [
    "## Vector Stores: A way to store embeddings (hidden states) of your data\n",
    "The choice of vector store influences how \"relevant\" documents can be identified, speed of document retrieval, and organization.\n",
    "\n",
    "Helpful resources:\n",
    "* **[Brief Langchain Reference](https://python.langchain.com/v0.1/docs/modules/data_connection/vectorstores/)**\n",
    "* **[Vector Store Integrations](https://python.langchain.com/v0.1/docs/integrations/vectorstores/)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2831d1dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.vectorstores import Chroma\n",
    "from langchain_openai import ChatOpenAI, OpenAIEmbeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "jmXDTgoqjCUs",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the vector store\n",
    "db = # code to create store"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "217d27ea",
   "metadata": {},
   "source": [
    "### Similarity Search for Documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "XC-5zEeioFmq",
   "metadata": {},
   "outputs": [],
   "source": [
    "# query the vector store\n",
    "query = 'When was a new hope released?'\n",
    "\n",
    "# use a similarity search between the vectors\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d1c197d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get cosine distance alongside results\n",
    "relevant_docs = \n",
    "relevant_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "CWJwyByyhoSB",
   "metadata": {},
   "outputs": [],
   "source": [
    "# another query, but instead use normalized score\n",
    "query = 'What is the plot of A New Hope?'\n",
    "relevant_docs = # use normalized score\n",
    "relevant_docs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lt_uoYfLjUdx",
   "metadata": {},
   "source": [
    "## Retrievers: How we select the most relevant data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "KTNxqQf7kiY_",
   "metadata": {},
   "outputs": [],
   "source": [
    "# or use the db as a retriever with lcel\n",
    "retriever = # create retriever\n",
    "retrieved_docs = retriever.invoke(query)\n",
    "retrieved_docs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8IuaToUPc0Hy",
   "metadata": {},
   "source": [
    "## RAG\n",
    "For when we want to actually do generation, but want there to be retrieved documents included in the generation. For this, we're going to switch to a different embedding model which will be downloaded on our machine (or if on Google Colab, there)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3298219",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.embeddings import HuggingFaceEmbeddings\n",
    "from langchain_community.vectorstores import FAISS\n",
    "\n",
    "from langchain_core.runnables import RunnableLambda, RunnablePassthrough, RunnableSequence, RunnableParallel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4f952c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# use different embedding model\n",
    "embeddings_fn = HuggingFaceEmbeddings(model_name=\"all-MiniLM-L6-v2\") #, model_kwargs={\"device\":'mps'})\n",
    "hf_db = FAISS.from_documents(web_chunks, embeddings_fn)\n",
    "hf_retriever = hf_db.as_retriever(search_kwargs={\"k\":1})\n",
    "\n",
    "# make sure it works\n",
    "query = 'What is the plot of A New Hope?'\n",
    "hf_retriever.invoke(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16cb408b",
   "metadata": {},
   "source": [
    "### Default RAG: Question Answering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "Fsewtgf9alyl",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Basic question answering template\n",
    "template = \"\"\"Answer the question based only on the following context:\n",
    "{context}\n",
    "\n",
    "Question: {question}\n",
    "\"\"\"\n",
    "\n",
    "# compose prompt\n",
    "rag_prompt = ChatPromptTemplate.from_template(template)\n",
    "\n",
    "# create model (so we don't have to depend on the model definition at the top of the notebook)\n",
    "model = ChatOpenAI(model_name='gpt-3.5-turbo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ExSC1rNNmCw",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need to format the retrieved documents better\n",
    "def format_docs(docs):\n",
    "    return \"\\n\\n\".join([f'Reference text:\\n{doc.page_content}\\n\\Citation Info: {doc.metadata}' for doc in docs])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73636824",
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect behavior of format_docs\n",
    "format_docs(web_chunks[:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21071aec",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compose the chain\n",
    "rag_chain = (\n",
    "    ## Add special rag part\n",
    "    | rag_prompt\n",
    "    | model\n",
    "    | StrOutputParser()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2cba6e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# run the chain\n",
    "rag_chain.with_config(run_name = 'basic_rag_chain').invoke('What is the plot of a new hope')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31c99c95",
   "metadata": {},
   "source": [
    "### RAG with Sources\n",
    "Resource: [Langchain: Returning Sources](https://python.langchain.com/v0.1/docs/use_cases/question_answering/sources/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d47c733e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.runnables import RunnableParallel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7fdb371f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Basic prompt -> model -> parser chain\n",
    "single_turn_chain = (\n",
    "    rag_prompt\n",
    "    | model\n",
    "    | StrOutputParser()\n",
    ")\n",
    "\n",
    "# Break previous chain in half to access context and question in returned response\n",
    "rag_chain_with_source = RunnableParallel(\n",
    "    {\"context\": hf_retriever | format_docs, \"question\": RunnablePassthrough()}\n",
    ").assign(answer=single_turn_chain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4b36d6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# invoke\n",
    "response = rag_chain_with_source.with_config(run_name = 'sources_rag_chain').invoke(\"What happened to Princess Leia in a New Hope?\")\n",
    "\n",
    "# print full response\n",
    "for key, value in response.items():\n",
    "    print(f\"{key}: {value}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afec178a",
   "metadata": {},
   "source": [
    "### RAG with Chat History?\n",
    "\n",
    "We will have a one-turn system with our RAG system. How do we add chat memory? See below for implementation guides:\n",
    "- [Use cases: Q&A with Rag: Add Chat History.](https://python.langchain.com/v0.1/docs/use_cases/question_answering/chat_history/)  Builds on a RAG system, so will be of interest."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9d3434e",
   "metadata": {},
   "source": [
    "## LLM System Metrics\n",
    "Resource: [Guides -> Evaluation](https://python.langchain.com/v0.1/docs/guides/productionization/evaluation/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92372d35",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.evaluation import load_evaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7bac358",
   "metadata": {},
   "outputs": [],
   "source": [
    "# configure what we want to evaluate\n",
    "rs_question = \"What happened to Princess Leia in a New Hope?\"\n",
    "rs_answer = rag_chain.with_config(run_name = 'basic_rag_chain').invoke(rs_question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1da42e15",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load an evaluator that uses the conciseness criteria\n",
    "evaluator = #load evaluator\n",
    "\n",
    "# evaluate whether our model was concise or not\n",
    "eval_result = evaluator.evaluate_strings(\n",
    "    # add inputs to evaluate\n",
    ")\n",
    "\n",
    "# print result\n",
    "eval_result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "933a5e2e",
   "metadata": {},
   "source": [
    "View other criteria available through LangChain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6dbb810",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.evaluation import Criteria\n",
    "list(Criteria)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34ea2044",
   "metadata": {},
   "source": [
    "# Homework\n",
    "The following exercises are designed to help you gain depth in what you've learned about RAG today.\n",
    "\n",
    "## [Required] Learning more about RAG\n",
    "### Splitting Text (Conceptual)\n",
    "There are so many ways to split the text, and each has an impact on the resultant RAG system. Below is a resource (with sidebar dropdown) for you to read over and then answer the following question for the text splitting approaches (as relevant to your application):\n",
    "* What is the proposed value in adopting this text splitting approach? What are some drawbacks?\n",
    "\n",
    "### Splitting Text (Programmatic)\n",
    "Above, we have adopted specific chunk sizes and splitting approaches. Choose one of the documents (or use your own) and:\n",
    "* Modify the chunk size. How does this impact the resulting RAG performance? The cost?\n",
    "* Implement a different type of text splitter (as applicable, i.e., not code text splitters if you're not splitting code). How does this impact the resulting RAG performance? The cost?\n",
    "\n",
    "### Customizing RAG\n",
    "There are many, many, many ways to improve results with RAG. Below are some resources for you to read over then complete the following:\n",
    "1. What is the proposed value in adopting this approach? In other words, what is the expected performance improvement by using this method?\n",
    "2. How might it apply to your work?\n",
    "\n",
    "* [**Query Analysis**](https://python.langchain.com/v0.1/docs/use_cases/query_analysis/). Make sure to peruse subtopics.\n",
    "* [**Synthetic Data Generation**](https://python.langchain.com/v0.1/docs/use_cases/data_generation/). \n",
    "* [**Tagging**](https://python.langchain.com/v0.1/docs/use_cases/tagging/).\n",
    "* [**Routing Chain Logic Based on inputs****](https://python.langchain.com/v0.1/docs/expression_language/how_to/routing/). \n",
    "* [**Chain Composition**](https://python.langchain.com/v0.1/docs/modules/chains/). Of particular interest here are the Legacy chains. Although they will probably be completely removed in the future, consider their behavior. In what cases might these behaviors be useful?\n",
    "\n",
    "** This is highly recommended reading, but may not be suitable for those who are novices in programming. Although there is text, the code demonstrates concretely by the text. For novices, it may be better to copy/paste the code as well to understand the behavior, although it is noted that such a task may be outside of the the time constraints of for some participants.\n",
    "\n",
    "## [Required] Learning more about Evaluation\n",
    "Read the following text and answer these questions:\n",
    "1. What is the purpose of the individual criterion? Does it require and external LLM for evaluation?\n",
    "2. In what cases might this criteria be useful?\n",
    "\n",
    "Depth Text: [**Evaluation, by Langchain**](https://python.langchain.com/v0.1/docs/guides/productionization/evaluation/)\n",
    "\n",
    "## [Highly recommended] Learning more about the LLM System Lifecycle\n",
    "There is more to an LLM-based system than a user interface and the LLM chain. There is a whole framework around inspecting, testing, and evaluating these systems. Read the following and answer the questions below:\n",
    "1. Summarize the purpose of the individual components of the langsmith system (they generalize to all LLM systems).\n",
    "2. Consider your favorite LLM UI (i.e., ChatGPT, Gemini, Claude, etc). Describe how you think these components are utilized the LLM system.\n",
    "\n",
    "Depth Text: [**LangSmith User Guide**](https://docs.smith.langchain.com/old/user_guide)\n",
    "\n",
    "## [Recommended] Practicing with RAG and Langchain\n",
    "### Exercise 1: Modify the RAG system\n",
    "Modify or create a new chain which:\n",
    "1. Uses a different LLM than the one used in this notebook.\n",
    "2. Uses a different document loader\n",
    "3. Uses a different splitter than the one used in this notebook.\n",
    "4. Uses a different vector store/retriever than the one used in this notebook.\n",
    "\n",
    "Use the resources provided in the relevant sections of the notebook for other options.\n",
    "\n",
    "### Exercise 2: Implement a new RAG system\n",
    "1. [More challenging] Add chat history to one of your RAG chains. Make sure to enable tracing and inspect langsmith to ensure that the chat history is used.\n",
    "2. Create a gradio user interface to use your chain in a more user-friendly way.\n",
    "3. [Challenging] Implement an additional chain which uses one of the strategies you read about in the \"Learning more about RAG\" section.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "name": "python3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0cdb4ccb06c1499e9714b81a255993d2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "16a18b98a5f54513b79aa3618ea6736e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fff6c8d96fa541e6aa5370dc37390cec",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_94162c2e8de94850b3da21e898ae3e9e",
      "value": 3
     }
    },
    "1f35015f30164fa3becb2b142d933cd6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "31dd85caa7434f9b9a7cbec22140b0e4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f4788fa6efa644a8b0c08c2f7026df22",
      "placeholder": "​",
      "style": "IPY_MODEL_3cb1ff1c69ee4ad89dc4f8848b4f24a6",
      "value": "Loading checkpoint shards: 100%"
     }
    },
    "3cb1ff1c69ee4ad89dc4f8848b4f24a6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "94162c2e8de94850b3da21e898ae3e9e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "9e6e1c4e60114b249e4ced3a81e87f7f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9fa2070bbbbc477c886b8569244cfc20": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0cdb4ccb06c1499e9714b81a255993d2",
      "placeholder": "​",
      "style": "IPY_MODEL_1f35015f30164fa3becb2b142d933cd6",
      "value": " 3/3 [00:03&lt;00:00,  1.07s/it]"
     }
    },
    "ad13e58f69fe45efb8e617bdb429463b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_31dd85caa7434f9b9a7cbec22140b0e4",
       "IPY_MODEL_16a18b98a5f54513b79aa3618ea6736e",
       "IPY_MODEL_9fa2070bbbbc477c886b8569244cfc20"
      ],
      "layout": "IPY_MODEL_9e6e1c4e60114b249e4ced3a81e87f7f"
     }
    },
    "f4788fa6efa644a8b0c08c2f7026df22": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fff6c8d96fa541e6aa5370dc37390cec": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}