Skip to content

Commit

Permalink
adjusted intro
Browse files Browse the repository at this point in the history
  • Loading branch information
@lkuligin
lkuligin committed Jan 13, 2025
1 parent 975c9c3 commit 0a567cc
Showing 1 changed file with 19 additions and 192 deletions.
211 changes: 19 additions & 192 deletions libs/vertexai/langchain_google_vertexai/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -1,37 +1,26 @@
"""
## langchain-google-vertexai
This module contains the LangChain integrations for Google Cloud generative models.
This module contains the LangChain integrations for Vertex AI service - Google foundational models, third-party foundational modela available on Vertex Model Garden and.
## Installation
```bash
pip install -U langchain-google-vertexai
```
## Supported Models (MaaS: Model-as-a-Service)
## Supported foundational models:
1. Llama
2. Mistral
3. Gemma
4. Vision models
1. Gemini family.
2. Other Google's foundational models: Imagen, embeddings, Codey models and others.
3. Third-party foundational models available as a an API (mdel-as-a-service) on Vertex Model Garden (Mistral, Llama, Anthropic)
4. Third-party foundational models deployed on Vertex AI endpoints from Vertex Model Garden or Hugginface.
5. Vector Search on Vertex AI.
6. Vertex AI evaluators for generative AI.
Integration on Google Cloud Vertex AI Model-as-a-Service.
### Setup
For more information, see:
https://cloud.google.com/blog/products/ai-machine-learning/llama-3-1-on-vertex-ai
#### Setup
You need to enable a corresponding MaaS model (Google Cloud UI console ->
Vertex AI -> Model Garden -> search for a model you need and click enable)
You must have the langchain-google-vertexai Python package installed
.. code-block:: bash
pip install -U langchain-google-vertexai
And either:
You need to enable needed Google Cloud APIs (depending on the integration you're using) and set up credentials by either:
- Have credentials configured for your environment
(gcloud, workload identity, etc...)
- Store the path to a service account JSON file as the
Expand All @@ -45,180 +34,18 @@
and
https://googleapis.dev/python/google-auth/latest/reference/google.auth.html#module-google.auth.
## Chat Language Model vs Base Language Models
A "base model" is a large language model trained on a wide variety of text data,
providing a general-purpose foundation for various tasks.
While a "chat model" is a specialized version of a base model specifically fine-tuned
to excel in conversational interactions, meaning it's optimized to understand and
respond to dialogue within a chat context, often with features like maintaining
conversational context and generating engaging responses.
## Gemma models
Gemma is a family of lightweight, state-of-the-art open models built from the same
research and technology used to create the Gemini models. Developed by Google DeepMind
and other teams across Google.
We currently support the following Gemma models.
1. GemmaChatLocalHF: Gemma Chat local model from Hugging Face.
2. GemmaChatLocalKaggle - Gemma chat local model from Kaggle.
3. GemmaLocalHF - Gemma Local model from Hugging Face
4. GemmaLocalKaggle - Gemma Local model from Kaggle
## Vision models
### Image Captioning
Generate Captions from Image. Implementation of the Image Captioning model
The model takes a list of prompts as an input where each prompt must be a
string that represents an image. Currently supported are:
1. Google Cloud Storage URI
2. B64 encoded string
3. Local file path
4. Remote url
The model returns Captions generated from every prompt (Image).
### Image Generation
Generates images from text prompt.
The message must be a list of only one element with one part i.e. the user prompt.
### Visual QnA Chat
Answers questions about an image. Chat implementation of a visual QnA model.
The model takes a list of messages. The first message should contain a string
representation of the image. Currently supported are:
1. Google Cloud Storage URI
2. B64 encoded string
3. Local file path
4. Remote url
There has to be at least another message with the first question.
The model returns the generated answer for the questions asked.
### Image Editor
Given an image and a prompt, edit the image. Currently only supports mask free editing.
The message must be a list of only one element with two part:
1. The image as a dictionary: { 'type': 'image_url', 'image_url': {'url': <message_str>} }
2. The user prompt.
## Chat Models
`ChatVertexAI` class exposes models such as `gemini-pro` and `chat-bison`.
To use, you should have Google Cloud project with APIs enabled, and configured
credentials. Initialize the model as:
```python
from langchain_google_vertexai import ChatVertexAI
llm = ChatVertexAI(model_name="gemini-pro")
llm.invoke("Sing a ballad of LangChain.")
```
You can use other models, e.g. `chat-bison`:
```python
from langchain_google_vertexai import ChatVertexAI
llm = ChatVertexAI(model_name="chat-bison", temperature=0.3)
llm.invoke("Sing a ballad of LangChain.")
```
#### Multimodal inputs
Gemini vision model supports image inputs when providing a single chat message. Example:
```python
from langchain_core.messages import HumanMessage
from langchain_google_vertexai import ChatVertexAI
llm = ChatVertexAI(model_name="gemini-pro-vision")
# example
message = HumanMessage(
content=[
{
"type": "text",
"text": "What's in this image?",
}, # You can optionally provide text parts
{"type": "image_url", "image_url": {"url": "https://picsum.photos/seed/picsum/200/300"}},
]
)
llm.invoke([message])
```
The value of `image_url` can be any of the following:
- A public image URL
- An accessible gcs file (e.g., "gcs://path/to/file.png")
- A base64 encoded image (e.g., `data:image/png;base64,abcd124`)
## Embeddings
You can use Google Cloud's embeddings models as:
```python
from langchain_google_vertexai import VertexAIEmbeddings
embeddings = VertexAIEmbeddings()
embeddings.embed_query("hello, world!")
```
## List of all integrations
## LLMs
- Gemini: `ChatVertexAI`, `VertexAI`, `VertexAIEmbeddings`
- Gemma on ModelGarden or deployed locally: `GemmaChatLocalHF`, `GemmaChatLocalKaggle`, `GemmaChatVertexAIModelGarden`, `GemmaLocalHF`, `GemmaLocalKaggle`, `GemmaVertexAIModelGarden`
- Imagen models: `VertexAIImageCaptioning`, `VertexAIImageCaptioningChat`, `VertexAIImageEditorChat`, `VertexAIImageGeneratorChat`, `VertexAIVisualQnAChat`
- Partner foundational models: `model_garden.ChatAnthropicVertex`, `model_garden_maas.VertexModelGardenLlama`, `model_garden_maas.VertexModelGardenMistral`
- Third-party Models from Model Garden: `VertexAIModelGarden`
- Vertex AI Evaluation: `VertexPairWiseStringEvaluator`, `VertexStringEvaluator`
- Vertex Vector Search: `VectorSearchVectorStore`, `VectorSearchVectorStoreDatastore`, `VectorSearchVectorStoreGCS`
You can use Google Cloud's generative AI models as Langchain LLMs:
Take a look at detailed documentation for each class for further details.
```python
from langchain_core.prompts import PromptTemplate
from langchain_google_vertexai import ChatVertexAI
template = \"""Question: {question}
Answer: Let's think step by step.\"""
prompt = PromptTemplate.from_template(template)
llm = ChatVertexAI(model_name="gemini-pro")
chain = prompt | llm
question = "Who was the president of the USA in 1994?"
print(chain.invoke({"question": question}))
```
You can use Gemini and Palm models, including code-generations ones:
```python
from langchain_google_vertexai import VertexAI
llm = VertexAI(model_name="code-bison", max_output_tokens=1000, temperature=0.3)
question = "Write a python function that checks if a string is a valid email address"
output = llm(question)
## Vector Stores
#### Vector Search Vector Store GCS
VertexAI VectorStore that handles the search and indexing using Vector Search
and stores the documents in Google Cloud Storage.
#### Vector Search Vector Store Datastore
VectorSearch with DatasTore document storage.
```
"""

from google.cloud.aiplatform_v1beta1.types import (
Expand Down

0 comments on commit 0a567cc

Please sign in to comment.