README_en.md

AITuberKit

Notice: From version v2.0.0, this project adopts a custom license. If you plan to use it for commercial purposes, please check the Usage Agreement section.

🌟 Demo Site 🌟

【Japanese】｜ 【Chinese】｜ 【Korean】

Overview

This repository has mainly the following 2 features:

1. Conversation with AI character
2. AITuber streaming

I've written a detailed usage guide in the article below:

⚠️ Important Security Notice

This repository is designed for personal use, local development, and commercial use with appropriate security measures. However, please note the following points when deploying to a web environment:

• API Key Handling: Since the specification requires API calls to AI services (OpenAI, Anthropic, etc.) and TTS services through a backend server, proper management of API keys is necessary.

Regarding Production Environment Usage

When using in a production environment, we recommend one of the following approaches:

1. Backend Server Implementation: Manage API keys on the server side and avoid direct API access from clients
2. Proper User Instructions: When users use their own API keys, explain the security considerations
3. Access Control Implementation: Implement appropriate authentication and authorization mechanisms as needed

Development Environment

This project is developed in the following environment:

• Node.js: ^20.0.0
• npm: 10.8.1

Common Preparations

1. Clone the repository to your local machine.

git clone https://github.com/tegnike/aituber-kit.git

2. Open the folder.

cd aituber-kit

3. Install packages.

npm install

4. Start the application in development mode.

npm run dev

5. Open the URL http://localhost:3000

6. Create the .env file if necessary.

cp .env.example .env

Conversation with AI Character

• This is a feature to converse with an AI character.
• It is an extended feature of pixiv/ChatVRM, which is the basis of this repository.
• It can be easily started as long as you have an API key for various LLMs.
• The recent conversation sentences are retained as memory.
• It is multimodal, capable of recognizing images from the camera or uploaded images to generate responses.

Usage

1. Enter your API key for various LLMs in the settings screen.
   • OpenAI
   • Anthropic
   • Google Gemini
   • Azure OpenAI
   • Groq
   • Cohere
   • Mistral AI
   • Perplexity
   • Fireworks
   • Local LLM
   • Dify (Chatbot or Agent)
2. Edit the character's setting prompt if necessary.
3. Load a VRM file and background file if needed.
4. Select a speech synthesis engine and configure voice settings if necessary.
   • VOICEVOX: You can select a speaker from multiple options. The VOICEVOX app needs to be running beforehand.
   • Koeiromap: You can finely adjust the voice. An API key is required.
   • Google TTS: Languages other than Japanese can also be selected. Credential information is required.
   • Style-Bert-VITS2: A local API server needs to be running.
   • AivisSpeech: The AivisSpeech app needs to be running beforehand.
   • GSVI TTS: A local API server needs to be running.
   • ElevenLabs: Various language selection is possible. Please enter the API key.
   • OpenAI: API key is required.
   • Azure OpenAI: API key is required.
   • Nijivoice: API key is required.
5. Start conversing with the character from the input form. Microphone input is also possible.

AITuber Streaming

• It is possible to retrieve YouTube streaming comments and have the character speak.
• A YouTube API key is required.
• Comments starting with '#' are not read.

Usage

1. Turn on YouTube mode in the settings screen.
2. Enter your YouTube API key and YouTube Live ID.
3. Configure other settings the same way as "Conversation with AI Character".
4. Start streaming on YouTube and confirm that the character reacts to comments.
5. Turn on the conversation continuity mode to be able to speak even if there are no comments.

Other Features

External Linkage Mode

• You can send requests to the server app via WebSocket and get responses.
• A separate server app needs to be prepared.

Usage

1. Start the server app and open the ws://127.0.0.1:8000/ws endpoint.
2. Turn on External Linkage Mode in the settings screen.
3. Configure other settings the same way as "Conversation with AI Character".
4. Send requests from the input form and confirm that responses are returned from the server app.

Related

• You can try it immediately with this server app repository. tegnike/aituber-server
• For detailed settings, please read "Let's develop with a beautiful girl!! [Open Interpreter]".

Slide Mode

• This is a mode where the AI character automatically presents slides.
• You need to prepare slides and script files in advance.

Usage

1. Proceed to the point where you can interact with the AI character.
2. Place the slide folder and script file in the designated folder.
3. Turn on Slide Mode in the settings screen.
4. Press the Start Slide button to begin the presentation.

Related

• For detailed settings, please read "AI Does Slide Presentations Now!!!!".

Realtime API Mode

• This is a mode where you can interact with the character with low latency using OpenAI's Realtime API.
• Function execution can be defined.

Usage

1. Select OpenAI or Azure OpenAI as the AI service.
2. Turn on Realtime API mode.
3. Use the microphone to talk to the character.

Function Execution

• Define new functions in src/components/realtimeAPITools.tsx and src/components/realtimeAPITools.json.
• Refer to the existing get_current_weather function as an example.

TIPS

Background Fixing Method

• Change the background image at public/bg-c.png. Do not change the name.

Setting Environment Variables

• Some configuration values can be referenced from the .env file contents.
• If entered in the settings screen, that value takes precedence.

Microphone Input Methods (2 Patterns)

1. Hold Alt (or option) key to record => Release to send
2. Click microphone button (click once to start recording) => Click again to send

Other

• Settings and conversation history can be reset in the settings screen.
• Various settings are stored in the browser's local storage.
• Elements enclosed in code blocks are not read by TTS.

Related Articles

• You are AITuber Developer from Today | Nike-chan
• Let's develop with a beautiful girl!! [Open Interpreter]
• AI Does Slide Presentations Now!!!!
• Added Multimodal Features to AITuberKit, So Let's Have a Drink at Home with AI Character
• AITuberKit × Dify for Super Easy Chatbot Building
• Publishing Dify on the Internet with Xserver
• Try the Advanced Voice Mode Called Realtime API

Seeking Sponsors

We are seeking sponsors to continue our development efforts.
Your support will greatly contribute to the development and improvement of the AITuber Kit.

Our Supporters (in order of support)

Plus multiple private sponsors

Usage Agreement

License

From version v2.0.0, this project adopts a custom license.

• Non-Commercial Use

  • Non-Commercial Use is available for personal use, educational purposes, and non-profit purposes that are not for commercial purposes.

• Commercial License

  • A separate commercial license is required for commercial use.
  • For details, please check About License.

Others

• Logo Usage Agreement
• VRM Model Usage Agreement