BossDB RAG System: A Language Interface to the BossDB Ecosystem

A Retrieval-Augmented Generation (RAG) system specifically designed for BossDB documentation and data queries. This system combines the power of large language models with contextual document retrieval (embedding models) to provide accurate, source-backed answers about BossDB, its tools, and related neuroscience data.

🌟 Features

• Intelligent Query Processing: Combines vector search with LLM-based response generation
• Conversation Memory: Maintains context across multiple queries with one of two memory modes:
  • Summary-based memory using Claude 3 Haiku for efficient compression
  • Window-based memory for maintaining recent conversation history
• Source Attribution: Every response includes references to the source documents used
• Multi-Source Knowledge Base: Automatically ingests and indexes content from:
  • Documentation websites
  • GitHub repositories
  • API specifications
  • Academic papers
  • Jupyter notebooks
• Token Management: Intelligent handling of token limits for both queries and conversation history
• Conversation Tracking: Database-backed system for tracking user interactions and chat sessions

🔧 Installation

1. Clone the repository:

git clone https://github.com/aplbrain/bossdb-rag.git
cd bossdb-rag

2. Create and activate a virtual environment:

   • Requires Python 3.10 and above

python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

   • May need to do the following on Ubuntu: sudo apt-get install build-essential python3-dev and pip install wheel.

pip install -r requirements.txt

4. Set up environment variables:

export AWS_ACCESS_KEY_ID="your_aws_access_key"
export AWS_SECRET_ACCESS_KEY="your_aws_secret_key"
export AWS_REGION="your_aws_region"
export GITHUB_TOKEN="your_github_token"

⚙️ Configuration

All system configuration is managed through config.yaml in the project root. The configuration is organized into three main sections:

LLM Configuration (llm_config)

llm_config:
  default_llm: "anthropic.claude-3-5-sonnet-20240620-v1:0"  # Main response generation model
  fast_llm: "anthropic.claude-3-haiku-20240307-v1:0"        # Memory summarization model
  embed_model: "cohere.embed-english-v3"                     # Text embedding model
  
  # AWS Bedrock credentials (loaded from environment variables)
  aws_region: "OS_ENV_AWS_REGION"
  aws_access_key_id: "OS_ENV_AWS_ACCESS_KEY_ID"
  aws_secret_access_key: "OS_ENV_AWS_SECRET_ACCESS_KEY"
  
  # GitHub authentication
  github_token: "OS_ENV_GITHUB_TOKEN"

Usage Limits (limits)

limits:
  max_questions: 1000          # Maximum questions per session
  max_words: 100000           # Maximum total words per session
  max_message_tokens: 4096    # Maximum tokens per message
  max_total_tokens: 8192      # Maximum conversation history tokens

Data Sources (sources)

sources:
  urls:                      # List of documentation sources
    - "https://docs.url"
    - "https://api.url"
  github_orgs:              # GitHub organizations to index
    - "org1"
    - "org2"

Environment Variables

Values prefixed with OS_ENV_ in the config file are loaded from environment variables. Required variables:

• AWS_REGION
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• GITHUB_TOKEN

Updating Configuration

1. Modify config.yaml as needed
2. Ensure required environment variables are set
3. Restart the application to apply changes

Storage

The system maintains its vector store in:

• Vector store files (./storage/)

This is automatically created on first run.

🚀 Usage

Starting the Application

Run the application using Chainlit:

chainlit run main.py

This will start the web interface on http://localhost:8000.

💾 Data Storage

The system uses MongoDB for storing:

• User information (profiles, usage metrics)
• Chat threads (conversation sessions)
• Message history (individual messages within threads)
• Usage statistics

MongoDB Setup

1. Start MongoDB (using Docker):

docker run -d \
    --name mongodb \
    -p 27017:27017 \
    -e MONGO_INITDB_ROOT_USERNAME=admin \
    -e MONGO_INITDB_ROOT_PASSWORD=password123 \
    mongodb/mongodb-community-server:latest

2. The system will automatically:
   • Create required collections
   • Set up indexes for efficient querying
   • Handle connection management

Collection Structure

• users: Stores user profiles and usage statistics

  {
    "user_identifier": "ip_sessionid",
    "question_count": 0,
    "word_count": 0,
    "created_at": "timestamp",
    "last_activity": "timestamp"
  }

• chat_threads: Tracks conversation sessions

  {
    "user_id": "ObjectId",
    "start_time": "timestamp",
    "end_time": "timestamp"
  }

• messages: Stores individual messages

  {
    "chat_thread_id": "ObjectId",
    "content": "message text",
    "is_user": true/false,
    "timestamp": "timestamp"
  }

Viewing Database Contents

The included view_database.py script provides a way to inspect the MongoDB collections:

1. Usage:

   python view_database.py

2. Configuration:

   • Default connection: mongodb://admin:password123@localhost:27017
   • Default database: bossdb_rag
   • To modify, update the connection string in the script or use environment variables

3. Features:

   • Displays all collections in the database
   • Shows the first 5 documents from each collection
   • Formats complex data types for readability
   • Displays total document count per collection
   • Handles nested objects and timestamps

4. Example Output:

   Database: bossdb_rag
   
   Collection: users (Total documents: 25)
   +------------------+----------------+-------------+-------------------------+-------------------------+
   | _id             | user_identifier | word_count  | created_at             | last_activity          |
   +------------------+----------------+-------------+-------------------------+-------------------------+
   | ...             | 127.0.0.1_abc  | 150         | 2024-03-15T10:30:00Z   | 2024-03-15T11:45:00Z   |
   [Additional rows...]
   
   Collection: chat_threads (Total documents: 30)
   [Thread information...]
   
   Collection: messages (Total documents: 145)
   [Message information...]
   

Storage Cleanup

To reset the database and storage:

1. Drop MongoDB collections:

   mongosh "mongodb://admin:password123@localhost:27017/bossdb_rag" --eval "db.dropDatabase()"

2. Remove vector store files:

   rm -rf ./storage/*

🔄 RAG Pipeline

1. Document Processing

   • Documents are loaded from configured sources
   • Content is split into chunks using appropriate splitters
   • Chunks are embedded and stored in a vector index

2. Query Processing

   • User query is processed
   • Relevant documents are retrieved using vector similarity
   • Context is combined with conversation history
   • Response is generated using Claude 3.5 Sonnet
   • Sources are processed and attached to response

3. Memory Management

   • Conversation history is maintained using either:
     • Summary-based memory (using Claude 3 Haiku)
     • Window-based memory (keeping recent messages)
   • Token limits are enforced for both individual messages and total context

🛠️ Development

Pipeline Structure

Adding New Data Sources

Adding new data sources is simple through the config.yaml file. No code changes are required - just update the configuration.

1. Open config.yaml in the project root directory
2. Navigate to the sources section:

sources:
  # Add new URLs to the urls list
  urls:
    - "https://existing-url.com"
    - "https://your-new-documentation-url.com"     # Add new URLs here
    - "https://github.com/org/repo"                # GitHub repositories are supported
    - "https://api-endpoint.com/openapi.json"      # API specifications
    
  # Add new GitHub organizations to github_orgs list
  github_orgs:
    - "existing-org"
    - "your-new-org"                              # Add new organizations here

The system supports various types of sources:

• Documentation websites
• GitHub repositories (including wikis, specific files, or entire repos)
• API specifications (OpenAPI/Swagger)
• Academic papers
• Jupyter notebooks
• Markdown files
• JSON endpoints

When adding new sources, ensure they:

• Are publicly accessible (or accessible with provided GitHub token)
• Contain relevant BossDB-related content
• Have stable URLs that won't frequently change

After updating the configuration:

1. Shutdown the application (if running)
2. Delete the old storage folder
3. Start up the application
4. The system will automatically process and index the new sources
5. New content will be immediately available for queries

Incremental Updates

The system supports incremental updates to the knowledge base:

1. Document Processing

   • New documents are processed and added to the vector store
   • Existing document hashes are compared to detect changes
   • Only modified content is reprocessed

2. Storage Management

   • Vector store files in ./storage/ contain embeddings and index

3. Update Process

   # Example: Incremental update
   app = BossDBRAGApplication(
       urls=["https://docs.example.com"],
       orgs=["example-org"],
       incremental=True
       force_reload=False,
   )
   await app.setup()

Customizing Document Processing

Modify Splitter class to add new document types:

def split(self, document: Document) -> List[Document]:
    file_extension = self._get_file_extension(document)
    if file_extension == ".new_type":
        return self.custom_splitter.get_nodes_from_documents([document])

Stress Testing

The stress testing script (stress_test.py) can simulate multiple concurrent users interacting with the chatbot. The script uses Playwright for browser automation and provides detailed metrics about system performance.

python stress_test.py http://localhost:8000 --sessions 5 --questions "What is BossDB?" "How do I download data?"

Features:

• Simulates multiple concurrent chat sessions
• Measures response times, success rates, and throughput
• Tracks concurrent session statistics
• Generates detailed CSV reports with timing analysis
• Provides comprehensive test results including:
  • Total requests and success rate
  • Maximum concurrent sessions
  • Response time percentiles
  • Requests per second
  • Session-level timing analysis

Results are saved to:

• stress_test_results_[timestamp].csv: Detailed metrics for each request
• session_timings_[timestamp].csv: Session-level timing analysis
• stress_test.log: Detailed test execution log

📊 Monitoring

This system includes comprehensive logging:

• Application logs in console and bossdb_rag.log
• Database viewer script view_database.py

⚠️ Limitations

• Requires valid AWS credentials for Bedrock access
• GitHub token required for repository access
• Protections against single user spamming
• Multiple users are supported through Chainlit but is limited
• Requires some level of local compute for calculating distances for vector search

📄 License

Apache-2.0 license

---

Made with 💙 at