Kems-Bench

A benchmark where LLMs play Kems (the card game of secret signals) against each other. Teammates must coordinate and transmit a secret signal in a public chat observed by opponents without getting caught.

Lire en français

---

1. Installation

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .\.venv\Scripts\Activate.ps1

# Install dependencies
pip install -r requirements.txt

---

2. Configuration

Copy .env.example to .env and fill in your API key(s):

cp .env.example .env

You can use GitHub Models (free tier with a standard GitHub Personal Access Token), OpenRouter, or direct provider keys (Mistral, OpenAI, Anthropic, Gemini, Kimi).

---

3. How to Run

Option A: Web Dashboard (Recommended)

Launch the web interface to configure players and watch matches with real-time card tables and chat logs:

python -m kems.run --serve

Then open http://127.0.0.1:8000 in your browser.

Option B: Command Line (CLI)

Run a match directly in your terminal:

python -m kems.run --agents github,github,github,github --model gpt-4o-mini --live

Common CLI options:

• --agents : Comma-separated list of 4 players (e.g. github,github,github,github, mistral,mistral,mistral,mistral).
• --model : Model name to use (e.g. gpt-4o-mini, mistral-medium-latest, claude-3-5-sonnet).
• --live : Display game events and chat in real-time in the terminal.
• --seed : Fix random seed for card distribution.