All benchmarks

Benchmark · AI on your own computer

Tholos-Bench

160 workspace tasks for small AI assistants, scored on what the workspace looks like when the assistant has finished.

Dataset (Hugging Face)

Each scenario sets up a Tholos workspace with assistants, tools, tables and notes, then gives one assistant a task. The run passes only if the final state matches the checks: the right row was added, the right teammate got the task, no forbidden tool was called.

The 14 categories run from table edits and handoffs to approvals, write conflicts, web research on saved pages, instructions hidden in pages or notes, and tasks where the right move is to do nothing.

Scenarios
160 in 14 categories
Language
English
Format
One JSON file per scenario, also as JSON Lines on Hugging Face
Settings
Temperature 0 (the same answer on every run), a context of 16,384 tokens, fixed clock
Licence
Apache-2.0
Released
1 October 2026, with Tholos 0.1.0

Results

One Kaggle T4 graphics card, llama.cpp, Q4_K_M model files (compressed to about 4 bits per weight), output held to a JSON schema

ModelFilePassed (of 160)Hidden instructions (of 12)
Qwen3.5-2B1.28 GB974
MiniCPM5-2B1.56 GB1128
Tholos-2B1.56 GB13710
Llama 3.2 3B Instruct2.02 GB371
Granite 4.2-3B2.24 GB1297
Qwen3.5-4B2.74 GB1416

Limits

Tholos-2B was trained on top of MiniCPM5-2B and is the only model in the table trained for this step format. On a computer’s main processor (CPU) with llama.cpp, MiniCPM5-2B passes 117 and Tholos-2B 134, so compare scores only within one setup.

Run it

Start a model server, install Tholos and run the full benchmark. Add --only with a category name to run part of it.

llama-server -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M --jinja -c 16384 -t 4 -a tholos-2b --host 127.0.0.1 --port 8080
uv tool install git+https://github.com/mertkayacs/tholos
tholos bench --base-url http://127.0.0.1:8080/v1 --model tholos-2b

Cite

@misc{kaya2026tholosbench,
  author = {Kaya, Mert},
  title = {Tholos-Bench: Workspace Tasks for Small AI Assistants},
  publisher = {Eschatia Labs},
  year = {2026},
  url = {https://huggingface.co/datasets/mertkayacs/tholos-bench}
}