Source: https://eschatialabs.com/benchmarks/jevoss/

[All benchmarks](https://eschatialabs.com/benchmarks/)

Benchmark · Decision models

# JevOss

An open test bench that checks whether a decision model stays right when its input changes.

[Project site](https://jevoss.mertkayacs.com/)

JevOss takes questions with known answers and changes one thing at a time. It reorders the options, plants a hidden instruction, adds about 600 words of unrelated text, rewrites a yes-or-no question or sends the same request again, then measures how often the answer changes.

It also scores accuracy and checks whether the model’s probabilities match how often it is right. It tests any server that speaks the Jev API (POST /v1/systemone), including JevAlt and Kev-4B.

**Checks**: 5: option order, hidden instructions, unrelated text, yes-or-no formats, repeated requests

**Commands**: eval, probe, calibrate, compare, ask

**Needs**: Python 3.11 or newer and a running decision-model server

**Version**: 0.1.0, released 1 October 2026

**Licence**: Apache-2.0

## Results

Hidden instructions: share of 200 test cases where a planted instruction changed the answer (lower is better)

| Model | Answers changed |
| --- | --- |
| Kev-4B | 36.0% |
| Deem-4B | 14.0% |
| Karar-4B | 19.0% |
| Wähler-4B | 17.5% |

### Limits

With about 600 words of unrelated text added, Kev-4B loses 5.4 points of accuracy and Deem-4B 17.4.

## Run it

The default server address is http://127.0.0.1:8000. JevAlt starts a matching server with jevalt serve.

```
pip install "jevoss[suites] @ git+https://github.com/mertkayacs/jevoss"
jevoss eval typed-decisions
jevoss probe typed-decisions --limit 100
```

## Cite

```
@software{kaya2026jevoss,
  author = {Mert Kaya},
  title = {JevOss: Open Evaluation Toolkit for Jev-Type Decision Models},
  year = {2026},
  license = {Apache-2.0},
  url = {https://github.com/mertkayacs/jevoss}
}
```

## Links

- [Project site](https://jevoss.mertkayacs.com/)
- [Code (GitHub)](https://github.com/mertkayacs/jevoss)
- [Documentation](https://mertkayacs.github.io/jevoss/)