Evaluating and Benchmarking the System One Model JEV

Problem

Jev is a recently released ‘system one’ model, evaluating it on ‘informed decision’ tasks.

Solution

This paper introduces evaluation insights on 37 publicly benchmark datasets on QWEN 27B (OS), GEMMA 4B (OS), JEV (proprietary). Whereas JEV outperforms around 22 on QWEN and all 37 on GEMMA. Firstly, the Jev token generation mechanism is different from typical LLMs. The mechanism is bypassing generation entirely by reading the model's next-token logits directly in a single forward pass. Which is also implemented in base models for fair comparison. The mechanism is below,

Single-Token Option Codes: They assigned each option a code that maps to a single token in the model's vocabulary (e.g., A, B, Yes, No, or single digits for scores).

Prompt Construction: They rendered the input state as JSON alongside the question and instructed the model to respond using only the option code.

Single Forward Pass: Token generation and thinking/reasoning steps were disabled. They ran a single forward pass to obtain the raw next-token probability distribution.

Logit Filtering & Renormalization: They restricted the next-token distribution strictly to the tokens representing the valid option codes, then renormalized those values so they summed to 1.

This forces the model's confidence scores to be distributed entirely among the valid choices without generating any text.

While JEV outperforms most of the models, JEV’s model param size has not been disclosed since it is a proprietary model. So I doubt if it is a valid comparison.