decision-eval: An Opinionated Benchmark for Zero-Shot Decision Models

October 16, 2026
research llm benchmark evaluation decision-models zero-shot classification calibration latency

Disclaimer. This is an opinionated, small-scale evaluation suite run by one person, not a leaderboard of record. The suites are hand-picked, each task uses one prompt wording, no model was tuned, and hosted models can change behind the same name. Read the numbers as a snapshot under the protocol stated on this page.

decision-eval compares zero-shot decision models on identical requests and reports accuracy, latency, cost and calibration per suite. A decision model is a zero-shot classifier with a typed interface: given a text and a set of questions, it returns calibrated probabilities over the answers you allow, in one of three modes: choice (pick one of k options), noul (yes/no) or score (a level on an ordered scale). It never generates text. Each suite asks every item in every mode it supports, so the same item is scored as a choice, a verification and a rating. Suites so far: synthetic STEM problems (one right answer per item) and two guardrail sets imported from reflex-guard (flag or pass a prompt). Code and per-item results: NISH1001/decision-eval.

Models.

  • Laya, english checkpoint: convaiinnovations/laya [1]. ModernBERT-large [2], ~395M parameters, local (PyTorch). STEM: laya 0.3.10, english checkpoint. Guardrails: laya 0.3.20 through its Router, which routes English text to the same english checkpoint (byte-identical weights).
  • Laya typed, typed-decisions checkpoint: convaiinnovations/laya-typed-decisions. Same backbone, different training mix; local.
  • GLiNER: fastino/GLiNER2.5-Decide [3], DeBERTa-v3-large [4], ~435M parameters, run locally through an fp32 ONNX export (nishparadox/gliner2.5-decide-onnx) with exact parity to PyTorch. In the guardrail suites its yes/no mode is averaged over 3 label orders (reflex-guard's default).
  • Jev 1.14: typesafe/jev-1.14 on OpenRouter's decisions API [5]. Hosted; architecture and size undisclosed; USD 0.042 per million input tokens, output free (list price at time of run). Exact version, as the API reports it: typesafe/jev-1.14-20261014, the snapshot dated 2026-10-14. Every Jev result on this page comes from that snapshot, run on 2026-10-16 (STEM here, the guardrail sets in reflex-guard).
  • GPT-6 Luna: gpt-6-luna on OpenAI's Decisions API (POST /v1/decisions, public beta) [7]. Hosted; architecture and size undisclosed; USD 0.10 per million input tokens, output not billed on this endpoint (list price at time of run). The API reports only the alias gpt-6-luna; no dated snapshot is published, so the run records the model's creation date from the models endpoint (2026-09-14) and the run date (2026-10-16). Our modes map to its question types: choice → choice, noul → predicate, score → score; the state is sent as its JSON text. When it refuses a question in the guardrail suites, reflex-guard scores the refusal as a flag, so a refusal never reads as safe.

Jev reports about 75 output tokens per request even though it returns only probabilities. That suggests a generative model behind the structured interface, but TypeSafe does not say. GPT-6 Luna reports zero output tokens.

Suite: STEM (synthetic). 180 generated problems, 15 in each of 12 tiers from single-digit arithmetic through complex analysis, calculus and linear algebra to mechanics and E&M. Each problem has one correct answer and three distractors, which are common mistakes where one exists (kinetic energy without the ½, a determinant with the cofactor signs dropped, i² = +1). noul shows the correct answer half the time; score asks which order of magnitude the answer’s absolute value falls in (five levels). Every model receives byte-identical requests.

Suite: Guardrails, core. reflex-guard’s core set [8], test split: 322 prompts, of which 104 are harmful, 94 benign and 124 hard negatives (safe prompts built to look risky: trigger words, the benign twin of a harmful prompt). Science-domain requests, jailbreaks and prompt injections over 8 harm categories. Every guard asks every category in each mode; a prompt’s P(flag) in a mode is its highest category score, and it is flagged at that model’s threshold for the mode, chosen on the dev split (best balanced accuracy) and never on these prompts. Imported from reflex-guard’s eval caches: nothing is re-run, and its own report’s numbers are reproduced exactly.

Suite: Guardrails, OOD. reflex-guard’s out-of-distribution set [8]: 988 prompts from three public safety sets, Aegis 2.0, XSTest v2 and OR-Bench: 519 harmful, 99 benign, 370 hard negatives (over-refusal bait), over 8 content categories. Nothing is tuned here: every model keeps the thresholds it got on the core dev split, so this measures how a guard transfers. Harmful prompts are over half the set, so flagging everything already scores 53% precision and accuracy; compare against the baselines in Table 1.

1. Accuracy

"GLiNER" here is GLiNER2.5-Decide (fastino), run from its fp32 ONNX export, which matches the PyTorch model exactly; the int8 and fp16 exports are not shown.
Loading…
Figure 1 (interactive). macro / micro / weighted squeeze the categories into one bar per model: macro is the plain mean of the per-category scores, micro pools all items (each counted once), and weighted averages the categories by their size. For a plain accuracy, weighted equals micro; they differ for balanced accuracy and when an item carries several categories. Accuracy per category for each model and question mode; switch modes in the panel. Dashed line: chance. In score mode, black ticks mark a baseline that always guesses the category's most common level. Hover a bar for counts.Recall per harm category (share of harmful prompts with that label that get flagged) and false-positive rate on safe prompts, per model, at each model's threshold for the selected mode. Hover a bar for counts. Jev is near ceiling in every mode; GPT-6 Luna is clearly above chance everywhere but often picks the common-mistake distractor (a dropped ½, a flipped sign, a near-miss sum), even on multi-digit addition. GLiNER clears chance only in choice mode; both Laya checkpoints scatter around chance in every tier. Every hosted-model miss was checked against the answer key. Jev and GPT-6 Luna flag most attacks with the fewest false positives; score mode is the best single mode for both. GLiNER's yes/no mode also flags about a quarter of benign and two in five hard-negative prompts. On unseen data every model's false positives rise, in every mode. Jev keeps the highest recall (90–97%) but flags half or more of the hard negatives; GPT-6 Luna holds recall near its core level with far fewer false positives, though its numbers include 251 prompts where a refused question counted as a flag.
Table 1. Overall scores. Brackets: 95% bootstrap interval over items (1,000 resamples). Verify uses balanced accuracy, the mean of the accuracies on correct and on wrong shown answers, so a model that always answers yes scores 50%. Click a column to sort.Overall scores for the selected mode. Positives are harmful prompts; negatives are benign and hard-negative prompts. Each model and mode is thresholded as tuned on the core dev split. Brackets: 95% bootstrap interval over prompts (1,000 resamples). The grey row gives baselines: flagging everything, or always predicting the majority class. AUC is threshold-free. Click a column to sort.
Figure 2 (interactive). P(flag) per prompt, in lanes by what the prompt actually is; the dashed tick is that model's threshold for the mode. A good guard puts harmful prompts right of its threshold and both safe lanes left of it. Hover a dot for the prompt.Verification in noul mode. Each dot is one item: the model's P(yes) that the answer shown with the problem is correct, in the lane for whether it actually was. A useful verifier puts the top lane right of the dashed line and the bottom lane left of it. Hover a dot for the item. Both Laya checkpoints say yes to about 90% of statements and GLiNER says no to all of them. Jev separates the lanes almost perfectly (AUC 0.99) and GPT-6 Luna well (AUC 0.92). Constant answers give exactly chance-level balanced accuracy, which is why three different behaviours share 50% in Table 1.

2. Latency and cost

Figure 3 (interactive). F1 for the selected mode against median latency per prompt in that mode (log scale). Vertical bars: 95% interval on F1. Horizontal bars: interquartile range of latency. Dashed: the F1 of flagging everything. Labels: cost per thousand prompts for all three modes. Local guards ran on an Apple-silicon laptop (Laya on MPS, GLiNER on CPU); hosted latency includes the network round trip.Score against median latency per item (log scale) with every question answered in one request. Vertical bars: 95% interval on the score. Horizontal bars: interquartile range of latency. Labels: cost per thousand items at the price billed. Up and to the left is better. Local models ran on an Apple-silicon laptop; hosted latency includes the network round trip (Jev via OpenRouter, GPT-6 Luna via OpenAI), measured from the same machine with 8 requests in flight.
Table 2. Median latency per prompt in each mode, and what each hosted model's run over this set cost, from reflex-guard's run logs (requests include one warm-up per mode pass). Jev's cost is OpenRouter's billed amount. OpenAI's native API reports no cost, so GPT-6 Luna's is input tokens at USD 0.10 per million; its totals include re-running requests that had refusals, and an earlier partial run through OpenRouter (about USD 0.22) was discarded. Output tokens were not recorded in this eval.Latency per item, tokens and what the run cost, with the exact model version each API reported. Tokens and cost come from each response's usage field; GPT-6 Luna's cost is its input tokens at the list price, since the Decisions endpoint bills input only. Local models cost only the hardware they run on. Jev was run twice; its two runs agree on 177 of 180 choice picks, so hosted results can move by a point between runs.

3. Calibration

Not shown for the guardrail suites: their thresholds are tuned per model, so P(flag) is used to rank prompts (AUC in Table 1), not read as a probability.

Figure 4 (interactive). Reliability diagram for choice mode. Items are binned by the probability of the option picked; each point is a bin's mean confidence against its accuracy. Dashed: perfect calibration. ECE [6]: the confidence–accuracy gap averaged over bins, weighted by size. Jev is accurate and calibrated. GPT-6 Luna is reliable when it is sure (93% right on its 103 answers given with ≥0.9 confidence) but overconfident in the middle range (ECE 0.10). Laya's curve is flat, so its confidence says nothing about correctness. Laya typed's low ECE comes from being unconfident and at chance, not from being right.

4. Dataset explorer

Figure 5 (interactive). Every item and every model's answer in every mode. Filter by category, by who got it wrong, or search. The outlined option is the correct answer.Every prompt with each model's P(flag) and decision in every mode. Filter by prompt kind, by who got it wrong, or by refusals, or search.

Notes

  • Prompting. One instruction wording, one seed and no tuning for any model. GLiNER’s prompt format takes the option text as the label, where Laya and Jev see A–D with the text as the description, and it reserves ( and ), which are swapped for brackets. Re-running Laya with option text as labels (on an earlier 100-item version) left it at chance.
  • Answer key. Answers are exact to two decimals. An early version rounded to three significant figures, which made a correct rejection count as a miss; all models were re-run.
  • Scope. Items are template-generated and per-tier n is 15; quote the overall intervals rather than single tiers.
  • Source. Imported from reflex-guard at commit 999d51d; the thresholds, recall and false-positive rates match its own report. Precision, F1 and accuracy are computed here from the same per-prompt flags.
  • Thin categories. Most harmful prompts are attacks (prompt injection, jailbreak); the content categories have 2 to 14 prompts each, so their recall bars are indicative only.
  • Laya typed was not part of the guardrail eval.
  • Source. Imported from reflex-guard at commit 999d51d (its report). “Out of distribution” means not named as training data in either local model’s card; all three source sets predate both models.
  • Refusals. GPT-6 Luna refused at least one question on 251 of 988 prompts, each counted as a flag (reflex-guard’s policy), which raises both its recall and its false positives. The explorer can filter to them.
  • Laya typed was not part of the guardrail eval.

Code and data: github.com/NISH1001/decision-eval, with the per-item results for every model. To reproduce a suite without cloning anything (results go to ./results/):

R="git+https://github.com/NISH1001/decision-eval"
uvx --from "decision-eval[laya] @ $R"   decision-eval run --suite stem-synth --backend laya        # and laya-typed
uvx --from "decision-eval[gliner] @ $R" decision-eval run --suite stem-synth --backend gliner
uvx --from "$R" decision-eval run --suite stem-synth --backend jev          # needs OPENROUTER_API_KEY
uvx --from "$R" decision-eval run --suite stem-synth --backend gpt-6-luna   # needs OPENAI_API_KEY
uvx --from "decision-eval[figures] @ $R" decision-eval figures --suite stem-synth   # paper figures + LaTeX table
uvx --from "$R" decision-eval import-reflexguard --repo <reflex-guard checkout>     # the guardrail suites

References

  1. convaiinnovations/laya. Hugging Face model repository (english and typed-decisions checkpoints); laya Python package 0.3.10.
  2. Warner, B., et al. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv:2412.13663.
  3. fastino/GLiNER2.5-Decide. Hugging Face model card.
  4. He, P., Gao, J., & Chen, W. (2021). DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. arXiv:2111.09543.
  5. TypeSafe. Jev 1.14 (typesafe/jev-1.14), OpenRouter decisions API. Snapshot typesafe/jev-1.14-20261014 (dated 2026-10-14), accessed 2026-10-16.
  6. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML 2017.
  7. OpenAI. Decisions API guide (POST /v1/decisions, beta). developers.openai.com/api/docs/guides/decisions. Model gpt-6-luna, created 2026-09-14, accessed 2026-10-16.
  8. Pantha, N. (2026). reflex-guard: multi-label guardrails on System One decision models. github.com/NISH1001/reflex-guard; eval sets, caches and report at commit 999d51d.

Citation

@techreport{nish2026decisioneval,
  author      = {Pantha, Nishan},
  title       = {decision-eval: An Opinionated Benchmark for Zero-Shot Decision Models},
  institution = {Bits and Paradoxes},
  type        = {Technical Report},
  year        = {2026},
  month       = oct,
  url         = {https://nishparadox.com/research/decision-eval/}
}

Plain text:

Pantha, N. (2026). decision-eval: An Opinionated Benchmark for Zero-Shot Decision Models. Technical Report. https://nishparadox.com/research/decision-eval/

Bits and Paradoxes