Beinex AI
Beinex is a data, analytics and AI company headquartered in Dubai. We work with finance, risk and audit teams, and we build our own models for the problems we see every day.
Noah: small, specialised models
Noah models are small enough to run on your own hardware, explain their answers, and say when they are unsure.
| Model | What it does | Status |
|---|---|---|
| Noah-Verify (Noah 2.5) | Is this claim supported by this document? Reads prose and tables, checks the numbers in the claim (scale, sign, transposed digits, simple arithmetic), points to the evidence, escalates when unsure. | Research preview |
| Noah-Decide | Typed decisions (yes/no, pick one of many, score on a scale) from a state and a set of options, independent of option order. | In development |
Latest results (2026-10-12)
Nutrient grounding benchmark (en, 5,000 rows), scored with the benchmark's own, unmodified score.py. The leaderboard maintainers pulled the public repo and reran the scorer on their hardware before merging: 0.8145 overall / 0.7041 number, identical to our run — the result is independently reproduced.
Nutrient grounding leaderboard: 6th of 15; first independent entry; above every permissively-licensed open model; open weights under CC BY-NC 4.0.
| ROC-AUC | Number (headline) | Overall |
|---|---|---|
| Noah 2.5, standard export (0.4B; open weights, CC BY-NC 4.0; non-commercial-use category) | 0.7041 | 0.8145 |
| DeBERTa-v3-large zero-shot (best open model not made by Nutrient) | 0.658 | 0.786 |
| OpenAI Decisions (gpt-6-luna) | 0.769 | 0.823 |
| Nutrient grounding-en | 0.923 | 0.882 |
- Sources we did not train on (3,551 rows): Noah 0.819 overall · DeBERTa 0.840 (our re-run of the same scorer). Noah's lead comes from the source families whose training splits we used; on the others DeBERTa zero-shot is ahead.
- The full Noah 2.5, with its number-checking channel and chunked reading, scores 0.755 number / 0.800 overall in our own pipeline. The channel adds +0.065 number AUC in a same-data, same-seed ablation. It needs custom code, so the official scorer can't run it.
- Guardrails AI hallucination index data (BUMP, 1,386 news summaries; self-run, as the index has no submission process; report only, no ranking claim):
- Our harness reproduces Vectara HHEM's published 0.632 exactly, as F1 with "faithful" as the positive class.
- On that metric, Noah 2.5 scores 0.673 (95% CI 0.641–0.700).
- A constant "everything is faithful" baseline scores 0.667 on it, above the published Microsoft (0.648) and Vectara (0.632) scores, so we make no ranking claim.
- On threshold-free ROC-AUC, Noah scores 0.689 vs HHEM's 0.604.
- Misses we are working on:
- LLM-AggreFact fell to 70.6 (our previous model: 72.7);
- our stretch goal of beating gpt-6-luna was not met;
- the calibration target was met only after post-hoc recalibration;
- on sources we did not train on, DeBERTa zero-shot is still ahead (see above).
- Self-run, single training seed, except where independent reproduction is stated.
How we work
- Pre-registered evaluation. We write the pass/fail bars before training and report misses as well as wins.
- Data provenance. We list every training source and licence. Three LLM-generated sources — WANLI (CC BY 4.0), MiniCheck (MIT), SummEdits (CC BY 4.0) — are under provenance review; the base model, ModernBERT-large, is Apache-2.0.
- Open where we can. The standard Noah 2.5 export has open weights for non-commercial use (CC BY-NC 4.0). The full models are research previews; contact us for evaluation or commercial access.