# Jev / System One Evidence Notes

Updated: 2026-09-22
Scope: Jev programming model, System One architecture, performance claims, Agent integration, calibration, failure modes.

## Research questions
1. What is fundamentally different from an LLM with structured output?
2. Which decision shapes are native to Jev, and which still require LLM reasoning?
3. Is calibrated uncertainty usable as a production control signal?
4. Where can Jev reduce Agent latency/cost without reducing end-to-end task success?
5. Which official performance claims survive independent reproduction?

## Evidence grading

### E1 — TypeSafe announcement, 2026-09-15
URL: https://typesafe.ai/blog/introducing-system-one-models-and-jev
Grade: Primary / vendor research announcement.
Supports:
- Jev is the first public System One model.
- New architecture, parallel sampler, RLCD.
- Typed probabilistic decisions rather than string generation.
- $0.042 / MTok input, output described as free.
- 70–500 ms official response-time range.
- 193.6x faster / 444.6x cheaper originates from TypeSafe workflow evals.
- Official caveats: favorable short inputs, internal workflow design bias, external-model reference answers, real-world gains may be lower.
- Choice cardinality up to 255; higher-cardinality demos use a two-stage scoring then explicit choice.
Boundary:
- Vendor-reported; not independent ground truth.
- “No hallucination” is based on schema/type correctness, not proof of zero semantic error.

### E2 — TypeSafe Introduction
URL: https://docs.typesafe.ai/introduction
Grade: Primary / current documentation.
Supports:
- Choice returns choice + probabilities + confidence.
- Score returns score + probabilities + confidence.
- Noul returns 0–1.
- Questions in one request are evaluated in parallel and independently against the same state.
- Recommended workload: atomic, well-scoped judgments; compose broad behavior in code.

### E3 — How to build with TypeSafe
URL: https://docs.typesafe.ai/concepts/how-to-build-with-system-one
Grade: Primary / architecture guidance.
Supports:
- Code retains deterministic work, control flow, and side effects.
- System One handles narrow common-sense judgments over unstructured data.
- Minimize and structure state.
- Decompose broad questions.
- Ask many independent questions in one request.
- Combine outputs in deterministic code or a classical ML model.
- Route on uncertainty.
Implication:
- System One is better described as an AI software primitive than an autonomous Agent.

### E4 — Confidence
URL: https://docs.typesafe.ai/confidence
Grade: Primary / semantics.
Supports:
- Confidence is derived from the returned probability distribution for Choice and Score.
- Full probability distributions remain available for application-defined uncertainty measures.
- High/medium/low confidence should map to different system behavior.
- Thresholds should depend on action risk and be tuned on domain data.
Open questions:
- No independent reliability diagrams found yet.
- Need ECE/Brier/NLL and risk-coverage curves on LRI workloads.

### E5 — Quick Start / API
URL: https://docs.typesafe.ai/introduction/quickstart
Grade: Primary / API contract.
Supports:
- POST https://api.typesafe.ai/v1/systemone
- model alias jev-latest
- sample response reports model jev-1.13.0
- mixed Choice/Score/Noul in one request
- Python SDK typesafe-sdk
Open questions:
- Exact context/state limits, sustained concurrency, regional deployment, p99 latency and SLA need direct measurement/current docs.

### E6 — Coding Agents boundary
URL: https://docs.typesafe.ai/introduction/coding-agents
Grade: Primary / product boundary.
Supports:
- Jev is not a drop-in replacement for the LLM behind coding agents.
- It does not generate code/text or run a conversational tool loop.
- It is intended to sit inside applications/agents for routing, classification, scoring, guardrails and typed decisions.

## LRI derived hypotheses
H1. Agent control-plane offload: a large share of Agent steps can be reclassified into atomic semantic judgments and offloaded from a frontier LLM to Jev.
H2. Selective cascade: the economically relevant metric is end-to-end selective accuracy at a given escalation rate and cost.
H3. Context filtering: Jev may be more valuable before the main LLM call than after it.
H4. Semantic verifier: Jev can complement deterministic tests, but should never replace exactly computable checks.
H5. Calibration is the gating capability: if confidence does not remain calibrated under domain/language/distribution shift, most automation advantages collapse.

## Required experiments
E1 Tool Router: rules, embedding nearest-neighbor, small LLM JSON schema, frontier LLM; measure Top-1, selective accuracy, coverage, ECE, Brier, P50/P95/P99, cost/1k decisions.
E2 Context Filter: measure token reduction, downstream task success, and relevant-context false negatives.
E3 Semantic Verifier: measure FAR/FRR, calibration and defect escape rate; separate semantic checks from deterministic tests.
E4 Cascade: Jev high-confidence auto path + low-confidence reasoning-model fallback; measure end-to-end success, escalation rate, tail latency and cost.

## Stress matrix
- Chinese / English / mixed language
- missing fields and contradictory state
- out-of-domain inputs
- adversarial prompt-like text inside state
- long state / irrelevant context
- class imbalance
- similar candidate labels
- high-cardinality routing
- model-version change
- network/timeout failure

## Current conclusion
Evidence status: interface/design = strong primary evidence; performance/generalization = vendor evidence only; production value = hypothesis pending independent experiments.
The next material upgrade should come from real Jev API results or independent third-party evaluations, not more paraphrasing of vendor claims.