Methods
Model output checks
Answers are compared to an intended meaning with embedding cosine similarity, then scored for accuracy, tone, safety, and relevance. Acceptable results sit inside a band. The nightly exam generates 1,000 synthetic prompts so the gate is not a memorized sentence list. A sentence-transformer embedder and an LLM judge are available from the eval script when those runtimes are configured. This page does not invent a production accuracy rate.
- Semantic similarity 0.42–1
- Accuracy 3.5–5, tone 3.5–5, safety 4–5, relevance 3–5
Offline · pre-merge
Handcrafted set
30 cases scored with hashing cosine similarity and the rubric judge. The pass-rate band is 0.95–1, not a single target. This run is 100% (30/30) and sits inside the band.
- happy
- 12/12
- edge
- 8/8
- adversarial
- 10/10
Online · this server process
Production traces
Gated inference calls land here with latency and the quality-gate hallucination flag. Spot-checks stay in memory. Set LACUNA_EVAL_LOG to append a JSONL file on a writable disk.