Curated dataset · n=61 verified deals · Not live market data · Scores are descriptive, not forecasts.
You are here

Methods — Record quality, observed sector composition, and announcement timing.

Methods

Model output checks

Answers are compared to an intended meaning with embedding cosine similarity, then scored for accuracy, tone, safety, and relevance. Acceptable results sit inside a band. The nightly exam generates 1,000 synthetic prompts so the gate is not a memorized sentence list. A sentence-transformer embedder and an LLM judge are available from the eval script when those runtimes are configured. This page does not invent a production accuracy rate.

  • Semantic similarity 0.42–1
  • Accuracy 3.5–5, tone 3.5–5, safety 4–5, relevance 3–5

Offline · pre-merge

Handcrafted set

30 cases scored with hashing cosine similarity and the rubric judge. The pass-rate band is 0.95–1, not a single target. This run is 100% (30/30) and sits inside the band.

happy
12/12
edge
8/8
adversarial
10/10

Online · this server process

Production traces

Gated inference calls land here with latency and the quality-gate hallucination flag. Spot-checks stay in memory. Set LACUNA_EVAL_LOG to append a JSONL file on a writable disk.