What EnigmaForge actually is

EnigmaForge is a benchmark where the model never gets an explicit question. Instead, it receives a stack of “old documents” — letters, receipts, logbook margins — inside which a small logic puzzle is buried [\[1\]](http://arxiv.org/abs/2609.30144v1). The puzzle has a *single, provably unique* solution. That uniqueness is not hand-waved: it’s proven at instance-generation time by a SAT solver, and backed by an ablation certificate demonstrating that every clue is load‑bearing [\[1\]](http://arxiv.org/abs/2609.30144v1).

Key properties:

  • No query: the system only sees the story.
  • Embedded puzzle: a small logic problem is implied by the documents.
  • Formal guarantee: a SAT solver proves there is exactly one solution.
  • Ablation certificate: if you remove any clue, uniqueness breaks; nothing is redundant.
  • Synthetic and renewable: instances are generated rather than collected, so the corpus can be refreshed indefinitely [\[1\]](http://arxiv.org/abs/2609.30144v1).

The main metric is *intuition*: can the model, given only the story, succeed at the underlying task? World reconstruction — recovering the underlying facts that define the puzzle’s world — is tracked as a secondary axis [\[1\]](http://arxiv.org/abs/2609.30144v1).

Twenty-five frontier models are evaluated on over 600 instances, for 17,400 scored records, under three matched conditions [\[1\]](http://arxiv.org/abs/2609.30144v1). Those conditions include variants with and without an explicit question, which exposes surprising behavior in how models use (or ignore) guidance.

How EnigmaForge works, step by step

The paper does not spell out the full pipeline, but it *does* describe the critical verification loop and what is being evaluated [\[1\]](http://arxiv.org/abs/2609.30144v1):

  1. Generate a world and puzzle

A small logic puzzle is embedded in a fictional world expressed as letters, receipts, and logbook margins. Each instance is constructed so that:

  • All puzzle-relevant information is contained within the documents.
  • The correct solution can be expressed as a concrete configuration of facts.
  1. Prove uniqueness with a SAT solver

For each instance, a SAT solver is used at generation time to prove that the puzzle has a *unique* solution [\[1\]](http://arxiv.org/abs/2609.30144v1). In effect:

  • The world constraints and clues are encoded as a satisfiability problem.
  • The solver is used to show there is exactly one satisfying assignment.
  • If multiple solutions exist, the instance is rejected or adjusted.

This gives you a binary, mechanically checkable ground truth: either the model’s world reconstruction matches the unique solution, or it does not.

  1. Generate an ablation certificate

EnigmaForge doesn’t stop at uniqueness: it generates an *ablation certificate* showing “every clue is load‑bearing” [\[1\]](http://arxiv.org/abs/2609.30144v1). Concretely:

  • For each clue, the system removes it from the constraint set.
  • The SAT solver re-checks uniqueness.
  • If removing a clue either introduces multiple solutions or makes the puzzle unsatisfiable, that clue is proven essential.

The end result is a certificate that there are no red herrings. Every line in the story that encodes a constraint is necessary for the puzzle to be well-posed.

  1. Serve to models under multiple conditions

25 frontier models are run over more than 600 instances, producing 17,400 scored records across three matched conditions [\[1\]](http://arxiv.org/abs/2609.30144v1). The conditions include:

  • Just the story (no explicit question).
  • Variants where the model is given the question explicitly.
  • A matched setup to compare what changes when the model is told what to focus on.

Performance is measured along:

  • Intuition: task success given only the story.
  • World reconstruction: accuracy on recovering the underlying facts.
  1. Score intuition vs reconstruction

The headline measure is *intuition*, defined as success when “handed only the story” [\[1\]](http://arxiv.org/abs/2609.30144v1). World reconstruction accuracy is a secondary axis — a more standard “did it reconstruct the ground-truth facts?” [\[1\]](http://arxiv.org/abs/2609.30144v1).

Because puzzles are SAT-verified and ablated, every success or failure is interpretable: the model either latched onto the necessary clues and built the unique solution, or it didn’t.

What EnigmaForge exposes about current models

The headline claim from the results: *intuition reshuffles the leaderboard*.

Across the 25 frontier models, EnigmaForge reports [\[1\]](http://arxiv.org/abs/2609.30144v1):

  • A 22× spread on the intuition measure — the best models are 22 times better than the worst on “figure out the hidden task.”
  • Only a 1.6× spread on fact recovery (world reconstruction).
  • The second-best fact-recoverer ranks fourteenth on intuition.
  • One model is *indifferent* to being told the question.
  • Another model is *significantly better without* being told the question.

This is the core punchline: traditional QA-style benchmarks, which always supply a clear question, mostly probe fact recovery once the task is framed for the model. EnigmaForge specifically stresses the *framing* step — can the model look at a bundle of text and infer what is being asked and what must be inferred to answer it?

The separation between the two axes (intuition vs fact recovery) shows that:

  • You can be very good at reconstructing facts given a clearly stated task, yet poor at inferring the task from raw context.
  • Adding explicit questions does not uniformly help; in some cases it doesn’t matter, and in at least one case performance improves when you *don’t* give the question [\[1\]](http://arxiv.org/abs/2609.30144v1).

For anyone building agents that operate in unstructured environments (logs, emails, EHR notes, etc.), this matters more than headline QA scores.

Filters, refusals, and what your evals are really measuring

A subtle but important finding is that “several models were blocked by their own content filters before reaching the puzzle” [\[1\]](http://arxiv.org/abs/2609.30144v1). Because EnigmaForge stories include heterogeneous old documents, some instances evidently trigger safety or content filters.

The paper points out a concrete consequence: “any benchmark scoring refusals as failure is quietly measuring filter behavior” [\[1\]](http://arxiv.org/abs/2609.30144v1).

From a stack-design perspective:

  • If your model refuses on certain narratives, your evaluation score folds together:
  • Capability (could it solve the puzzle?)
  • Filter behavior (did it run at all on this input?)
  • Safety-hardening a model can look like a capability regression on such benchmarks, even if raw reasoning didn’t change.
  • Conversely, disabling or weakening filters can “improve” the benchmark score even if reasoning is unchanged.

For agent builders, this is a warning: if you evaluate with tasks that look like real-world, messy text, and you count refusals as incorrect, your metrics are as much about your filter policy as your agent capability.

How this fits into an agentic stack

EnigmaForge isn’t an agent framework; it’s a benchmark. But the way it’s constructed maps cleanly onto the kinds of cognitive steps you want from robust agents:

  1. Sense-making over raw logs

The input is a stack of letters, receipts, logbooks [\[1\]](http://arxiv.org/abs/2609.30144v1) — structurally similar to:

  • Issue trackers and commit messages.
  • Customer support threads.
  • Audit logs and receipts.

The agent’s first job is to *discover* what matters, not just answer a stated query.

  1. Latent task inference

Intuition in EnigmaForge is exactly this: find the hidden puzzle in the story and solve it [\[1\]](http://arxiv.org/abs/2609.30144v1). That is close to:

  • Extracting implicit requirements from messy tickets.
  • Inferring “what is the real problem?” from a stack of errors.
  • Deriving investigation questions from an incident log.
  1. World modeling with formal ground truth

World reconstruction maps to the agent’s internal state model. In EnigmaForge this is verifiable via SAT [\[1\]](http://arxiv.org/abs/2609.30144v1), giving a strict notion of correctness that most production environments don’t have — but the logical pattern is the same.

  1. Separation of “what to ask” vs “how to solve”

The benchmark empirically demonstrates that “being good at solving once the question is known” and “figuring out the question” are not the same capability [\[1\]](http://arxiv.org/abs/2609.30144v1). That suggests a stack pattern:

  • One component or mode devoted to *inferring the latent task* from raw context.
  • Another devoted to efficient factual reasoning and retrieval given a framed query.

EnigmaForge gives you a way to evaluate that first stage independently.

  1. Renewable, synthetic eval

Because instances are generated rather than collected, “the corpus renews forever” [\[1\]](http://arxiv.org/abs/2609.30144v1). This plays well with continuous deployment:

  • You can re-evaluate regularly without exhausting a fixed test set.
  • You can monitor regressions on intuition separately from classic QA and retrieval tasks.

Why this matters now

Several concurrent works in the same batch of papers point in a similar direction: *how systems behave in novel, less-structured settings, and how we can verify what they’re doing*.

Examples from the provided sources:

  • ExplorationBench builds “verifiable Alien Worlds” with executable rules that conflict with familiar knowledge, so recall alone cannot solve the tasks [\[7\]](http://arxiv.org/abs/2609.30199v1). That’s another move away from static, QA-style datasets toward environments where exploration and rule-discovery are required.
  • BRIE, a living benchmark for information retrieval from electronic health records, automatically generates and updates question–answer pairs from longitudinal notes [\[6\]](http://arxiv.org/abs/2609.30205v1), echoing EnigmaForge’s emphasis on renewable evaluations.
  • VeriSpeak targets speech-based fact verification, showing that retrieval alone has limited gains and that explicit reasoning over retrieved evidence is needed to compare claims to evidence [\[3\]](http://arxiv.org/abs/2609.30227v1). Again, the theme is grounded reasoning, not just recall.

EnigmaForge contributes a complementary dimension: *can your model even figure out what is being asked of it* in a pile of context, before retrieval and chain-of-thought kick in?

For production agents that must:

  • Triage tickets.
  • Read unstructured logs and infer incidents.
  • Digest long narrative reports and extract action items.

— the EnigmaForge-style capability is closer to what you actually need than yet another multiple-choice QA benchmark.

How to use these ideas in your own evaluation

The paper doesn’t prescribe evaluation practice, but some direct implications for builders follow from its documented results [\[1\]](http://arxiv.org/abs/2609.30144v1):

  1. Add “hidden-task” evals alongside explicit QA

Don’t just feed your agent explicit prompts like “Summarize X” or “Find Y.” Also evaluate:

  • Give it entire threads, logs, or document bundles with *no explicit question*.
  • Score it on whether it can:
  • Identify the implicit question (e.g., “What should I do next?”).
  • Recover the correct underlying facts or state.
  • Propose the right resolution.

EnigmaForge shows that these are distinct skills, with a 22× spread on intuition vs 1.6× on fact recovery and a reshuffled leaderboard [\[1\]](http://arxiv.org/abs/2609.30144v1).

  1. Isolate filter behavior in your metrics

Since EnigmaForge finds that several models are blocked by their own filters before reaching the puzzle and warns that counting refusals as failure quietly measures filter behavior [\[1\]](http://arxiv.org/abs/2609.30144v1), you should:

  • Log refusals separately from incorrect reasoning.
  • Run dual evals (with full safety filters, and with relaxed filters in a sandbox) to see where safety, not capability, is driving failures.
  • Be explicit in dashboards: which metrics are “can it reason?” vs “will it respond?”
  1. Treat “question discovery” as a first-class capability

Given that the second-best fact-recoverer ranks fourteenth on intuition [\[1\]](http://arxiv.org/abs/2609.30144v1), you shouldn’t assume that bigger or better retrieval stacks solve the “what is being asked?” problem.

In agent architectures, you may want:

  • A dedicated pass for “intent discovery” over raw context.
  • Training or prompting modes that reward discovering latent tasks and articulating them explicitly before acting.
  1. Expect non-monotonic effects from more guidance

One model in EnigmaForge is indifferent to being told the question, and another is *worse* when given the question [\[1\]](http://arxiv.org/abs/2609.30144v1). That should make you cautious about:

  • Assuming that more explicit prompting always helps.
  • Over-constraining agents with heavy meta-instructions, which may interfere with their own internal framing.

In practice, you may want A/B evaluations of:

  • “Just the documents; do what seems most useful.”
  • “Here is the explicit question. Answer it.”

And compare both performance and failure modes.

What is not documented

From the available text, several implementation and result details are *not* established:

  • The exact structure of the logic puzzles (e.g., specific constraint languages, domains, or sizes) beyond being “small logic puzzles.”
  • The nature of the three matched evaluation conditions beyond the presence/absence of explicit questions.
  • Which specific 25 frontier models were evaluated, their parameter counts, or vendors.
  • The precise scoring methodology for intuition and world reconstruction (e.g., metric definitions, thresholds).
  • Any per-model results, rankings, or numerical scores beyond the relative spreads (22× for intuition, 1.6× for fact recovery) and the qualitative statements about rank reshuffling [\[1\]](http://arxiv.org/abs/2609.30144v1).
  • The exact trigger content that caused some models’ filters to block access to the puzzles.
  • Implementation details of the SAT encoding, solver choice, or the exact form of the ablation certificates.

Any further specifics about EnigmaForge’s design or outcomes would require information beyond what is present in the cited abstract and metadata.