What SolBehave is

SolBehave is a benchmark for checking whether LLM-generated Solidity smart contracts behave like the real contracts they’re supposed to reproduce, not just whether they compile SolBehave.

The core context:

  • Smart contracts on Ethereum are effectively immutable; once deployed, you cannot patch in-place. A single latent fault is permanent and exploitable SolBehave.
  • Most current evaluations of LLM-generated code stop at compilation success SolBehave.
  • SolBehave instead measures *behavioral fidelity* to the original contract across 250 real, reproducible Ethereum contracts SolBehave.

For builders shipping agentic systems that write or modify Solidity, SolBehave is less “yet another benchmark” and more “a reference architecture for how your test oracle should look.”

The three behavioral oracles

SolBehave composes three distinct checks that all compare a generated contract to the original one SolBehave:

  1. Differential fuzzing
  2. Historical transaction replay
  3. DCR-guided synthetic transaction replay

Each has a specific role in the control flow of evaluation.

1. Differential fuzzing: function-level equivalence

Goal: compare each generated function against the original on many automatically generated inputs SolBehave.

Mechanically:

  • Take one function f in the original contract and the corresponding f' in the LLM-generated contract.
  • Generate a large set of input tuples that are type-correct and within some value ranges.
  • Execute both f and f' on each input.
  • Flag divergence if any observable result (return values, state changes, reverts) differs.

Behaviorally equivalent functions pass; everything else fails.

SolBehave’s pooled results across seven state-of-the-art open-weight models: only 31.4% of all generated function bodies that compile are equivalent to the original under differential fuzzing SolBehave. That’s after excluding non-compiling outputs.

For your own agents, that number is a strong signal: if your evaluation mainly looks like “does it compile?” you’re overestimating correctness by roughly a factor of three in the regimes SolBehave tests.

2. Historical transaction replay: contract-level reality check

Goal: replay what actually happened on-chain to see if the generated contract matches the original on the *exact* historical transaction stream SolBehave.

Control flow:

  • For each of the 250 real Ethereum contracts, gather the historical transactions they actually received SolBehave.
  • Feed that sequence of calls (addresses, function selectors, calldata, values) into:
  • The original contract.
  • The generated contract.
  • Compare contract-level outcomes over the entire replay (state, balances, events, success/failure).

If they ever diverge on an observable effect during that historical sequence, the generated contract fails this test.

In SolBehave, divergence at the contract level is the norm rather than the exception: for six of the seven models, *between 13% and 31% of generated contracts that compile* diverge from the original under historical replay SolBehave.

So even when models reproduce something that passes real-world usage traces, roughly a sixth to a third of compiled outputs still behave differently.

3. DCR-guided synthetic replay: exploring valid but unseen paths

This is the core contribution: synthetic transaction replay guided by a Dynamic Condition Response (DCR) graph SolBehave.

What’s important, as documented:

  • A DCR graph enumerates the space of *legitimate function-call sequences* for a given contract SolBehave.
  • SolBehave uses this DCR graph to generate sequences of function calls that:
  • Follow intended usage patterns (valid sequences).
  • Include orderings that never occurred on-chain in the historical data SolBehave.

Evaluation flow:

  • Use the DCR graph as a generator of valid call sequences.
  • Replay these sequences against both original and generated contracts.
  • Detect divergence in behavior on these *new but legitimate* paths.

This is where many naive testing setups fail: they only replay historical data or random fuzzing, which tends to skew toward simple or already-seen traces. The DCR-guided approach explicitly targets the “unknown-but-valid” region your LLM code is likely to mishandle.

Under this stricter lens, SolBehave finds that for six of the seven models, between 13% and 44% of generated contracts that compile diverge from the original under DCR-guided replay SolBehave.

One model, Gemma-4-31B, is a clear outlier: it stays *below 3% divergence* on both historical and DCR-guided replay SolBehave.

The compilation trap

SolBehave evaluates seven state-of-the-art open-weight models after one round of automated repair SolBehave. At that point:

  • Compilation success ranges from 26.4% to 92.7% across models SolBehave.
  • But only 31.4% of compiled function bodies are equivalent under differential fuzzing when pooling all models and samples SolBehave.
  • And at the contract level, compiled artifacts diverge 13–31% of the time on historical data and 13–44% under DCR-guided replay for six models SolBehave.

The authors explicitly conclude: compilation is a necessary condition for correctness and far from a sufficient one SolBehave.

In agent terms: using “compiles” as your done-condition for a code-generation loop is equivalent to declaring success as soon as your agent stops throwing syntax errors. SolBehave quantifies just how misleading that is.

How this fits into an agentic Solidity stack

If you’re building an agent that writes or edits smart contracts, SolBehave’s structure gives you a template for your own orchestrations:

  1. Synthesis phase
  • Agent drafts or edits the Solidity contract.
  1. Syntactic gate
  • Check that the contract compiles. This is the same “necessary but not sufficient” gate SolBehave uses SolBehave.
  1. Function-level behavioral filter (differential fuzzing)
  • If you have a spec or a reference version, compare each function’s behavior across a battery of inputs, rather than just checking that tests pass.
  • Adopt the mindset that initially, only on the order of a third of compiled function bodies may be actually correct, as seen in SolBehave’s pooled numbers SolBehave.
  1. Real-usage regression (historical replay)
  • For upgrades or regenerations of existing contracts, treat historical on-chain transaction sequences as a regression suite and run them side-by-side on original vs. new.
  1. Specification-aware exploration (DCR-style replay)
  • If you have an explicit or inferred model of valid function-call sequences, use it to generate new transaction sequences that are:
  • Valid under the intended workflow.
  • Not trivially covered by the historical data.
  1. Decision logic
  • Block deployment if any divergence appears in steps 3–5.
  • Optionally, feed failures back into your agent as examples for another repair round, mirroring that SolBehave evaluates models *after* one automated repair SolBehave.

Crucially, SolBehave’s design makes one architectural point: “contract correctness” is inherently multi-view. If your deployment or CI pipeline has only one of these three oracles, you’re blind in important regions of the behavior space.

Why this matters now

Two facts from SolBehave combine into a strong operational warning:

  • Smart contracts are immutable once deployed on Ethereum SolBehave.
  • Compilable, even seemingly plausible contracts frequently diverge in behavior under realistic and spec-guided usage patterns SolBehave.

For agent builders, this is an instance of a broader pattern: models are now good enough to produce “looks right” artifacts that are wrong in subtle, systemic ways.

SolBehave shows this concretely:

  • Even after an automated repair round, broad ranges of compiled outputs hide behavioral faults revealed only by:
  • Function-level fuzzing.
  • Real-world historical usage.
  • Spec-guided exploration of unseen but valid paths SolBehave.

And the Gemma-4-31B result is equally important: model choice interacts strongly with behavioral robustness. While six models sit in that 13–44% divergence band for compiled contracts, one remains below 3% on both historical and DCR-guided replay SolBehave. That’s not enough to skip testing, but it suggests that smart-contract workloads are highly sensitive to training and architecture choices.

What to watch and adapt

Based on just what’s documented, there are a few concrete patterns you can port into your own stack:

  • Differential testing as a first-class tool: Treat “generate-and-compare against a reference implementation under many inputs” as the default evaluation pattern for LLM-written code when any reference exists SolBehave.
  • Trace-based regression: If you’re refactoring or upgrading deployed systems, accumulate and replay real usage traces (like SolBehave’s historical replay) rather than relying only on tests you wrote pre-upgrade SolBehave.
  • Workflow models for exploration: Even a rough approximation of valid call sequences can drive more targeted synthetic tests, analogous to SolBehave’s DCR-guided replay over legitimate function-call sequences SolBehave.

Finally, treat SolBehave-style evaluation as an *online* component of agent systems, not just an offline benchmark:

  • Loop your agent through “synthesize → evaluate (SolBehave-style) → repair” before any human ever sees a PR.
  • Surface behavioral divergence metrics (e.g., percent of historical traces that diverge) as feedback for humans deciding whether to trust an auto-generated contract.

SolBehave quantifies how far compilation is from correctness; your systems need to bridge that gap in their control flow, not via hope.

What is not documented

From the available text, several implementation details of SolBehave are *not* specified:

  • How the 250 Ethereum contracts were selected beyond being “real, reproducible” SolBehave.
  • The exact procedure or tooling used to perform differential fuzzing (input domains, distributions, or limits) SolBehave.
  • How historical transaction traces were collected, filtered, or normalized before replay SolBehave.
  • The method used to construct or infer the Dynamic Condition Response (DCR) graphs for each contract, beyond the fact that they enumerate legitimate function-call sequences SolBehave.
  • The identities of the seven evaluated models apart from mentioning Gemma-4-31B and that they are open-weight SolBehave.
  • Details of the “one round of automated repair,” including how patches are proposed and validated SolBehave.

Any concrete implementation choices beyond these high-level descriptions are not established by the cited source and would require consulting the full paper or associated code.