The problem: multi-step agents fail on the missing edge case

If you’re running serious agentic workflows, you’ve seen this failure mode:

  • The agent plans 5–10 steps.
  • It calls a couple of APIs, does some retrieval, maybe runs code.
  • Somewhere in the middle it needs a very specific tool integration that… you never exposed.
  • The agent either stalls, hallucinates a result, or silently degrades the workflow.

For multi-step tasks, *one* missing capability is enough to sink the whole run. The system in Tool Recommendation for Multi-Step Agents is built around that observation: “A multi-step agent task fails when a single tool it needs is missing” [1].

At small scale you can hand-pick tools into an MCP server or similar tool host. At platform scale you can’t:

  • The platform evaluated in the paper has “roughly 8,200 tools” [1].
  • The *definitions* of those tools “together exceed the context window of any model” they tested [1].
  • Yet multi-step agents may need any of them on demand.

So the core engineering question becomes:

How do you give an agent *access to everything it might need* without dumping thousands of tool specs into context on every call?

The Tool Recommendation system in [1] is one production answer.

The core idea: a fixed three-tool gateway to thousands of tools

Instead of wiring thousands of tools directly into an agent, the authors interpose a *gateway* between agent and tools.

Key properties of that gateway, as documented in [1]:

  • It “runs at the gateway between agent and tools” [1].
  • It exposes a “fixed interface of three tools” to the agent [1].
  • Behind that fixed interface, it can give the agent “all of them” — the full set of ~8,200 tools, “hundreds of times more than a typical MCP server exposes” [1].
  • Every downstream tool call is executed “under the end user’s own credentials” [1].
  • Those calls are checked “against the user’s consent for each service and the tools the agent may use” [1].

Functionally, you can think of it as:

  • The agent only ever knows about three gateway tools.
  • The gateway is responsible for:
  • Finding the actual concrete tools relevant to the current task.
  • Mediating permissions and credentials.
  • Forwarding and translating calls into real API invocations.

From an orchestration standpoint, this is a clean separation:

  • Agent harness: prompts, planning, reasoning, high-level tool usage.
  • Tool gateway: search over tool universe, selection, auth, consent, execution.

Where most MCP and tool-hosting systems shove selection into the LLM prompt (long lists of tools, inline descriptions), this system moves selection into an explicit retrieval and policy layer.

What Tool Recommendation demonstrably achieves

The paper gives several quantitative results on both retrieval quality and end-to-end cost [1]:

  • On the public ToolRet benchmark, Tool Recommendation:
  • Places “every tool a query needs in the top ten 57% more often than BM25” [1].
  • BM25 is the lexical baseline used in commercial tool search [1].
  • It “compares favorably with published systems up to nine times its size” [1].
  • In paired multi-step workflows:
  • Its first recommendation “covered every required tool in 93% of Codex sessions” [1].
  • Against a fixed list of 100 tools:
  • It cut *input tokens* by “17% with Codex and 40% with Claude Code” [1].
  • It reduced the *total bill* in “all 14 runs, by up to 40%” [1].
  • These cost wins came “with no change in task completion” [1].

For a production builder, that last line is the important one: you can shrink tool metadata in context by double-digit percentages and trim up to 40% off session cost *without* losing completion rate — if your retrieval layer actually finds the tools needed for the workflow.

How it fits into an agent stack

Based on the description in [1], the stack looks conceptually like this:

  1. User-facing agent harness
  • Multi-step reasoning: planning, decomposition, looping.
  • Uses a small fixed toolset (three gateway tools).
  • Model weights are standard (Codex, Claude Code etc. are used in evaluation [1]).
  1. Tool Recommendation gateway
  • Exposes three abstract tools to the agent.
  • For each call, retrieves the subset of concrete tools relevant to the query.
  • Executes calls under user credentials.
  • Enforces per-service and per-tool consent [1].
  1. Tool universe
  • About 8,200 tools on the platform studied [1].
  • Individual tool definitions are not all shipped to the model – they live outside the LLM context.
  • Aggregate tool definitions exceed context windows of all evaluated models [1].
  1. External systems
  • Services/APIs that the tools wrap.
  • Authentication and auditing tied to the “end user’s own credentials” [1].

The notable thing: the *agent harness* doesn’t grow linearly with tool count. You keep a constant, bounded interface from the agent’s point of view, and move all the growth to the gateway.

This mirrors a separate but related trend in “harnessing” work like Harness², where the *harness* (prompts, tools, memory, subagents, plugins) is treated as a first-class object that can be optimized independently of model weights [3]. Harness² exposes editable components for “system prompt, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins” [3]. Tool Recommendation is effectively a specialized tool-guidance and tool-execution harness focused on retrieval coverage and consent.

Execution and consent: why the gateway matters for security

The gateway in [1] does more than search:

  • “Executing every call under the end user’s own credentials” [1].
  • “Checking it against the user’s consent for each service and the tools the agent may use” [1].

In a world where attackers are already coordinating “an army of AI agents” to attack websites with “little involvement from the human hacker” [2], putting your agents directly on raw credentials and unfenced tools is asking for trouble.

The Norforge case study described in [2] is instructive:

  • A hacker used an open-source agent platform to break into “more than 100 websites” [2].
  • The agents mostly “operated on their own” given high-level preferences [2].
  • Attacks were orchestrated through an engine that lets users “set a start and end point for the AI agents, which document their progress” [2].

In that environment, a gateway that consistently:

  • Mediates access at the *tool* layer,
  • Checks each call against explicit per-tool consent,
  • Runs under end-user credentials instead of shared super-tokens,

is not a nice-to-have. It’s the only sane way to scale tool exposure to thousands of capabilities and thousands of users.

For production systems, that suggests a pattern:

  • Keep tools behind a broker that is policy-aware.
  • Never let the agent directly own standing credentials.
  • Treat tool selection as a controlled retrieval problem, not just “list these in the prompt”.

What to copy into your own system

The paper is sparse on internals, but the externally visible behavior points to several patterns you can adopt.

1. Decouple tool discovery from tool use

Instead of binding the agent to a fixed static list of tools:

  • Give the agent a small set of *meta-tools* (“find tools”, “call tools”, “inspect tools”).
  • Implement those as services that:
  • Search over a large tool registry.
  • Return only a small, relevant subset.
  • Hide implementation details from the agent.

This is exactly what the three-tool gateway achieves: the agent is given “all of them” — the ~8,200 tools — *through* that interface, without being directly aware of each one [1].

For your system, that might look like:

  • A service that returns top-k candidate tools for a textual description of the subtask.
  • A wrapper that, given a selected tool id and arguments, executes and returns results.
  • Optional: a “describe tool” endpoint for rich tool inspection when needed.

2. Minimize tool metadata in-context

The empirical results in [1] show:

  • Compared to a fixed list of 100 tools, the system could cut *input tokens* by 17–40% depending on model [1].
  • That translated into up to 40% total cost reduction across 14 runs, with no drop in task completion [1].

The engineering implication: don’t put large, static tool catalogs in every prompt.

Better patterns:

  • Keep the full tool registry outside the LLM.
  • For each query or subtask, bring in:
  • Only the *few* candidate tools you think are relevant.
  • Only *just-enough* descriptions (e.g. input schema and short summary).

That’s exactly what retrieval-based gateway designs enable.

3. Treat tool coverage as a first-class metric

On the ToolRet benchmark, Tool Recommendation is judged by whether it “places every tool a query needs in the top ten” [1]. It beats BM25 on that metric by 57% [1].

This is a different orientation than generic relevance metrics. For multi-step agents, *coverage* (did we surface all required tools?) is more important than ranking perfection: missing a required tool breaks the workflow, while adding a few extra plausible tools usually just adds some noise.

Implications for your evaluation loop:

  • Construct task sets where the necessary tools are known.
  • Measure: “did the retrieval layer include all required tools in its candidate set?”
  • Optimize for this “every required tool in top-k” metric, not just generic retrieval scores.

4. Push auth and consent down to the tool broker

The gateway in [1]:

  • Runs calls “under the end user’s own credentials” [1].
  • Enforces user consent for “each service and the tools the agent may use” [1].

From a platform perspective, that’s non-negotiable:

  • Different tools map to different external systems and risk levels.
  • Different users have different entitlements and data scopes.
  • Regulatory and audit requirements apply at the service boundary, not the LLM boundary.

So, in your own infrastructure, put:

  • Identity-aware, audited execution in the broker or tool gateway.
  • Simple, stateless tool usage in the agent harness.

Why this matters now: task diversity and open-world agents

Work like Harness² frames the challenge as “recursive agent harnessing in an open world, where agents encounter tasks unseen during harness design” [3]. Harness² improves task performance by editing and refining the harness (system prompt, guardrails, tool guidance, subagents, etc.) via a “Propose–Probe–Compose” loop, and then reusing “evidence-supported lessons from each task” in future runs [3].

The Tool Recommendation system is compatible with that view:

  • It lets the agent confront new tasks without enumerating all tools upfront.
  • It makes the tool layer discoverable and scalable.
  • It focuses explicitly on the “open world” of tools — thousands of APIs and integrations that can’t fit into a single prompt.

As agents are used in broader environments — from “professional work” and “legal work” benchmarks in Harness² [3] to real-world systems that touch sensitive infrastructure (as in the Norforge incident [2]) — the number and diversity of tools will only grow.

A design that scales *tool exposure* without sacrificing safety or cost is a prerequisite, not an optimization.

What to watch next

From a builder’s perspective, a few directions stand out:

  • Better retrieval for tools. ToolRet is one benchmark, but internal company tool registries and vendor ecosystems will be messy. Expect more work on embeddings, structured metadata, and usage signals specifically for tools.
  • Harness-level learning. Harness² already retains “adopted harnesses and evidence-supported lessons” after each task [3]. Similar ideas could apply to tool recommendation: learning which classes of tasks need which tools, across users and sessions.
  • Stronger policy layers. As attacks like the Norforge case proliferate [2], the tool gateway becomes not just a router but a firewall: rate limiting, anomaly detection, sandboxing will need to live close to the tool execution point.
  • Cross-stack standards. Today, every platform seems to invent its own tool schema and consent model. A common, policy-rich interface between agents and tool gateways would make plug-and-play safer.

What is not documented

The sources do not establish:

  • The internal retrieval algorithms, data structures, or model types used inside Tool Recommendation beyond comparisons against BM25 and unspecified “published systems up to nine times its size” [1].
  • The exact specification of the three gateway tools (their names, arguments, or protocols) [1].
  • Any concrete architecture diagrams, dataflow, latency measurements, or throughput numbers for the deployed system [1].
  • How user consent policies are represented, updated, or audited inside the gateway beyond the fact that they are checked per service and tool [1].
  • Interactions, integrations, or comparisons between Tool Recommendation and other frameworks such as Harness²; any cross-reference here is conceptual only, not empirical [1][3].