Tool Selection Optimization Preview
Tool Selection Optimization tests and improves tool definitions that you paste into Promptic. It is designed for agents whose tools live in application code or another format that is not exposed by an MCP server.
Use it when
- You can provide tool names, descriptions, and input schemas directly.
- The model often selects the wrong tool or misses a relevant tool.
- You want to improve selection without changing tool implementation.
- You can label queries with the expected tool or a no-tool outcome.
Use MCP Optimization when Promptic should discover tool definitions from a running MCP server.
What gets optimized
Promptic keeps tool names, input schemas, the selected model, and test cases fixed while it proposes clearer tool descriptions. It can also propose an updated shared system prompt when that option is enabled.
| Held constant | Candidate output | Measured outcome |
|---|---|---|
| Tool names and input schemas | Clearer, less-overlapping descriptions | Correct tool selected |
| Representative labelled queries | Optional revised shared system prompt | Correctly chooses no tool |
| Model used for the evaluation | Multiple candidate configurations | Regressions by test case |
| Definitions pasted into Promptic |
Tool Optimization does not change tool code, names, or input schemas, and it does not invoke the tools during optimization. It measures whether the model selects the expected tool definition.
Why descriptions matter
Useful descriptions state what a tool does, when it should be used, when it should not be used, and the meaning of important arguments. Vague or overlapping descriptions force the model to infer those boundaries.
search
Search for information.search_knowledge_base
Search the internal product knowledge base for product behavior, setup, and troubleshooting.
Use this for product-specific questions. Do not use it for account data or general web research.How it works
Add tool definitions
Create a Tool Selection Optimization AI Component and paste the definitions from the agent configuration. Keep names and schemas identical to the runtime configuration.
Review the imported shape
Confirm that descriptions, required arguments, and input schemas match what the production model receives. Correct stale definitions before starting an evaluation.
Build representative test cases
For every query, select the exact expected tool or mark it as a no-tool case. Include common, ambiguous, and business-critical requests.
Establish the baseline
Run the original definitions against the dataset to identify confused tool pairs, missed calls, and unnecessary tool use.
Optimize and review
Promptic proposes candidate descriptions and, when enabled, a revised shared system prompt. It scores every candidate against the same model and test cases.
Apply and validate
Copy accepted descriptions into the agent configuration, then test the complete trajectory—not only tool selection—in the production-like runtime.
Design a useful dataset
A strong dataset teaches the boundaries between tools rather than repeating only obvious requests.
| Case type | Why include it | Example |
|---|---|---|
| Clear positive | Confirms each tool can be selected | “Reset the password for user 42” |
| Near overlap | Distinguishes tools with similar purposes | Search documentation vs search account data |
| No tool | Prevents unnecessary calls | “Explain what OAuth is” |
| Noisy phrasing | Reflects abbreviations and production language | “cant login pls help” |
| High-impact edge case | Protects rare cases where a wrong call is costly | Refund vs cancel an order |
Include at least one case for every tool in scope. Use the exact tool name as the expected result,
or none when the model should not call a tool.
Review results
The primary score measures whether the selected tool name matches the expected result. Read it together with:
- accuracy by tool, especially for tools with fewer examples;
- confusion between tools with overlapping responsibilities;
- false-positive calls on no-tool cases;
- regressions from the original definitions;
- changes to the optional system prompt;
- individual failures on business-critical cases.
Do not accept a candidate based only on aggregate accuracy. An apparent improvement can still make a rare but important tool less reliable.
What you ship
Copy the winning descriptions and optional system prompt into the source agent configuration, review the change, and rerun held-out cases through the complete agent. Re-evaluate after changing tool names, schemas, or the model used by the agent.
Security considerations
Pasted tool definitions can contain internal service names, argument descriptions, and schema details. Remove secrets and unnecessary sensitive metadata before adding them to Promptic. Keep the definitions aligned with production without including credentials, customer data, or private runtime configuration.
Current availability
You can create tool-selection experiments through the public API and Python SDK by supplying your tool definitions and test cases directly (see the Create a tool-selection experiment endpoint and the SDK client reference). The dashboard remains the place to review runs and their optimized descriptions, which are not yet returned through the public API. The workflow may change while it remains in Preview.