MCP Optimization Preview

MCP Optimization connects to a Model Context Protocol server, discovers its tools, and tests whether a model selects the expected tool for representative queries.

Use it when

Your tools are exposed by an MCP server.
Tool names and schemas are stable, but descriptions can be improved.
The model often chooses the wrong tool or no tool.
You can label each representative query with the expected tool.

Use Tool Selection Optimization when you want to paste tool definitions directly without connecting an MCP server.

Promptic keeps tool names, input schemas, the selected model, and test cases fixed while it proposes clearer tool descriptions. It can also propose an updated shared system prompt when that option is enabled.

Held constant	Candidate output	Measured outcome
Tool names and input schemas	Clearer, less-overlapping descriptions	Correct tool selected
Representative labelled queries	Optional revised shared system prompt	Correctly chooses no tool
Model used for the evaluation	Multiple candidate configurations	Regressions by test case
Tools discovered from the server

Tool Optimization does not change tool code, names, or input schemas, and it does not invoke the tools during optimization. It measures whether the model selects the expected tool definition.

Why descriptions matter

Useful descriptions state what a tool does, when it should be used, when it should not be used, and the meaning of important arguments. Vague or overlapping descriptions force the model to infer those boundaries.

search

Search for information.

search_knowledge_base

Search the internal product knowledge base for product behavior, setup, and troubleshooting.
Use this for product-specific questions. Do not use it for account data or general web research.

How it works

Connect the server

Create an MCP Optimization AI Component, provide the MCP server URL and authentication when required, then test the connection. Promptic imports the advertised tool names, descriptions, and input schemas as the baseline.

A successful connection confirms discovery—not that the tools themselves will succeed when invoked. Resolve connectivity and authentication failures before starting an evaluation.

Review discovered definitions

Confirm that the imported names and schemas match what the production model receives. Fix incorrect server metadata before optimizing descriptions.

Build representative test cases

For every query, select the exact expected tool or mark it as a no-tool case. Include common, ambiguous, and business-critical requests.

Establish the baseline

Run the original definitions against the dataset to identify confused tool pairs, missed calls, and unnecessary tool use.

Optimize and review

Promptic proposes candidate descriptions and, when enabled, a revised shared system prompt. It scores every candidate against the same model and test cases.

Apply and validate

Copy accepted descriptions back to the MCP server, then test the complete agent in its real runtime.

Design a useful dataset

A strong dataset teaches the boundaries between tools rather than repeating only obvious requests.

Case type	Why include it	Example
Clear positive	Confirms each tool can be selected	“Reset the password for user 42”
Near overlap	Distinguishes tools with similar purposes	Search documentation vs search account data
No tool	Prevents unnecessary calls	“Explain what OAuth is”
Noisy phrasing	Reflects abbreviations and production language	“cant login pls help”
High-impact edge case	Protects rare cases where a wrong call is costly	Refund vs cancel an order

Include at least one case for every tool in scope. Use the exact tool name as the expected result, or none when the model should not call a tool.

Review results

The primary score measures whether the selected tool name matches the expected result. Read it together with:

accuracy by tool, especially for tools with fewer examples;
confusion between tools with overlapping responsibilities;
false-positive calls on no-tool cases;
regressions from the original definitions;
changes to the optional system prompt;
individual failures on business-critical cases.

Do not accept a candidate based only on aggregate accuracy. An apparent improvement can still make a rare but important tool less reliable.

What you ship

The result is not a hosted runtime deployment. Copy the winning descriptions and optional system prompt back to the MCP server, review the change, and rerun held-out cases through the complete agent. Re-evaluate after changing tool names, schemas, or the model used by the agent.

Security considerations

Connecting an MCP server gives Promptic access to the endpoint and its advertised tool metadata. Review the authentication method, network exposure, and whether descriptions or schemas contain sensitive information before connecting production infrastructure. A successful connection test does not prove that a server is safe to connect or that its tools are safe to invoke.

Current availability

You can create tool-selection experiments through the public API and Python SDK by supplying the tool definitions and test cases directly (see the Create a tool-selection experiment endpoint and the SDK client reference). Auto-discovering tools from an MCP server, and reviewing the optimized descriptions, remain in the dashboard—optimized descriptions are not yet returned through the public API. The workflow may change while it remains in Preview.